v0.29.0 fails to push to distribution v2.8.1 in some operations #3446
Description
Contributing guidelines
- I've read the contributing guidelines and wholeheartedly agree
I've found a bug and checked that ...
- ... the documentation does not mention anything about my problem
- ... there are no open or closed issues that are related to my problem
Description
I am running registry version v2.8.1. With the release of buildx plugin v0.29.0, some operations that previously worked are failing.
I am creating and pushing a multi-arch image with the following command (sensitive details removed):
docker buildx imagetools create \
--tag private-registry.mydomain.com/sometagname/my-image-name:my-tag-name \
private-registry.mydomain.com/sometagname/x86_64/my-image-name:my-tag-name \
private-registry.mydomain.com/sometagname/aarch64/my-image-name:my-tag-name
private-registry.mydomain.com is several instances of https://github.com/distribution/distribution v2.8.1 behind a load balancer, and configured to store data on Amazon S3.
The above command logs the following:
#1 [internal] pushing private-registry.mydomain.com/sometagname/my-image-name:my-tag-name
#1 0.000 copying sha256:03ed94b30c46b16399e0810e6b1b138ec337896d144c42ac76881a9bf21fb595 from private-registry.mydomain.com/sometagname/x86_64/my-image-name:my-tag-name to private-registry.mydomain.com/sometagname/my-image-name:my-tag-name
#1 2.114 pushing sha256:03ed94b30c46b16399e0810e6b1b138ec337896d144c42ac76881a9bf21fb595 to private-registry.mydomain.com/sometagname/my-image-name:my-tag-name
#1 ERROR: failed commit on ref "index-sha256:03ed94b30c46b16399e0810e6b1b138ec337896d144c42ac76881a9bf21fb595": unexpected status from PUT request to https://private-registry.mydomain.com/v2/sometagname/my-image-name/manifests/my-tag-name: 400 Bad Request
unknown: blob unknown to registry
Distribution itself doesn't log anything particularly interesting beyond the blob unknown message.
Expected behaviour
Buildx plugin is able to push to the registry just like the previous version is able to, as I don't see any changes to requirements listed in the release notes.
Actual behaviour
Error per above
Buildx version
github.com/docker/buildx v0.29.0 9c5897f
Docker info
Client: Docker Engine - Community
Version: 28.4.0
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.29.0
Path: /usr/libexec/docker/cli-plugins/docker-buildx
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.1
Path: /usr/local/lib/docker/cli-plugins/docker-sbom
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 1
Server Version: 27.4.1
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 88bf19b2105c8b17560993bee28a01ddc2f97182
runc version: v1.2.2-0-g7cb3632
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.8.0-1021-aws
Operating System: Ubuntu 24.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 30.65GiB
Name: ci-jenkinsworker-20251002154255013171
ID: 0e1330a3-7ae2-4632-bea5-481f374aa499
Docker Root Dir: /datadisk/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Builders list
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
default* docker
\_ default \_ default running v0.17.3 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/arm64, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64, linux/loong64, linux/arm/v7, linux/arm/v6
Configuration
N/A
Build logs
Additional info
No response