v0.10.0 and up pushes multiple small images to ECR

[emmahsax]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

When building with Buildx version < 0.10.0, when I use docker buildx bake to push my multi-arch image to ECR, I get one Image Index, which then is a manifest which contains images for arm64 and amd64. This is what I'd expect:

But when I try to build the same with Buildx v0.10.0 or higher, my push to ECR results in multiple untagged images, some of which have 0 MB. Upon checking further, it's not quite 0, but more like 0.001374 MB, so it's just rounded to 0.

Example 1:

Example 2:

Note: I am running this on a GitHub Actions self-hosted runner

Expected behaviour

The behavior I'd expect wouldn't have so many untagged images and so many images with 0 MB present for each push to ECR. Ideally, all of them would have tags, and they'd be as succinct as possible to not create bloat in the ECR repositories.

Actual behaviour

A bunch of untagged images are being created, some of which have 0 MB in size.

Buildx version

github.com/docker/buildx v0.10.0 8764628

Docker info

Client: Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc., v0.10.4) compose: Docker Compose (Docker Inc., v2.16.0) Server: Containers: 1 Running: 1 Paused: 0 Stopped: 0 Images: 1 Server Version: 24.0.0 Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 1677a17964311325ed1c31e2c0a3589ce6d5c30d runc version: v1.1.7-0-g860f061 init version: de40ad0 Security Options: seccomp Profile: builtin Kernel Version: 5.10.178-162.673.amzn2.x86_64 Operating System: Alpine Linux v3.18 (containerized) OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 7.651GiB Name: SELF-HOSTED-RUNNER-TAG ID: 6984ffaf-4fbb-406d-9403-be0d792d0071 Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

Builders list

NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS builder-0a78176d-bab0-43e1-a9c6-99048eeccbdb Dockerfile README.md docker-bake.hcl renovate.json docker-container builder-0a78176d-bab0-43e1-a9c6-99048eeccbdb0 builders running v0.11.6 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/386 builders docker builders builders running v0.11.6 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/386 default docker default default running v0.11.6 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/386

Configuration

Dockerfile:

# syntax=docker/dockerfile:experimental
FROM ruby:3.1-alpine
RUN apk add --no-cache \
  openssl-dev \
  build-base \
  cmake \
  git
RUN gem install pronto pronto-rubocop rubocop-rails
WORKDIR /app

docker-bake.hcl:

group "default" {
  targets = ["pronto", "pronto-arm64", "pronto-amd64"]
}

variable "TEST_MODE" {
  default = false
}

variable "AWS_REGION" {
  default = "us-east-2"
}

variable "COMMIT_SHORT_SHA" {
  default = ""
}

target "pronto" {
  output = [TEST_MODE ? "" : "type=registry"]
  cache-from = ["type=registry,ref=ghcr.io/GITHUB-ORG/pronto:cache"]
  cache-to = ["type=registry,ref=ghcr.io/GITHUB-ORG/pronto:cache"]
  tags = ["123456789012.dkr.ecr.${AWS_REGION}.amazonaws.com/pronto-test:v0.10.4-${COMMIT_SHORT_SHA}"]
  platforms = ["linux/amd64", "linux/arm64"]
}

target "pronto-arm64" {
  output = [TEST_MODE ? "" : "type=registry"]
  cache-from = ["type=registry,ref=ghcr.io/GITHUB-ORG/pronto:cache-arm64"]
  cache-to = ["type=registry,ref=ghcr.io/GITHUB-ORG/pronto:cache-arm64"]
  tags = ["123456789012.dkr.ecr.${AWS_REGION}.amazonaws.com/pronto-test:v0.10.4-${COMMIT_SHORT_SHA}-arm64"]
  platforms = ["linux/arm64"]
}

target "pronto-amd64" {
  output = [TEST_MODE ? "" : "type=registry"]
  cache-from = ["type=registry,ref=ghcr.io/GITHUB-ORG/pronto:cache-amd64"]
  cache-to = ["type=registry,ref=ghcr.io/GITHUB-ORG/pronto:cache-amd64"]
  tags = ["123456789012.dkr.ecr.${AWS_REGION}.amazonaws.com/pronto-test:v0.10.4-${COMMIT_SHORT_SHA}-amd64"]
  platforms = ["linux/amd64"]
}

GitHub Action jobs:

- name: Set up Docker Context for Buildx
  shell: bash
  run: docker context create builders

- name: Set up Docker Buildx
  uses: docker/setup-buildx-action@v2
  with:
    version: v0.10.4
    endpoint: builders

- name: Login to GitHub Packages
  uses: docker/login-action@v2
  with:
    registry: ghcr.io
    username: ${{ github.actor }}
    password: ${{ secrets.GITHUB_TOKEN }}

- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v2
  with:
    aws-region: us-east-2

- name: Login to Amazon ECR
  uses: aws-actions/amazon-ecr-login@v1
  with:
    registries: "123456789012"

- run: echo $(docker --version)
- run: echo $(docker buildx version)
- run: echo $(docker info)
- run: echo $(docker buildx ls)

- name: Build on main
  run: COMMIT_SHORT_SHA=abc123def456 docker buildx bake --progress plain

Build logs

#1 [pronto-arm64 internal] load build definition from Dockerfile
#1 ...

#2 [pronto-amd64 internal] load build definition from Dockerfile
#2 transferring dockerfile: 279B 0.0s done
#2 DONE 0.1s

#3 [pronto-amd64 internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.1s

#1 [pronto-arm64 internal] load build definition from Dockerfile
#1 transferring dockerfile: 279B done
#1 DONE 0.2s

#4 [pronto internal] load build definition from Dockerfile
#4 transferring dockerfile: 279B done
#4 DONE 0.1s

#5 [pronto internal] load .dockerignore
#5 transferring context: 2B done
#5 DONE 0.1s

#6 [pronto-arm64 internal] load .dockerignore
#6 transferring context: 2B done
#6 DONE 0.2s

#7 [pronto] resolve image config for docker.io/docker/dockerfile:experimental
#7 DONE 0.4s

#8 [pronto] docker-image://docker.io/docker/dockerfile:experimental@sha256:600e5c62eedff338b3f7a0850beb7c05866e0ef27b2d2e8c02aa468e78496ff5
#8 resolve docker.io/docker/dockerfile:experimental@sha256:600e5c62eedff338b3f7a0850beb7c05866e0ef27b2d2e8c02aa468e78496ff5 0.0s done
#8 sha256:d7f0373ffb1d5ac3477b10ab7f91cba7c5df586c72e7e4a12649024efdc0d531 9.64MB / 9.64MB 0.2s done
#8 sha256:d7f0373ffb1d5ac3477b10ab7f91cba7c5df586c72e7e4a12649024efdc0d531 9.64MB / 9.64MB 0.2s done
#8 extracting sha256:d7f0373ffb1d5ac3477b10ab7f91cba7c5df586c72e7e4a12649024efdc0d531
#8 extracting sha256:d7f0373ffb1d5ac3477b10ab7f91cba7c5df586c72e7e4a12649024efdc0d531 0.2s done
#8 DONE 0.5s

#9 [pronto internal] load build definition from Dockerfile
#9 DONE 0.0s

#10 [pronto internal] load .dockerignore
#10 DONE 0.0s

#11 [pronto-arm64 internal] load .dockerignore
#11 DONE 0.0s

#12 [pronto-arm64 internal] load build definition from Dockerfile
#12 DONE 0.0s

#13 [pronto-amd64 internal] load build definition from Dockerfile
#13 DONE 0.0s

#14 [pronto-amd64 internal] load .dockerignore
#14 DONE 0.0s

#15 [pronto-amd64 internal] load metadata for docker.io/library/ruby:3.1-alpine
#15 DONE 0.8s

#16 [pronto-arm64 internal] load metadata for docker.io/library/ruby:3.1-alpine
#16 DONE 0.9s

#17 [pronto 1/4] FROM docker.io/library/ruby:3.1-alpine@sha256:2400ea9bf369d24b29874ec69c08bc3a9b44f27969eda6e3e9eb31289c55e3db
#17 resolve docker.io/library/ruby:3.1-alpine@sha256:2400ea9bf369d24b29874ec69c08bc3a9b44f27969eda6e3e9eb31289c55e3db 0.0s done
#17 DONE 0.0s

#18 [auth] dispatchitinc/pronto:pull token for ghcr.io
#18 DONE 0.0s

#19 [pronto-arm64] importing cache manifest from ghcr.io/dispatchitinc/pronto:cache-arm64
#19 DONE 0.7s

#20 [pronto] importing cache manifest from ghcr.io/dispatchitinc/pronto:cache
#20 DONE 0.7s

#21 [pronto-amd64] importing cache manifest from ghcr.io/dispatchitinc/pronto:cache-amd64
#21 DONE 0.8s

#22 [pronto-amd64 2/4] RUN apk add --no-cache   openssl-dev   build-base   cmake   git
#22 CACHED

#23 [pronto-amd64 3/4] RUN gem install pronto pronto-rubocop rubocop-rails
#23 CACHED

#24 [pronto linux/arm64 1/4] FROM docker.io/library/ruby:3.1-alpine@sha256:2400ea9bf369d24b29874ec69c08bc3a9b44f27969eda6e3e9eb31289c55e3db
#24 resolve docker.io/library/ruby:3.1-alpine@sha256:2400ea9bf369d24b29874ec69c08bc3a9b44f27969eda6e3e9eb31289c55e3db 0.0s done
#24 DONE 0.0s

#25 [pronto-arm64 linux/arm64 2/4] RUN apk add --no-cache   openssl-dev   build-base   cmake   git
#25 CACHED

#26 [pronto-arm64 linux/arm64 3/4] RUN gem install pronto pronto-rubocop rubocop-rails
#26 CACHED

#27 [pronto-arm64 linux/arm64 4/4] WORKDIR /app
#27 CACHED

#28 [pronto-amd64 4/4] WORKDIR /app
#28 CACHED

#29 [pronto-amd64] exporting to image
#29 exporting layers done
#29 exporting manifest sha256:4c7ab0b0778cad4431730f8b7b5aeb29fda03d801ecee2ec3dfbab3ccd028f86 0.0s done
#29 exporting config sha256:ed0aab476d7a95777e8233a8bab6f20030d51406fa57fdc6523b3083901a9054 0.0s done
#29 exporting attestation manifest sha256:2b96e95e18ef73814a6569fc10ad67e573c08e9c49475961f6129a82b8e71565
#29 ...

#30 [auth] sharing credentials for 123456789012.dkr.ecr.us-east-2.amazonaws.com
#30 DONE 0.0s

#29 [pronto-amd64] exporting to image
#29 exporting attestation manifest sha256:2b96e95e18ef73814a6569fc10ad67e573c08e9c49475961f6129a82b8e71565 0.1s done
#29 exporting manifest list sha256:fc4cad659aa7e90d226fe484dd65f3e4d447a478873223d066424fee68457ea0 0.0s done
#29 pushing layers
#29 ...

#31 [auth] sharing credentials for 123456789012.dkr.ecr.us-east-2.amazonaws.com
#31 DONE 0.0s

#32 [auth] sharing credentials for 123456789012.dkr.ecr.us-east-2.amazonaws.com
#32 DONE 0.0s

#33 [pronto] exporting to image
#33 exporting layers done
#33 exporting manifest sha256:4c7ab0b0778cad4431730f8b7b5aeb29fda03d801ecee2ec3dfbab3ccd028f86 done
#33 exporting config sha256:ed0aab476d7a95777e8233a8bab6f20030d51406fa57fdc6523b3083901a9054 done
#33 exporting attestation manifest sha256:2e0178cb90e72f70bdf8b035fd84d9507597daa67ef8a8f924e82ed30e49ce85 0.1s done
#33 exporting manifest sha256:2aeffdeb611151af870546aa9faced49941d91d34bb40a975fc3cce712307ba2 0.0s done
#33 exporting config sha256:e0610427f19f049c5d64ac79efce7ee14bab16850748b320669d032291b46671 0.0s done
#33 exporting attestation manifest sha256:47359e755b292e149728cd0f0da04f7a499f1b761b235116250d6305494f9373 0.0s done
#33 exporting manifest list sha256:4ffb174fd761353dd26a0f24458806f8cfbf4ded55b9f87da8bcf91fd6d01bdc done
#33 pushing layers 0.7s done
#33 pushing manifest for 123456789012.dkr.ecr.us-east-2.amazonaws.com/pronto-test:v0.10.4-74cfd08@sha256:4ffb174fd761353dd26a0f24458806f8cfbf4ded55b9f87da8bcf91fd6d01bdc
#33 ...

#29 [pronto-amd64] exporting to image
#29 pushing layers 0.4s done
#29 pushing manifest for 123456789012.dkr.ecr.us-east-2.amazonaws.com/pronto-test:v0.10.4-74cfd08-amd64@sha256:fc4cad659aa7e90d226fe484dd65f3e4d447a478873223d066424fee68457ea0 0.5s done
#29 DONE 1.0s

#34 [auth] dispatchitinc/pronto:pull,push token for ghcr.io
#34 DONE 0.0s

#35 [pronto-arm64] exporting to image
#35 exporting layers done
#35 exporting manifest sha256:2aeffdeb611151af870546aa9faced49941d91d34bb40a975fc3cce712307ba2 0.0s done
#35 exporting config sha256:e0610427f19f049c5d64ac79efce7ee14bab16850748b320669d032291b46671 0.0s done
#35 exporting attestation manifest sha256:4c5067c262589d05f41c538b19401324bb0bd7715645788dad47059cfcf662e7 0.0s done
#35 exporting manifest list sha256:698b57f04d2f14a5902cad886dcae4539face0da33c28a9a7d680fdb7cbb3c2b 0.0s done
#35 pushing layers 0.6s done
#35 pushing manifest for 123456789012.dkr.ecr.us-east-2.amazonaws.com/pronto-test:v0.10.4-74cfd08-arm64@sha256:698b57f04d2f14a5902cad886dcae4539face0da33c28a9a7d680fdb7cbb3c2b 0.3s done
#35 DONE 1.1s

#36 [pronto-arm64] exporting content cache
#36 preparing build cache for export
#36 writing layer sha256:08409d4172603f40b56eb6b76240a1e6bd78baa0e96590dc7ff76c5f1a093af2
#36 writing layer sha256:08409d4172603f40b56eb6b76240a1e6bd78baa0e96590dc7ff76c5f1a093af2 0.1s done
#36 writing layer sha256:335fc4470cecf31504e0536041c5daec3a87d7803e76d8508296a808a37b46ec 0.1s done
#36 writing layer sha256:59b492009e1697af0a35d8344ae929d156f9d5e2d3ba72b1853aaee24f6bfa95
#36 ...

#33 [pronto] exporting to image
#33 pushing manifest for 123456789012.dkr.ecr.us-east-2.amazonaws.com/pronto-test:v0.10.4-74cfd08@sha256:4ffb174fd761353dd26a0f24458806f8cfbf4ded55b9f87da8bcf91fd6d01bdc 0.6s done
#33 DONE 1.5s

#36 [pronto-arm64] exporting content cache
#36 writing layer sha256:59b492009e1697af0a35d8344ae929d156f9d5e2d3ba72b1853aaee24f6bfa95 0.1s done
#36 writing layer sha256:75acee0b64a8e73a13f847e35298675dcac4bc0f73e630db4f236ca0a9f6f1c1
#36 writing layer sha256:75acee0b64a8e73a13f847e35298675dcac4bc0f73e630db4f236ca0a9f6f1c1 0.1s done
#36 writing layer sha256:7e668614de827af0be2644643a7bc9c6546f6bc38106ab1819cd7e5a6376ff40
#36 writing layer sha256:7e668614de827af0be2644643a7bc9c6546f6bc38106ab1819cd7e5a6376ff40 0.1s done
#36 writing layer sha256:831bb4769cd713dde88dccba20919c346eb5a49bdeebfb7b2d034c667bec9ea3
#36 writing layer sha256:831bb4769cd713dde88dccba20919c346eb5a49bdeebfb7b2d034c667bec9ea3 0.1s done
#36 writing layer sha256:c904c85f639aee0f916c284937bc4e2cf9cf883319456c954a23f6f9e4c99f3e
#36 writing layer sha256:c904c85f639aee0f916c284937bc4e2cf9cf883319456c954a23f6f9e4c99f3e 0.1s done
#36 writing layer sha256:ed8d08b30450c12a72939d3227c817154531f453653453d46d7d2a0708f4497d
#36 writing layer sha256:ed8d08b30450c12a72939d3227c817154531f453653453d46d7d2a0708f4497d 0.1s done
#36 writing config sha256:69263ed0883b70454c6e244d5c1421d71771112682c658d432e18ef04316f071
#36 writing config sha256:69263ed0883b70454c6e244d5c1421d71771112682c658d432e18ef04316f071 0.1s done
#36 writing manifest sha256:2131269077258d71142daae05596fe3087b0c3d9844fda38807b72946fe7f139
#36 preparing build cache for export 1.7s done
#36 writing manifest sha256:2131269077258d71142daae05596fe3087b0c3d9844fda38807b72946fe7f139 0.5s done
#36 DONE 1.7s

#37 [pronto-amd64] exporting content cache
#37 preparing build cache for export 1.8s done
#37 writing layer sha256:17b2a3a3ef6eecb7f8a8575ebae60d72ab5b63a312eaf2da612c224585984271 0.2s done
#37 writing layer sha256:43449cce2b2af7cae5eefe22798316153ac88417fa6dcf9f0dc5390d209ed367 0.2s done
#37 writing layer sha256:4d3d25e339e798c7606304d1a00cff7f31a5876c66dec58866278476db2c83ea 0.1s done
#37 writing layer sha256:73b3bf5b28e8f5da8ff5a7a25b3854fb94ab5da530332903411313878ca778f2 0.1s done
#37 writing layer sha256:75dd67f3ccc89561ff13877120b513941dc1e14c5ab32c4ce6d5585ee6ef9189 0.1s done
#37 writing layer sha256:8543f06a28fde480630d4b61f2b1a367b6e3bd3684cd09179aa18914936d957c 0.1s done
#37 writing layer sha256:8a49fdb3b6a5ff2bd8ec6a86c05b2922a0f7454579ecc07637e94dfd1d0639b6 0.1s done
#37 writing layer sha256:de1335ad17a0a796f474e851b11015d08d6cdb44bb1826b942e981a8ae9297fa 0.1s done
#37 writing config sha256:acb6ae06fa2cb1c239310442abded55e6ac360dafac4c3bd28b4483838d24b85 0.1s done
#37 writing manifest sha256:7003bd2facf79ecb7a7dfc6c808b9d5166c1620218dd0b1ecf412c6ea6ac68b8 0.5s done
#37 DONE 1.8s

#38 [pronto] exporting content cache
#38 writing layer sha256:08409d4172603f40b56eb6b76240a1e6bd78baa0e96590dc7ff76c5f1a093af2 0.1s done
#38 writing layer sha256:17b2a3a3ef6eecb7f8a8575ebae60d72ab5b63a312eaf2da612c224585984271 0.1s done
#38 writing layer sha256:335fc4470cecf31504e0536041c5daec3a87d7803e76d8508296a808a37b46ec 0.1s done
#38 writing layer sha256:43449cce2b2af7cae5eefe22798316153ac88417fa6dcf9f0dc5390d209ed367 0.1s done
#38 writing layer sha256:4d3d25e339e798c7606304d1a00cff7f31a5876c66dec58866278476db2c83ea 0.1s done
#38 writing layer sha256:59b492009e1697af0a35d8344ae929d156f9d5e2d3ba72b1853aaee24f6bfa95 0.1s done
#38 writing layer sha256:73b3bf5b28e8f5da8ff5a7a25b3854fb94ab5da530332903411313878ca778f2 0.1s done
#38 writing layer sha256:75acee0b64a8e73a13f847e35298675dcac4bc0f73e630db4f236ca0a9f6f1c1 0.1s done
#38 writing layer sha256:75dd67f3ccc89561ff13877120b513941dc1e14c5ab32c4ce6d5585ee6ef9189 0.1s done
#38 writing layer sha256:7e668614de827af0be2644643a7bc9c6546f6bc38106ab1819cd7e5a6376ff40 0.1s done
#38 writing layer sha256:831bb4769cd713dde88dccba20919c346eb5a49bdeebfb7b2d034c667bec9ea3 0.1s done
#38 writing layer sha256:8543f06a28fde480630d4b61f2b1a367b6e3bd3684cd09179aa18914936d957c 0.1s done
#38 writing layer sha256:8a49fdb3b6a5ff2bd8ec6a86c05b2922a0f7454579ecc07637e94dfd1d0639b6 0.1s done
#38 writing layer sha256:c904c85f639aee0f916c284937bc4e2cf9cf883319456c954a23f6f9e4c99f3e 0.1s done
#38 writing layer sha256:de1335ad17a0a796f474e851b11015d08d6cdb44bb1826b942e981a8ae9297fa 0.1s done
#38 writing layer sha256:ed8d08b30450c12a72939d3227c817154531f453653453d46d7d2a0708f4497d 0.1s done
#38 writing config sha256:c023a85ad692b3165bd9965a2564d4d92965007a5bfc0f869bbc2a38c8667934 0.1s done
#38 writing manifest sha256:64639ff66c3d36fdc54b24782391e87affb5e520435c41dfcc43cbe75bd5bda1
#38 preparing build cache for export 2.6s done
#38 writing manifest sha256:64639ff66c3d36fdc54b24782391e87affb5e520435c41dfcc43cbe75bd5bda1 0.5s done
#38 DONE 2.6s

### Additional info

_No response_

[emmahsax]

I stand corrected, the way around this issue is using --provenance false (#1507 (comment)). My apologies for the kinda duplicated issue.