docker manifest does not work with output from buildx v0.10

[debu99]

How to reproduce:
$ uname -a
Darwin xxxxxx 21.6.0 Darwin Kernel Version 21.6.0: Thu Sep 29 20:13:46 PDT 2022; root:xnu-8020.240.7~1/RELEASE_ARM64_T8101 arm64

$ docker buildx version
github.com/docker/buildx v0.10.0 8764628

$ docker buildx create --driver=docker-container --name=buildkit-builder --use
$ docker buildx inspect --bootstrap
$ docker buildx ls

$docker buildx build -t yourname/test:2-arm64 --platform=linux/arm64 --push .

$ docker manifest inspect yourname/test:2-arm64
no such manifest: docker.io/yourname/test:2-arm64

[crazy-max]

Since buildx 0.10, a minimal provenance attestation is also pushed (unknown/unknown):

$ docker buildx build --tag crazymax/test --push .
...
#7 exporting to image
#7 exporting layers
#7 exporting layers 0.2s done
#7 exporting manifest sha256:b52fc2edb674b9a3fc945addd13e468984c91058eba02dd45c9827df5299dddb 0.0s done
#7 exporting config sha256:0355c7b04a8e614b6de7c038a04a063dab0655c77280e54a488979bf52b28b43 0.0s done
#7 exporting attestation manifest sha256:3032911c04a048a7e1d148e6505ed00d5690d8ae12fef878ad05ce77311afa4f 0.0s done
#7 exporting manifest list sha256:055420aed05f4dcfeab12c64dbb4dbcdfa28bd9f9647fe1a63e0813c3694e14f
#7 exporting manifest list sha256:055420aed05f4dcfeab12c64dbb4dbcdfa28bd9f9647fe1a63e0813c3694e14f 0.0s done
#7 pushing layers
#7 pushing layers 1.5s done
#7 pushing manifest for docker.io/crazymax/test:latest@sha256:055420aed05f4dcfeab12c64dbb4dbcdfa28bd9f9647fe1a63e0813c3694e14f
#7 pushing manifest for docker.io/crazymax/test:latest@sha256:055420aed05f4dcfeab12c64dbb4dbcdfa28bd9f9647fe1a63e0813c3694e14f 1.5s done
#7 DONE 3.3s

But don't think docker manifest supports OCI manifest. Debugging this command:

$ docker --debug manifest inspect crazymax/test
DEBU[0000] endpoints for docker.io/crazymax/test:latest: [{false https://registry-1.docker.io v2 false true true 0xc0003fd380}] 
DEBU[0000] not continuing on error (*errors.fundamental) docker.io/crazymax/test:latest is a manifest list 
DEBU[0000] endpoints for docker.io/crazymax/test:latest: [{false https://registry-1.docker.io v2 false true true 0xc00015ca80}]
DEBU[0002] continuing on error (errcode.Errors) manifest unknown: OCI manifest found, but accept header does not support OCI manifests 
no such manifest: docker.io/crazymax/test:latest

If you disable provenance it should work:

$ docker buildx build --tag crazymax/test --push --provenance=false .
#7 exporting to image
#7 exporting layers
#7 exporting layers done
#7 exporting manifest sha256:cf680f47b231ad1be1443c475cd45510213e7c05b839ee194d6593bb06ff8de9 0.0s done
#7 exporting config sha256:0355c7b04a8e614b6de7c038a04a063dab0655c77280e54a488979bf52b28b43 done
#7 pushing layers
#7 pushing layers 0.6s done
#7 pushing manifest for docker.io/crazymax/test:latest@sha256:cf680f47b231ad1be1443c475cd45510213e7c05b839ee194d6593bb06ff8de9
#7 pushing manifest for docker.io/crazymax/test:latest@sha256:cf680f47b231ad1be1443c475cd45510213e7c05b839ee194d6593bb06ff8de9 0.5s done
#7 DONE 1.1s

$ docker --debug manifest inspect crazymax/test
{
        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
        "schemaVersion": 2,
        "config": {
                "mediaType": "application/vnd.docker.container.image.v1+json",
                "digest": "sha256:0355c7b04a8e614b6de7c038a04a063dab0655c77280e54a488979bf52b28b43",
                "size": 1341
        },
        "layers": [
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "digest": "sha256:2461e8255644c236fb21f20d482f3ca7017d69b52f3177737502fbcfbdc82f55",
                        "size": 2588339
                },
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "digest": "sha256:dd0d0c2c5c3d6127c4e2126db4658d49f9f0336f2f197f0ff48fb3ee89604719",
                        "size": 154
                },
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "digest": "sha256:6d769fe3e59f51cf3f930102006edd896f257e4103cccd176fda77c5b02beaeb",
                        "size": 159
                }
        ]
}

cc @tonistiigi @jedevc

[jedevc]

Copying my workaround suggestions from another issue:

1. Use docker buildx imagetools inspect --raw instead of docker manifest inspect - it should work similarly, and supports all the different media types in the registry. See the docs for docker buildx imagetools inspect..
   Hopefully, this will just work, and should be a drop-in replacement!
2. Set oci-mediatypes=false in your --output flag (to use the docker distribution manifest list instead of an OCI index).
   This might cause some issues with the generated provenance, which means you'd probably prefer 3 instead.
3. Set --provenance=false to not generate the provenance (which is what causes the multi-platform index to be generated, even for a single platform).
   Just remove the generated provenance entirely, this means that only a single manifest is created, no index needed, which sidesteps the problem.

[item4]

If I use imagetools instead of default one, how can I use manifest push with imagetools?

[jedevc]

To merge manifests using imagetools you can use imagetools create to construct and push the manifest all in one step. If you're doing multi-platform builds, you can also use the new --platform flag to handle this automatically, without needing to have separate manifest merging logic, etc: https://docs.docker.com/build/building/multi-platform/.

[jedevc]

Looks related to moby/moby#43126 👀

[ddelange]

Hi 👋

I'd like to use docker manifest inspect -v to sum up layer sizes: https://stackoverflow.com/a/73108928/5511061

When --provenance is specified during build/push, I'll get a no such manifest error when querying docker manifest inspect -v for any image that includes an attestation manifest.

Additionally, docker buildx imagetools inspect --raw does not include the verbosity needed to sum up layer sizes.

Leaving this here as suggested in docker/build-push-action#764