[Docker Engine] Add multi-platform image support (migrate to the containerd content store and snapshotters)

[chris-crone]

Tell us about your request
Docker started the containerd project with the plan for the Docker Engine to be migrated to use it. The Docker Engine already uses containerd to manage container lifecycle but it still uses its own implementation for image and filesystem management.

Docker is working with the Moby project contributors on completing the migration and moving Docker Engine to use containerd's content store and snapshotters. This will enable low-level features like multi-platform image support. It will also reduce the code complexity of the Docker Engine.

Which service(s) is this request for?
Docker Engine which is a core part of Docker Desktop and Docker for Linux.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
There are many user facing features that this will enable. To list a few:

• Native multi-platform image build support (i.e.: do not require containerized BuildKit)
• Resource namespacing
• More OCI support
• Support for containerd’s snapshotters

[sam-thibault]

The containerd image store feature is now in beta in Docker Desktop 4.12.0 🎉
More info is available here: https://docs.docker.com/desktop/containerd/

We’d love to hear from you! Please feel free to provide us feedback or report any bugs you may find through the issues tracker on the feedback form.

[NiklasBr]

Building with --platform works, but gives this warning:

WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

Using--load as suggested by the warning results in this error:

ERROR: docker exporter does not currently support exporting manifest lists

Can you explain what is supposed to happen here and why following the instructions does not work?

[TBBle]

Which instructions were you following? https://docs.docker.com/desktop/containerd/#building-multi-platform-images ?

There's not really that much detail in your comment, I'd suggest opening a Docker Desktop bug report for the failure you're seeing, and hence (by populating the issue template) including details like the commands you ran, the Dockerfile, and the versions of the various components in-use.

I suspect that something in your configuration (the use of the buildx Docker container driver, perhaps) is incompatible with the containerd image store for the Docker component versions currently installed, and hence guess that you will need to manually upgrade to buildx 0.10 and/or wait for a new release of Docker Desktop to include buildx 0.10 and hence BuildKit 0.11, which I think fixes this particular incompatibility.

To be clear, that's several guesses stacked together, so I still recommend opening a new bug report for the problem with all the details on the likely chance I'm giving incorrect or faulty suggestions.

[NiklasBr]

Which instructions were you following?

These instructions:

WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

So, I append --load to the command, and then I get this message instead:

ERROR: docker exporter does not currently support exporting manifest lists

[TBBle]

Okay, if you weren't following a specific tutorial (sorry, I misunderstood) then the answer is there in the output: If using the docker-container driver, you must provide either --push or --load in order to get the built manifest or manifest list out of the build cache (this is true irrespective of the containerd integration, and is as-expected), and the docker exporter (triggered by --load) does not handle manifest lists in that version of buildx (which is also not related to the containerd image store; the cause here is that without the containerd content store, docker doesn't know how to store manifest lists at all, so the docker exporter didn't support them either).

Basically, you've hit docker/buildx#59 (comment). I still guess that the fix is to enable both the containerd content store beta (as you are) so that manifest lists can be stored and also the newer version of buildx I mentioned earlier, which knows how to store manifest lists in Docker. (Note: I have not tested this, just looked at some PRs)

[jacobwgillespie]

Does any version of buildx support multi-platform exports with the beta containerd store? My understanding is that exporting an image to docker (--load) still explicitly blocks exporting multi-platform images:

https://github.com/docker/buildx/blob/fb27e3f919dcbf614d7126b10c2bc2d0b1927eb6/build/build.go#L507-L509

[TBBle]

Drat. That's why it was a guess. 🐍👀.

My guess was based on moby/buildkit#3389 having been merged for BuildKit 0.11, and BuildKit 0.11 having been vendored into buildx 0.10. However, on closer inspection, it (and moby/buildkit#3380 which is how I got there) are not solving this issue, but were preventing the same error message being triggered by other changes.

So I guess there's still some work to go in buildx to be able to push a manifest list via the Docker exporter (--load) into the content store when containerd is enabled, and perhaps even work needed in the Docker Engine side and/or BuildKit too.

I wasn't able to find an open issue tracking this: docker/buildx#59 was the first one, and it was closed by documenting the limitation with a direction towards this roadmap issue as the blocker. docker/buildx#166 is also close, but it is related to the image exporter used by the docker driver, and that's apparently working with this beta already.

Again, it's possible that a specific tracking issue exists, and I just failed to locate it.

If I'm reading this correctly, then BuildKit would also reject --load (shorthand for --output=type=docker) if the relevant check was removed from buildx. That block is only hit for the Docker on-disk image format, as it looks like the OCI on-disk image format does support manifest lists (see this block) and so the required work is make this code edit: mistake, it should be this branch work for an OCI format tarball (probably needs to be done in Docker Engine? Maybe already works?) and change this block a bit so that you don't need to do --output=type=oci,dest= (or whatever trick is needed to get an empty-string "dest" value into the CLI parser) to cause the previous-linked block to be hit in OCI mode.

And just to clarify, this is only affecting the docker-container buildx driver. The docker buildx driver uses the image exporter instead, which should already work for the equivalent of --load with the containerd beta storage as demonstrated at https://docs.docker.com/desktop/containerd/#building-multi-platform-images.

[NiklasBr]

Excellent write-up @TBBle, thanks! So, just for the record, this is the current setup:

docker version
Client:
 Cloud integration: v1.0.29
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:28:41 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.16.1 (95567)
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:25:43 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.14
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker buildx version
github.com/docker/buildx v0.10.0 876462897612d36679153c3414f7689626251501

And the failing command is:

docker buildx build --platform=linux/amd64,linux/arm64 --no-cache --pull--load --tag ourregistry/ourproject:ourrevision

[TBBle]

The only other thing I'd suggest, in order to validate one of my assumptions, is the output of

docker buildx ls

Depending on the details of your setup and Dockerfile,

docker buildx --builder default build --platform=linux/amd64,linux/arm64 --no-cache --pull --load --tag ourregistry/ourproject:ourrevision

might work, but I don't know for sure. The docker driver is documented to not support multi-platform, but the containerd image store docs appear to show it working, although it's not explicitly using the docker driver, and the Dockerfile there does not require a functional RUN environment.

It also might be possible replace --load in your command with --output=type=oci,dest= or --output=type=oci,dest='' or something similar, but that'll depend on the CLI parser, which I haven't checked. If that worked, it would bypass both of the current error-outs, as they are specifically tied to type=docker. That's assuming there's no missing or incomplete code in that codepath.

[NiklasBr]

docker buildx ls
NAME/NODE       DRIVER/ENDPOINT             STATUS  BUILDKIT PLATFORMS
epic_wilbur *   docker-container                             
  epic_wilbur0  unix:///var/run/docker.sock running v0.11.0  linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
default         docker                                       
  default       default                     running 20.10.22 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
desktop-linux   docker                                       
  desktop-linux desktop-linux               running 20.10.22 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

Both docker buildx --builder default … and docker buildx --builder epic_wilbur … fails with the same error ERROR: docker exporter does not currently support exporting manifest lists.

[TBBle]

Thought I should probably make some effort on my own behalf.

On a reset-to-defaults and then containerd-beta-activated Docker Desktop for Windows:

> docker buildx ls
NAME/NODE       DRIVER/ENDPOINT STATUS  BUILDKIT                         PLATFORMS
default *       docker
  default       default         running 22.06.0-beta.0-913-g6be7129598.m linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
desktop-linux   docker
  desktop-linux desktop-linux   running 22.06.0-beta.0-913-g6be7129598.m linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64

So looks like the docker driver is multiplatform now? From this state, the trivial Dockerfile from the documentation worked.

So to recreate the observed issue:

docker buildx create --name docker-container --driver docker-container
docker buildx --builder docker-container build --platform linux/amd64,linux/arm64 -t hello-friends .

indeed produces

WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

and adding --load or --output=type=docker produces

ERROR: docker exporter does not currently support exporting manifest lists

and then trying the workaround:

docker buildx --builder docker-container build --output=type=oci,dest= --platform linux/amd64,linux/arm64 -t hello-friends .

produces the following error:

ERROR: failed to solve: method /moby.filesync.v1.FileSend/diffcopy not supported by the client

which is hitting this code from here, as the next expected line was "sending tarball". So either this code-path is not yet implemented, or I've done something that broke it. (I'm not sure which code is "the client" here, my guess is buildx, and maybe it's doing something specifically in docker output mode to implement FileSend that isn't implemented in oci mode, e.g. this code).

The same error happens if you use a single platform with --output=type=oci,dest=, while --output=type=docker,dest= in that case happily exports. So not being able to load oci-format images is not a multi-platform-specific problem, but only matters in multi-platform circumstances, because docker-format images are feasible in the single-platform case.

And for sanity, actually adding a filename for all the following cases exports fine, so the limitation is specifically for loading back into Docker automatically.

• docker buildx --builder docker-container build --output=type=docker,dest=bob.docker.tar --platform linux/amd64 -t hello-friends .
• docker buildx --builder docker-container build --output=type=oci,dest=bob.oci.tar --platform linux/amd64 -t hello-friends .
• docker buildx --builder docker-container build --output=type=oci,dest=bob.oci.multi.tar --platform linux/amd64,linux/arm64 -t hello-friends .

And interestingly, the output of all three appears to work with docker load -i, so that's another possible workaround, if doing the docker load manually is a feasible option. (Although I haven't checked that the actual loaded images are rational, as I docker image isn't dealing with manifest lists correctly, and I don't know where the containerd socket is for ctr).

---

So in the end, a workaround for --load for multi-platform builds with the docker-container buildx driver:

docker buildx --builder docker-container build --quiet --output=type=oci --platform linux/amd64,linux/arm64 -t hello-friends . | docker load

appears to work in Git Bash (MSYS2) but not Powershell 7.3.0; some kind of unclean binary pipeline going on in the latter, I fear.

(--quiet is needed, it looks like docker buildx build is putting stuff on stdout which interferes with the output from the OCI exporter... But that's a different issue. moby/buildkit#1186 suggests that BuildKit isn't doing this, so it must be buildx's reporting of the BuildKit output. Poking around a bit, this might be a Windows issue, since in MSYS2, 2>/dev/null works without --quiet, suggesting that | even in Git Bash is picking up both stdout and stderr. I'm sure that's not how it's supposed to work. And redirecting either one stops the build output, so perhaps buildx is doing some output autodetection and it's doing the wrong thing here.)

Bonus edit: The redirect thing is a Windows platform (or Windows buildx build) issue, as

wsl -e sh -c 'docker buildx build --output=type=oci --platform linux/amd64,linux/arm64 -t hello-friends . | docker load'

doesn't suffer this issue.

[TBBle]

Both docker buildx --builder default … and docker buildx --builder epic_wilbur … fails with the same error ERROR: docker exporter does not currently support exporting manifest lists.

Oh, with --builder default, take out --load, that's overriding the exporter from image to docker.

[TBBle]

Based on my last batch of testing, I went ahead and opened docker/buildx#1522 for #371 (comment) and ensuing discussion, since it looks like all the plumbing is there in BuildKit and Docker for something like --load that supports multi-platform images, it's just the glue in buildx that's missing.

[zhixinwen]

I am compiling x86 on my M2 machine, and I enabled containerd on Docker.
This gives me error parent snapshot sha256:b6bab4b00958273f1ba360d4fbb20cf6b4ee829508f4b4985e16d51a28b3a0f0 does not exist: not found

[AaronFriel]

@zhixinwen The containerd backend for Docker is still considered experimental, so I'm afraid that's likely an issue with Docker for Mac.

[NiklasBr]

Any updates here?

[NiklasBr]

What's the timeline until we will be able to use multi-platform builds using docker buildx build --platform=linux/amd64,linux/arm64 it's been a while since you wrote about it at https://www.docker.com/blog/multi-platform-docker-builds/

[savannahostrowski]

Containerd image store support is now generally available with the release of Moby 25 (shipped in DD 4.27). There will be a blog post that covers the DD 4.27 soon. Docs can be found at https://docs.docker.com/desktop/containerd/.

[NiklasBr]

Looking forward to additional documentation because the link did not work with containerd and SBOM enabled:

❯ docker buildx create --bootstrap

[+] Building 5.5s (1/1) FINISHED                                                                                                                                                                                                                                                      
 => [internal] booting buildkit                                                                                                                                                                                                                                                  5.5s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                                                                                                                               4.9s
 => => creating container buildx_buildkit_pedantic_brattain0                                                                                                                                                                                                                     0.6s
pedantic_brattain
❯ docker info --format="{{ .Driver }}"

overlay2
❯ docker buildx build --platform=linux/arm64,linux/amd64 --tag=user/containerd-multiplatform ./docker/proxy-nginx
[+] Building 0.0s (0/0)                                                                                                                                                                                                                                          docker:desktop-linux
ERROR: Multi-platform build is not supported for the docker driver.

[dvdksn]

hi @NiklasBr. From your docker info command it looks like you aren't using the containerd storage backend.

docker info --format="{{ .Driver }}"
overlay2

overlay2 is the default graph driver. If you enable the containerd image store feature in Docker Desktop and try again, building multi-platform images and loading them to the local image store should work fine, even with the default buildx driver.

[TBBle]

Assuming there were more lines to that error

Switch to a different driver, or turn on the containerd image store, and try again.
Learn more at https://docs.docker.com/go/build-multi-platform/

then it appears your docker buildx doesn't believe you're using the containerd image store.

If no such lines appeared, then I have no idea what version of buildx you're using, but I can't see any way in the buildx 0.12.1 release to generate that error without those extra lines, and 0.11 and older had different phrasing for that error.

---

I just upgraded my Docker Desktop to 4.27.1 (I already had the containerd image store enabled from the beta), and trivially built the container from https://docs.docker.com/get-started/02_our_app/ using the given command:

git clone https://github.com/docker/getting-started-app.git
cd .\getting-started-app
# Populate the Dockerfile from the example
gvim .\Dockerfile
docker buildx build --platform=linux/arm64,linux/amd64 --tag=user/containerd-multiplatform .

and it appears to have worked as expected:

> docker image list
REPOSITORY                      TAG       IMAGE ID       CREATED              SIZE
user/containerd-multiplatform   latest    03d0489bce84   About a minute ago   86.6MB
user/containerd-multiplatform   latest    03d0489bce84   2 minutes ago        352MB