Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker-in-docker fail to start with amd64 image on macOS M1 #219

Open
zephyros-dev opened this issue Oct 13, 2022 · 8 comments
Open

docker-in-docker fail to start with amd64 image on macOS M1 #219

zephyros-dev opened this issue Oct 13, 2022 · 8 comments
Labels
documentation Improvements or additions to documentation

Comments

@zephyros-dev
Copy link
Contributor

zephyros-dev commented Oct 13, 2022

Description

  • dockerd installed using docker-in-docker features fails to run on macOS M1 machine with amd64 image
  • The features works normally using arm64 image

Steps to reproduce

  1. Build .devcontainer from the following file:
    devcontainer.json
{
    "name": "DIND container",
    "dockerFile": "Dockerfile",
    "containerUser": "node",
    "remoteUser": "node",
    "features": {
        "ghcr.io/devcontainers/features/docker-from-docker:1": {},
    }
}

Dockerfile

FROM --platform=linux/amd64 mcr.microsoft.com/devcontainers/typescript-node:16

Issues

  • dockerd is unable to start. Here's the logs if I run sudo dockerd:
INFO[2022-10-13T03:09:21.395301838Z] Starting up                                  
INFO[2022-10-13T03:09:21.412464546Z] libcontainerd: started new containerd process  pid=3080
INFO[2022-10-13T03:09:21.414531504Z] parsed scheme: "unix"                         module=grpc
INFO[2022-10-13T03:09:21.414705338Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-10-13T03:09:21.415490796Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-10-13T03:09:21.415749879Z] ClientConn switching balancer to "pick_first"  module=grpc
WARN[0000] containerd config version `1` has been deprecated and will be removed in containerd v2.0, please switch to version `2`, see https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#version-header 
INFO[2022-10-13T03:09:21.657304379Z] starting containerd                           revision=a17ec496a95e55601607ca50828147e8ccaeebf1 version=1.5.13+azure-2
INFO[2022-10-13T03:09:21.775481546Z] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
INFO[2022-10-13T03:09:21.776213463Z] loading plugin "io.containerd.snapshotter.v1.aufs"...  type=io.containerd.snapshotter.v1
INFO[2022-10-13T03:09:21.777676171Z] skip loading plugin "io.containerd.snapshotter.v1.aufs"...  error="aufs is not supported (modprobe aufs failed: exec: \"modprobe\": executable file not found in $PATH \"\"): skip plugin" type=io.containerd.snapshotter.v1
INFO[2022-10-13T03:09:21.778019421Z] loading plugin "io.containerd.snapshotter.v1.btrfs"...  type=io.containerd.snapshotter.v1
INFO[2022-10-13T03:09:21.778673296Z] skip loading plugin "io.containerd.snapshotter.v1.btrfs"...  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2022-10-13T03:09:21.778758379Z] loading plugin "io.containerd.snapshotter.v1.devmapper"...  type=io.containerd.snapshotter.v1
WARN[2022-10-13T03:09:21.779001879Z] failed to load plugin io.containerd.snapshotter.v1.devmapper  error="devmapper not configured"
INFO[2022-10-13T03:09:21.779105296Z] loading plugin "io.containerd.snapshotter.v1.native"...  type=io.containerd.snapshotter.v1
INFO[2022-10-13T03:09:21.779538629Z] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  type=io.containerd.snapshotter.v1
INFO[2022-10-13T03:09:21.781382129Z] loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
INFO[2022-10-13T03:09:21.781861588Z] skip loading plugin "io.containerd.snapshotter.v1.zfs"...  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2022-10-13T03:09:21.781940254Z] loading plugin "io.containerd.metadata.v1.bolt"...  type=io.containerd.metadata.v1
WARN[2022-10-13T03:09:21.782172754Z] could not use snapshotter devmapper in metadata plugin  error="devmapper not configured"
INFO[2022-10-13T03:09:21.782304463Z] metadata content store policy set             policy=shared
INFO[2022-10-13T03:09:21.785518338Z] loading plugin "io.containerd.differ.v1.walking"...  type=io.containerd.differ.v1
INFO[2022-10-13T03:09:21.785782713Z] loading plugin "io.containerd.gc.v1.scheduler"...  type=io.containerd.gc.v1
INFO[2022-10-13T03:09:21.787059838Z] loading plugin "io.containerd.service.v1.introspection-service"...  type=io.containerd.service.v1
INFO[2022-10-13T03:09:21.788375129Z] loading plugin "io.containerd.service.v1.containers-service"...  type=io.containerd.service.v1
INFO[2022-10-13T03:09:21.788565129Z] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2022-10-13T03:09:21.788827463Z] loading plugin "io.containerd.service.v1.diff-service"...  type=io.containerd.service.v1
INFO[2022-10-13T03:09:21.789024129Z] loading plugin "io.containerd.service.v1.images-service"...  type=io.containerd.service.v1
INFO[2022-10-13T03:09:21.789192338Z] loading plugin "io.containerd.service.v1.leases-service"...  type=io.containerd.service.v1
INFO[2022-10-13T03:09:21.789330088Z] loading plugin "io.containerd.service.v1.namespaces-service"...  type=io.containerd.service.v1
INFO[2022-10-13T03:09:21.789434338Z] loading plugin "io.containerd.service.v1.snapshots-service"...  type=io.containerd.service.v1
INFO[2022-10-13T03:09:21.789628213Z] loading plugin "io.containerd.runtime.v1.linux"...  type=io.containerd.runtime.v1
INFO[2022-10-13T03:09:21.790231171Z] loading plugin "io.containerd.runtime.v2.task"...  type=io.containerd.runtime.v2
INFO[2022-10-13T03:09:21.791182546Z] loading plugin "io.containerd.monitor.v1.cgroups"...  type=io.containerd.monitor.v1
INFO[2022-10-13T03:09:21.793136796Z] loading plugin "io.containerd.service.v1.tasks-service"...  type=io.containerd.service.v1
INFO[2022-10-13T03:09:21.793658754Z] loading plugin "io.containerd.internal.v1.restart"...  type=io.containerd.internal.v1
INFO[2022-10-13T03:09:21.795432421Z] loading plugin "io.containerd.grpc.v1.containers"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.795857504Z] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.796000879Z] loading plugin "io.containerd.grpc.v1.diff"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.796269838Z] loading plugin "io.containerd.grpc.v1.events"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.796690879Z] loading plugin "io.containerd.grpc.v1.healthcheck"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.796906588Z] loading plugin "io.containerd.grpc.v1.images"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.797034796Z] loading plugin "io.containerd.grpc.v1.leases"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.797202588Z] loading plugin "io.containerd.grpc.v1.namespaces"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.797557629Z] loading plugin "io.containerd.internal.v1.opt"...  type=io.containerd.internal.v1
INFO[2022-10-13T03:09:21.799339254Z] loading plugin "io.containerd.grpc.v1.snapshots"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.799557879Z] loading plugin "io.containerd.grpc.v1.tasks"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.799694629Z] loading plugin "io.containerd.grpc.v1.version"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.799756796Z] loading plugin "io.containerd.grpc.v1.introspection"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.805491421Z] serving...                                    address=/var/run/docker/containerd/containerd-debug.sock
INFO[2022-10-13T03:09:21.806265588Z] serving...                                    address=/var/run/docker/containerd/containerd.sock.ttrpc
INFO[2022-10-13T03:09:21.807189921Z] serving...                                    address=/var/run/docker/containerd/containerd.sock
INFO[2022-10-13T03:09:21.821722546Z] containerd successfully booted in 0.184633s  
INFO[2022-10-13T03:09:21.887599004Z] parsed scheme: "unix"                         module=grpc
INFO[2022-10-13T03:09:21.887708796Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-10-13T03:09:21.887778879Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-10-13T03:09:21.887836546Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-10-13T03:09:21.893983879Z] parsed scheme: "unix"                         module=grpc
INFO[2022-10-13T03:09:21.894081379Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-10-13T03:09:21.894154963Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-10-13T03:09:21.894206296Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-10-13T03:09:21.907646796Z] [graphdriver] using prior storage driver: overlay2 
INFO[2022-10-13T03:09:21.925756838Z] Loading containers: start.                   
WARN[2022-10-13T03:09:21.946484213Z] Running iptables --wait -t nat -L -n failed with message: `iptables v1.8.7 (legacy): can't initialize iptables table `nat': iptables who? (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.`, error: exit status 3 
INFO[2022-10-13T03:09:22.252070713Z] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
INFO[2022-10-13T03:09:22.253173921Z] stopping healthcheck following graceful shutdown  module=libcontainerd
INFO[2022-10-13T03:09:22.254884796Z] stopping event stream following graceful shutdown  error="context canceled" module=libcontainerd namespace=plugins.moby
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.7 (legacy): can't initialize iptables table `nat': iptables who? (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
 (exit status 3)

Information

  • macOS version: Monterey 12.6
  • Docker Desktop version: 4.12.0 (85629)
  • Docker Engine version: 20.10.17
@zephyros-dev zephyros-dev changed the title docker-in-docker fail to start on macOS docker-in-docker fail to start if bind mount localWorkspaceFolder on macOS Oct 13, 2022
@zephyros-dev zephyros-dev changed the title docker-in-docker fail to start if bind mount localWorkspaceFolder on macOS docker-in-docker fail to start if bind mount localWorkspaceFolder on macOS Oct 13, 2022
@zephyros-dev zephyros-dev changed the title docker-in-docker fail to start if bind mount localWorkspaceFolder on macOS docker-in-docker fail to start with amd64 image on macOS M1 Oct 13, 2022
@zephyros-dev zephyros-dev reopened this Oct 13, 2022
@zephyros-dev zephyros-dev reopened this Oct 26, 2022
@bamurtaugh
Copy link
Member

Thanks for reporting. There seem to have been some transient package issues around this time. Are you still experiencing this issue?

@Chuxel @chrmarti I see containerd in the logs above. Would microsoft/vscode-remote-release#6014 pose issues with using containerd?

@Chuxel
Copy link
Member

Chuxel commented Oct 26, 2022

No this is a problem with mixing architectures - x86 and arm64.

Cross-architecture emulation has a number of limitations. I am guessing that prevents access to the needed underlying devices for it to work. This may not be possible at all.

Here's how docker talks about emulation:

Some images do not support the ARM64 architecture. You can add --platform linux/amd64 to run (or build) an Intel image using emulation.

However, attempts to run Intel-based containers on Apple silicon machines under emulation can crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs (inotify) do not work under qemu emulation. Even when the containers do run correctly under emulation, they will be slower and use more memory than the native equivalent.

In summary, running Intel-based containers on Arm-based machines should be regarded as “best effort” only. We recommend running arm64 containers on Apple silicon machines whenever possible, and encouraging container authors to produce arm64, or multi-arch, versions of their containers. We expect this issue to become less common over time, as more and more images are rebuilt supporting multiple architectures.

@zephyros-dev
Copy link
Contributor Author

zephyros-dev commented Oct 27, 2022

No this is a problem with mixing architectures - x86 and arm64.

Cross-architecture emulation has a number of limitations. I am guessing that prevents access to the needed underlying devices for it to work. This may not be possible at all.

Here's how docker talks about emulation:

Some images do not support the ARM64 architecture. You can add --platform linux/amd64 to run (or build) an Intel image using emulation.

However, attempts to run Intel-based containers on Apple silicon machines under emulation can crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs (inotify) do not work under qemu emulation. Even when the containers do run correctly under emulation, they will be slower and use more memory than the native equivalent.

In summary, running Intel-based containers on Arm-based machines should be regarded as “best effort” only. We recommend running arm64 containers on Apple silicon machines whenever possible, and encouraging container authors to produce arm64, or multi-arch, versions of their containers. We expect this issue to become less common over time, as more and more images are rebuilt supporting multiple architectures.

According to this issue docker/for-mac#6284 (comment), it seems like dind will only works on arm64 image since the M1 amd64 emulator does not support ip routing. While this issue is not related to devcontainer, I think adding a note to the README would be useful for future user. What do you think?

@Chuxel Chuxel added the documentation Improvements or additions to documentation label Oct 27, 2022
@Chuxel
Copy link
Member

Chuxel commented Oct 27, 2022

Yep, that seems reasonable. Adding a NOTES.md to the feature will cause it to appear in the README.

@ci-vamp
Copy link

ci-vamp commented Jan 31, 2023

it may be worth updating this note. as of macOS ventura (13.1) and docker desktop 4.16.1 there is an option to emulate using rosetta in the Features in development options of docker desktop.

docker desktop macOS features in development docs

i almost gave up when reading this message and thread but decided to try it anyways (since it was just released a few weeks ago).

its working great! just had to mount my docker socket from host to devcontainer and the rest worked.

reference for others:

Dockerfile

FROM --platform=linux/amd64 mcr.microsoft.com/devcontainers/base:ubuntu

RUN ENABLE_NONROOT_DOCKER=false \
  /bin/bash \
  -c "$(curl -fsSL https://raw.githubusercontent.com/devcontainers/features/main/src/docker-outside-of-docker/install.sh)"

...

devcontainer.json

...
  "mounts": [
    // mount the host docker socket so docker can be used in the devcontainer
    "source=/var/run/docker.sock,target=/var/run/docker-host.sock,type=bind",
  ],
...

@antspy
Copy link

antspy commented Nov 5, 2024

@ci-vamp Thank you for your message! This still doesn't work for me, unfortunately. I install docker-in-docker using devcontainer features, I don't know if that makes a difference:

// devcontainer.json
"features": {
    "ghcr.io/devcontainers/features/docker-in-docker:2": {},

I have added the mounts and I have enabled the Allow the default Docker socket to be used (requires password) setting in docker desktop, but I still see the same error:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer: unknown.

@dschneiderch
Copy link

dschneiderch commented Nov 6, 2024

@antspy i think you need "remoteUser": "root" as well.
And, I had to change the mount so:
"source=/var/run/docker.sock,target=/var/run/docker-host.sock,type=bind",

@antspy
Copy link

antspy commented Nov 10, 2024

@dschneiderch Thank you for your message! Isn't your mount the same as in the previous comment?

Unfortunately even when setting remoteUser to root, I still have the same issue. Do you change any docker configuration in the devcontainer? Or maybe you need to add some flags while running? I am simply running with docker run --rm --net=host mybin:latest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

No branches or pull requests

7 participants
@Chuxel @antspy @dschneiderch @bamurtaugh @zephyros-dev @ci-vamp and others