Can I execute docker in docker in docker？ #752

xlw712 · 2023-11-29T09:47:19Z

xlw712
Nov 29, 2023

There is a question that needs to be confirmed, can sysbox implement a dind to achieve multi-layer dind? like this

contariner    <-- Is there a way to continue creating docker by creating containers with Pods below? 
----
pod (enable sysbox)   <-- this pod can execute docker commands and implement dind by starting docker
----
k8s (daemon sysbox)

I continued to execute the start docker command in container and returned the following error

WARN[2023-11-29T17:38:05.565601018+08:00] could not use snapshotter devmapper in metadata plugin  error="devmapper not configured"
INFO[2023-11-29T17:38:05.565613591+08:00] metadata content store policy set             policy=shared
INFO[2023-11-29T17:38:05.566102397+08:00] loading plugin "io.containerd.differ.v1.walking"...  type=io.containerd.differ.v1
INFO[2023-11-29T17:38:05.566119168+08:00] loading plugin "io.containerd.event.v1.exchange"...  type=io.containerd.event.v1
INFO[2023-11-29T17:38:05.566134760+08:00] loading plugin "io.containerd.gc.v1.scheduler"...  type=io.containerd.gc.v1
INFO[2023-11-29T17:38:05.566165349+08:00] loading plugin "io.containerd.service.v1.introspection-service"...  type=io.containerd.service.v1
INFO[2023-11-29T17:38:05.566185339+08:00] loading plugin "io.containerd.service.v1.containers-service"...  type=io.containerd.service.v1
INFO[2023-11-29T17:38:05.566195743+08:00] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2023-11-29T17:38:05.566206304+08:00] loading plugin "io.containerd.service.v1.diff-service"...  type=io.containerd.service.v1
INFO[2023-11-29T17:38:05.566227969+08:00] loading plugin "io.containerd.service.v1.images-service"...  type=io.containerd.service.v1
INFO[2023-11-29T17:38:05.566248557+08:00] loading plugin "io.containerd.service.v1.leases-service"...  type=io.containerd.service.v1
INFO[2023-11-29T17:38:05.566261564+08:00] loading plugin "io.containerd.service.v1.namespaces-service"...  type=io.containerd.service.v1
INFO[2023-11-29T17:38:05.566272821+08:00] loading plugin "io.containerd.service.v1.snapshots-service"...  type=io.containerd.service.v1
INFO[2023-11-29T17:38:05.566282841+08:00] loading plugin "io.containerd.runtime.v1.linux"...  type=io.containerd.runtime.v1
INFO[2023-11-29T17:38:05.566632539+08:00] loading plugin "io.containerd.runtime.v2.task"...  type=io.containerd.runtime.v2
INFO[2023-11-29T17:38:05.566965179+08:00] loading plugin "io.containerd.monitor.v1.cgroups"...  type=io.containerd.monitor.v1
INFO[2023-11-29T17:38:05.567433697+08:00] loading plugin "io.containerd.service.v1.tasks-service"...  type=io.containerd.service.v1
INFO[2023-11-29T17:38:05.567465725+08:00] loading plugin "io.containerd.grpc.v1.introspection"...  type=io.containerd.grpc.v1
INFO[2023-11-29T17:38:05.567478679+08:00] loading plugin "io.containerd.internal.v1.restart"...  type=io.containerd.internal.v1
INFO[2023-11-29T17:38:05.567524598+08:00] loading plugin "io.containerd.grpc.v1.containers"...  type=io.containerd.grpc.v1
INFO[2023-11-29T17:38:05.567535648+08:00] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1
INFO[2023-11-29T17:38:05.567546506+08:00] loading plugin "io.containerd.grpc.v1.diff"...  type=io.containerd.grpc.v1
INFO[2023-11-29T17:38:05.567556019+08:00] loading plugin "io.containerd.grpc.v1.events"...  type=io.containerd.grpc.v1
INFO[2023-11-29T17:38:05.567565697+08:00] loading plugin "io.containerd.grpc.v1.healthcheck"...  type=io.containerd.grpc.v1
INFO[2023-11-29T17:38:05.567579525+08:00] loading plugin "io.containerd.grpc.v1.images"...  type=io.containerd.grpc.v1
INFO[2023-11-29T17:38:05.567589784+08:00] loading plugin "io.containerd.grpc.v1.leases"...  type=io.containerd.grpc.v1
INFO[2023-11-29T17:38:05.567599608+08:00] loading plugin "io.containerd.grpc.v1.namespaces"...  type=io.containerd.grpc.v1
INFO[2023-11-29T17:38:05.567610594+08:00] loading plugin "io.containerd.internal.v1.opt"...  type=io.containerd.internal.v1
INFO[2023-11-29T17:38:05.568265189+08:00] loading plugin "io.containerd.grpc.v1.snapshots"...  type=io.containerd.grpc.v1
INFO[2023-11-29T17:38:05.568276844+08:00] loading plugin "io.containerd.grpc.v1.tasks"...  type=io.containerd.grpc.v1
INFO[2023-11-29T17:38:05.568286564+08:00] loading plugin "io.containerd.grpc.v1.version"...  type=io.containerd.grpc.v1
INFO[2023-11-29T17:38:05.568296710+08:00] loading plugin "io.containerd.tracing.processor.v1.otlp"...  type=io.containerd.tracing.processor.v1
INFO[2023-11-29T17:38:05.568308190+08:00] skip loading plugin "io.containerd.tracing.processor.v1.otlp"...  error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
INFO[2023-11-29T17:38:05.568317082+08:00] loading plugin "io.containerd.internal.v1.tracing"...  type=io.containerd.internal.v1
ERRO[2023-11-29T17:38:05.568478133+08:00] failed to initialize a tracing processor "otlp"  error="no OpenTelemetry endpoint: skip plugin"
INFO[2023-11-29T17:38:05.569204980+08:00] serving...                                    address=/var/run/docker/containerd/containerd-debug.sock
INFO[2023-11-29T17:38:05.569480586+08:00] serving...                                    address=/var/run/docker/containerd/containerd.sock.ttrpc
INFO[2023-11-29T17:38:05.569714822+08:00] serving...                                    address=/var/run/docker/containerd/containerd.sock
INFO[2023-11-29T17:38:05.569735711+08:00] containerd successfully booted in 0.023429s
ERRO[2023-11-29T17:38:05.584605008+08:00] failed to mount overlay: operation not permitted  storage-driver=overlay2
INFO[2023-11-29T17:38:05.586237374+08:00] Loading containers: start.
INFO[2023-11-29T17:38:05.591842203+08:00] unable to detect if iptables supports xlock: 'iptables --wait -L -n': `iptables v1.8.4 (nf_tables): Could not fetch rule set generation id: Permission denied (you must be root)`  error="exit status 4"
INFO[2023-11-29T17:38:05.639416941+08:00] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
INFO[2023-11-29T17:38:05.639661542+08:00] stopping healthcheck following graceful shutdown  module=libcontainerd
INFO[2023-11-29T17:38:05.639693145+08:00] stopping event stream following graceful shutdown  error="context canceled" module=libcontainerd namespace=plugins.moby
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.4 (nf_tables): Could not fetch rule set generation id: Permission denied (you must be root)

 (exit status 4)

I can execute the docker ps command after adding the --privileged flag,But none of the other commands can be executed
Like below
docker version is 24.0.7 for image
1.

bash-4.4# docker pull alibaba-cloud-linux-3-registry.cn-hangzhou.cr.aliyuncs.com/alinux3/alinux3:3.8
Error response from daemon: Get "https://alibaba-cloud-linux-3-registry.cn-hangzhou.cr.aliyuncs.com/v2/": net/http: TLS handshake timeout

bash-4.4# docker run -itd curlimages/curl:latest /bin/bash
docker: Error response from daemon: mkdir /var/lib/docker/fuse-overlayfs/15a1345bd67dfe21d86ee4b328bb1031200da706fb7eb111d306226e3f3a7e45-init/merged/dev/pts: operation not permitted.
See 'docker run --help'.

Is there any good way? Or not at all?

@ctalledo Thank you so much for your time

ctalledo · 2023-12-05T01:24:32Z

ctalledo
Dec 5, 2023
Maintainer

Hi @xlw712,

Yes, it is possible to run "docker-in-docker" inside a Sysbox pod/container (i.e., docker-in-docker-in-(k8s/docker)).

In fact we have a test in the sysbox test suite that verifies this works.

For clarity, let's call the sysbox pod "level 0", and the the docker container inside the sysbox pod "level 1". Make sure that level 1 container runs with --privileged (i.e., it will be privileged inside the sysbox pod, but not at host level since the sysbox pod is rootless).

I looked quickly at the errors you got above, and I am not sure why you are seeing them.

As a baseline, try running Docker + Sysbox on a Ubuntu host, and do the docker-in-docker-in-docker experiment (for reference you can follow the steps in the test I linked above) and see if that works. If it does, then give me more info about your K8s + Sysbox setup to see if I can repro.

Hope that helps!

4 replies

xlw712 Dec 6, 2023
Author

@ctalledo Thanks for your reply. I tried according to what you said. I found a server of ubunut system separately, installed sysbox through.deb, and triggered docker in docker logic through docker, which would fail to create level 2 container. Here are my steps

root@host:~# docker run --privileged --runtime=sysbox-runc -itd --hostname level-0 gocd-agent-docker-dind:aliunx3.8_adbmysql_20231205  /bin/bash
80479c20648692a61713fdbc942709c0ba82e615befa5630526d3667bacde2ba

root@host:~# docker exec --privileged -it 80479c206486 /bin/bash
[root@level-0 /]# dockerd >/var/log/dockerd.log 2>&1 &
[root@level-0 /] docker run --privileged  -itd --hostname level-1 gocd-agent-docker-dind:aliunx3.8_adbmysql_20231205  /bin/bash
9cda6053675ce8e941dc58a73b37c100fe0f9c114fee8afb00716b93d9245ea9
[root@level-0 /]# docker exec --privileged -it 9cda6053675c /bin/bash

[root@level-1 /]# dockerd >/var/log/dockerd.log 2>&1 &
[root@level-1 /]# docker run --privileged  -itd --hostname level-2 gocd-agent-docker-dind:aliunx3.8_adbmysql_20231205 /bin/bash
docker: Error response from daemon: using mount program fuse-overlayfs: unknown argument ignored: lazytime

level-1 docker log

time="2023-12-06T11:37:22.883931802+08:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/5.15.0-86-generic\\n\"): skip plugin" type=io.containerd.snapshotter.v1
time="2023-12-06T11:37:22.883958617+08:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
time="2023-12-06T11:37:22.883972344+08:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2023-12-06T11:37:22.883982315+08:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
time="2023-12-06T11:37:22.884100949+08:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
time="2023-12-06T11:37:22.884302802+08:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
time="2023-12-06T11:37:22.884825432+08:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2023-12-06T11:37:22.884840384+08:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
time="2023-12-06T11:37:22.884884568+08:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
time="2023-12-06T11:37:22.884894197+08:00" level=info msg="metadata content store policy set" policy=shared
time="2023-12-06T11:37:22.887850289+08:00" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
time="2023-12-06T11:37:22.887870972+08:00" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
time="2023-12-06T11:37:22.887881502+08:00" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
time="2023-12-06T11:37:22.887903546+08:00" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
time="2023-12-06T11:37:22.887915261+08:00" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
time="2023-12-06T11:37:22.887926195+08:00" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
time="2023-12-06T11:37:22.887940002+08:00" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
time="2023-12-06T11:37:22.887951553+08:00" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
time="2023-12-06T11:37:22.887965817+08:00" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
time="2023-12-06T11:37:22.887985123+08:00" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
time="2023-12-06T11:37:22.888008657+08:00" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
time="2023-12-06T11:37:22.888020419+08:00" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
time="2023-12-06T11:37:22.888133377+08:00" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
time="2023-12-06T11:37:22.888216225+08:00" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
time="2023-12-06T11:37:22.888422608+08:00" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
time="2023-12-06T11:37:22.888458096+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
time="2023-12-06T11:37:22.888471572+08:00" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
time="2023-12-06T11:37:22.888515286+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
time="2023-12-06T11:37:22.888527168+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
time="2023-12-06T11:37:22.888543686+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
time="2023-12-06T11:37:22.888554873+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
time="2023-12-06T11:37:22.888565488+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
time="2023-12-06T11:37:22.888575930+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
time="2023-12-06T11:37:22.888586195+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
time="2023-12-06T11:37:22.888596894+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
time="2023-12-06T11:37:22.888608294+08:00" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
time="2023-12-06T11:37:22.888753587+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
time="2023-12-06T11:37:22.888766392+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
time="2023-12-06T11:37:22.888776882+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
time="2023-12-06T11:37:22.888786935+08:00" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
time="2023-12-06T11:37:22.888799871+08:00" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
time="2023-12-06T11:37:22.888809216+08:00" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
time="2023-12-06T11:37:22.888850576+08:00" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
time="2023-12-06T11:37:22.889823440+08:00" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
time="2023-12-06T11:37:22.889902177+08:00" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
time="2023-12-06T11:37:22.889948941+08:00" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
time="2023-12-06T11:37:22.889967931+08:00" level=info msg="containerd successfully booted in 0.027299s"
time="2023-12-06T11:37:22.954710382+08:00" level=error msg="failed to mount overlay: invalid argument" storage-driver=overlay2
time="2023-12-06T11:37:22.956169557+08:00" level=info msg="Loading containers: start."
time="2023-12-06T11:37:23.141080314+08:00" level=info msg="Loading containers: done."
time="2023-12-06T11:37:23.194422596+08:00" level=info msg="Docker daemon" commit=311b9ff graphdriver=fuse-overlayfs version=24.0.7
time="2023-12-06T11:37:23.194529468+08:00" level=info msg="Daemon has completed initialization"
time="2023-12-06T11:37:23.311261989+08:00" level=info msg="API listen on /var/run/docker.sock"
time="2023-12-06T11:48:26.785297970+08:00" level=error msg="error unmounting /var/lib/docker/fuse-overlayfs/891107edfe190f6dda0964a94d058194c787d9d6b054526fd6b1ae612a25a6af-init/merged: invalid argument" storage-driver=fuse-overlayfs
time="2023-12-06T11:48:26.786122667+08:00" level=error msg="Handler for POST /v1.43/containers/create returned error: using mount program fuse-overlayfs: unknown argument ignored: lazytime\nfuse-overlayfs: cannot read upper dir: Permission denied\n: exit status 1"

System configuration

host.docker.version= Docker version 24.0.5, build 24.0.5-0ubuntu1~22.04.1
host.sysbox.version= sysbox-ce_0.6.2-0.linux_amd64.deb
host.kernel= Linux olap-cicd011158129134.na620 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

image.docker.version=Docker version 24.0.7, build afdd53b

xlw712 Dec 8, 2023
Author

@ctalledo How about that? Can it be repeated?

ctalledo Dec 12, 2023
Maintainer

Hi @xlw712,

I just tried locally and things worked; here's what I did:

Start a container with docker + sysbox:

$ docker run --runtime=sysbox-runc -it --rm nestybox/ubuntu-jammy-systemd-docker
Welcome to Ubuntu 22.04.1 LTS!                                                                                                                                
                                                                               
[  OK  ] Created slice Slice /system/getty.                                                                                                                   
[  OK  ] Created slice Slice /system/modprobe.                                                                                                                
[  OK  ] Created slice User and Session Slice.                                                                                                                
...
Ubuntu 22.04.1 LTS aca1a222dd4b console                                                                                                                                                                                                                                                                                      
                                                                               
aca1a222dd4b login: admin                                                                                                                                                                                                                                                                                                    
Password:  admin                                                                                                                                                                                                                                                                                                        
Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 6.2.0-37-generic x86_64)
...
admin@aca1a222dd4b:~$

Inside the Sysbox container, start a docker-in-docker using Docker's official image:

admin@aca1a222dd4b:~$ docker create network some-network

admin@aca1a222dd4b:~$ docker run --privileged --name some-docker -d     --network some-network --network-alias docker   -e DOCKER_TLS_CERTDIR=/certs    -v some-docker-certs-ca:/certs/ca       -v some-docker-certs-client:/certs/client       docker:dind

admin@aca1a222dd4b:~$ docker run -it --rm --network some-network        -e DOCKER_TLS_CERTDIR=/certs    -v some-docker-certs-client:/certs/client:ro    docker:latest sh

/ # docker run -it --rm alpine
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
661ff4d9561e: Pull complete 
Digest: sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48
Status: Downloaded newer image for alpine:latest
/ #

The last step shows the double-nesting working (i.e., docker-in-docker-in-docker).

Can you try on your host with the procedure above, so we can level-sync?

Then we can try to figure out why the procedure you showed before failed.

xlw712 Dec 13, 2023
Author

@ctalledo Thanks for your reply. I found a new environment record and tried it according to your steps. The following is the error I reported when installing sysbox,N: Download is performed unsandboxed as root as file '/root/sysbox-ce_0.6.2-0.linux_amd64.deb' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)** root@olap-cicd011158142220:~# systemctl status sysbox -n20 ,But viewing the service is normal

root@olap-cicd011158142220:~# sudo apt-get install ./sysbox-ce_0.6.2-0.linux_amd64.deb
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'sysbox-ce' instead of './sysbox-ce_0.6.2-0.linux_amd64.deb'
The following NEW packages will be installed:
  sysbox-ce
0 upgraded, 1 newly installed, 0 to remove and 3 not upgraded.
Need to get 0 B/11.7 MB of archives.
After this operation, 39.7 MB of additional disk space will be used.
Get:1 /root/sysbox-ce_0.6.2-0.linux_amd64.deb sysbox-ce amd64 0.6.2.linux [11.7 MB]
Selecting previously unselected package sysbox-ce.
(Reading database ... 80938 files and directories currently installed.)
Preparing to unpack .../sysbox-ce_0.6.2-0.linux_amd64.deb ...
Unpacking sysbox-ce (0.6.2.linux) ...
Setting up sysbox-ce (0.6.2.linux) ...
Created symlink /etc/systemd/system/sysbox.service.wants/sysbox-fs.service → /lib/systemd/system/sysbox-fs.service.
Created symlink /etc/systemd/system/sysbox.service.wants/sysbox-mgr.service → /lib/systemd/system/sysbox-mgr.service.
Created symlink /etc/systemd/system/multi-user.target.wants/sysbox.service → /lib/systemd/system/sysbox.service.
Scanning processes...
Scanning linux images...

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

**No VM guests are running outdated hypervisor (qemu) binaries on this host.
N: Download is performed unsandboxed as root as file '/root/sysbox-ce_0.6.2-0.linux_amd64.deb' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)**
root@olap-cicd011158142220:~# systemctl status sysbox -n20
● sysbox.service - Sysbox container runtime
     Loaded: loaded (/lib/systemd/system/sysbox.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-12-13 10:55:07 CST; 28s ago
       Docs: https://github.com/nestybox/sysbox
   Main PID: 6260 (sh)
      Tasks: 2 (limit: 151578)
     Memory: 480.0K
        CPU: 22ms
     CGroup: /system.slice/sysbox.service
             ├─6260 /bin/sh -c "/usr/bin/sysbox-runc --version && /usr/bin/sysbox-mgr --version && /usr/bin/sysbox-fs --version && /bin/sleep infinity"
             └─6279 /bin/sleep infinity

Dec 13 10:55:07 olap-cicd011158142220.na61 systemd[1]: Started Sysbox container runtime.
Dec 13 10:55:07 olap-cicd011158142220.na61 sh[6261]: sysbox-runc
Dec 13 10:55:07 olap-cicd011158142220.na61 sh[6261]:         edition:         Community Edition (CE)
Dec 13 10:55:07 olap-cicd011158142220.na61 sh[6261]:         version:         0.6.2
Dec 13 10:55:07 olap-cicd011158142220.na61 sh[6261]:         commit:         60ca93c783b19c63581e34aa183421ce0b9b26b7
Dec 13 10:55:07 olap-cicd011158142220.na61 sh[6261]:         built at:         Mon Jun 12 03:49:19 UTC 2023
Dec 13 10:55:07 olap-cicd011158142220.na61 sh[6261]:         built by:         Cesar Talledo
Dec 13 10:55:07 olap-cicd011158142220.na61 sh[6261]:         oci-specs:         1.0.2-dev
Dec 13 10:55:07 olap-cicd011158142220.na61 sh[6268]: sysbox-mgr
Dec 13 10:55:07 olap-cicd011158142220.na61 sh[6268]:         edition:         Community Edition (CE)
Dec 13 10:55:07 olap-cicd011158142220.na61 sh[6268]:         version:         0.6.2
Dec 13 10:55:07 olap-cicd011158142220.na61 sh[6268]:         commit:         4b5fb1def9abe6a256cfe62bacaf2a7d333d81d2
Dec 13 10:55:07 olap-cicd011158142220.na61 sh[6268]:         built at:         Mon Jun 12 03:49:55 UTC 2023
Dec 13 10:55:07 olap-cicd011158142220.na61 sh[6268]:         built by:         Cesar Talledo
Dec 13 10:55:07 olap-cicd011158142220.na61 sh[6273]: sysbox-fs
Dec 13 10:55:07 olap-cicd011158142220.na61 sh[6273]:         edition:         Community Edition (CE)
Dec 13 10:55:07 olap-cicd011158142220.na61 sh[6273]:         version:         0.6.2
Dec 13 10:55:07 olap-cicd011158142220.na61 sh[6273]:         commit:         30fd49edbd51048fed8b2ad0af327598d30b29eb
Dec 13 10:55:07 olap-cicd011158142220.na61 sh[6273]:         built at:         Mon Jun 12 03:49:46 UTC 2023
Dec 13 10:55:07 olap-cicd011158142220.na61 sh[6273]:         built by:         Cesar Talledo

xlw712 · 2023-12-13T05:53:20Z

xlw712
Dec 13, 2023
Author

@ctalledo Following your steps, my execution on my host will work fine. I tried it again in the k8s pod and it still didn't work. Is there any log or other means for me to check this double-nesting?Do I need to add a network?

3 replies

xlw712 Dec 13, 2023
Author

Hi @ctalledo I had an idea. I tried sysbox double nesting with a public docker image and found that the level2 I started docker, the error iptables -t nat-N DOCKER: iptables v1.4.21: can't initialize iptables table nat': Permission denied (you must be root) Perhaps iptables or your kernel needs to be upgraded. It's the same error as the image I built myself. Can you help me repeat through my steps?
Here are my steps, described as simply as I can。I always use the image gocd/gocd-agent-centos-7:v23.3.0 for dind
First step at ubuntu host start sysbox container && install docker && start docker

$ docker run --runtime=sysbox-runc -itd gocd/gocd-agent-centos-7:v23.3.0 /bin/bash\

$ docker exec -u root -it a3321f993927 /bin/bash

[root@a3321f993927 /] yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

[root@a3321f993927 /] yum install docker-ce -y

[root@a3321f993927 /] dockerd >/var/log/dockerd.log 2>&1 &

[root@a3321f993927 /] dockerd -v
Docker version 24.0.7, build afdd53b

Second setp start level-1 container && install docker && start docker failed

[root@a3321f993927 /] docker run -itd gocd/gocd-agent-centos-7:v23.3.0 /bin/bash

[root@a3321f993927 /] docker exec -u root -it 00a821c74e71 /bin/bash

[root@00a821c74e71 /] yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

[root@00a821c74e71 /] yum install docker-ce -y

[root@00a821c74e71 /] dockerd
INFO[2023-12-13T07:36:57.013105996Z] Starting up
INFO[2023-12-13T07:36:57.013858025Z] containerd not running, starting managed containerd
INFO[2023-12-13T07:36:57.014928875Z] started new containerd process                address=/var/run/docker/containerd/containerd.sock module=libcontainerd pid=647
INFO[2023-12-13T07:36:57.033842309Z] starting containerd                           revision=3dd1e886e55dd695541fdcd67420c2888645a495 version=1.6.26
INFO[2023-12-13T07:36:57.047333917Z] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
INFO[2023-12-13T07:36:57.047397981Z] loading plugin "io.containerd.snapshotter.v1.aufs"...  type=io.containerd.snapshotter.v1
INFO[2023-12-13T07:36:57.048455045Z] skip loading plugin "io.containerd.snapshotter.v1.aufs"...  error="aufs is not supported (modprobe aufs failed: exit status 1 \"\"): skip plugin" type=io.containerd.snapshotter.v1
INFO[2023-12-13T07:36:57.048473456Z] loading plugin "io.containerd.snapshotter.v1.btrfs"...  type=io.containerd.snapshotter.v1
INFO[2023-12-13T07:36:57.048609306Z] skip loading plugin "io.containerd.snapshotter.v1.btrfs"...  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (overlay) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2023-12-13T07:36:57.048623538Z] loading plugin "io.containerd.snapshotter.v1.devmapper"...  type=io.containerd.snapshotter.v1
WARN[2023-12-13T07:36:57.048633952Z] failed to load plugin io.containerd.snapshotter.v1.devmapper  error="devmapper not configured"
INFO[2023-12-13T07:36:57.048642202Z] loading plugin "io.containerd.snapshotter.v1.native"...  type=io.containerd.snapshotter.v1
INFO[2023-12-13T07:36:57.048663995Z] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  type=io.containerd.snapshotter.v1
INFO[2023-12-13T07:36:57.048770991Z] loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
INFO[2023-12-13T07:36:57.048874057Z] skip loading plugin "io.containerd.snapshotter.v1.zfs"...  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2023-12-13T07:36:57.048886066Z] loading plugin "io.containerd.metadata.v1.bolt"...  type=io.containerd.metadata.v1
WARN[2023-12-13T07:36:57.048901148Z] could not use snapshotter devmapper in metadata plugin  error="devmapper not configured"
INFO[2023-12-13T07:36:57.048908879Z] metadata content store policy set             policy=shared
INFO[2023-12-13T07:36:57.048993360Z] loading plugin "io.containerd.differ.v1.walking"...  type=io.containerd.differ.v1
INFO[2023-12-13T07:36:57.049007131Z] loading plugin "io.containerd.event.v1.exchange"...  type=io.containerd.event.v1
INFO[2023-12-13T07:36:57.049016446Z] loading plugin "io.containerd.gc.v1.scheduler"...  type=io.containerd.gc.v1
INFO[2023-12-13T07:36:57.049040833Z] loading plugin "io.containerd.warning.v1.deprecations"...  type=io.containerd.warning.v1
INFO[2023-12-13T07:36:57.049062000Z] loading plugin "io.containerd.service.v1.introspection-service"...  type=io.containerd.service.v1
INFO[2023-12-13T07:36:57.049086829Z] loading plugin "io.containerd.service.v1.containers-service"...  type=io.containerd.service.v1
INFO[2023-12-13T07:36:57.049097832Z] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2023-12-13T07:36:57.049107411Z] loading plugin "io.containerd.service.v1.diff-service"...  type=io.containerd.service.v1
INFO[2023-12-13T07:36:57.049117926Z] loading plugin "io.containerd.service.v1.images-service"...  type=io.containerd.service.v1
INFO[2023-12-13T07:36:57.049129571Z] loading plugin "io.containerd.service.v1.leases-service"...  type=io.containerd.service.v1
INFO[2023-12-13T07:36:57.049140423Z] loading plugin "io.containerd.service.v1.namespaces-service"...  type=io.containerd.service.v1
INFO[2023-12-13T07:36:57.049150212Z] loading plugin "io.containerd.service.v1.snapshots-service"...  type=io.containerd.service.v1
INFO[2023-12-13T07:36:57.049163804Z] loading plugin "io.containerd.runtime.v1.linux"...  type=io.containerd.runtime.v1
INFO[2023-12-13T07:36:57.049211982Z] loading plugin "io.containerd.runtime.v2.task"...  type=io.containerd.runtime.v2
INFO[2023-12-13T07:36:57.049252022Z] loading plugin "io.containerd.monitor.v1.cgroups"...  type=io.containerd.monitor.v1
INFO[2023-12-13T07:36:57.049479538Z] loading plugin "io.containerd.service.v1.tasks-service"...  type=io.containerd.service.v1
INFO[2023-12-13T07:36:57.049499559Z] loading plugin "io.containerd.grpc.v1.introspection"...  type=io.containerd.grpc.v1
INFO[2023-12-13T07:36:57.049510976Z] loading plugin "io.containerd.internal.v1.restart"...  type=io.containerd.internal.v1
INFO[2023-12-13T07:36:57.049553306Z] loading plugin "io.containerd.grpc.v1.containers"...  type=io.containerd.grpc.v1
INFO[2023-12-13T07:36:57.049564567Z] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1
INFO[2023-12-13T07:36:57.049574302Z] loading plugin "io.containerd.grpc.v1.diff"...  type=io.containerd.grpc.v1
INFO[2023-12-13T07:36:57.049583562Z] loading plugin "io.containerd.grpc.v1.events"...  type=io.containerd.grpc.v1
INFO[2023-12-13T07:36:57.049593807Z] loading plugin "io.containerd.grpc.v1.healthcheck"...  type=io.containerd.grpc.v1
INFO[2023-12-13T07:36:57.049603745Z] loading plugin "io.containerd.grpc.v1.images"...  type=io.containerd.grpc.v1
INFO[2023-12-13T07:36:57.049612927Z] loading plugin "io.containerd.grpc.v1.leases"...  type=io.containerd.grpc.v1
INFO[2023-12-13T07:36:57.049623815Z] loading plugin "io.containerd.grpc.v1.namespaces"...  type=io.containerd.grpc.v1
INFO[2023-12-13T07:36:57.049635084Z] loading plugin "io.containerd.internal.v1.opt"...  type=io.containerd.internal.v1
INFO[2023-12-13T07:36:57.049673156Z] loading plugin "io.containerd.grpc.v1.snapshots"...  type=io.containerd.grpc.v1
INFO[2023-12-13T07:36:57.049690898Z] loading plugin "io.containerd.grpc.v1.tasks"...  type=io.containerd.grpc.v1
INFO[2023-12-13T07:36:57.049701209Z] loading plugin "io.containerd.grpc.v1.version"...  type=io.containerd.grpc.v1
INFO[2023-12-13T07:36:57.049710334Z] loading plugin "io.containerd.tracing.processor.v1.otlp"...  type=io.containerd.tracing.processor.v1
INFO[2023-12-13T07:36:57.049723665Z] skip loading plugin "io.containerd.tracing.processor.v1.otlp"...  error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
INFO[2023-12-13T07:36:57.049732550Z] loading plugin "io.containerd.internal.v1.tracing"...  type=io.containerd.internal.v1
ERRO[2023-12-13T07:36:57.049746487Z] failed to initialize a tracing processor "otlp"  error="no OpenTelemetry endpoint: skip plugin"
INFO[2023-12-13T07:36:57.050237804Z] serving...                                    address=/var/run/docker/containerd/containerd-debug.sock
INFO[2023-12-13T07:36:57.050301584Z] serving...                                    address=/var/run/docker/containerd/containerd.sock.ttrpc
INFO[2023-12-13T07:36:57.050345143Z] serving...                                    address=/var/run/docker/containerd/containerd.sock
INFO[2023-12-13T07:36:57.050363227Z] containerd successfully booted in 0.017014s
INFO[2023-12-13T07:36:57.105527717Z] [graphdriver] using prior storage driver: fuse-overlayfs
INFO[2023-12-13T07:36:57.105665752Z] Loading containers: start.
INFO[2023-12-13T07:36:57.113344553Z] unable to detect if iptables supports xlock: 'iptables --wait -L -n': `iptables v1.4.21: can't initialize iptables table `filter': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.`  error="exit status 3"
INFO[2023-12-13T07:36:57.128826052Z] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
INFO[2023-12-13T07:36:57.129081959Z] stopping healthcheck following graceful shutdown  module=libcontainerd
INFO[2023-12-13T07:36:57.129108937Z] stopping event stream following graceful shutdown  error="context canceled" module=libcontainerd namespace=plugins.moby
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.4.21: can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.
 (exit status 3)

Looking forward to your reply !

ctalledo Dec 15, 2023
Maintainer

Hi @xlw712,

Second setp start level-1 container && install docker && start docker failed

[root@a3321f993927 /] docker run -itd gocd/gocd-agent-centos-7:v23.3.0 /bin/bash

Don't you need a --privileged for that level 1 container?

In other words, think of the Sysbox container as a VM-like environment, and then run Docker-in-Docker inside as you would in a VM (without Sysbox).

Let me know if that works.

ctalledo Dec 15, 2023
Maintainer

Following-up, notice how I used --privileged to launch the level 1 container in my comment above:

admin@aca1a222dd4b:~$ docker run --privileged --name some-docker -d     --network some-network --network-alias docker   -e DOCKER_TLS_CERTDIR=/certs    -v some-docker-certs-ca:/certs/ca       -v some-docker-certs-client:/certs/client       docker:dind

xlw712 · 2023-12-18T06:32:21Z

xlw712
Dec 18, 2023
Author

Hi @ctalledo

Second setp start level-1 container && install docker && start docker failed
[root@a3321f993927 /] docker run -itd gocd/gocd-agent-centos-7:v23.3.0 /bin/bash

Don't you need a --privileged for that level 1 container?

In other words, think of the Sysbox container as a VM-like environment, and then run Docker-in-Docker inside as you would in a VM (without Sysbox).

Let me know if that works.

After I tried to add the keyword --privileged,docker could be executed, but the following error occurred when I executed other commands of docker (I executed it inside the sysbox container)

[root@a3321f993927 /]# docker run --privileged -itd gocd/gocd-agent-centos-7:v23.3.0 /bin/bash
5285e2ffbccb99e1d46b2a9336b710cfa7ee929e973ebbbde59054bd914babf7

[root@a3321f993927 /]# docker exec -u root -it 5285e2ffbccb /bin/bash
[root@5285e2ffbccb /]# yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[root@5285e2ffbccb /]# yum install docker-ce -y
[root@5285e2ffbccb /]# dockerd
INFO[2023-12-18T06:11:10.132250386Z] Starting up
INFO[2023-12-18T06:11:10.133977494Z] containerd not running, starting managed containerd
INFO[2023-12-18T06:11:10.134693967Z] started new containerd process                address=/var/run/docker/containerd/containerd.sock module=libcontainerd pid=531
INFO[2023-12-18T06:11:10.153642960Z] starting containerd                           revision=3dd1e886e55dd695541fdcd67420c2888645a495 version=1.6.26
INFO[2023-12-18T06:11:10.167043064Z] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
INFO[2023-12-18T06:11:10.167143303Z] loading plugin "io.containerd.snapshotter.v1.aufs"...  type=io.containerd.snapshotter.v1
INFO[2023-12-18T06:11:10.168243852Z] skip loading plugin "io.containerd.snapshotter.v1.aufs"...  error="aufs is not supported (modprobe aufs failed: exit status 1 \"\"): skip plugin" type=io.containerd.snapshotter.v1
INFO[2023-12-18T06:11:10.168263206Z] loading plugin "io.containerd.snapshotter.v1.btrfs"...  type=io.containerd.snapshotter.v1
INFO[2023-12-18T06:11:10.168420065Z] skip loading plugin "io.containerd.snapshotter.v1.btrfs"...  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (overlay) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2023-12-18T06:11:10.168435040Z] loading plugin "io.containerd.snapshotter.v1.devmapper"...  type=io.containerd.snapshotter.v1
WARN[2023-12-18T06:11:10.168445719Z] failed to load plugin io.containerd.snapshotter.v1.devmapper  error="devmapper not configured"
INFO[2023-12-18T06:11:10.168453770Z] loading plugin "io.containerd.snapshotter.v1.native"...  type=io.containerd.snapshotter.v1
INFO[2023-12-18T06:11:10.168509763Z] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  type=io.containerd.snapshotter.v1
INFO[2023-12-18T06:11:10.168690181Z] loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
INFO[2023-12-18T06:11:10.168824915Z] skip loading plugin "io.containerd.snapshotter.v1.zfs"...  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2023-12-18T06:11:10.168837854Z] loading plugin "io.containerd.metadata.v1.bolt"...  type=io.containerd.metadata.v1
WARN[2023-12-18T06:11:10.168884080Z] could not use snapshotter devmapper in metadata plugin  error="devmapper not configured"
INFO[2023-12-18T06:11:10.168892823Z] metadata content store policy set             policy=shared
INFO[2023-12-18T06:11:10.172721623Z] loading plugin "io.containerd.differ.v1.walking"...  type=io.containerd.differ.v1
INFO[2023-12-18T06:11:10.172745049Z] loading plugin "io.containerd.event.v1.exchange"...  type=io.containerd.event.v1
INFO[2023-12-18T06:11:10.172756802Z] loading plugin "io.containerd.gc.v1.scheduler"...  type=io.containerd.gc.v1
INFO[2023-12-18T06:11:10.172793332Z] loading plugin "io.containerd.warning.v1.deprecations"...  type=io.containerd.warning.v1
INFO[2023-12-18T06:11:10.172805209Z] loading plugin "io.containerd.service.v1.introspection-service"...  type=io.containerd.service.v1
INFO[2023-12-18T06:11:10.172816113Z] loading plugin "io.containerd.service.v1.containers-service"...  type=io.containerd.service.v1
INFO[2023-12-18T06:11:10.172826482Z] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2023-12-18T06:11:10.172840702Z] loading plugin "io.containerd.service.v1.diff-service"...  type=io.containerd.service.v1
INFO[2023-12-18T06:11:10.172850568Z] loading plugin "io.containerd.service.v1.images-service"...  type=io.containerd.service.v1
INFO[2023-12-18T06:11:10.172861718Z] loading plugin "io.containerd.service.v1.leases-service"...  type=io.containerd.service.v1
INFO[2023-12-18T06:11:10.172871815Z] loading plugin "io.containerd.service.v1.namespaces-service"...  type=io.containerd.service.v1
INFO[2023-12-18T06:11:10.172881712Z] loading plugin "io.containerd.service.v1.snapshots-service"...  type=io.containerd.service.v1
INFO[2023-12-18T06:11:10.172891417Z] loading plugin "io.containerd.runtime.v1.linux"...  type=io.containerd.runtime.v1
INFO[2023-12-18T06:11:10.172997582Z] loading plugin "io.containerd.runtime.v2.task"...  type=io.containerd.runtime.v2
INFO[2023-12-18T06:11:10.173104784Z] loading plugin "io.containerd.monitor.v1.cgroups"...  type=io.containerd.monitor.v1
INFO[2023-12-18T06:11:10.173311771Z] loading plugin "io.containerd.service.v1.tasks-service"...  type=io.containerd.service.v1
INFO[2023-12-18T06:11:10.173330387Z] loading plugin "io.containerd.grpc.v1.introspection"...  type=io.containerd.grpc.v1
INFO[2023-12-18T06:11:10.173342338Z] loading plugin "io.containerd.internal.v1.restart"...  type=io.containerd.internal.v1
INFO[2023-12-18T06:11:10.173405782Z] loading plugin "io.containerd.grpc.v1.containers"...  type=io.containerd.grpc.v1
INFO[2023-12-18T06:11:10.173419685Z] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1
INFO[2023-12-18T06:11:10.173429505Z] loading plugin "io.containerd.grpc.v1.diff"...  type=io.containerd.grpc.v1
INFO[2023-12-18T06:11:10.173444019Z] loading plugin "io.containerd.grpc.v1.events"...  type=io.containerd.grpc.v1
INFO[2023-12-18T06:11:10.173455013Z] loading plugin "io.containerd.grpc.v1.healthcheck"...  type=io.containerd.grpc.v1
INFO[2023-12-18T06:11:10.173464747Z] loading plugin "io.containerd.grpc.v1.images"...  type=io.containerd.grpc.v1
INFO[2023-12-18T06:11:10.173474029Z] loading plugin "io.containerd.grpc.v1.leases"...  type=io.containerd.grpc.v1
INFO[2023-12-18T06:11:10.173484184Z] loading plugin "io.containerd.grpc.v1.namespaces"...  type=io.containerd.grpc.v1
INFO[2023-12-18T06:11:10.173511034Z] loading plugin "io.containerd.internal.v1.opt"...  type=io.containerd.internal.v1
INFO[2023-12-18T06:11:10.173630274Z] loading plugin "io.containerd.grpc.v1.snapshots"...  type=io.containerd.grpc.v1
INFO[2023-12-18T06:11:10.173651805Z] loading plugin "io.containerd.grpc.v1.tasks"...  type=io.containerd.grpc.v1
INFO[2023-12-18T06:11:10.173663104Z] loading plugin "io.containerd.grpc.v1.version"...  type=io.containerd.grpc.v1
INFO[2023-12-18T06:11:10.173673015Z] loading plugin "io.containerd.tracing.processor.v1.otlp"...  type=io.containerd.tracing.processor.v1
INFO[2023-12-18T06:11:10.173686282Z] skip loading plugin "io.containerd.tracing.processor.v1.otlp"...  error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
INFO[2023-12-18T06:11:10.173695290Z] loading plugin "io.containerd.internal.v1.tracing"...  type=io.containerd.internal.v1
ERRO[2023-12-18T06:11:10.173732828Z] failed to initialize a tracing processor "otlp"  error="no OpenTelemetry endpoint: skip plugin"
INFO[2023-12-18T06:11:10.174259216Z] serving...                                    address=/var/run/docker/containerd/containerd-debug.sock
INFO[2023-12-18T06:11:10.174313691Z] serving...                                    address=/var/run/docker/containerd/containerd.sock.ttrpc
INFO[2023-12-18T06:11:10.174360010Z] serving...                                    address=/var/run/docker/containerd/containerd.sock
INFO[2023-12-18T06:11:10.174378684Z] containerd successfully booted in 0.021353s
ERRO[2023-12-18T06:11:10.237997156Z] failed to mount overlay: invalid argument     storage-driver=overlay2
INFO[2023-12-18T06:11:10.239487037Z] Loading containers: start.
INFO[2023-12-18T06:11:10.360036909Z] Loading containers: done.
INFO[2023-12-18T06:11:10.413629511Z] Docker daemon                                 commit=311b9ff graphdriver=fuse-overlayfs version=24.0.7
INFO[2023-12-18T06:11:10.413743040Z] Daemon has completed initialization
INFO[2023-12-18T06:11:10.530404304Z] API listen on /var/run/docker.sock

[root@5285e2ffbccb /]#   dockerd >/var/log/dockerd.log 2>&1 &

[root@5285e2ffbccb /]# docker run --privileged -itd gocd/gocd-agent-centos-7:v23.3.0 /bin/bash
docker: Error response from daemon: using mount program fuse-overlayfs: fuse: device not found, try 'modprobe fuse' first
fuse-overlayfs: cannot mount: No such file or directory
: exit status 1.

1 reply

ctalledo Dec 18, 2023
Maintainer

Hi @xlw712,

docker: Error response from daemon: using mount program fuse-overlayfs: fuse: device not found, try 'modprobe fuse' first

Looks like Docker is looking for the fuse kernel module; check if you have it lsmod | grep fuse and if not try modprobe fuse (if you haven't already).

xlw712 · 2023-12-19T03:36:56Z

xlw712
Dec 19, 2023
Author

Hi @ctalledo

Looks like Docker is looking for the fuse kernel module; check if you have it lsmod | grep fuse and if not try modprobe fuse (if you haven't already).

Sorry I'm not familiar with this. I ran the lsmod | grep fuse command and it seems that fuse is not found. What do I need to install?

[root@5285e2ffbccb /]# lsmod | grep fuse
[root@5285e2ffbccb /]# modprobe fuse

1 reply

ctalledo Dec 19, 2023
Maintainer

No worries @xlw712; at host level, you need to do sudo modprobe fuse, and then lsmod | grep fuse to confirm the module is loaded.

If that doesn't work, search the web for how to install the fuse module in the linux kernel, on your particular distro.

The requirement for fuse is coming from Docker (running inside the unprivileged Sysbox container), not from Sysbox itself. I am actually surprised that the fuse module is not present in your kernel, it usually is in most distros.

Hope that helps.

xlw712 · 2023-12-19T12:27:47Z

xlw712
Dec 19, 2023
Author

Hi @ctalledo

No worries @xlw712; at host level, you need to do sudo modprobe fuse, and then lsmod | grep fuse to confirm the module is loaded.

If that doesn't work, search the web for how to install the fuse module in the linux kernel, on your particular distro.

The requirement for fuse is coming from Docker (running inside the unprivileged Sysbox container), not from Sysbox itself. I am actually surprised that the fuse module is not present in your kernel, it usually is in most distros.

Hope that helps.

I did some research and got the message that ubuntu installs fuse into the kernel by default, and I tried a few commands to verify that I installed it. But strangely, I use command lsmod | grep fuse can't display this module. Do you have any good ideas? I tried a lot of things

root@olap-cicd011158142163:/var/log# uname -a
Linux olap-cicd011158142163.na610 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

root@olap-cicd011158142163:/var/log# dpkg --get-selections | grep fuse
fuse						install
fuse2fs						install
fuse3						deinstall
libfuse-dev					install
libfuse2:amd64					install
libfuse3-3:amd64				install

6 replies

xlw712 Dec 29, 2023
Author

@ctalledo Happy New Year, if you have time to help look at this problem. ^_^

ctalledo Jan 2, 2024
Maintainer

Hi @xlw712,

Sounds like there's something in the gocd/gocd-agent-centos-7:v23.3.0 image that's causing the problem then, but I have no clue what it is right now. I'll try to spend some time on it this week to see if I can figure it out. If you don't hear from me by Friday, please ping me again.

xlw712 Jan 4, 2024
Author

Hi @ctalledo
Excuse me,Let me ask. Any progress?

xlw712 Jan 17, 2024
Author

@ctalledo Hey, any updates on the issue we discussed before?

ctalledo Jan 18, 2024
Maintainer

My apologies @xlw712, I've not had a chance to look at it yet. Will try my best this week.

Can I execute docker in docker in docker？ #752

xlw712 Nov 29, 2023

Replies: 5 comments · 15 replies

ctalledo Dec 5, 2023 Maintainer

xlw712 Dec 6, 2023 Author

xlw712 Dec 8, 2023 Author

ctalledo Dec 12, 2023 Maintainer

xlw712 Dec 13, 2023 Author

xlw712 Dec 13, 2023 Author

xlw712 Dec 13, 2023 Author

ctalledo Dec 15, 2023 Maintainer

ctalledo Dec 15, 2023 Maintainer

xlw712 Dec 18, 2023 Author

ctalledo Dec 18, 2023 Maintainer

xlw712 Dec 19, 2023 Author

ctalledo Dec 19, 2023 Maintainer

xlw712 Dec 19, 2023 Author

xlw712 Dec 29, 2023 Author

ctalledo Jan 2, 2024 Maintainer

xlw712 Jan 4, 2024 Author

xlw712 Jan 17, 2024 Author

ctalledo Jan 18, 2024 Maintainer

xlw712
Nov 29, 2023

Replies: 5 comments 15 replies

ctalledo
Dec 5, 2023
Maintainer

xlw712 Dec 6, 2023
Author

xlw712 Dec 8, 2023
Author

ctalledo Dec 12, 2023
Maintainer

xlw712 Dec 13, 2023
Author

xlw712
Dec 13, 2023
Author

xlw712 Dec 13, 2023
Author

ctalledo Dec 15, 2023
Maintainer

ctalledo Dec 15, 2023
Maintainer

xlw712
Dec 18, 2023
Author

ctalledo Dec 18, 2023
Maintainer

xlw712
Dec 19, 2023
Author

ctalledo Dec 19, 2023
Maintainer

xlw712
Dec 19, 2023
Author

xlw712 Dec 29, 2023
Author

ctalledo Jan 2, 2024
Maintainer

xlw712 Jan 4, 2024
Author

xlw712 Jan 17, 2024
Author

ctalledo Jan 18, 2024
Maintainer