Supporting runtimes that support Docker-in-Docker

[blitss]

Creating this issue/discussion here for visibility that maybe someone had overcome this problem or maybe maintainers of Talos have any insights into this issue.

Docker in Docker (dind) can be used to safely run Docker inside containers, which is used by remote development envinroments like Coder.

The most obvious choice for doing that is sysbox which is currently not supported by Talos so I was researching other solutions.

The closest you can get to that currently is by using kata runtimes via extension that was recently aded. It works, but it's not perfect and comes at a price:

kataShared mount types are powered by virtio-fs, a marked improvement over virtio-9p, thanks to kata-containers/runtime#1016. While virtio-fs is normally an excellent choice, in the case of DinD workloads virtio-fs causes an issue -- it cannot be used as a "upper layer" of overlayfs without a custom patch.

Kata maintainers suggest using loop mounted disks which is something that doesn't work in Talos, specifically mount part because it's unable to set up loop device:

mount: can't setup loop device: No space left on device

spec:
  containers:
    - name: docker
      image: docker:20.10-dind
      command: ["sh", "-c"]
      args:
      - if [[ $(df -PT /var/lib/docker | awk 'NR==2 {print $2}') == virtiofs ]]; then
          apk add e2fsprogs &&
          truncate -s 20G /tmp/disk.img &&
          mkfs.ext4 /tmp/disk.img &&
          mount /tmp/disk.img /var/lib/docker; fi &&
        dockerd-entrypoint.sh;
      securityContext:
        privileged: true

So docker in kata runtime works but it's slow because it's using slow storage driver. And to overcome that you have to create a loop mounted disk which currently doesn't work in Talos.

Relevant issues: #5803 #4385 #3922

[smira]

We run dind on Talos ourselves a lot, and the only thing you need is to make sure /var/lib/docker is a volume mount in a pod (e.g. emptyDir work, or any other volume option in Kubernetes). As container's rootfs is overlayfs, you can't run Docker on top of that, as nested overlayfs mounts don't work.