Mix runtime in a swarm OR run a Portainer agent in SysBox

[Szaamaan]

I've got a Docker Swarm which hosts a number of MS DevOps agents. Some of those agents are configured to have access to the Docker pipeline, performing Docker In Docker operations. I'd like to make use of Sysbox to isolate those agents, so that they don't "mess up" the host by accident (by leaving containers running or leaving images / cache).

Sysbox seems perfect for the job, and I was able to install it and confirm that these agents work in isolation as expected - but to do this I had to switch the default Docker runtime to Sysbox (I can't see a way to customize the runtime in a swarm compose file; from what I can tell this isn't possible any more).

Except... I also use Portainer to handle the swarm, and that includes running the Portainer Agent on each swarm node. And here it seems the Portainer agent stops working when it's running from inside Sysbox. It makes sense, really - it's now isolated; it expects to use the shared host docker.sock which is now missing.

Except I'm in this weird situation where on one Docker host I'd like to have both containers running with and without Sysbox.

Is there some way to perhaps get Sysbox to share the docker pipeline with a specific container via config? I mean, changing the runtime seems like the most straightforward approach, but it's not supported AFAIK... Or has anyone managed to get a Portainer Agent working alongside Sysbox?

EDIT: I suppose this might be related to #404? I'm not sure if there's a way around this however, and I cannot find anything more recent. The compose definition for the agent is as follows:

services:
  agent_linux:
    image: portainer/agent:2.20.3
    environment:
      - AGENT_CLUSTER_ADDR=tasks.agent
    volumes:
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
      - type: bind
        source: /var/lib/docker/volumes
        target: /var/lib/docker/volumes
    networks:
      agent_network:
        aliases:
          - agent
    deploy:
      mode: global
      placement:
        constraints:
          - node.platform.os == linux

So it's using a "standard" mechanism for mounting the docker.sock, except that doesn't seem to work when Sysbox is used.

[ctalledo]

Hi @Szaamaan, thanks for filing the issue and apologies for the belated reply.

I can't see a way to customize the runtime in a swarm compose file; from what I can tell this isn't possible any more

I think ideally you would be able to select the runtime for each service in the swarm cluster, so that you could run some of them with the default runc runtime and others with sysbox-runc. However it seems in the Compose YAML v3 spec support for the "runtime" clause was dropped (even though the docs indicate it's possible, see here). I'll inquire about this.

I also use Portainer to handle the swarm, and that includes running the Portainer Agent on each swarm node. And here it seems the Portainer agent stops working when it's running from inside Sysbox. It makes sense, really - it's now isolated; it expects to use the shared host docker.sock which is now missing

Mmm ... that should have worked; even though Sysbox allows you to run "true" Docker-in-Docker using unprivileged containers, you can also still mount the host Docker socket into a Sysbox container if you wish. For example this worked for me:

docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -v /var/run/docker/volumes:/var/run/docker/volumes --runtime=sysbox-runc  -p 8000:8000  -p 9000:9000 portainer/portainer-ce

docker run -d   -p 9001:9001   --name portainer_agent   --runtime=sysbox-runc   -v /var/run/docker.sock:/var/run/docker.sock   -v /var/lib/docker/volumes:/var/lib/docker/volumes   -v /:/host   portainer/agent:2.21.2

And then add the agent to the Portainer via the UI at localhost:9000.

[Szaamaan]

Hmm... I'll play around some more then. Also you're seemingly running both Portainer and the Agent on a single docker host which may potentially simplify some stuff.

To be fair, I provided a "short" version of how the Portainer Agent is set up. It's actually part of single config file for the whole docker swarm with mixed Windows and Linux hosts:

networks:
  agent_network:
    driver: overlay
    attachable: true

volumes:
  portainer_data:

services:
  agent_linux:
# at the moment the only valid host for this service is the one running SysBox by default!
    image: portainer/agent:2.21.0
    environment:
      - AGENT_CLUSTER_ADDR=tasks.agent
    volumes:
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
      - type: bind
        source: /var/lib/docker/volumes
        target: /var/lib/docker/volumes
    networks:
      agent_network:
        aliases:
          - agent
    deploy:
      mode: global
      placement:
        constraints:
          - node.platform.os == linux

  agent_windows:
    image: portainer/agent:2.21.0
    environment:
      - AGENT_CLUSTER_ADDR=tasks.agent
    volumes:
      - type: npipe
        source: \\.\pipe\docker_engine
        target: \\.\pipe\docker_engine
      - C:\ProgramData\docker\volumes:C:\ProgramData\docker\volumes
    networks:
      agent_network:
        aliases:
          - agent
    deploy:
      mode: global
      placement:
        constraints:
          - node.platform.os == windows

  portainer:
    image: portainer/portainer-ee:2.21.0
    command: -H tcp://tasks.agent:9001 --tlsskipverify
    ports:
      - target: 9443
        published: 9443
      - target: 9000
        published: 9000
      - target: 8000
        published: 8000
    volumes:
      - portainer_data:/data
    networks:
      - agent_network
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.platform.os == linux

Could it be the lack of explicit port definitions on the agents?

[ctalledo]

Hmm... I'll play around some more then.

Sounds good.

Also you're seemingly running both Portainer and the Agent on a single docker host which may potentially simplify some stuff.

Yes, but that shouldn't make a difference; the fact that I was able to run both Portainer and the agent in Sysbox containers means they will run fine whether on the same host or not.

Could it be the lack of explicit port definitions on the agents?

I doubt it, since Sysbox is pretty much agnostic to the container networking config.

[Szaamaan]

I've retested my initial original setup and... it just started working?

Oh well, I won't complain about this outcome. Perhaps some component / image was updated... 😅

[ctalledo]

Great to hear @Szaamaan ... hope you find Sysbox helpful.

[ctalledo]

Closing discussion now.