/init hangs when starting a container using non-native image (QEMU)

[midzelis]

I'm not exactly sure, but launching these images using QEMU I think has the same problem as described in the linked.

I think its possible to do with S6 wanting to be PID 1. I verified that the "unshare" workaround works for me, but its unideal as it needs a privileged container, and it doesn't respond to SIGTERM (but at least it starts).

I plan on using this just for testing the images overriding the --entrypoint. The images I'm creating will continue to use /init. But I would like to understand what is going wrong here.

Workaround from: pi-hole/docker-pi-hole#1176
Dockerfile

FROM alpine:3.15 as base
ARG TARGETARCH
ARG TARGETVARIANT

FROM base AS base-amd64
ENV S6_OVERLAY_ARCH=x86_64

FROM base AS base-386
ENV S6_OVERLAY_ARCH=x86

FROM base AS base-arm64
ENV S6_OVERLAY_ARCH=aarch64

FROM base AS base-armv7
ENV S6_OVERLAY_ARCH=armhf

FROM base AS base-armv6
ENV S6_OVERLAY_ARCH=arm

FROM base AS base-ppc64le
ENV S6_OVERLAY_ARCH=ppc64le

FROM base-${TARGETARCH}${TARGETVARIANT}

ARG S6_OVERLAY_VERSION=v3.1.5.0

ADD https://github.com/just-containers/s6-overlay/releases/download/${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz /tmp
ADD https://github.com/just-containers/s6-overlay/releases/download/${S6_OVERLAY_VERSION}/s6-overlay-${S6_OVERLAY_ARCH}.tar.xz /tmp
ADD https://github.com/just-containers/s6-overlay/releases/download/${S6_OVERLAY_VERSION}/syslogd-overlay-noarch.tar.xz /tmp

RUN apk add util-linux perl

RUN set -ex; \
	tar -C / -Jxpf /tmp/s6-overlay-noarch.tar.xz \
	; \
	tar -C / -Jxpf /tmp/s6-overlay-${S6_OVERLAY_ARCH}.tar.xz \
	; \
	tar -C / -Jxpf /tmp/syslogd-overlay-noarch.tar.xz \
	; \
	rm -rf /tmp/* \
	; \
	mkdir -p /etc/fix-attrs.d \
	; \
    mkdir -p /etc/services.d 
# Above inspired by https://github.com/padhi-homelab/docker_s6-overlay/blob/master/Dockerfile

ENTRYPOINT [ "/init" ]
# Workaround: https://github.com/pi-hole/docker-pi-hole/issues/1176#issuecomment-1232363970
# ENTRYPOINT [ \
#     "unshare", "--pid", "--fork", "--kill-child=SIGTERM", "--mount-proc", \
#     "perl", "-e", "$SIG{INT}=''; $SIG{TERM}=''; exec @ARGV;", "--", \
#     "/init" ]

CMD [ "sh" ]

Test script

#!/bin/bash
set -euo pipefail
IFS=$'\n\t'

# This is a one time setup script to setup the build env needed by the build and debug scripts. 
# Assumes this is running on Linux (any distro) using (intel/amd) - I'm using Alpine running on Proxmox as QEMU VM. 

# Create a multi-arch buildx builder named MyBuilder (if it doesn't exist)
if ! docker buildx inspect MyBuilder 1> /dev/null 2>& 1; then
    echo Creating MyBuilder
    # --use will make it automatically use this builder
    docker buildx create --name MyBuilder --platform linux/arm64,linux/arm/v7,linux/386 --use
    # this is needed to register the arch-specific images with QEMU to be able to test
    # these without native hardware
    docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
fi

echo "Building..."
docker buildx build -o type=docker,name=s6-test --load --platform linux/arm/v7 .
echo "Starting..."
docker run --privileged --platform linux/arm/v7  -it s6-test:latest "sh"

[midzelis]

I was just digging through issues, and I found something that helps! /init REALLY does not like it when you run the container with a tty (-it). Dropping this will cause it to work.

See
#399
#396 (comment)

The most frustrating thing was that there was no output at all when this happens. Is there a way to /init can detect this situation and give us a headsup?

[skarnet]

It's unfortunately impossible to make s6-overlay work with all cases by default when your container needs a terminal. But there should always be a setting that works for you.

When you run the container with -it, do you experience anything different when the S6_CMD_USE_TERMINAL variable is set to 1?

[midzelis]

That did not work. It still hangs.

It doesn't matter if the command is sh or something that does NOT need stdio.

I tried a new test where I tried to launch touch /test/testing as the command and it did not execute. If I remote -it from docker run then it creates /test/testing.

Basically, if I supply -it (even if the command is non-interactive and does not need stdio) then /init hangs.

Actually, specifying no command at all causes /init to hang too. The key is if -it is specified or not AND if QEMU is running. It works fine if its not using emulation.

[skarnet]

Oh. Yeah. I'm afraid it's yet another of these "don't do that" cases.

Terminals are a quirky parts of Unix, with lots of moving parts; it is difficult to support them in s6-overlay for a variety of reasons. I am, unfortunately, not at all surprised that it doesn't work when you add qemu into the mix - qemu also has to do its own terminal shenanigans, and it is to be expected that they don't mesh well with containers underneath.

I'm sorry I don't have a better answer for you, but try fiddling with qemu's setting? This is not something I can help with at the s6-overlay level.

[midzelis]

I was able to workaround this issue by just making sure to not docker run with it. later exec it into that container work fine.

Here is script I use to run and enter one a s6 overlay based container using multi-platform via QEMU.

#!/bin/bash
# Note - S6 overlay does NOT like it when you give it a tty (-it) under QEMU. Run it in background (so we get log output) then
# exec a bash process to enter the container. 
# Install a trap to monitor when bash script exits, and stop the running container with its done

trap "trap - SIGTERM && docker stop insert_your_container_name" SIGINT SIGTERM EXIT
docker run --rm --name insert_your_container_name--platform linux/arm/v7 insert_your_container_name:insert_tag &
sleep 5
docker exec -it insert_your_container_name bash

Do you think you could add a section to your FAQ specifically about avoiding -it when invoking containers using QEMU emulation?

[skarnet]

Yeah, I'll add a note about that.

[skarnet]

Note added in the README.md, under the Notes section, and the new release is out. Can I close this issue?