S6 Overlay 3.0.0.x hangs on executing interactive terminal on M1 Macs #396

jaydrogers · 2022-02-04T22:38:32Z

Issue Description

When I attempt to run a simple S6 Overlay container with the -it flag on a Macbook Pro M1 Max, the container hangs and no output is displayed. I cannot exit the container from CLI either.

Important notes:

This is not able to be replicated on Intel-based Macs.
Version 2.x of S6 Overlay works fine on M1 Macs.

Steps to reproduce the issue

Find someone with an M1 Max computer
Run docker run --platform linux/amd64 --rm -it serversideup/docker-baseimage-s6-overlay-ubuntu:edge-3-20.04 bash

You will notice that a bash shell does not appear, nor can you exit the container with CTRL + C.

Related issues on other repositories

I have a PR that I am working on for my own open source repo. You can see the full conversation if you'd like here: serversideup/s6-overlay#3

The serversideup/docker-baseimage-s6-overlay-ubuntu:edge-3-20.04 is a prebuilt image of the Dockerfile below:

My Dockerfile

Dockerfile

# This pipeline will keep image size as small as possible.
################################################################################

# base image and platform
# E.g: FROM --platform=linux/amd64 ubuntu:20.04 as base
FROM ubuntu:20.04 as base

################################################################################

# extracts s6-overlay bundles into a temporary image
FROM base as source

# sets version for s6 overlay
ARG DEBIAN_FRONTEND="noninteractive"
ARG S6_OVERLAY_BUILD_DEP="wget xz-utils"
ARG S6_OVERLAY_SRC="https://github.com/just-containers/s6-overlay/releases/download"
ARG S6_OVERLAY_VERSION="3.0.0.2"
ARG S6_OVERLAY_ARCH="x86_64"

# installation pipeline
RUN apt-get update \
    && apt-get install -y --no-install-recommends --no-install-suggests \
        $S6_OVERLAY_BUILD_DEP \
    \
    # adds s6 overlay
    && mkdir -p /usr/src/s6 \
    && untar(){ wget --no-check-certificate -O- $1 2>/dev/null | tar Jxp -C ${2:-/usr/src/s6/}; }\
    && untar ${S6_OVERLAY_SRC}/v${S6_OVERLAY_VERSION}/s6-overlay-noarch-${S6_OVERLAY_VERSION}.tar.xz \
    && untar ${S6_OVERLAY_SRC}/v${S6_OVERLAY_VERSION}/s6-overlay-${S6_OVERLAY_ARCH}-${S6_OVERLAY_VERSION}.tar.xz

################################################################################

# main image
FROM  base
LABEL maintainer="@jaydrogers"
LABEL contributor="@shinsenter"

# copies s6-overlay from source
COPY --from=source /usr/src/s6/ /

# sets environment variables
ENV DEBIAN_FRONTEND="noninteractive"
ENV HOME="/root"
ENV LANG="en_US.UTF-8"
ENV LANGUAGE="en_US.UTF-8"
ENV LC_ALL="C"
ENV TERM="xterm"

################################################################################

ENTRYPOINT ["/init"]

Environment

✅ Works fine on an Intel Mac

Operating System Version:
ProductName:	macOS
ProductVersion:	11.6.1
BuildVersion:	20G224

Docker Info:
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.7.1)
  compose: Docker Compose (Docker Inc., v2.2.3)
  scan: Docker Scan (Docker Inc., v0.16.0)

Server:
 Containers: 81
  Running: 8
  Paused: 0
  Stopped: 73
 Images: 138
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.774GiB
 Name: docker-desktop
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 123
  Goroutines: 131
  System Time: 2022-02-04T22:02:10.3709249Z
  EventsListeners: 5
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

❌ DOES NOT work on M1 Mac

Operating System Version:
ProductName:	macOS
ProductVersion:	12.1
BuildVersion:	21C52

Docker Info:
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.7.1)
  compose: Docker Compose (Docker Inc., v2.2.3)
  scan: Docker Scan (Docker Inc., v0.16.0)

Server:
 Containers: 7
  Running: 1
  Paused: 0
  Stopped: 6
 Images: 9
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 3.841GiB
 Name: docker-desktop
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Next steps

Thanks for maintaining this project. Please let me know if you need any more information! Would greatly appreciate any insight on this! 🙌

The text was updated successfully, but these errors were encountered:

skarnet · 2022-02-04T23:37:55Z

Basic sanity check: when you're running the container on a M1 Mac, are you changing S6_OVERLAY_ARCH to aarch64?

If you are, it's probably because our aarch64 binaries are optimized for the Raspberry Pi 4, and not suited to the Firestorm and Icestorm, in which case I'll try and make a toolchain for the M1 so we can explicitly support it.

shinsenter · 2022-02-05T00:58:51Z

@skarnet @jaydrogers
Changing S6_OVERLAY_ARCH to aarch64 works for me. Thank you!

jaydrogers · 2022-02-07T15:08:44Z

Thank you @skarnet and @shinsenter!

I confirmed changing S6_OVERLAY_ARCH to aarch64 worked.

Just out of curiosity

In S6 Overlay v2.x, I was able to run this on my M1 Mac:

docker run --platform linux/amd64 --rm -it {{my-image}} bash

With --platform linux/amd64, I assume it would emulate things with Rosetta through Docker Desktop? Is there any reason this would not work in S6 Overlay 3.x?

I'm a novice when it comes to the differences in architectures, so I would appreciate any insight if you could share some. Thanks again for all your work!

skarnet · 2022-02-07T15:55:03Z

I have no idea! All I know is that our prebuilt -${arch} tarballs contain architecture-specific binaries and things work better if you use the binaries that are suited to your architecture. ;-) I am totally unacquainted with Docker Desktop or the mechanisms of Apple products, including the Rosetta emulator.
Maybe @jprjr knows why it worked with the previous versions.

jprjr · 2022-02-07T23:49:18Z

I honestly have no idea how the amd64 version was working on Intel macs.

I do know that on macOS, Docker Desktop is running a Linux VM. From that output above, it looks like Docker Desktop runs a VM that matches your host architecture (presumably on amd64, it runs an amd64 VM, on aarch64, it runs an aarch64 VM, etc).

Docker Desktop is supposed to have transparent support for multiple architectures, and it mentions QEMU on this page but it's scant on details.

So, I'm unsure if the translation is happening in:

The VM layer, with something like QEMU user mode
The Docker Desktop layer, by launching multiple/different VMs
The MacOS layer, via Rosetta.

I'm sure there's a document somewhere with better details, but on Docker Desktop it seems to be pretty much magic.

skarnet · 2022-02-27T03:08:34Z

Closing this - we'll solve the mysteries of Docker Desktop another day in another issue. ;-)

icereval · 2022-04-11T22:00:32Z

When running under QEMU on an M1 w/ --platform linux/amd64, if you drop the tty option when executing docker run -i ... s6 overlay seems to startup.

-t, --tty Allocate a pseudo-TTY

jaydrogers mentioned this issue Feb 4, 2022

[ ❌ ABANDONED] Upgrade S6 Overlay to v3.0.0.0 serversideup/s6-overlay#3

Closed

3 tasks

skarnet closed this as completed Feb 27, 2022

midzelis mentioned this issue Jun 22, 2023

/init hangs when starting a container using non-native image (QEMU) #537

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

S6 Overlay 3.0.0.x hangs on executing interactive terminal on M1 Macs #396

S6 Overlay 3.0.0.x hangs on executing interactive terminal on M1 Macs #396

jaydrogers commented Feb 4, 2022 •

edited

Loading

skarnet commented Feb 4, 2022

shinsenter commented Feb 5, 2022

jaydrogers commented Feb 7, 2022

skarnet commented Feb 7, 2022

jprjr commented Feb 7, 2022 •

edited

Loading

skarnet commented Feb 27, 2022

icereval commented Apr 11, 2022

S6 Overlay 3.0.0.x hangs on executing interactive terminal on M1 Macs #396

S6 Overlay 3.0.0.x hangs on executing interactive terminal on M1 Macs #396

Comments

jaydrogers commented Feb 4, 2022 • edited Loading

Issue Description

Steps to reproduce the issue

Related issues on other repositories

My Dockerfile

Environment

✅ Works fine on an Intel Mac

❌ DOES NOT work on M1 Mac

Next steps

skarnet commented Feb 4, 2022

shinsenter commented Feb 5, 2022

jaydrogers commented Feb 7, 2022

Just out of curiosity

skarnet commented Feb 7, 2022

jprjr commented Feb 7, 2022 • edited Loading

skarnet commented Feb 27, 2022

icereval commented Apr 11, 2022

jaydrogers commented Feb 4, 2022 •

edited

Loading

jprjr commented Feb 7, 2022 •

edited

Loading