ERGURNT: s6-overlay failed to start service on docker-1.10.3

[pubyun]

after we upgrade docker 1.9.1 to docker-1.10.3 on centos 7, all docker container failed to start service:

s6-supervise (child): fatal: unable to exec run: Permission denied
s6-supervise (child): fatal: unable to exec run: Permission denied
s6-supervise supervisord: warning: unable to spawn ./run - waiting 10 seconds
s6-supervise sshd: warning: unable to spawn ./run - waiting 10 seconds
s6-supervise (child): fatal: unable to exec run: Permission denied
s6-supervise cron: warning: unable to spawn ./run - waiting 10 seconds

[pubyun]

downgrade to docker 1.9.1, then it's OK

[jprjr]

@pubyun can you provide an example Dockerfile and run script?

[pubyun]

FROM docker.pubyun.com/pubyun/centos

ENV S6_OVERLAY_VERSION v1.18.1.0

ADD file/ /

ADD https://github.com/just-containers/s6-overlay/releases/download/v1.11.0.1/s6-overlay-amd64.tar.gz /tmp/
RUN tar xzf /tmp/s6-overlay-amd64.tar.gz -C /
&& mv /bin /bin-s6
&& rm /usr/bin/execlineb
&& /bin-s6/s6-update-symlinks /bin /bin-s6 /usr/bin
&& ln -s /bin/execlineb /usr/bin/execlineb
&& mv /sbin /sbin-s6
&& /bin-s6/s6-update-symlinks /sbin /sbin-s6 /usr/sbin
&& rm /tmp/s6-overlay-amd64.tar.gz

EXPOSE 22

ENTRYPOINT ["/init"]

[pubyun]

$ cat file/etc/services.d/sshd/run
#!/usr/bin/execlineb -P

/usr/sbin/sshd -D

[pubyun]

type error in Dockfile, i have update to latest s6-overlay, same error:
ADD https://github.com/just-containers/s6-overlay/releases/download/v1.18.1.1/s6-overlay-amd64.tar.gz

[jprjr]

I can't pull from docker.pubyun.com/pubyun/centos, but here's a something to try out.

I think you're doing a lot to work around a known issue: https://github.com/just-containers/s6-overlay#bin-and-sbin-are-symlinks - try just splitting up your tar extraction into two steps

I made this Dockerfile:

FROM centos:7

ENV S6_OVERLAY_VERSION v1.18.1.1

ADD file/ /

ADD https://github.com/just-containers/s6-overlay/releases/download/v1.18.1.1/s6-overlay-amd64.tar.gz /tmp/
RUN tar xzf /tmp/s6-overlay-amd64.tar.gz -C / --exclude="./bin" --exclude="./sbin" \
&& tar xzf /tmp/s6-overlay-amd64.tar.gz -C /usr ./bin ./sbin \
&& yum install -y openssh-server

EXPOSE 22

ENTRYPOINT ["/init"]

I was able to build and run it just fine on docker 1.11.2

[pubyun]

i am runing docker on CentOS 7.2, which have many patches from RedHat.

maybe this patch trigger the problem. it will merge into docker 1.12, but have patched for docker 1.10.3 of redhat:
moby/moby#23301

https://github.com/projectatomic/docker.git

i add one line to Dockerfile, it works now.

VOLUME ["/run"]

can you test it on Docker 1.12?

[jprjr]

It sounds like your problem might be with the CentOS version of Docker? I'm unable to replicate your issue so I'm not sure if there's anything I can do.

[pubyun]

i test package from docker.com, version 1.11 and 1.12, it's OK.
https://yum.dockerproject.org/repo/main/centos/7/
https://yum.dockerproject.org/repo/testing/centos/7/

it fails when i use official package from centos:
http://mirror.centos.org/centos/7/extras

the "/run" directory was locked, and any files can't be executed under it.

i add one line to Dockerfile, then it works:

VOLUME ["/run"]

[pubyun]

i find the reason, there is noexec flag on /run:

sudo docker run --rm=true -ti centos:s6 /bin/bash -c 'mount | grep run'

/dev/mapper/vg0-var on /run/secrets type xfs (rw,relatime,attr2,inode64,noquota)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,mode=755)

[glerchundi]

@pubyun is this something centos+s6 related? Or just your custom docker image which overrides the /run folder? Thanks

[nomadalex]

@glerchundi when use without /init as ENTRYPOINT, like /bin/bash, the /run is not mount use tmpfs.
but when use /init as ENTRYPOINT, the /run is mount use tmpfs with noexec flag. I don't know why, but it make s6-overlay useless on CentOS 7, my host is CentOS 7 with official docker, docker base container is centos:7. Can you change the service dir for better compatible?

[pubyun]

@glerchundi this is centos7 + s6 related. the script can't run for the noexec flag of /run

[danielunderwood]

I'm having this issue as well on CentOS 7.2 with Docker 1.10.3. It seems to be resolved after updating to Docker 1.12.1 with the directions in the Docker docs. I also forced recreation of the images after updating docker, which may have had some effect on SELinux flags (z on volumes).

[pubyun]

ping

[pubyun]

moby/moby#26528

if we rename /init script to other name, then all is OK.

[pubyun]

https://bugzilla.redhat.com/show_bug.cgi?id=1375661

[pubyun]

i fix the issue:

ADD https://github.com/just-containers/s6-overlay/releases/download/v1.18.1.1/s6-overlay-amd64.tar.gz
RUN tar xzf /tmp/s6-overlay-amd64.tar.gz -C / --exclude="./bin" --exclude="./sbin"
&& tar xzf /tmp/s6-overlay-amd64.tar.gz -C /usr ./bin ./sbin
&& ln -s /init /init.rhel
&& rm /tmp/s6-overlay-amd64.tar.gz

ENTRYPOINT ["/init.rhel"]

[simw]

I have the same issue using Docker packages in Fedora 25. You have most of the info above, but just to put my understanding of it together:

• Fedora / RHEL / CentOS run a patched version of Docker.
• One of the patches runs hooks before / after a container is launched, including 'oci-systemd-hook'.
• This hook assumes that if the command is init or systemd, then the container is running systemd, and does a few extra things, including mounting /run (linked to /var/run) as tmpfs and noexec.
• init-stage2 in s6 copies services scripts to /var/run, then tries to execute, and fails

While the patch adding hooks hasn't yet been accepted upstream in docker, there is on-going discussion.

As mentioned, changing the name from init to something else would 'fix' this. The workaround VOLUME ["/run"] works for me, as does running docker run -v /run ... There aren't offical docker packages for Fedora 25 yet (I haven't tried using the Fedora 24 repo in 25).

[jprjr]

Look I'll be real, if this is still an issue and somebody has a fix, open a PR. I'm not going to fix it.

[adambkaplan]

I just tested this via podman run on Fedora 33 Silverblue. The /run issue isn't a problem, Plex seems to run just fine.

Note that docker is not supported in RHEL 8. Docker can be run on Fedora 32 and higher, but support there comes from the Docker community, not Fedora's.