run --rootfs is broken, (at least) 2 different ways

[edsantiago]

Background: this system test

podman/test/system/030-run.bats

Lines 624 to 632 in 0c0d68e

	# Make sure that running on a read-only rootfs works (#9230).
	if ! is_rootless && ! is_remote; then
	# image mount is hard to test as a rootless user
	# and does not work remotely
	run_podman image mount $IMAGE
	romount="$output"
	run_podman run --rm --rootfs $romount echo "Hello world"
	is "$output" "Hello world"

History PR #9231, which claims to fix #9230

Problem: the test works with quay.io/libpod/testimage:20210610, but does not work with any other image that I can find. These are the two three failure modes:

Error: OCI runtime error: crun: mkdir `secrets`: Read-only file system

or

Error: crun: open `/var/lib/..../merged/.containerenv`: No such file or directory: 
    -> OCI runtime attempted to invoke a command that was not found

or

(same as above, but with resolv.conf instead of .containerenv)

The error is not deterministic: a simple !! of the reproducer can fail in any of those three ways. (Usually the .containerenv one, but not always).

Reproducer:

#!/bin/bash
sudo bin/podman pull -q "$1"
x=$(sudo bin/podman image mount "$1")
sudo bin/podman run --rm --rootfs $x echo hi
sudo bin/podman image unmount "$1"

Save as test-mounting, chmod it +x, and run it with:

$ ./test-mounting quay.io/libpod/testimage:20210610     <---- this always works
$ ./test-mounting docker.io/library/alpine:3.13.5  (or 3.14, or 3.16)
     ^------ this always fails, even though 3.13.5 is the image that testimage:20210610 is based on

@vrothberg PTAL

[vrothberg]

The described symptoms suggest a race condition. My queue is quite full today but maybe I find time.

[edsantiago]

Oh, no hurry: if noone has noticed in more than a year, it's not that big a deal. (It's just blocking me from fixing #14456, but I can always just disable the --rootfs. And I'm OOTO today, so I can't do anything until Wednesday).

[Luap99]

Note that the difference in the image is likely the /run and /etc directory because podman tries always tries to bind mount certain files such as /run/.containerenv, /etc/resolv.conf, /etc/hosts and more. In order to bind mount to a target, the file/directory must exists. Normally this is not a problem since the OCI runtime does this for us (see the open/mkdir error). However in this case the fs is readonly so creating the file/directory cannot work.

[rhatdan]

Add an Overlay :O mount to the --rootfs object.

 mnt=$(podman image mount alpine)
sh-5.1# podman run --rm --rootfs $mnt ls / 
Error: OCI runtime error: crun: mkdir `secrets`: Read-only file system
sh-5.1# podman run --rm --rootfs $mnt:O ls / 
bin
dev
etc
home
lib
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var

[Luap99]

The point of the test is to use --rootfs with an readonly rootfs and not overlay.

[vrothberg]

The point of the test is to use --rootfs with an readonly rootfs and not overlay.

I think we should check whether the --rootfs is read-only and carry that information along. If it's RO, then Podman shouldn't attempt to write on it. Probably worth emitting a warning and pointing to the overlay mount alternative.

[edsantiago]

While this is getting resolved, may I just submit my PR with this test skipped?

[vrothberg]

While this is getting resolved, may I just submit my PR with this test skipped?

It's probably a good idea. Alternatively, we can use :O for now. Once the issue is fixed, the test should pass as is now.

[rhatdan]

I do not know what the general fix for this is, other then to state that all of the mount points need to be precreated in the original image for --rootfs to work on a read-only file system. We could attempt to give a list of these locations, but I am not sure how complete it would be and difficult to maintain going forward.

[vrothberg]

I do not know what the general fix for this is, other then to state that all of the mount points need to be precreated in the original image for --rootfs to work on a read-only file system. We could attempt to give a list of these locations, but I am not sure how complete it would be and difficult to maintain going forward.

So should we detect a RO rootfs and error out?

[Luap99]

No, this clearly works as long as you have the correct paths that podman bind mounts already in the rootfs. There are options to turn off the bind mounts for /etc/{hosts,resolv.conf}. I do not know if there is a option for the /run mounts? Although I wonder why does podman tries to create /run/secrets when the container does not use any secrets.

[rhatdan]

We could, but this use case might work if /dev, /run, /etc/hosts, /etc/resolv.conf exist in the image.

[rhatdan]

/run/secrets is created for RHEL stuff as well. Not sure if it is always created. A container without read/write /run would fail is most cases as well.

[giuseppe]

I think we should not support a read-only rootfs, many things can break. e.g. what to do if the rootfs is missing a /proc directory?
If the rootfs must be read-only then it can be done only after all the mounts are created by the OCI runtime and we specify --read-only to run.

[rhatdan]

I agree, lets just force people to use Overlay for this use case.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Since we agreed to force overlay for this use case I am going to close.