fuse-overlayfs on Ubuntu 22.04 hangs when remounting on top of lowerdir

[DrDaveD]

When attempting to import #1730 into an apptainer feature branch via vzokay/apptainer#335 we experienced unit tests getting stuck indefinitely. I was able to reproduce that interactively on an Ubuntu 22.04 VM with

cd pkg/ocibundle
sudo go test -v -count=1 ./sif

Each time it ran, fuse-overlayfs would get into an unkillable state, even with kill -9. I noticed that both the lowerdir and the mountpoint were the same, ending in "rootfs". I first tried to change the lowerdir name, but got a complaint about it needing to be named "rootfs", so instead in this commit I changed the name of the mountpoint, and that made it pass all tests. I'm not sure that it is actually functioning as expected (I'm not sure if anything is looking at the new mountpoint or if the tests just happen to be happy with the read-only original mountpoint), however. Apparently the kernel overlayfs module has no problem with keeping a handle to the underlying mount point and then mounting on top of it, because that's what pkg/ocibundle/tools/overlay_linux.go has always done, but apparently fuse-overlayfs at least on Ubuntu 22.04 can't handle it.

[dtrudg]

Hi @DrDaveD - I'm not clear whether you have replicated this error with the sylabs/singularity codebase, or the code as it is in your PR - vzokay/apptainer#335

Our CI runs on Ubuntu 22.04 and does not show any issue:

I am also unable to reproduce on our main branch in an Ubuntu 22.04 VM:

root@dev-ubuntu2204:/home/dtrudg-sylabs/Git_Sylabs/singularity# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.2 LTS"

root@dev-ubuntu2204:/home/dtrudg-sylabs/Git_Sylabs/singularity# git describe
v3.11.0-426-g825b20b70

root@dev-ubuntu2204:/home/dtrudg-sylabs/Git_Sylabs/singularity# go test -v -count=1 ./pkg/ocibundle/sif
=== RUN   TestFromSif
=== RUN   TestFromSif/FromSif
time="2023-07-07T09:49:51+01:00" level=warning msg="Checking linux mount types without --host-specific is not supported yet"
=== RUN   TestFromSif/FromSifWritable
time="2023-07-07T09:49:52+01:00" level=warning msg="Checking linux mount types without --host-specific is not supported yet"
--- PASS: TestFromSif (1.71s)
    --- PASS: TestFromSif/FromSif (1.03s)
    --- PASS: TestFromSif/FromSifWritable (0.68s)
PASS
ok  	github.com/sylabs/singularity/pkg/ocibundle/sif	1.715s

I have confirmed there are no leftover namespaces, or stuck processes on completion of these passing tests on my VM.

If you are replicating this error on sylabs/singularity please let us know, with any more detail you can give about the VM config / how you are running the test.

If it is only seen on your cherry-pick PR then this will be out of scope for us to look at.

I'd ask that if you report issues found when cherry-picking from sylabs/singularity that they are only reported if they have been reproduced in sylabs/singularity. It is out of scope for our project to consider issues for cherry-picks where the code has been adapted, and not necessarily applied in the same order etc.

Thanks!

[dtrudg]

From a quick read-around it appears that what you are seeing may be due to a bug that has been fixed upstream in fuse-overlayfs:

containers/fuse-overlayfs#364
containers/fuse-overlayfs#366

... but is perhaps present in the version that is in Ubuntu 22.04.

However... given that our tests don't appear to hit this bug path on Ubuntu 22.04, this isn't something we'll be looking to workaround unless it is possible to replicate it on sylabs/singularity main rather than the apptainer cherry-picking.

If you can exercise the bug on sylabs/singularity main, via the tests or interactive usage of singularity oci / singularity --oci, then it's likely something we would want to work around given the lack of updates to fuse-overlayfs that make it to Ubuntu LTS.

Thanks!

[DrDaveD]

Good find on that issue, DT.

I tried to reproduce it now in singularity-ce and found that fuse-overlayfs isn't getting called at all. And that actually makes sense, because it is only supposed to be called when unprivileged overlayfs isn't available, and it is available on ubuntu 22.04. So the real problem seems to be with the cherry-pick's identification of the availability of unprivileged overlayfs. I'll look into it.