-
Notifications
You must be signed in to change notification settings - Fork 98
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fuse-overlayfs on Ubuntu 22.04 hangs when remounting on top of lowerdir #1851
Comments
Hi @DrDaveD - I'm not clear whether you have replicated this error with the sylabs/singularity codebase, or the code as it is in your PR - vzokay/apptainer#335 Our CI runs on Ubuntu 22.04 and does not show any issue: I am also unable to reproduce on our main branch in an Ubuntu 22.04 VM:
I have confirmed there are no leftover namespaces, or stuck processes on completion of these passing tests on my VM. If you are replicating this error on If it is only seen on your cherry-pick PR then this will be out of scope for us to look at. I'd ask that if you report issues found when cherry-picking from sylabs/singularity that they are only reported if they have been reproduced in sylabs/singularity. It is out of scope for our project to consider issues for cherry-picks where the code has been adapted, and not necessarily applied in the same order etc. Thanks! |
From a quick read-around it appears that what you are seeing may be due to a bug that has been fixed upstream in fuse-overlayfs: containers/fuse-overlayfs#364 ... but is perhaps present in the version that is in Ubuntu 22.04. However... given that our tests don't appear to hit this bug path on Ubuntu 22.04, this isn't something we'll be looking to workaround unless it is possible to replicate it on sylabs/singularity main rather than the apptainer cherry-picking. If you can exercise the bug on sylabs/singularity main, via the tests or interactive usage of Thanks! |
Good find on that issue, DT. I tried to reproduce it now in singularity-ce and found that fuse-overlayfs isn't getting called at all. And that actually makes sense, because it is only supposed to be called when unprivileged overlayfs isn't available, and it is available on ubuntu 22.04. So the real problem seems to be with the cherry-pick's identification of the availability of unprivileged overlayfs. I'll look into it. |
When attempting to import #1730 into an apptainer feature branch via vzokay/apptainer#335 we experienced unit tests getting stuck indefinitely. I was able to reproduce that interactively on an Ubuntu 22.04 VM with
Each time it ran, fuse-overlayfs would get into an unkillable state, even with kill -9. I noticed that both the lowerdir and the mountpoint were the same, ending in "rootfs". I first tried to change the lowerdir name, but got a complaint about it needing to be named "rootfs", so instead in this commit I changed the name of the mountpoint, and that made it pass all tests. I'm not sure that it is actually functioning as expected (I'm not sure if anything is looking at the new mountpoint or if the tests just happen to be happy with the read-only original mountpoint), however. Apparently the kernel overlayfs module has no problem with keeping a handle to the underlying mount point and then mounting on top of it, because that's what pkg/ocibundle/tools/overlay_linux.go has always done, but apparently fuse-overlayfs at least on Ubuntu 22.04 can't handle it.
The text was updated successfully, but these errors were encountered: