Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to configure container ... overlay mount for '/' failed #776

Closed
PhilippWendler opened this issue Oct 19, 2021 · 2 comments · Fixed by #1062
Closed

Failed to configure container ... overlay mount for '/' failed #776

PhilippWendler opened this issue Oct 19, 2021 · 2 comments · Fixed by #1062
Labels
container related to container mode high priority

Comments

@PhilippWendler
Copy link
Member

PhilippWendler commented Oct 19, 2021

On Ubuntu 18.04 with kernel 4.15.0-159 and on Ubuntu 20.04 with kernel 5.4.0-89-generic (and likely similarly recent versions) BenchExec fails by default with the error

CRITICAL - Failed to configure container: [Errno 22] Creating overlay mount for '/' failed: Invalid argument. Please use other directory modes, for example '--read-only-dir /'.

Avoiding the use of overlay for / (e.g., with --read-only-dir /) indeed makes it work. Overlay mode for other directories works.

According to the changelog (here for 18.04) the respective patch is very likely the one named "ovl: prevent private clone if bind mount is not allowed", which is originally here and was backported to older kernels as well.

We have to investigate what exactly is the condition that makes it fail for / and lets it work for other directories, and then consider either reporting this as a regression or find a workaround. As the patch is a security fix (announcements from Ubuntu, Debian), we might need the latter.

@PhilippWendler PhilippWendler added high priority container related to container mode labels Oct 19, 2021
@PhilippWendler
Copy link
Member Author

The problem for overlay mounts affects all directories that are non-leaf mount points (i.e., directories below which there is another mount point somewhere). This means that for most people a suitable workaround should be something like --read-only-dir / --overlay-dir . or --read-only-dir / --overlay-dir ~, assuming that write access is needed only below . or ~, respectively, and that there are no submounts in these directories.

I have reported an Ubuntu bug in their tracker.

@PhilippWendler PhilippWendler pinned this issue Oct 19, 2021
PhilippWendler added a commit that referenced this issue Feb 22, 2022
If directory modes for /foo and /foo/bar are given,
and these two modes are the same,
and /foo/bar is not already a mountpoint,
there is no point in doing something for /foo/bar:
The directory mode for /foo will apply to /foo/bar as well anyway.

Avoiding these useless nested mounts is especially good for overlay mounts:
Overlayfs complains (in the kernel log) if we first use .../temp/foo as
upper directory of an overlayfs instance
and then use .../temp/foo/bar as an upper directory of another overlayfs
instance as well.
The message says that we should not create overlayfs instances
with overlapping upper directories.

With the change of this commit we do this less often,
we only create overlapping overlayfs mounts in the following two
situations:
a) if /foo and /foo/bar are both overlays and on different file systems,
but currently this fails anyway due to #776; and
b) if /foo is an overlay, /foo/bar is something else, and /foo/bar/child
is an overlay again.
However, so far this has worked without problems.
This is likely because if we use both .../temp/foo and
.../temp/foo/bar/child as upper directories for two overlayfs instances,
we will never actually access the bar/child directory through the former
instance. Each actual directory in the upper directory is accessed only
via a single overlayfs instance.
thaprau pushed a commit to thaprau/benchexec that referenced this issue Mar 28, 2022
@dbeyer dbeyer unpinned this issue Feb 3, 2023
@dbeyer dbeyer pinned this issue Feb 3, 2023
@PhilippWendler PhilippWendler changed the title Overlay for / fails on recent kernels (Ubuntu and likely others) Failed to configure container ... overlay mount for '/' failed Jun 28, 2024
@PhilippWendler PhilippWendler linked a pull request Sep 19, 2024 that will close this issue
@PhilippWendler
Copy link
Member Author

For those affected by this issue, you can install BenchExec 3.25 and fuse-overlayfs 1.10 or newer (available for example in Ubuntu 24.04) and then the default directory works again without the need for --read-only-dir / or similar.

@PhilippWendler PhilippWendler unpinned this issue Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
container related to container mode high priority
Development

Successfully merging a pull request may close this issue.

1 participant