Progress Log for "Overlay Handling with fuse-overlayfs"

[younghojan]

I am the accepted contributor for the GSoC 2024 project "overlay handling with fuse-overlayfs #928". This discussion will serve as the project progress log during GSoC 2024. I will post real-time updates on the current progress in this discussion, and I will also respond to any related questions as soon as possible.

Welcome everyone to follow the project and its progress!

[younghojan]

Progress Made up (July 2nd)

All the code is on the branch gsoc-overlay-handling-with-fuse-overlayfs-dev.

Current Progress

So far, we have achieved the following:

• By preserving all capabilities granted in the parent user namespace for the child process, we successfully utilize fuse-overlayfs (fusermount) to perform overlay mounts in containerexec and runexec (benchexec not supported currently).
• Another way to make fuse-overlayfs work is to avoid using the cap_permitted_to_ambient function to preserve capabilities. Instead, by using the --root parameter to run containerexec and runexec, a fake root is used within the container, ensuring that fuse-overlayfs no longer faces any permission issues.

Comparison to Plan

I must admit that the current progress is slower than the planned schedule. According to the original plan, fuse-overlayfs should now be fully replacing the overlay mechanism in BenchExec, and fuse-overlayfs should be used as a fallback for kernel overlayfs, which is milestone 3, "Improve the prototype to incorporate the fallback mechanism required by the goal". Due to academic matters that delayed me in the past week or two, I will catch up with the planned progress as soon as possible.

Challenges and Roadblocks

There are currently two issues, with the first being more urgent and challenging:

1. Using fuse-overlayfs in benchexec results in the error "Transport endpoint is not connected".

In the code, we first overlay mount the root directory to the corresponding mount_path. Then, we mount other mount points under the root directory to the corresponding locations under mount_path, obtaining a copy of the system's mount hierarchy. Mounting the root directory using fuse-overlayfs does not cause problems, but immediately after, binding mounts of other mount points under the root directory, such as /dev which should be mounted in read-only mode, result in the error "Transport endpoint is not connected". This issue is not specific to a particular mount point or a specific mounting mode, but rather a general problem. It may be related to benchexec running logic, which uses unshare() to put the current process into new namespaces, whereas containerexec and runexec use clone(), so they can run normally. According to my rough tests, if the libc.CLONE_NEWPID parameter is removed from unshare(), the duplicate_mount_hierarchy() can complete successfully. However, libc.CLONE_NEWPID is necessary as it creates a new pid namespace, and not using it will cause other fatal errors, so it should not be removed. The root cause and solution to this issue are still unclear, and it might be related to bind mounting a fuse mountpoint, as discussed in a related discussion on StackOverflow. I am currently further discussing this with my mentor, Dr. Philipp Wendler.

2. Will using cap_permitted_to_ambient() to preserve caps for child processes cause any unintended side effects?

In containerexec and runexec, executing container.cap_permitted_to_ambient() in child() ensures that capabilities granted in the user namespace are preserved in the child process. This approach is borrowed from the implementation of the Linux unshare command and is equivalent to unshare --keep-caps. We have not distinguished which caps are necessary and which are not, but I currently believe that container.drop_capabilities() ensures that unnecessary caps are dropped.

Future Plans

First, we need to quickly resolve the first issue mentioned above to ensure that fuse-overlayfs functions correctly in all scenarios. Next, we will set it as a fallback for kernel overlayfs. After that, we will address testing and documentation related matters.

[PhilippWendler]

2. Will using cap_permitted_to_ambient() to preserve caps for child processes cause any unintended side effects?

We have not distinguished which caps are necessary and which are not,

I think as one of the future steps it would be nice to do this (but it is not on the critical path).

but I currently believe that container.drop_capabilities() ensures that unnecessary caps are dropped.

I agree. However, in some cases we deliberately do not drop all capabilities, and in this case they would currently also remain as ambient capabilities. So I would say we should simply ensure that all ambient capabilities are cleared, no matter which capabilities are kept in other sets. This could be done either in drop_capabilities() or as part of the mounting code (for example, one could use a context manager that adds some ambient capabilities on enter and removes them when exiting the context).

[younghojan]

Progress Made up (July 10th)

All the code is on the branch gsoc-overlay-handling-with-fuse-overlayfs-dev.

Current Progress

All functions of BenchExec work correctly, with the fuse-based overlay mount serving as a fallback for the kernel overlay mount and providing appropriate, comprehensive messages. This was tested on an Ubuntu Server 24.04/Linux 6.8.0-36-generic.

Comparison to Plan

The current progress is consistent with the plan, with no significant delays or advancements.

Challenges and Roadblocks

On Ubuntu 22.04, containerexec hangs after Mounting '/run' as hidden, showing "Child not ready after 60s," and the grandchild process cannot start executing the benchmarked tool. This may be due to a deadlock in the child process. There is a related issue at #656. It is currently unclear what is causing this issue, as everything runs fine on another Ubuntu Server 24.04/Linux 6.8.0-36-generic. We may need more machines with different versions to determine if this is caused by version differences.

Here is the full log:

(benchexec-venv) dadayang@haoranyang:~$ containerexec --debug ls
2024-07-10 15:05:51 - DEBUG - This is containerexec 3.22-dev.
2024-07-10 15:05:51 - INFO - Starting command ls
2024-07-10 15:05:51 - DEBUG - Available Cgroups: {}
2024-07-10 15:05:51 - DEBUG - Starting process.
2024-07-10 15:05:51 - DEBUG - Parent: child process of RunExecutor with PID 1509 started.
2024-07-10 15:05:51 - DEBUG - Child: child process of RunExecutor with PID 1509 started
2024-07-10 15:05:51 - DEBUG - Failed to make b'/tmp/BenchExec_run_wqsljv85/mount/home/benchexec' a bind mount: [Errno 2] mount(b'/tmp/BenchExec_run_wqsljv85/mount/home/benchexec', b'/tmp/BenchExec_run_wqsljv85/mount/home/benchexec', None, 4096, None) failed: No such file or directory
2024-07-10 15:05:51 - DEBUG - Mounting '/' as overlay
2024-07-10 15:05:51 - DEBUG - [Errno 16] umount(b'/tmp/BenchExec_run_wqsljv85/mount/') failed: Device or resource busy
2024-07-10 15:05:51 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_wqsljv85/mount/', lower=b'/', upper=b'/tmp/BenchExec_run_wqsljv85/temp/', work=b'/tmp/BenchExec_run_wqsljv85/overlayfs/1'
2024-07-10 15:05:51 - DEBUG - Cannot use kernel overlay for /: [Errno 22] mount(b'none', b'/tmp/BenchExec_run_wqsljv85/mount/', b'overlay', 0, b'lowerdir=/,upperdir=/tmp/BenchExec_run_wqsljv85/temp/,workdir=/tmp/BenchExec_run_wqsljv85/overlayfs/1') failed: Invalid argument. Trying to use fuse-overlayfs instead.
2024-07-10 15:05:51 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_wqsljv85/mount/', lower=b'/', upper=b'/tmp/BenchExec_run_wqsljv85/temp/', work=b'/tmp/BenchExec_run_wqsljv85/overlayfs/1'
2024-07-10 15:05:51 - DEBUG - Mounting '/dev' as read-only
2024-07-10 15:05:51 - DEBUG - Mounting '/dev/pts' as read-only
2024-07-10 15:05:51 - DEBUG - Mounting '/dev/shm' as read-only
2024-07-10 15:05:51 - DEBUG - Mounting '/dev/hugepages' as read-only
2024-07-10 15:05:51 - DEBUG - Mounting '/dev/mqueue' as read-only
2024-07-10 15:05:51 - DEBUG - Mounting '/run' as hidden
2024-07-10 15:05:51 - DEBUG - Mounting '/sys' as read-only
2024-07-10 15:05:51 - DEBUG - Mounting '/sys/kernel/security' as read-only
2024-07-10 15:05:51 - DEBUG - Mounting '/sys/fs/cgroup' as read-only
2024-07-10 15:05:51 - DEBUG - Mounting '/sys/fs/pstore' as read-only
2024-07-10 15:05:51 - DEBUG - Mounting '/sys/firmware/efi/efivars' as read-only
2024-07-10 15:05:51 - DEBUG - Mounting '/sys/fs/bpf' as read-only
2024-07-10 15:05:51 - DEBUG - Mounting '/sys/kernel/debug' as read-only
2024-07-10 15:05:51 - DEBUG - Marking inaccessible directory '/sys/kernel/debug' as hidden because it contains a mountpoint at '/sys/kernel/debug/tracing'
2024-07-10 15:05:51 - DEBUG - [Errno 22] umount(b'/tmp/BenchExec_run_wqsljv85/mount/sys/kernel/debug') failed: Invalid argument
2024-07-10 15:05:51 - DEBUG - Mounting '/sys/kernel/tracing' as read-only
2024-07-10 15:05:51 - DEBUG - Mounting '/sys/fs/fuse/connections' as read-only
2024-07-10 15:05:51 - DEBUG - Mounting '/sys/kernel/config' as read-only
2024-07-10 15:05:51 - DEBUG - Mounting '/proc' as read-only
2024-07-10 15:05:51 - DEBUG - Mounting '/snap/core20/2264' as overlay
2024-07-10 15:05:51 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_wqsljv85/mount/snap/core20/2264', lower=b'/snap/core20/2264', upper=b'/tmp/BenchExec_run_wqsljv85/temp/snap/core20/2264', work=b'/tmp/BenchExec_run_wqsljv85/overlayfs/2'
2024-07-10 15:05:51 - DEBUG - Mounting '/snap/core20/2318' as overlay
2024-07-10 15:05:51 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_wqsljv85/mount/snap/core20/2318', lower=b'/snap/core20/2318', upper=b'/tmp/BenchExec_run_wqsljv85/temp/snap/core20/2318', work=b'/tmp/BenchExec_run_wqsljv85/overlayfs/3'
2024-07-10 15:05:51 - DEBUG - Mounting '/snap/lxd/27948' as overlay
2024-07-10 15:05:51 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_wqsljv85/mount/snap/lxd/27948', lower=b'/snap/lxd/27948', upper=b'/tmp/BenchExec_run_wqsljv85/temp/snap/lxd/27948', work=b'/tmp/BenchExec_run_wqsljv85/overlayfs/4'
2024-07-10 15:05:51 - DEBUG - Mounting '/snap/lxd/28373' as overlay
2024-07-10 15:05:51 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_wqsljv85/mount/snap/lxd/28373', lower=b'/snap/lxd/28373', upper=b'/tmp/BenchExec_run_wqsljv85/temp/snap/lxd/28373', work=b'/tmp/BenchExec_run_wqsljv85/overlayfs/5'
2024-07-10 15:05:51 - DEBUG - Mounting '/snap/snapd/21465' as overlay
2024-07-10 15:05:51 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_wqsljv85/mount/snap/snapd/21465', lower=b'/snap/snapd/21465', upper=b'/tmp/BenchExec_run_wqsljv85/temp/snap/snapd/21465', work=b'/tmp/BenchExec_run_wqsljv85/overlayfs/6'
2024-07-10 15:05:51 - DEBUG - Mounting '/snap/snapd/21759' as overlay
2024-07-10 15:05:51 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_wqsljv85/mount/snap/snapd/21759', lower=b'/snap/snapd/21759', upper=b'/tmp/BenchExec_run_wqsljv85/temp/snap/snapd/21759', work=b'/tmp/BenchExec_run_wqsljv85/overlayfs/7'
2024-07-10 15:05:51 - DEBUG - Cannot use overlay mode for /boot/efi because it has file system vfat. Using read-only mode instead. You can override this by specifying a different directory mode.
2024-07-10 15:05:51 - DEBUG - Mounting '/boot/efi' as read-only
2024-07-10 15:05:51 - DEBUG - Mounting '/var/lib/lxcfs' as read-only
2024-07-10 15:05:51 - DEBUG - Mounting '/tmp' as hidden
2024-07-10 15:05:51 - DEBUG - Mounting '/run' as hidden
2024-07-10 15:06:51 - WARNING - Child 1509 not ready after 60s, likely https://github.com/sosy-lab/benchexec/issues/656 occurred. Killing it and trying again

Future Plans

Next, we need to identify the cause of the error and ensure everything works correctly. I will conduct further tests on machines with other versions of Ubuntu first.

[younghojan]

Progress Made up (Aug 1st)

The relevant scripts used for evalutation and results are available in the benchexec-eval repo.

Current Progress

Performance evalutions have beed conducted on BenchExec, comparing the performance metrics of BenchExec using kernel overlayfs and fuse-overlayfs. The conclusion is that using fuse-overlayfs has only a minor impact on performance, though it is still measuarble.

Environment

The experiment was conducted on a PC equipped with the following specifications:

• 4-core Intel(R) Core(TM) i3-9100 CPU @ 3.60GHz processor
• 16GiB of RAM
• 1TiB HDD
• Linux 6.8.0-39-generic x86_64
• Ubuntu 24.04

BenchExec Version

Consistent with gsoc-overlay-handling-with-fuse-overlayfs-dev.

Workload

Some benchmarks from the SPEC CPU 2017 test suite were selected as the workload. SPEC designed these suites to provide a comparative measure of compute-intensive performance across the widest practical range of hardware using workloads developed from real user applications. The selected benchmarks are 600.perlbench_s, 602.gcc_s, 620.omnetpp_s, 631.deepsjeng_s, 641.leela_s, and 648.exchange2_s. In addition to being compute-intensive, these benchmarks also involve a significant amount of I/O.

Metrics

BenchExec provides the following data as metrics:

• walltime
• cputime
• memory
• blkio-read
• blkio-write
• pressure-cpu-some
• pressure-io-some
• pressure-memory-some

Among these, we are most concerned with walltime, cputime, and memory.

Results

Comparison to Plan

The progress is currently slightly behind schedule, but this was anticipated, I had allocated some extra time in the research plan to address this situation.

Challenges and Roadblocks

The current issue remains that BenchExec using a fuse-based overlay causes a deadlock problem on Ubuntu 22.04, which has not been resolved. The problem only occurs on Ubuntu 22.04 among all current LTS versions (20.04, 22.04, 24.04). Both 20.04 and 24.04 have been tested and can run all of BenchExec’s functionalities correctly. Deadlock issues are typically difficult to diagnose and resolve. However, since BenchExec using a fuse-based overlay works correctly on most Ubuntu LTS versions, the worst-case scenario would be to issue a warning in BenchExec to users running Ubuntu 22.04.

Future Plans

Further organize the relevant documents and eval data, refine the code, and issue warning messages to users running BenchExec on Ubuntu 22.04 (until we can resolve the deadlock issue).

Aside

My mentor Dr. Philipp Wendler has raised several points regarding the current fuse-based overlay mechanism, which are worth discussing. These discussions have been very meaningful, and we have reached conclusions through communication and experimentation. The topic of discussion was whether fuse-overlayfs might incorrectly affect BenchExec's performance measurements (cputime and memory). For details, please refer to #1062 (comment), and we will not elaborate further here.

[younghojan]

Progress Made up (Aug 5th)

All the code is on the branch gsoc-overlay-handling-with-fuse-overlayfs-dev.

Current Progress

We resolved the deadlock issue that occurred with BenchExec based on fuse-overlayfs on Ubuntu 22.04. Additionally, we provided and implemented a solution to ensure that all ambient capabilities are cleared, as discussed here.

Deadlock

On Ubuntu 22.04, when BenchExec uses fuse-overlayfs to overlay mount /run, a deadlock occurs, as shown in the log. As mentioned in the previous discussions, this only occurs on Ubuntu 22.04 and does not appear on 20.04 or 24.04.

We compared the system call traces obtained via strace tool when running containerexec --debug -- ls on Ubuntu 22.04 and 24.04:

• Ubuntu 22.04 (dead lock, the version of fuse-overlayfs is 1.7.1)

  2781  close(10)                         = 0
  2781  getdents64(9, 0x55de091727d0 /* 0 entries */, 32768) = 0
  2781  lgetxattr("//home/ecs-user", "trusted.overlay.opaque",  <unfinished ...>
  2777  <... pselect6 resumed>)           = 0 (Timeout)
  2777  getpid()                          = 2777
  2777  write(2, "\33[32m2024-08-04 22:31:59\33[0m - \33[1;30mWARNING\33[0m - \33[33mChild 2778 not ready after 60s, likely https://github.com/sosy-lab/benchexec/issues/656 occurred. Killing it and trying again.\33[0m\n", 188) = 188
  

• Ubuntu 24.04 (successful, the version of fuse-overlayfs is 1.13-dev)

  19405 close(10)                         = 0
  19405 getdents64(9, 0x5b1e336eaad0 /* 0 entries */, 32768) = 0
  19405 lgetxattr("/proc/self/fd/3/home/haoranyang", "trusted.overlay.opaque", 0x7ffc1088a510, 16) = -1 ENODATA (No data available)
  19405 lgetxattr("/proc/self/fd/3/home/haoranyang", "user.overlay.opaque", 0x7ffc1088a510, 16) = -1 ENODATA (No data available)
  19405 lgetxattr("/proc/self/fd/3/home/haoranyang", "user.fuseoverlayfs.opaque", 0x7ffc1088a510, 16) = -1 ENODATA (No data available)
  19405 faccessat2(3, "home/haoranyang/.wh..wh..opq", F_OK, AT_SYMLINK_NOFOLLOW|AT_EACCESS) = -1 ENOENT (No such file or directory)
  19405 close(9)                          = 0
  19405 statx(3, "home/haoranyang", AT_STATX_DONT_SYNC|AT_SYMLINK_NOFOLLOW, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0750, stx_size=4096, ...}) = 0
  19405 writev(6, [{iov_base="\220\0\0\0\0\0\0\0002\0\0\0\0\0\0\0", iov_len=16}, {iov_base="\20qt3\36[\0\0\0\0\0\0\0\0\0\0\0\312\232;\0\0\0\0\0\312\232;\0\0\0\0\0\0\0\0\0\0\0\0\2\0,\0\0\0\0\0\0\20\0\0\0\0\0\0\10\0\0\0\0\0\0\0C\217\257f\0\0\0\0C\217\257f\0\0\0\0C\217\257f\0\0\0\0\267#\263(\317C&\25\317C&\25\350A\0\0\27\0\0\0\350\3\0\0\350\3\0\0\0\0\0\0\0\20\0\0\0\0\0\0", iov_len=128}], 2) = 144
  19403 <... chdir resumed>)              = 0
  

By comparing these logs, it can be observed that containerexec times out when executing lgetxattr (line 3).
If use a fuse-based overlay mount solely on /boot, it becomes clearer where the deadlock occurs, specifically at lgetxattr("/boot/grub", "trusted.overlay.opaque"). If using an updated version (1.13-dev) of fuse-overlayfs, the corresponding system call would be lgetxattr("/proc/self/fd/3/grub", "trusted.overlay.opaque").

When we mount an overlayfs to /boot, an issue arises where fuse-overlayfs attempts to access /boot/grub. This access then goes through the /boot mount into fuse-overlayfs itself, resulting in recursive access where the request loops back into the same fuse-overlayfs process. fuse-overlayfs cannot handle this recursive access (i.e., an access made through fuse-overlayfs should not end up inside the same fuse-overlayfs process again).

This issue and its solution were mentioned in a PR for fuse-overlayfs. The PR modifies the behavior by using /proc/self/fd/3/ instead of directly accessing /boot. The path /proc/self/fd/3/ is a special symlink that points to the original /boot directory (the one obscured by the overlay mount), which is necessary for fuse-overlayfs to access. This was then merged and included as part of the v1.10 release of fuse-overlayfs.

I previously mentioned that BenchExec using fuse-overlayfs works correctly on Ubuntu 20.04. I checked the version of fuse-overlayfs installed in that environment, which is 0.7.6, released on February 4, 2020. The release notes mention "do not look in lower layers for the ino if there is no origin xattr set", which may be one of the reasons it functions properly.

It is certain that, in the context of BenchExec, we must use a version of fuse-overlayfs higher than 1.10.

Drop Ambient Capabilities

This issue was mentioned in previous discussions. I simply added libc.prctl(libc.PR_CAP_AMBIENT, libc.PR_CAP_AMBIENT_CLEAR_ALL, 0, 0, 0) in the drop_capabilities function.

Comparison to Plan

We are almost there, and the code in the development branch is mergeable. Next, we need to write the relevant documentation to make all the changes user-friendly.

Future Plans

Write user-friendly documentation and make additional optimizations to BenchExec's use of fuse-overlayfs in terms of process and logic (this part has not been detailed yet, as I need to discuss and confirm it further with my mentor).

[younghojan]

Final State: GSoC '24

• Contributor: Hojan Young
• Org: SoSy-Lab
• Mentors: Philipp Wendler, Thomas Lemberger
  
• Project: https://summerofcode.withgoogle.com/programs/2024/projects/UzhlnEel

For relevant discussions and code, please refer to the PR #1062.

About the Project

In this GSoC project, I implemented a mechanism in BenchExec to use fuse-overlayfs as an alternative to kernel overlayfs, addressing the issue where kernel security updates prevented mounting non-leaf nodes with kernel overlayfs.

Current Progress Compared to Plan

With the help of my mentor and the community, I have nearly completed all the goals originally planned, which include:

1. Validate fuse-overlayfs as an alternative to kernel overlayfs.

2. Modify/improve containerexec's file-system overlay mechanism: First attempt kernel overlayfs, and if unsuccessful, fallback to an implementation based on fuse-overlayfs.

   Both of the above points have been correctly implemented. Now, BenchExec can utilize fuse-overlayfs even when running in an unprivileged Podman container. When any user-specified path requiring an overlay mount is a non-leaf node, BenchExec will use fuse-overlayfs for all overlay mounts to avoid kernel restrictions on overlays. Additionally, if the kernel overlayfs encounters runtime errors, fuse-overlayfs will be used as a fallback overlay mount method, ensuring that we can achieve triple-nested container nesting. I drew inspiration from the implementation of Linux unshare and manipulated the process's capability set to enable fuse-overlayfs to function correctly in any unprivileged environment, providing BenchExec with a lower-level overlay file system.

3. Validate the correctness and reliability of the modified containerexec's file-system overlay mechanism, and perform performance evaluation and analysis.

   I used SPEC CPU 2017 to conduct performance testing on BenchExec, comparing the performance metrics of BenchExec using kernel overlayfs and fuse-overlayfs. I selected both compute-intensive and I/O-intensive benchmarks, ultimately proving that when fully utilizing fuse-overlayfs, BenchExec's overall performance showed almost no significant decline. Additionally, fuse-overlayfs does not affect BenchExec's ability to collect performance data, such as memory usage and I/O throughput. For related content, please refer to this post: Progress Log for "Overlay Handling with fuse-overlayfs" #1036 (comment).

4. Extend the documentation for users regarding the new mode, which involves detailing its requirements, outlining its effective scenarios, and explaining the trade-offs between the two overlay modes. This could enable users to make informed choices.

   Dr. Philipp Wendler has handled all related documentation writing, and I will continue to make possible additions and improvements in the future. Therefore, I have not yet completed the documentation aspect of the plan. In addition to the documentation, we also provide users with detailed error messages. When BenchExec encounters runtime errors, users can derive possible solutions based on the outputted error information. Overall, these error messages are quite user-friendly.

5. Fix any other undiscovered bugs that may be encountered during the project's debugging process.

   We have addressed the restrictions on the use of user namespaces in Ubuntu 24.04. Please refer to issue BenchExec not working after upgrading from Ubuntu 23.10 to 24.04 #1041 and PR fix: Add error message output to respond to the impact of Ubuntu 24.04 #1042.

Future Work

Next, I will continue to make some minor improvements to the project, which will not affect the existing functionality.

Thank You

I am grateful to Dr. Philipp Wendler for his thoughtful guidance, including his revisions to my proposal and the solutions he provided for the various issues I encountered. I also want to thank Google for offering this opportunity to participate in the open-source community.