Sandbox doesn't work on macOS Big Sur when the sandbox directory is in a firm linked directory

[ob]

macOS Catalina (10.*) introduced a security feature where the root filesystem is mounted read-only, and a mechanism for creating directories or symlinks at boot time called firmlinks. Since the root filesystem is read-only, a new volume mounted at /System/Volumes/Data is overlaid on top of the root filesystem and contains the mutable portions of the filesystem (more info here).

When building with Bazel on a location that is firm linked from the root, and moving the sandbox to that same location by using the --experimental_sandbox_base flag:

In macOS Catalina (10.15.7 to be precise), it works as expected.

$ cat /etc/synthetic.conf
export	/System/Volumes/Data/export
$ ls -l /export
lrwxr-xr-x  1 root  wheel  27 May 30 22:42 /export -> /System/Volumes/Data/export
$ cd /export/home/tester
$ git clone https://github.com/bazelbuild/bazel.git
Cloning into 'bazel'...
remote: Enumerating objects: 567875, done.
remote: Counting objects: 100% (1761/1761), done.
remote: Compressing objects: 100% (924/924), done.
remote: Total 567875 (delta 783), reused 1374 (delta 586), pack-reused 566114
Receiving objects: 100% (567875/567875), 757.61 MiB | 27.05 MiB/s, done.
Resolving deltas: 100% (354168/354168), done.
Updating files: 100% (9123/9123), done.
$ cd bazel
$ bazel build --experimental_sandbox_base=$(pwd)/sandbox-dir //src:bazel
INFO: Elapsed time: 92.474s, Critical Path: 29.31s
INFO: 2915 processes: 2034 remote cache hit, 167 internal, 533 darwin-sandbox, 1 local, 180 worker.
INFO: Build completed successfully, 2915 total actions

However, that exact same sequence of operations on macOS Big Sur (11.4), produces the following error:

$ bazel build --experimental_sandbox_base=$(pwd)/sandbox-dir //src:bazel
<output elided>
ERROR: /System/Volumes/Data/export/home/tester/bazel/third_party/BUILD:467:20: Extracting interface //third_party:flogger_checked_in failed: (Segmentation fault): ijar failed: error executing command external/remote_java_tools_darwin/java_tools/ijar/ijar third_party/flogger/flogger-0.5.1.jar ... (remaining 3 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox ijar failed: error executing command external/remote_java_tools_darwin/java_tools/ijar/ijar third_party/flogger/flogger-0.5.1.jar ... (remaining 3 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
open(): Operation not permitted
Target //src:bazel failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /System/Volumes/Data/export/home/tester/bazel/src/main/java/com/google/devtools/build/lib/analysis/BUILD:2085:13 Building src/main/java/com/google/devtools/build/lib/analysis/libextra/extra_action_info_file_write_action.jar (1 source file) and running annotation processors (AutoCodecProcessor) failed: (Segmentation fault): ijar failed: error executing command external/remote_java_tools_darwin/java_tools/ijar/ijar third_party/tomcat_annotations_api/tomcat-annotations-api-8.0.5.jar ... (remaining 3 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox ijar failed: error executing command external/remote_java_tools_darwin/java_tools/ijar/ijar third_party/tomcat_annotations_api/tomcat-annotations-api-8.0.5.jar ... (remaining 3 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
INFO: Elapsed time: 14.914s, Critical Path: 0.17s
INFO: 23 processes: 1 remote cache hit, 22 internal.
FAILED: Build did NOT complete successfully

I have traced the problem to a difference on how sandbox-exec works in Big Sur when firm links are involved.

If I create the following sandbox.sb file (modified from the actual sandbox.sb that Bazel creates):

(version 1)
(debug deny)
(allow default)
(deny file-write*)
(allow file-write*
    (subpath "/dev")
    (subpath "/System/Volumes/Data/export/home/tester/Library/Logs")
    (subpath "/private/var/tmp")
    (subpath "/private/var/folders/p1/5rdx8yks6wl56p49qhzlnbv40000gq/C")
    (subpath "/private/var/folders/p1/5rdx8yks6wl56p49qhzlnbv40000gq/T")
    (subpath "/System/Volumes/Data/export/home/tester/sb/sandbox-dir")
    (subpath "/System/Volumes/Data/export/home/tester/TEMP")
    (subpath "/private/tmp")
    (subpath "/System/Volumes/Data/export/home/tester/Library/Developer")
)

And put it in an empty directory in /export/home/tester/sb, on macOS Catalina:

$ mkdir sandbox-dir
$ /usr/bin/sandbox-exec -f sandbox.sb /bin/bash -c "touch sandbox-dir/foobar"
$ echo $?
0
$ ls -l sandbox-dir/
total 0
-rw-r--r--  1 tester  staff  0 Jul 28 10:08 foobar

Everything works as expected. However on macOS Big Sur:

$ mkdir sandbox-dir
$ /usr/bin/sandbox-exec -f sandbox.sb /bin/bash -c "touch sandbox-dir/foobar"
touch: sandbox-dir/foobar: Operation not permitted
$ ls -l sandbox-dir/

However, if I change the line:

    (subpath "/System/Volumes/Data/export/home/tester/sb/sandbox-dir")

to remove the /System/Volumes/Data prefix (since I'm using that firm link through the symlink), I get the expected results in macOS Big Sur:

Now, this could be a bug in how macOS handles firm links but Bazel could also avoid resolving symlinks (or firm links) on macOS to work around this. I will file a bug with Apple but the man page of sandbox-exec says the command is deprecated so I don't have high hopes there.

[ob]

A followup from my conversation with Apple:

Sandbox-exec is deprecated and the sandbox profile language is not public or supported API.

[keith]

A bit amusing since SwiftPM is using it as well

[ob]

This diff fixes it for me:

diff --git a/src/main/java/com/google/devtools/build/lib/sandbox/SandboxModule.java b/src/main/java/com/google/devtools/build/lib/sandbox/SandboxModule.java
index 4058f4bd5c..f4ead5c1eb 100644
--- a/src/main/java/com/google/devtools/build/lib/sandbox/SandboxModule.java
+++ b/src/main/java/com/google/devtools/build/lib/sandbox/SandboxModule.java
@@ -125,7 +125,7 @@ public final class SandboxModule extends BlazeModule {
               env.getRuntime().getProductName(),
               Fingerprint.getHexDigest(env.getOutputBase().toString()));
       FileSystem fileSystem = env.getRuntime().getFileSystem();
-      Path resolvedSandboxBase = fileSystem.getPath(options.sandboxBase).resolveSymbolicLinks();
+      Path resolvedSandboxBase = fileSystem.getPath(options.sandboxBase); // .resolveSymbolicLinks();
       return resolvedSandboxBase.getRelative(dirName);
     }
   }

I can start a PR if no one has any objections to it but it essentially reverts a piece of this commit. Namely,

When using --experimental_sandbox_base, ensure that symlinks in the path are resolved. Before this, you had to check whether on your system /dev/shm is a symlink to /run/shm and then use that instead. Now it no longer matters, as symlinks are resolved.

I don't know if that would be considered a regression or if the diff should be conditioned to macOS...

[larsrc-google]

That does sound like a regression, yes. Doing this only for MacOS would be better.

[ob]

Any tips on how to check if it's macOS?

[larsrc-google]

OS.getCurrent() will tell you.

[ob]

PR updated. Thanks for the tip @larsrc-google!

[Wyverald]

Is this fixed by #13984?

[ob]

Yeah

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[github-actions]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please reach out to the triage team (@bazelbuild/triage). Thanks!