Allow the exec root to be placed outside the output base

[EdSchouten]

In an attempt to achieve 'Builds without the Bytes' without losing
access to build outputs, I am experimenting with a FUSE file system that
gives access to objects stored in the CAS. In PR #11703, I added a
command line flag to let Bazel emit symbolic links pointing into this
FUSE file system, as opposed to downloading files into the exec root.

Though this change has allowed me to get quite a lot of stuff working,
there are also many build actions that break. For example, Python calls
realpath(argv[0]) to figure out its installation path. Because the FUSE
file system does not mimic the execroot, Python won't be able to find
its site packages. Similar problems hold with shared library resolution
in general.

This is why I think the only proper way we can get this to work is by
using hard links instead of symbolic links. That way the usual file
hierarchy remains intact. This, however, requires that the exec root
itself is placed on a FUSE file system. It is already possible to
achieve this by setting --output_base, but that has the downside of also
placing many other files on FUSE (e.g., the sandbox directories), which
is detrimental to performance.

This change adds a new command line flag, --exec_root_base, which can be
used to leave the output base at the regular place, but host the
exec root directory on a FUSE file system.

This change originally seemed to work all right with Bazel 3.4-3.7. In
order to make this work with Bazel master, I had to make a slight tweak
to the changes in 0c249d5. That code
added the assumption that "${output_base}/external" is always placed at
"${exec_root_base}/../external". I suspect that already causes a
regression in case a BlazeModule overrides the exec root base. While
there, rename 'execRootParent' to 'execRootBase', as it corresponds to
the exec root itself; not its parent directory.

[EdSchouten]

Cc @coeuvre @janakdr

[coeuvre]

Thanks for your PR!

I am not familiar with this part so might not be the one to review the code. But overall, the changes LGTM.

[janakdr]

cc reviewer @lberki for the issue with /external, since I think the author is not part of our github org.

[janakdr]

Internally, the "bazel-out" directory is a symlink to the FUSE filesystem. It's created inside the "startBuild" method here:

bazel/src/main/java/com/google/devtools/build/lib/buildtool/ExecutionTool.java

Line 266 in 0999cf1

outputService.startBuild(

. I think it might be a smaller change and more consistent to have just the bazel-out directory be on the FUSE filesystem, and make it a symbolic link. Does that sound ok? It's conceivable that we could open-source the abstract FUSE logic as an OutputService, configured via command-line flags, that would allow sharing of this code (although it's not very complex code).

[EdSchouten]

Internally, the "bazel-out" directory is a symlink to the FUSE filesystem. It's created inside the "startBuild" method here:

bazel/src/main/java/com/google/devtools/build/lib/buildtool/ExecutionTool.java

Line 266 in 0999cf1

outputService.startBuild(

. I think it might be a smaller change and more consistent to have just the bazel-out directory be on the FUSE filesystem, and make it a symbolic link. Does that sound ok?

Ah, nice. The difference between putting the exec root or just bazel-out on FUSE should be fairly small, so that would work for me. Thanks for pointing me to that part of the code!

It's conceivable that we could open-source the abstract FUSE logic as an OutputService, configured via command-line flags, that would allow sharing of this code (although it's not very complex code).

That would be awesome! Anything I could do to help out with that (review, test, etc.)?

[janakdr]

@alexjski separating out an abstract FuseOutputService might be one entry point to work we discussed around rethinking the output tree, although it kind of goes in the opposite direction!

[EdSchouten]

Hey @janakdr,

Thanks again for pointing me to OutputService a couple of days ago. The last couple of days I've been experimenting with it, and at the same time doing some beard scratching. Maybe for Bazel (not Blaze) it would make sense to have a completely separate implementation of OutputService, specifically aimed at REv2.

Just as an experiment, I'm working on adding an implementation of OutputService to lib/remote that transforms many of the operations into gRPC calls, to be sent to a FUSE daemon. At the same time, I'm trying to see whether this gRPC protocol can incorporate the features that I (attempted to) add in #11703, #11662 and #12566 into something cohesive. More concretely: we could let Bazel just do gRPC calls against the FUSE daemon to do getDigest(), and to create CAS-backed pseudo-files.

I'll keep you posted on how that works out.

[janakdr]

That sounds very similar in spirit to our internal implementation, although I will admit that we still go through normal FUSE to retrieve digests (retrieving digests is typically not a big expense in the build because Bazel already has the digests in memory from remote execution).

[EdSchouten]

Superseded by #12823. Thanks for sharing your insights, @janakdr. Hope to get your opinions on #12823!