Windows: BAZEL_SH breaks remote caching

[aherrmann]

Description of the problem:

On Windows Bazel requires the BAZEL_SH environment variable to point to an msys2 installed bash.exe. The content of this variable affects the cache key of run_shell actions. This breaks remote caching of such actions if BAZEL_SH differs between machines. E.g. if it contains the username - as can be the case with a scoop installed msys2.

Reproduction

Minimal Example

Setup a new Bazel workspace such that

> cat .\WORKSPACE
> cat .\BUILD
load("//:myrule.bzl", "myrule")

myrule(name = "test")
> cat .\myrule.bzl
def _myrule_impl(ctx):
    out = ctx.actions.declare_file("{}_out".format(ctx.attr.name))
    ctx.actions.run_shell(
        command = "echo test > {}".format(out.path),
        outputs = [out],
    )
    return DefaultInfo(files = depset(items = [out]))

myrule = rule(
    implementation = _myrule_impl,
)

Setup two separate msys2 installations <msys_a> and <msys_b>.

Build twice with disk cache enabled using <msys_a> and observe cache hit on second build:

> bazel build //:test -s --disk_cache=./disk_cache --shell_executable=<msys_a>\usr\bin\bash.exe --experimental_execution_log_file=msys_a_1.log
> bazel clean
> bazel build //:test -s --disk_cache=./disk_cache --shell_executable=<msys_a>\usr\bin\bash.exe --experimental_execution_log_file=msys_a_2.log

Build with disk cache enabled using <msys_b> and observe cache miss:

> bazel clean
> bazel build //:test -s --disk_cache=./disk_cache --shell_executable=<msys_b>\usr\bin\bash.exe --experimental_execution_log_file=msys_b_1.log

(--shell_executable has the same effect as BAZEL_SH wrt this issue.)

Full Reproduction

Enable long path support (requires admin privileges)

> Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem' -Name LongPathsEnabled -Type DWord -Value 1

Clone repository and setup scoop installation

> git clone github.com/digital-asset/daml
> cd daml
> git checkout a6215f1ceb2261953f9d4855e5c5b73273b9fd8a
> .\dev-env\windows\bin\dadew.ps1 install
> .\dev-env\windows\bin\dadew.ps1 sync
> .\dev-env\windows\bin\dadew.ps1 enable

Build target - observe many cache misses (assuming your username is not VssAdministrator)

> bazel build --config windows //pipeline/samples/bazel/haskell:sop-example --experimental_execution_log_file=cache_misses.execlog  --remote_accept_cached=true

Copy msys2 to location matching remote cache (requires admin privileges)

> mkdir -p C:\Users\VssAdministrator\dadew\scoop\apps\msys2-20180531
> cp -r ~\dadew\scoop\apps\msys2-20180531\current C:\Users\VssAdministrator\dadew\scoop\apps\msys2-20180531\

Retry build and observe cache hits.

> bazel clean
> bazel build --config windows //pipeline/samples/bazel/haskell:sop-example --experimental_execution_log_file=cache_hits.execlog  --remote_accept_cached=true

What operating system are you running Bazel on?

Windows 10

What's the output of bazel info release?

release 0.23.2

Have you found anything relevant by searching the web?

I found this issue about removing the dependency on bash on Windows. That issue points to this design document. However, I did not find explicit mention of the remote caching issue described here.

[laszlocsomor]

@buchgr : FYI

@aherrmann : Thanks for the detailed report! The BAZEL_* envvars serve Bazel, so I think Bazel could (and should) unset them.

[aherrmann]

@laszlocsomor I'm not sure if it's the fact that BAZEL_SH is defined, or if it's that the full command_args enter the cache key - including the path to bash.exe. The fact that the "minimal example" above triggers the issue by setting --shell_executable suggests that it's the latter. But I don't know the internals of --shell_executable.

[buchgr]

Afaik it's the fact that the full path enters the cache key.

[laszlocsomor]

For the record, setting BAZEL_SH is no longer necessary.
Bazel 0.27 works fine without this envvar. (Bazel still uses the envvar if it's set, but won't fail if not, just falls back on checking common install paths and the registry.)

[sgowroji]

Hi there! We're doing a clean up of old issues and will be closing this one. Please reopen if you’d like to discuss anything further. We’ll respond as soon as we have the bandwidth/resources to do so.