[request] robust way to specify LD_LIBRARY_PATH for java/javac

[divanorama]

Description of the problem / feature request:

Bazel builds aren't fully hermetic in many cases

• some build rules rely on #!/usr/bin/env bash + common utilities
• repository rules rely on host commands (env, git, gcloud, go, tar, zip, python, grep, which, sed, bash, node,...)
• compilers (like remote_jdk11 javac) may be dynamically linked and relying on host libraries (like libz.so.1)
• dynamic linkers may rely on ld.so.cache in default locations, or shared libraries in default locations (in case executable doesn't specify search paths, which it usually doesn't for non-bundled dependencies)

But on the other hand bazel tries to cleanup environment variables and sandbox actions, which in turn may break non-hermetic cases, not always with an easy fix.

• --action_env may not work for rules without use_default_shell_env=True including some built-in rules (like DumpPlatformClassPath

  bazel/tools/jdk/default_java_toolchain.bzl

  Line 222 in 667ae2a

  executable = "%s/bin/javac" % host_javabase.java_home,

  )
• --repo_env seems to cover repo rules
• --run_under doesn't seem to work for build or repo rules (could try --run_under='env PATH=.. LD_LIBRARY_PATH=..')
• --sandbox_add_mount_pair refuses to create paths, only overrides existing
• supplying per-host toolchains feels non-hermetic and code-intrusive (can't just edit .bazelrc, also need to patch bzl code in the project)
• --experimental_sandboxfs_path may be an option, but may not cover repository rules(?) and requires patching sandboxfs

As an initial crude fix, maybe being able to non-conditionally set/extend PATH&LD_LIBRARY_PATH for rules&actions would work.

Feature requests: what underlying problem are you trying to solve with this feature?

Be able to build bazel projects (on NixOS) without a lot of patching bazel or doing brittle host (or container) setup.

Starting with semi-hermetic solutions like --action_env sounds ok for a start

• remote cache won't be easily reused, but local cache will work
• in case with NixOS setting *PATH=/nix/store/looonghash-smth/{bin,lib} is actually safe to cache and should be reproducible, assuming other bits are

Ideally of course it would be nice to be able to specify all dependencies explicitly, at least in built-in and core rules. If not as build targets/bazel dependencies, then at least like a ./configure step with ability to always easily specify custom paths.

What operating system are you running Bazel on?

NixOS

What's the output of bazel info release?

3.3.1, but should apply to 4.0.0 too

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

NixOS packages

Have you found anything relevant by searching the web?

NixPkgs already have few patches to bazel code like

• replacing /usr/bin/env something with nix path of something
• making bazel shell rules use patched bash that has special default search path (bash/env/glibc go to /bin, /usr/bin and such when PATH is empty; that won't help finding binaries on NixOS, patch is to add few common utils to bazel-bash default search path)
  https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/tools/build-managers/bazel/bazel_3/default.nix#L324

There's a buildFHSUserEnv helper in NixPkgs to create FHS sandboxes/containers https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments . buildFHSUserEnv + ..._env=PATH=... + ..._env=LD_LIBRARY_PATH=... may be used to make most of rules work in NixOS, but FHS environment feels like an overkill

• sandbox filesystem customization request Configurable sandbox root #6994
• on actions ignoring *_env Most actions ignore --action_env #3320
• also on sandbox filesystem Make the sandboxed file system more strict #7313
• non-hermiticity wrt system libs remote/correctness: Bazel doesn't track system libraries. #4558

[divanorama]

Another a bit involved case is https://github.com/bazelbuild/rules_nodejs/blob/stable/internal/providers/node_runtime_deps_info.bzl#L148 - here a rule does some env setup, so it can't(?) even apply use_default_shell_env=True or --action_env=.../--repo_env=.... When it is used to call typescript compiler wrapper tsc.sh which has #!/usr/bin/env bash the script uses tools like grep, which in turn assumes env - A=B C=D bash -c grep works on the host (which isn't necessarily true on NixOS with/without FHS wrapper)

[divanorama]

One more thing that can negate fix attempts is that rules can disable sandboxing (I'd say an .bazelrc option to enforce sandboxing can be useful) via execution_requirements or tags, also not very clear if fixes via sandbox setup would apply to process workers when --worker_sandboxing isn't set

[meisterT]

cc @larsrc-google

[larsrc-google]

There's an implicit fallback from sandboxed to local that since recently can be controlled with the --[no]incompatible_legacy_local_fallback flag. I've not yet managed to switch the default due to some subtle test interactions with the local tag.

Workers without --worker_sandboxing would indeed not benefit from any fixes to the sandbox setup.

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[github-actions]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please reach out to the triage team (@bazelbuild/triage). Thanks!