shell toolchain, design: including common bintools

[laszlocsomor]

Description of the problem / feature request:

Design a mechanism to express that a rule executing a shell command depends on certain command line tools.

Feature requests: what underlying problem are you trying to solve with this feature?

Not every machine has Bash installed, and those that do don't always install Bash at the same location, or have another shell interpreter installed. To accommodate all scenarios, I designed how to turn Bash, or broadly speaking, the Shell, into a toolchain. The efforts is tracked as #4319.

However what Bazel treats as "the shell" is more than just the shell interpreter. Bazel assumes that not only Bash is the shell and it's installed at a fixed location, but also that common GNU binutils and other common programs such as "zip" or "grep" are also installed and are on the PATH.

Rules that execute shell commands also expect these tools to exist and be on the PATH. Bazel currently has no way to express that such rules depend on these tools.

[laszlocsomor]

FYI, I don't plan to work on this for the foreseeable future (in the next few months).

[jin]

@laszlocsomor could you add a priority to this issue please?

[laszlocsomor]

Setting priority to P3 for now -- we will probably not see much advancement with this issue before Bazel 1.0, and while it is an important issue, it's not as critical as something like #6351 would be.

[robinp]

FYI if anyone is actively troubled by this, recently I successfully use rules_nixpkgs to source pinned version of binaries from nixpkgs (a reproducible set of linux/mac packages). No more depending on binaries being installed and put on PATH upfront.

See this target example for depending on bison from a nixpkgs import. See this WORKSPACE entry for how can you add rules_nixpkgs to your workspace.

[laszlocsomor]

FYI @asuffield

[aherrmann]

However what Bazel treats as "the shell" is more than just the shell interpreter. Bazel assumes that not only Bash is the shell and it's installed at a fixed location, but also that common GNU binutils and other common programs such as "zip" or "grep" are also installed and are on the PATH.

Rules that execute shell commands also expect these tools to exist and be on the PATH. Bazel currently has no way to express that such rules depend on these tools.

rules_sh implements such a toolchain for Bazel, usage described in the README.

[meteorcloudy]

Looks like this issue is resolved by rules_sh and the modern toolchain resolution mechanism.

[uri-canva]

I think this is still relevant, rules_sh requires genrules to be written with support for it in mind, so it cannot be used with third party genrules. Can this be reopened?

[fmeum]

Doesn't rules_sh just define additional make variables via a toolchain that any genrule-like rule could use without supporting rules_sh specifically (but rather toolchains in general)? If that's the case, it may be sufficient to get the building blocks for doing that in a convenient location (e.g. skylib) or simply document good design choices for such rules

[uri-canva]

Yeah but that still requires the whole ecosystem to rewrite their genrules to use those make variables.

[fmeum]

Yeah but that still requires the whole ecosystem to rewrite their genrules to use those make variables.

The effort wouldn't be specific to the make variables offered by rules_sh, but should apply to any make variable defined by any toolchain. Even if Bazel itself provided a shell toolchain, I think that third-party genrules would still need to support toolchain and the make variables to expose to make use of it.

@alexeagle Do the rules_js genrules support toolchains and make variables? Maybe bazel-lib already has helpers for doing that properly and evaluating the make variables?

[uri-canva]

Even if Bazel itself provided a shell toolchain, I think that third-party genrules would still need to support toolchain and the make variables to expose to make use of it.

Yeah absolutely, but there's things bazel itself can do that toolchains cannot to ease this transition or to support both ways of using it, for example by providing a shell toolchain, but also automatically adding all the locations of the binaries in the toolchain into the PATH that is passed to genrule actions.

Alternatively bazel could provide a remote prebuilt shell toolchain, like it provides a remote jdk, and the user could specify their own or autodetect a locally installed one.

This would allow bazel to use a wider range of systems as both its host environment and exec environment (very relevant in remote execution environments, where the exec environment could be a very barebones one), and improve the default hermeticity of bazel builds.

[uri-canva]

Is this it?

bazel/tools/sh/sh_configure.bzl

Line 84 in 4148241

native.register_toolchains("@local_config_sh//:local_sh_toolchain")

I can see @bazel_tools//tools/sh:toolchain_type referenced in various parts of the bazel ecosystem: https://sourcegraph.com/search?q=context%3Aglobal+%40bazel_tools%2F%2Ftools%2Fsh%3Atoolchain_type&patternType=standard&sm=1&groupBy=repo.