feat(bzlmod): expose platform-agnostic repo target for toolchain interpreter

[chrislovecnm]

This exposes a new repo and target, @{name}_host_interpreter//:python, created by python.toolchain(), that points to the host OS's Python interpreter for that particular toolchain.

This solves two problems:

1. pip.parse() can now refer to the same interpreter used in the toolchains
2. There is now a canonical, public, way to refer to the host OS Python interpreter
   for repository rules.

The above were sort of possible for users to do already, but it required them to write much more configuration and extension code to do so. This moves that sort of boilerplate into our code so they have a simpler configuration.

Also:

• removing bzlmod support in the build_file_generation example; making examples work
  with both WORKSPACE and MODULE is a pain, hence splitting them.
• adding an example of bzlmod and gazelle
• improved documentation in the pip arguments

Closes: #1161

CI failing

[aignas]

Suggested change

	# NOTE: we can use this flag in order to make our setup compatible with
	# bzlmod.
	use_pip_repository_aliases = True,
	use_pip_repository_aliases = True,

Since this is a bzlmod only example, we don't need this comment

[chrislovecnm]

Do we need the flag when we are not using bzlmod?

[chrislovecnm]

We are having a few flakes, but the building on Windows OS is not working. See #1163. I have recreated the problem locally.

[chrislovecnm]

@linzhp, maybe I do not understand what you recommend, but this is what I did as a test:

diff --git a/python/extensions.bzl b/python/extensions.bzl
index 3bcbb50..3c3d582 100644
--- a/python/extensions.bzl
+++ b/python/extensions.bzl
@@ -151,11 +151,12 @@ def _repo_mapped_label(module_label, extension_name, apparent):
           name used by the extension named by `ext_name` (i.e. the value of the
           `name` arg the extension passes to repository rules)
     """
-    return Label("@@{module}~{extension_name}~{apparent}".format(
-        module = module_label.workspace_name,
-        extension_name = extension_name,
-        apparent = apparent,
-    ))
+    return Label("@python3_x86_64-unknown-linux-gnu")
+    #return Label("@@{module}~{extension_name}~{apparent}".format(
+    #    module = module_label.workspace_name,
+    #    extension_name = extension_name,
+    #    apparent = apparent,
+    #))

 # We are doing some bazel stuff here that could use an explanation.
 # The basis of this function is that we need to create a symlink to

I used return Label("@python3_x86_64-unknown-linux-gnu") on Linux instead of building the label in the fashion that is working.

We then get the error:

$ bazel clean --expunge && bazel build //...
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
Starting local Bazel server and connecting to it...
DEBUG: gazelle@0.27.0/MODULE.bazel:7:6: WARNING: The bazel_gazelle Bazel module is still highly experimental and subject to change at any time. Only use it to try out bzlmod for now.
DEBUG: rules_go@0.37.0/MODULE.bazel:8:6: WARNING: The rules_go Bazel module is still highly experimental and subject to change at any time. Only use it to try out bzlmod for now.
DEBUG: rules_go@0.33.0/MODULE.bazel:7:6: WARNING: The rules_go Bazel module is still highly experimental and subject to change at any time. Only use it to try out bzlmod for now.
INFO: Repository rules_python~override~python~python3_host_interpreter instantiated at:
  callstack not available
Repository rule _host_hub defined at:
  /home/clove/.cache/bazel/_bazel_clove/e19dab5c65f1ab29c26be4ab0039d0eb/external/rules_python~override/python/extensions.bzl:209:28: in <toplevel>
ERROR: An error occurred during the fetch of repository 'rules_python~override~python~python3_host_interpreter':
   Traceback (most recent call last):
        File "/home/clove/.cache/bazel/_bazel_clove/e19dab5c65f1ab29c26be4ab0039d0eb/external/rules_python~override/python/extensions.bzl", line 205, column 21, in _host_hub_impl
                repo_ctx.symlink(label, "python")
Error in symlink: Unable to load package for @[unknown repo 'python3_x86_64-unknown-linux-gnu' requested from @rules_python~override]//:python3_x86_64-unknown-linux-gnu: The repository '@[unknown repo 'python3_x86_64-unknown-linux-gnu' requested from @rules_python~override]' could not be resolved: No repository visible as '@python3_x86_64-unknown-linux-gnu' from repository '@rules_python~override'
ERROR: <builtin>: fetching _host_hub rule @rules_python~override//python:rules_python~override~python~python3_host_interpreter: Traceback (most recent call last):
        File "/home/clove/.cache/bazel/_bazel_clove/e19dab5c65f1ab29c26be4ab0039d0eb/external/rules_python~override/python/extensions.bzl", line 205, column 21, in _host_hub_impl
                repo_ctx.symlink(label, "python")
Error in symlink: Unable to load package for @[unknown repo 'python3_x86_64-unknown-linux-gnu' requested from @rules_python~override]//:python3_x86_64-unknown-linux-gnu: The repository '@[unknown repo 'python3_x86_64-unknown-linux-gnu' requested from @rules_python~override]' could not be resolved: No repository visible as '@python3_x86_64-unknown-linux-gnu' from repository '@rules_python~override'
INFO: Repository bazel_tools~remote_coverage_tools_extension~remote_coverage_tools instantiated at:
  callstack not available
Repository rule http_archive defined at:
  /home/clove/.cache/bazel/_bazel_clove/e19dab5c65f1ab29c26be4ab0039d0eb/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in <toplevel>
INFO: Repository rules_cc~0.0.2 instantiated at:
  callstack not available
Repository rule http_archive defined at:
  /home/clove/.cache/bazel/_bazel_clove/e19dab5c65f1ab29c26be4ab0039d0eb/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in <toplevel>
INFO: Repository rules_java~5.3.5 instantiated at:
  callstack not available
Repository rule http_archive defined at:
  /home/clove/.cache/bazel/_bazel_clove/e19dab5c65f1ab29c26be4ab0039d0eb/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in <toplevel>
ERROR: /home/clove/.cache/bazel/_bazel_clove/e19dab5c65f1ab29c26be4ab0039d0eb/external/rules_python~override~pip~pip/tabulate/BUILD.bazel:3:6: no such package '@rules_python~override~pip~pip_tabulate//': no such package '@rules_python~override~python~python3_host_interpreter//': Unable to load package for @[unknown repo 'python3_x86_64-unknown-linux-gnu' requested from @rules_python~override]//:python3_x86_64-unknown-linux-gnu: The repository '@[unknown repo 'python3_x86_64-unknown-linux-gnu' requested from @rules_python~override]' could not be resolved: No repository visible as '@python3_x86_64-unknown-linux-gnu' from repository '@rules_python~override' and referenced by '@rules_python~override~pip~pip//tabulate:tabulate'
ERROR: Analysis of target '//:bzlmod_build_file_generation' failed; build aborted:
INFO: Elapsed time: 3.694s
INFO: 0 processes.

Let me know if there is a different approach. We can always iterate to improve this, but we have already spent many hours of dev time on this.

[rickeylev]

I think I remember why calling Label() there doesn't work: It's being called in the repo implementation function, not as part of evaluating the code generated by the repo.

When Label() is called in the host_hub_impl implementation function in extensions.bzl, it's current context is the rules_python module. Since the rules_python module itself never called use_repo() on anything (it can't; the repo names are user selected), it's repo visibility mapping is empty.

If host_hub generated a .bzl file with e.g. Label("@python3_x86_64-linux"), that would work, because when someone loads the host hub repo, that Label call will be evaluated in the context of the host hub repo, which does have access to the other repos that the extension generated.

[fmeum]

@rickeylev That's correct. This situation is especially tricky since the resolved label is needed in the implementation function of the repo rule. This may require introducing a helper extension.

@Wyverald and I were discussing a module_ctx.label_in_repo function that would drastically simplify such situations. If you would like to see something like this realized, could you file an issue for it?