Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rules_python 0.29.0 started enforcing symlink usage on Windows (was: broke Bazel on Windows) #1723

Closed
meteorcloudy opened this issue Jan 26, 2024 · 18 comments · Fixed by #1745
Assignees
Labels
type: pip type: toolchain Related to the toolchains provided by rules_python

Comments

@meteorcloudy
Copy link
Member

meteorcloudy commented Jan 26, 2024

🐞 bug report

Affected Rule

rules_python 0.29.0

Is this a regression?

Yes, rules_python 0.26.0 worked fine for Bazel

Description

I tried to upgrade rules_python for Bazel, which caused the following breakage on Windows:
https://buildkite.com/bazel/google-bazel-presubmit/builds/76496#018d45ca-4b9c-4572-83dc-9c605e169298

🔬 Minimal Reproduction

  1. Upgrade rules_python in Bazel's MODULE.bazel file
  2. bazel test //src/test/py/bazel:first_time_use_test

🔥 Exception or Error


(12:46:03) ERROR: An error occurred during the fetch of repository 'rules_python~0.29.0~pip~bazel_pip_dev_deps_38_bazel_runfiles':
   Traceback (most recent call last):
	File "C:/b/umz2tcci/external/rules_python~0.29.0/python/pip_install/pip_repository.bzl", line 723, column 13, in _whl_library_impl
		fail("whl_library %s failed: %s (%s) error code: '%s'" % (rctx.attr.name, result.stdout, result.stderr, result.return_code))
Error in fail: whl_library rules_python~0.29.0~pip~bazel_pip_dev_deps_38_bazel_runfiles failed:  () error code: '-1073741515'
(12:46:03) ERROR: : fetching whl_library rule //:rules_python~0.29.0~pip~bazel_pip_dev_deps_38_bazel_runfiles: Traceback (most recent call last):
	File "C:/b/umz2tcci/external/rules_python~0.29.0/python/pip_install/pip_repository.bzl", line 723, column 13, in _whl_library_impl
		fail("whl_library %s failed: %s (%s) error code: '%s'" % (rctx.attr.name, result.stdout, result.stderr, result.return_code))
Error in fail: whl_library rules_python~0.29.0~pip~bazel_pip_dev_deps_38_bazel_runfiles failed:  () error code: '-1073741515'

🌍 Your Environment

Operating System:

  
Windows
  

Output of bazel version:

  
7.0.2
  

Rules_python version:

  
0.29.0
  

Anything else relevant?

@meteorcloudy
Copy link
Member Author

Looks like this doesn't happen with 0.28.0

@rickeylev
Copy link
Contributor

Weird. 0.29.0 doesn't have many changes: 0.28.0...0.29.0

This commit looks like a likely culprit: 4fe0db3

It has some mentions of some windows-specific hacks that had to be done.

@aignas

Error code -1073741515 seems to be something about corrupt files or missing files.

The error says its happening during the pip phase, i.e when it's trying to go download the packages and install them.

Being unable to find some dll's because it's confused by the symlink/realpath-of-the-symlink seems plausible

@aignas
Copy link
Collaborator

aignas commented Jan 26, 2024

I'm curious in the env differences between rules_python CI and bazel. Are there any? What kind of Windows python is installed/setup on bazel CI? Or is it on local? I agree with the judgment that that commit is the most likely culprit, but curious why our CI did not complain.

@meteorcloudy
Copy link
Member Author

@aignas aignas added help wanted type: pip type: toolchain Related to the toolchains provided by rules_python labels Jan 29, 2024
@aignas
Copy link
Collaborator

aignas commented Jan 29, 2024

I don't have access to a Windows machine, but I think it would be interesting to see how the get interpreter function which does .realpath to resolve the interpreter in your case. In both cases, CI and local the resolution should use the host interpreter toolchain and I am not sure why it works only in CI. Is the Windows version different? Are we missing some .bazelrc flags on the bazel end that may affect symlink behaviour?

@meteorcloudy
Copy link
Member Author

@aignas Can you help me compose a single command that I can use to reproduce this issue? Basically, what's been executed at

result = rctx.execute(
args + [
"--whl-file",
whl_path,
] + ["--platform={}".format(p) for p in target_platforms],

aignas added a commit that referenced this issue Jan 31, 2024
… third party deps

Use this by running `bazel build @repro//..` in the `rules_python` module.
The build will fail all the time because I do `sys.exit(1)` in the
script but the expectation that on Windows we can reproduce the #1723
and the python script will exit before `sys.exit(1)` is called with a
different error.
@aignas
Copy link
Collaborator

aignas commented Jan 31, 2024

@meteorcloudy, from what I can understand, #1736 might reproduce the error seen here, but if it doesn't, let me know.

@meteorcloudy
Copy link
Member Author

@aignas Thank you very much, here is the output on a Windows VM:

pcloudy@bazel-windows-playground:~/workdir/rules_python ((HEAD detached at FETCH_HEAD))
$ bazel build @repro//...
INFO: Reading 'startup' options from c:\users\pcloudy\workdir\rules_python\.bazelrc: --windows_enable_symlinks
INFO: Reading 'startup' options from c:\users\pcloudy\.bazelrc: --output_user_root=C:/src/tmp
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=240
INFO: Reading rc options for 'build' from c:\users\pcloudy\workdir\rules_python\.bazelrc:
  Inherited 'common' options: --enable_bzlmod
INFO: Options provided by the client:
  'build' options: --python_path=C:/python3/python.exe
INFO: Reading rc options for 'build' from c:\users\pcloudy\workdir\rules_python\.bazelrc:
  'build' options: --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_modul
e/pkg,examples/bzlmod_build_file_generation/runfiles,examples/bzlmod/entry_points,examples/bzlmod/entry_points/tests,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/patc
hes,examples/bzlmod/py_proto_library,examples/bzlmod/py_proto_library/example.com/another_proto,examples/bzlmod/py_proto_library/example.com/proto,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod/tests/dupe_requirements,exampl
es/bzlmod/tests/other_module,examples/bzlmod/whl_mods,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_parse,examples/pip_parse_vendored,examples/pip_re
pository_annotations,examples/py_proto_library,examples/py_proto_library/example.com/another_proto,examples/py_proto_library/example.com/proto,gazelle,gazelle/manifest,gazelle/manifest/generate,gazelle/manifest/hasher,gazelle/manifest/test,
gazelle/modules_mapping,gazelle/python,gazelle/pythonconfig,tests/integration/compile_pip_requirements,tests/integration/compile_pip_requirements_test_from_external_repo,tests/integration/ignore_root_user_error,tests/integration/ignore_root
_user_error/submodule,tests/integration/pip_parse,tests/integration/pip_parse/empty,tests/integration/pip_repository_entry_points,tests/integration/py_cc_toolchain_registered --incompatible_default_to_explicit_init_py --enable_runfiles --lo
ckfile_mode=off
INFO: Reading rc options for 'build' from c:\users\pcloudy\.bazelrc:
  'build' options: --curses=yes --color=yes --verbose_failures --announce_rc
23.1
INFO: Repository _main~_repo_rules~repro instantiated at:
  <builtin>: in <toplevel>
Repository rule repro_rule defined at:
  C:/users/pcloudy/workdir/rules_python/python/private/repro_rule.bzl:134:29: in <toplevel>
ERROR: An error occurred during the fetch of repository '_main~_repo_rules~repro':
   Traceback (most recent call last):
        File "C:/users/pcloudy/workdir/rules_python/python/private/repro_rule.bzl", line 116, column 13, in _impl
                fail((
Error in fail: repro_rule '_main~_repo_rules~repro' failed:
  command: C:/src/tmp/ht22ikia/external/_main~python~python_3_9_x86_64-pc-windows-msvc/python.exe -c # Import from stdlib
import sys

# Add a third party package to ensure PYTHONPATH setting works
import packaging

print(packaging.__version__)
sys.exit(1)

  environment:
PYTHONPATH=C:/users/pcloudy/workdir/rules_python;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__build;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__click;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__colorama;C:/sr
c/tmp/ht22ikia/external/_main~internal_deps~pypi__importlib_metadata;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__installer;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__more_itertools;C:/src/tmp/ht22ikia/external/_main~i
nternal_deps~pypi__packaging;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__pep517;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__pip;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__pip_tools;C:/src/tmp/ht22ikia/exter
nal/_main~internal_deps~pypi__pyproject_hooks;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__setuptools;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__tomli;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__wheel;C:/src
/tmp/ht22ikia/external/_main~internal_deps~pypi__zipp
  return code: 1
===== stdout start ====
23.1

===== stdout end===
===== stderr start ====

===== stderr end===
ERROR: <builtin>: fetching repro_rule rule //:_main~_repo_rules~repro: Traceback (most recent call last):
        File "C:/users/pcloudy/workdir/rules_python/python/private/repro_rule.bzl", line 116, column 13, in _impl
                fail((
Error in fail: repro_rule '_main~_repo_rules~repro' failed:
  command: C:/src/tmp/ht22ikia/external/_main~python~python_3_9_x86_64-pc-windows-msvc/python.exe -c # Import from stdlib
import sys

# Add a third party package to ensure PYTHONPATH setting works
import packaging

print(packaging.__version__)
sys.exit(1)

  environment:
PYTHONPATH=C:/users/pcloudy/workdir/rules_python;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__build;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__click;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__colorama;C:/sr
c/tmp/ht22ikia/external/_main~internal_deps~pypi__importlib_metadata;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__installer;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__more_itertools;C:/src/tmp/ht22ikia/external/_main~i
nternal_deps~pypi__packaging;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__pep517;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__pip;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__pip_tools;C:/src/tmp/ht22ikia/exter
nal/_main~internal_deps~pypi__pyproject_hooks;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__setuptools;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__tomli;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__wheel;C:/src
/tmp/ht22ikia/external/_main~internal_deps~pypi__zipp
  return code: 1
===== stdout start ====
23.1

===== stdout end===
===== stderr start ====

===== stderr end===
ERROR: repro_rule '_main~_repo_rules~repro' failed:
  command: C:/src/tmp/ht22ikia/external/_main~python~python_3_9_x86_64-pc-windows-msvc/python.exe -c # Import from stdlib
import sys

# Add a third party package to ensure PYTHONPATH setting works
import packaging

print(packaging.__version__)
sys.exit(1)

  environment:
PYTHONPATH=C:/users/pcloudy/workdir/rules_python;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__build;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__click;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__colorama;C:/sr
c/tmp/ht22ikia/external/_main~internal_deps~pypi__importlib_metadata;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__installer;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__more_itertools;C:/src/tmp/ht22ikia/external/_main~i
nternal_deps~pypi__packaging;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__pep517;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__pip;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__pip_tools;C:/src/tmp/ht22ikia/exter
nal/_main~internal_deps~pypi__pyproject_hooks;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__setuptools;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__tomli;C:/src/tmp/ht22ikia/external/_main~internal_deps~pypi__wheel;C:/src
/tmp/ht22ikia/external/_main~internal_deps~pypi__zipp
  return code: 1
===== stdout start ====
23.1

===== stdout end===
===== stderr start ====

===== stderr end===
INFO: Elapsed time: 9.120s
INFO: 0 processes.
ERROR: Build did NOT complete successfully

@meteorcloudy
Copy link
Member Author

Hmm, but I didn't see the exact error code. Let me know if you want me to test any python command with the host Python.

aaliddell added a commit to rules-proto-grpc/rules_proto_grpc that referenced this issue Jan 31, 2024
aignas added a commit that referenced this issue Feb 5, 2024
Use the same binary path that the regular toolchain is using
in an attempt to see if the behaviour is any different.

Attempt to fix #1723
@aignas
Copy link
Collaborator

aignas commented Feb 5, 2024

I was reading the code that is executed once more and I was wondering if the need for #1745 might be causing the errors. I don't have a Windows machine but the CI still says that it is green: https://buildkite.com/bazel/rules-python-python/builds/7106#018d7924-16c6-4995-9995-e91fbdd7f9ac @meteorcloudy, if you have time, could you try and see if the following patch fixes bazel?

$ gh pr diff 1745

Gives the output

diff --git a/python/private/toolchains_repo.bzl b/python/private/toolchains_repo.bzl
index c586208a7..4c039ea84 100644
--- a/python/private/toolchains_repo.bzl
+++ b/python/private/toolchains_repo.bzl
@@ -238,11 +238,16 @@ exports_files(["python"], visibility = ["//visibility:public"])

     (os_name, arch) = get_host_os_arch(rctx)
     host_platform = get_host_platform(os_name, arch)
+
+    is_windows = (os_name == WINDOWS_NAME)
+    python3_binary_path = "python.exe" if is_windows else "bin/python3"
+
     host_python = rctx.path(
         Label(
-            "@@{py_repository}_{host_platform}//:python".format(
+            "@@{py_repository}_{host_platform}//:{python}".format(
                 py_repository = rctx.attr.name[:-len("_host")],
                 host_platform = host_platform,
+                python = python3_binary_path,
             ),
         ),
     )

@meteorcloudy
Copy link
Member Author

I'm overriding rules_python to a version 0.29.0 with this patch applied, but still got:

ERROR: An error occurred during the fetch of repository 'rules_python~override~pip~bazel_pip_dev_deps_38_bazel_runfiles':
   Traceback (most recent call last):
        File "C:/src/tmp/74bsjlkt/external/rules_python~override/python/pip_install/pip_repository.bzl", line 723, column 13, in _whl_library_impl
                fail("whl_library %s failed: %s (%s) error code: '%s'" % (rctx.attr.name, result.stdout, result.stderr, result.return_code))
Error in fail: whl_library rules_python~override~pip~bazel_pip_dev_deps_38_bazel_runfiles failed:  () error code: '-1073741515'
ERROR: <builtin>: fetching whl_library rule //:rules_python~override~pip~bazel_pip_dev_deps_38_bazel_runfiles: Traceback (most recent call last):
        File "C:/src/tmp/74bsjlkt/external/rules_python~override/python/pip_install/pip_repository.bzl", line 723, column 13, in _whl_library_impl
                fail("whl_library %s failed: %s (%s) error code: '%s'" % (rctx.attr.name, result.stdout, result.stderr, result.return_code))
Error in fail: whl_library rules_python~override~pip~bazel_pip_dev_deps_38_bazel_runfiles failed:  () error code: '-1073741515'
ERROR: no such package '@@rules_python~override~pip~bazel_pip_dev_deps_38_bazel_runfiles//': whl_library rules_python~override~pip~bazel_pip_dev_deps_38_bazel_runfiles failed:  () error code: '-1073741515'
ERROR: C:/src/tmp/74bsjlkt/external/rules_python~override~pip~bazel_pip_dev_deps/bazel_runfiles/BUILD.bazel:8:6: @@rules_python~override~pip~bazel_pip_dev_deps//bazel_runfiles:pkg depends on @@rules_python~over
ride~pip~bazel_pip_dev_deps_38_bazel_runfiles//:pkg in repository @@rules_python~override~pip~bazel_pip_dev_deps_38_bazel_runfiles which failed to fetch. no such package '@@rules_python~override~pip~bazel_pip_d
ev_deps_38_bazel_runfiles//': whl_library rules_python~override~pip~bazel_pip_dev_deps_38_bazel_runfiles failed:  () error code: '-1073741515'
INFO: Analyzed target //src/test/py/bazel:first_time_use_test (483 packages loaded, 11390 targets configured).
WARNING: errors encountered while analyzing target '//src/test/py/bazel:first_time_use_test', it will not be built.

@meteorcloudy
Copy link
Member Author

The failing command is

DEBUG: C:/src/tmp/74bsjlkt/external/rules_python~override/python/pip_install/pip_repository.bzl:722:10: command:  [C:/src/tmp/74bsjlkt/external/rules_python~override~python~python_3_8_host/python, "-m", "python
.pip_install.tools.wheel_installer.wheel_installer", "--requirement", "bazel-runfiles==0.24.0", "--isolated", "--extra_pip_args", "{\"arg\":[]}", "--pip_data_exclude", "{\"arg\":[]}", "--environment", "{\"arg\"
:{}}"]

@meteorcloudy
Copy link
Member Author

Manually running the python binary resulted:

pcloudy@bazel-windows-playground:~/workdir/bazel (master)
$ C:/src/tmp/74bsjlkt/external/rules_python~override~python~python_3_8_host/python
C:/src/tmp/74bsjlkt/external/rules_python~override~python~python_3_8_host/python: error while loading shared libraries: python38.dll: cannot open shared object file: No such file or directory

@meteorcloudy
Copy link
Member Author

I think I know what's going on. I assume C:/src/tmp/74bsjlkt/external/rules_python~override~python~python_3_8_host/python is suppose to be a symlink to the host python binary, but on Windows, it's actually a copy (because file symlink is not available by default). However, the dll files required by the python binary isn't copied over, which caused the runtime failure.

If I do bazel clean --expunge and then rebuild with bazel --windows_enable_symlinks test //src/test/py/bazel:first_time_use_test, everything works.

So basically, 4fe0db3 requires all windows users to turn on symlink support (https://bazel.build/configure/windows#symlink), which requires admin privilege, is there any way around this?

@meteorcloudy
Copy link
Member Author

Also explains why CI is green for rules_python:

startup --windows_enable_symlinks

@aignas
Copy link
Collaborator

aignas commented Feb 6, 2024

Thanks for the digging. That finally makes sense, I have one workaround in mind which I'll try.

@aignas aignas changed the title rules_python 0.29.0 broke Bazel on Windows rules_python 0.29.0 started enforcing symlink usage on Windows (was: broke Bazel on Windows) Feb 6, 2024
@aignas aignas removed the help wanted label Feb 6, 2024
@aignas
Copy link
Collaborator

aignas commented Feb 6, 2024

@meteorcloudy, I think #1745 should fix your issue, could you test bazel with that PR please? It relies on a few things on main so just applying the patch may be a tad more difficult.

@aignas aignas self-assigned this Feb 6, 2024
@meteorcloudy
Copy link
Member Author

meteorcloudy commented Feb 6, 2024

@aignas I can confirm #1745 fixes the issue for Bazel! Thanks!

$ ll C:/src/tmp/74bsjlkt/external/rules_python~override~python~python_3_8_host
total 17583
drwxr-xr-x 1 pcloudy None        0 Feb  6 10:30 ./
drwxr-xr-x 1 pcloudy None        0 Feb  6 10:30 ../
-rw-r--r-- 1 pcloudy None      115 Feb  6 10:30 BUILD.bazel
lrwxrwxrwx 1 pcloudy None       96 Feb  6 10:30 DLLs -> /c/src/tmp/74bsjlkt/external/rules_python~override~python~python_3_8_x86
_64-pc-windows-msvc/DLLs/
-rw-r--r-- 1 pcloudy None    25067 Jan  1  2021 LICENSE.txt
lrwxrwxrwx 1 pcloudy None       95 Feb  6 10:30 Lib -> /c/src/tmp/74bsjlkt/external/rules_python~override~python~python_3_8_x86_
64-pc-windows-msvc/Lib/
-rw-r--r-- 1 pcloudy None      100 Feb  6 10:30 STANDALONE_INTERPRETER
lrwxrwxrwx 1 pcloudy None       99 Feb  6 10:30 Scripts -> /c/src/tmp/74bsjlkt/external/rules_python~override~python~python_3_8_
x86_64-pc-windows-msvc/Scripts/
-rw-r--r-- 1 pcloudy None      143 Feb  6 10:30 WORKSPACE
lrwxrwxrwx 1 pcloudy None       99 Feb  6 10:30 include -> /c/src/tmp/74bsjlkt/external/rules_python~override~python~python_3_8_
x86_64-pc-windows-msvc/include/
lrwxrwxrwx 1 pcloudy None       96 Feb  6 10:30 libs -> /c/src/tmp/74bsjlkt/external/rules_python~override~python~python_3_8_x86
_64-pc-windows-msvc/libs/
-rwxr-xr-x 1 pcloudy None    94208 Jan  1  2021 python*
-rwxr-xr-x 1 pcloudy None    94208 Jan  1  2021 python.exe*
-rw-r--r-- 1 pcloudy None   438272 Jan  1  2021 python.pdb
-rwxr-xr-x 1 pcloudy None    51712 Jan  1  2021 python3.dll*
-rw-r--r-- 1 pcloudy None   159744 Jan  1  2021 python3.pdb
-rwxr-xr-x 1 pcloudy None  4227584 Jan  1  2021 python38.dll*
-rw-r--r-- 1 pcloudy None 12038144 Jan  1  2021 python38.pdb
-rwxr-xr-x 1 pcloudy None    92672 Jan  1  2021 pythonw.exe*
-rw-r--r-- 1 pcloudy None   430080 Jan  1  2021 pythonw.pdb
lrwxrwxrwx 1 pcloudy None       95 Feb  6 10:30 tcl -> /c/src/tmp/74bsjlkt/external/rules_python~override~python~python_3_8_x86_
64-pc-windows-msvc/tcl/
-rwxr-xr-x 1 pcloudy None    98224 Jan  1  2021 vcruntime140.dll*
-rwxr-xr-x 1 pcloudy None    37256 Jan  1  2021 vcruntime140_1.dll*

github-merge-queue bot pushed a commit that referenced this issue Feb 8, 2024
…1745)

Previously we would only symlink the interpreter binary itself
when creating the hermetic host toolchain used in setting up the
`whl_library` repositories. This works on UNIX platforms and Windows
if they have the following in their `.bazelrc`:
```
startup --windows_enable_symlinks
```

In our CI we had the same lines but the users did not need to add them
until the `0.29.0` forced them to have them because we actually started
using symlinks on Windows. If the symlinks are not enabled on the host
platform `bazel` tries to be helpful and copies the files over instead
of making the links. We are leveraging this to just symlink all of the
contents of the python interpreter repository for the host platform to
the `_host` toolchain repository.

Fixes #1723
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: pip type: toolchain Related to the toolchains provided by rules_python
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants