Black execution fails (missing 'typing_extensions')

[ph03]

Trying to run v0.2.2 with

python_register_toolchains(
    name = "python3",
    python_version = "3.10",
)

fails with

Traceback (most recent call last):
  File "/home/ph03/.cache/bazel/_bazel_ph03/2db8cbce6fe365e36a9715d20aab49e9/execroot/dsai/bazel-out/k8-opt/bin/external/aspect_rules_format_pypi_black/rules_python_wheel_entry_point_black.runfiles/aspect_rules_format_pypi_black/rules_python_wheel_entry_point_black.py", line 3, in <module>
    from black import patched_main
  File "/home/ph03/.cache/bazel/_bazel_ph03/2db8cbce6fe365e36a9715d20aab49e9/execroot/dsai/bazel-out/k8-opt/bin/external/aspect_rules_format_pypi_black/rules_python_wheel_entry_point_black.runfiles/aspect_rules_format_pypi_black/site-packages/black/__init__.py", line 42, in <module>
    from black.nodes import STARS, syms, is_simple_decorator_expression
  File "/home/ph03/.cache/bazel/_bazel_ph03/2db8cbce6fe365e36a9715d20aab49e9/execroot/dsai/bazel-out/k8-opt/bin/external/aspect_rules_format_pypi_black/rules_python_wheel_entry_point_black.runfiles/aspect_rules_format_pypi_black/site-packages/black/nodes.py", line 24, in <module>
    from typing_extensions import TypeGuard
ModuleNotFoundError: No module named 'typing_extensions'
FAILED: A formatter tool exited with code 123
Try running 'bazel run @aspect_rules_format//format' to fix this.

while running with

python_register_toolchains(
    name = "python3",
    python_version = "3.10",
)

already fails at the configuration step with

INFO: Invocation ID: c11114e4-935a-4a07-b80a-3b9cddb4bf13
ERROR: /home/ph03/.cache/bazel/_bazel_ph03/2db8cbce6fe365e36a9715d20aab49e9/external/aspect_rules_format_pypi_black/BUILD.bazel:22:11: no such package '@aspect_rules_format_pypi_typing_extensions//': The repository '@aspect_rules_format_pypi_typing_extensions' could not be resolved: Repository '@aspect_rules_format_pypi_typing_extensions' is not defined and referenced by '@aspect_rules_format_pypi_black//:pkg'
ERROR: Analysis of target '@aspect_rules_format//format:format' failed; build aborted: 

which looks like an inconsistency to me. Note that I have a python toolchain registered before, i.e.,

python_register_toolchains(
    name = "python39",
    # Available versions are listed in @rules_python//python:versions.bzl
    python_version = "3.9",
)

but that one is not picked up by bazel-super-formatter as it's expecting the python toolchain to be called python3 not python39 (could that be the source of the issue?)

[alexeagle]

Thanks for the issue. What version of rules_python is that running with?

You're right that python3 is hard-coded here

bazel-super-formatter/format/dependencies.bzl

Line 2 in 7803a5f

load("@python3//:defs.bzl", "interpreter")

could you try locally changing that to the one you intend, and confirm if it fixes the issue?

[ph03]

Thanks for the issue. What version of rules_python is that running with?

This is with rules_python-0.10.0 - I the original python_register_toolchains registration to also use python3 and with python_version = "3.9", it's still failing with

ERROR: /home/ph03/.cache/bazel/_bazel_janickm/2db8cbce6fe365e36a9715d20aab49e9/external/aspect_rules_format_pypi_black/BUILD.bazel:22:11: no such package '@aspect_rules_format_pypi_typing_extensions//': The repository '@aspect_rules_format_pypi_typing_extensions' could not be resolved: Repository '@aspect_rules_format_pypi_typing_extensions' is not defined and referenced by '@aspect_rules_format_pypi_black//:pkg'

To me it feels like the locked dependencies are evaluated for python3.10 (see This file is autogenerated by pip-compile with python 3.10 in https://github.com/aspect-build/bazel-super-formatter/blob/main/requirements_lock.txt), however, we are kind of locked into python3.9 and can't use python3.10 - is it perhaps possible to have different requirement files for each python version?

[ewhauser]

I hacked around this by patching the rules to use the version of black installed with our repository requirements file as a workaround. Obviously, not the best long term decision but we use the auto-detecting toolchain so needed the workaround:

WORKSPACE

http_archive(
    name = "aspect_rules_format",
    patch_args = ["-p1"],
    patches = [
        "//bazel/bazel-super-formatter:0001-use-autodetecting-python-toolchain.patch",
    ],
    sha256 = "3eaa9851e09e3d2813e6c4ac1b6153e64ee44b494fa4ddf36fff2ba8b105e8db",
    strip_prefix = "bazel-super-formatter-0.3.1",
    url = "https://github.com/aspect-build/bazel-super-formatter/archive/refs/tags/v0.3.1.tar.gz",
)

0001-use-autodetecting-python-toolchain.patch

diff --git a/format/BUILD.bazel b/format/BUILD.bazel
index 0e19609..8275d5c 100644
--- a/format/BUILD.bazel
+++ b/format/BUILD.bazel
@@ -1,7 +1,7 @@
 load("@bazel_skylib//:bzl_library.bzl", "bzl_library")
 load("@bazel_skylib//rules:common_settings.bzl", "bool_flag")
 load("@aspect_rules_format_npm//:prettier/package_json.bzl", prettier = "bin")
-load("@aspect_rules_format_pypi//:requirements.bzl", "entry_point")
+load("@pypi//:requirements.bzl", "entry_point")
 load(":langs.bzl", "LANGS")

 bzl_library(
diff --git a/format/dependencies.bzl b/format/dependencies.bzl
index e9ba437..1807e3a 100644
--- a/format/dependencies.bzl
+++ b/format/dependencies.bzl
@@ -1,5 +1,4 @@
 load("@aspect_rules_js//npm:npm_import.bzl", "npm_translate_lock")
-load("@python3//:defs.bzl", "interpreter")
 load("@rules_python//python:pip.bzl", "pip_parse")

 def parse_dependencies():
@@ -8,9 +7,3 @@ def parse_dependencies():
         link_workspace = "aspect_rules_format",
         pnpm_lock = "@aspect_rules_format//:pnpm-lock.yaml",
     )
-
-    pip_parse(
-        name = "aspect_rules_format_pypi",
-        python_interpreter_target = interpreter,
-        requirements_lock = "@aspect_rules_format//:requirements_lock.txt",
-    )
diff --git a/format/format.sh b/format/format.sh
index 60ee4bb..68fb340 100755
--- a/format/format.sh
+++ b/format/format.sh
@@ -96,7 +96,7 @@ if [ "$#" -eq 0 ]; then
 else
   files=$(find "$@" -name '*.py' -or -name '*.pyi')
 fi
-bin=$(rlocation aspect_rules_format_pypi_black/rules_python_wheel_entry_point_black)
+bin=$(rlocation pypi_black/rules_python_wheel_entry_point_black)
 if [ -n "$files" ] && [ -n "$bin" ]; then
   echo "Running black..."
   echo "$files" | tr \\n \\0 | xargs -0 $bin $blackmode
@@ -131,7 +131,7 @@ if [ -n "$files" ] && [ -n "$bin" ]; then
        echo "$NEED_FMT"
        exit 1
     fi
-  else 
+  else
     echo "$files" | tr \\n \\0 | xargs -0 $bin $gofmtmode
   fi
 fi
diff --git a/format/toolchains.bzl b/format/toolchains.bzl
index e04b7c0..b7c6803 100644
--- a/format/toolchains.bzl
+++ b/format/toolchains.bzl
@@ -1,5 +1,4 @@
 load("@buildifier_prebuilt//:defs.bzl", "buildifier_prebuilt_register_toolchains")
-load("@aspect_rules_format_pypi//:requirements.bzl", "install_deps")
 load("@aspect_rules_format_npm//:repositories.bzl", "npm_repositories")
 load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains")

@@ -10,8 +9,5 @@ def format_register_toolchains():
     if "go_sdk" not in native.existing_rules():
         go_register_toolchains(version = "1.19.1")

-    # Initialize repositories for all packages in requirements_lock.txt.
-    install_deps()
-
     # Initialize repositories for all packages in pnpm-lock.yaml.
     npm_repositories()

[alexeagle]

I just did an overhaul of the python setup to make it optional, I think that also means you could install black yourself so long as it appears at @aspect_rules_format_pypi_black//:rules_python_wheel_entry_point_black

[ewhauser]

Confirmed that works. For those following along at home, you just need to do this:

pip_parse(
    name = "aspect_rules_format_pypi",
    requirements_lock = "//tools/black:requirements.txt",
)

load("@aspect_rules_format_pypi//:requirements.bzl", _aspect_rules_format_pypi_install_deps = "install_deps")

_aspect_rules_format_pypi_install_deps()

Thanks!