fix(whl_library): remove `--no-index` and add `--no-build-isolation` when build sdist #2126

ewianda · 2024-08-18T18:13:18Z

Building sdist results in Could not find a version that satisfies the requirement setuptool this regressed when a fix in parameter handling
got introduced in #2091.

Before this change the building from sdist when using experimental_index_url
would break because --no-index is passed to pip. This means that pip
would fail to locate build time dependencies needed for the packages and would
just not work. In whl_library we setup PYTHONPATH to have some build
dependencies available (like setuptools) and we could use them during
building from sdist and to do so we need to add --no-build-isolation flag.
However, for some cases we need to also add other build-time dependencies (e.g.
flit_core) so that the building of the wheel in the repository_rule context
is successfuly. Removing --no-index allows pip to silently fetch the needed
build dependencies from PyPI if they are missing and continue with the build.

This is not a perfect solution, but it does unblock users to use the sdist
distributions with the experimental feature enabled by using
experimental_index_url (see #260 for tracking of the completion).

Fixes #2118
Fixes #2152

aignas

Thank you for the proposed fix!

I guess if we want to ensure that no PyPI index is contacted when we build wheels, we would have to also have to include steuptools path here at the minimum. But then some wheels may require more than that.

The correct long term fix would be to move the build of the wheels to the build phase and use proper toolchains, but for now it is good enough.

@dougthor42 could you let me know if that breaks/fixes build with 0.35 in your project? Since you are using private indexes, I am would love to have more data before merging and tagging 0.35.1.

python/private/pypi/whl_library.bzl

ewianda · 2024-08-18T19:24:04Z

@dougthor42 could you let me know if that breaks/fixes build with 0.35 in your project? Since you are using private indexes, I am would love to have more data before merging and tagging 0.35.1.

I am also using a private Google artifact registry with 0.35.0., and so far, no issues.

tkinz27 · 2024-08-18T22:12:07Z

I just ran into this issue as well after upgrading to 0.35. Using

git_override(module_name="rules_python", remote="https://github.com/bazelbuild/rules_python", commit="a239635661445e41d0fdb1ee56040279412798a1")

worked for me. I am also using a private Google Artifact Registry pypi index.

ewianda · 2024-08-19T22:29:26Z

Seeing a strange behaviour, this works locally but fails in CI with hosted Github actions runner

Looking in links: .
Collecting future==0.18.3 (from -r /tmp/tmpr9vp7aw7 (line 1))
  File was already downloaded /home/runner/.cache/bazel/_bazel_runner/5a611e380dfcc606a729a75fb4b58f73/external/rules_python~~pip~py_deps_311_future_sdist_34a17436/future-0.18.3.tar.gz
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'
===== stdout end =====
===== stderr start =====
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-wheel-c_cquw40/future_5a2bd15b1e6d40c9badc12cd38128df2/setup.py", line 86, in <module>
          import src.future
      ModuleNotFoundError: No module named 'src'
      [end of output]
      
      ````

dougthor42 · 2024-08-20T23:54:15Z

@dougthor42 could you let me know if that breaks/fixes build with 0.35 in your project?

Ack 👍 . I hope to test this by the end of the week.

aignas · 2024-08-22T15:31:37Z

By the way, to test if the PR fixes #2118, one should be able to do the following:

bazel build --@rules_python//python/config_settings:pip_whl=no //:all_requirements

Within the examples/bzlmod example. It successfully fails on the main branch:

$ bazel build --@rules_python//python/config_settings:pip_whl=no //:all_requirements
WARNING: For repository 'bazel_skylib', the root module requires module version bazel_skylib@1.4.1, but got bazel_skylib@1.6.1 in the resolved dependency graph.
WARNING: For repository 'rules_proto', the root module requires module version rules_proto@5.3.0-21.7, but got rules_proto@6.0.0-rc1 in the resolved dependency graph.
WARNING: Build option --@@rules_python~//python/config_settings:pip_whl has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
INFO: Repository rules_python~~pip~pip_39_pathspec_sdist_56200de4 instantiated at:
  <builtin>: in <toplevel>
Repository rule whl_library defined at:
  /private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~/python/private/pypi/whl_library.bzl:451:30: in <toplevel>
ERROR: An error occurred during the fetch of repository 'rules_python~~pip~pip_39_pathspec_sdist_56200de4':
   Traceback (most recent call last):
        File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~/python/private/pypi/whl_library.bzl", line 243, column 35, in _whl_library_impl
                repo_utils.execute_checked(
        File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~/python/private/repo_utils.bzl", line 212, column 29, in _execute_checked
                return _execute_internal(fail_on_error = True, *args, **kwargs)
        File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~/python/private/repo_utils.bzl", line 143, column 20, in _execute_internal
                logger.fail((
        File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~/python/private/repo_utils.bzl", line 85, column 39, in lambda
                fail = lambda message_cb: _log(-1, "FAIL", message_cb, fail),
        File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~/python/private/repo_utils.bzl", line 75, column 16, in _log
                printer("\nrules_python:{} {}:".format(
Error in fail:
rules_python:whl_library(@@rules_python~~pip~pip_39_pathspec_sdist_56200de4) FAIL: repo.execute: whl_library.BuildWheelFromSource(rules_python~~pip~pip_39_pathspec_sdist_56200de4, pathspec==0.10.3): end: f
ailure:
  command: /private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~python~python_3_9_host/python -m python.private.pypi.whl_installer.wheel_installer --requirement
 pathspec==0.10.3 --isolated --extra_pip_args "{\"arg\":[\"--no-index\",\"--find-links\",\".\"]}" --pip_data_exclude "{\"arg\":[]}" --environment "{\"arg\":{}}"
  return code: 1
  working dir: <default: /private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~pip~pip_39_pathspec_sdist_56200de4>
  timeout: 600
  environment:
PYTHONPATH="/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_pytho
n~~internal_deps~pypi__build:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__click:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d
08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__colorama:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__importlib_me
tadata:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__installer:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb
5025/external/rules_python~~internal_deps~pypi__more_itertools:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__packaging:/private/var/t
mp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__pep517:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_pyth
on~~internal_deps~pypi__pip:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__pip_tools:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5
d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__pyproject_hooks:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__se
tuptools:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__tomli:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb50
25/external/rules_python~~internal_deps~pypi__wheel:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__zipp"
CPPFLAGS="-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -isystem /private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~python~python_3_9_host/inc
lude/python3.9"
===== stdout start =====
Looking in links: .
Processing ./pathspec-0.10.3.tar.gz (from -r /var/folders/nq/dvvx_cwj0d1bqv6jjntc63r40000gr/T/tmphctvbdpw (line 1))
  File was already downloaded /private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~pip~pip_39_pathspec_sdist_56200de4/pathspec-0.10.3.tar.gz
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
===== stdout end =====
===== stderr start =====
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [3 lines of output]
      Looking in links: .
      ERROR: Could not find a version that satisfies the requirement setuptools>=40.8.0 (from versions: none)
      ERROR: No matching distribution found for setuptools>=40.8.0
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~python~python_3_9_x86_64-apple-darwin/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~python~python_3_9_x86_64-apple-darwin/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~/python/private/pypi/whl_installer/wheel_installer.py", line 205, in <module>
    main()
  File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~/python/private/pypi/whl_installer/wheel_installer.py", line 190, in main
    subprocess.run(pip_args, check=True, env=env)
  File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~python~python_3_9_x86_64-apple-darwin/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~python~python_3_9_host/python', '-m', 'pip', '--isolated', 'wheel
', '--no-deps', '--no-index', '--find-links', '.', '-r', '/var/folders/nq/dvvx_cwj0d1bqv6jjntc63r40000gr/T/tmphctvbdpw']' returned non-zero exit status 1.
===== stderr end =====
ERROR: no such package '@@rules_python~~pip~pip_39_pathspec_sdist_56200de4//':
rules_python:whl_library(@@rules_python~~pip~pip_39_pathspec_sdist_56200de4) FAIL: repo.execute: whl_library.BuildWheelFromSource(rules_python~~pip~pip_39_pathspec_sdist_56200de4, pathspec==0.10.3): end: f
ailure:
  command: /private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~python~python_3_9_host/python -m python.private.pypi.whl_installer.wheel_installer --requirement
 pathspec==0.10.3 --isolated --extra_pip_args "{\"arg\":[\"--no-index\",\"--find-links\",\".\"]}" --pip_data_exclude "{\"arg\":[]}" --environment "{\"arg\":{}}"
  return code: 1
  working dir: <default: /private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~pip~pip_39_pathspec_sdist_56200de4>
  timeout: 600
  environment:
PYTHONPATH="/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_pytho
n~~internal_deps~pypi__build:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__click:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d
08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__colorama:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__importlib_me
tadata:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__installer:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb
5025/external/rules_python~~internal_deps~pypi__more_itertools:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__packaging:/private/var/t
mp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__pep517:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_pyth
on~~internal_deps~pypi__pip:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__pip_tools:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5
d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__pyproject_hooks:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__se
tuptools:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__tomli:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb50
25/external/rules_python~~internal_deps~pypi__wheel:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__zipp"
CPPFLAGS="-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -isystem /private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~python~python_3_9_host/inc
lude/python3.9"
===== stdout start =====
Looking in links: .
Processing ./pathspec-0.10.3.tar.gz (from -r /var/folders/nq/dvvx_cwj0d1bqv6jjntc63r40000gr/T/tmphctvbdpw (line 1))
  File was already downloaded /private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~pip~pip_39_pathspec_sdist_56200de4/pathspec-0.10.3.tar.gz
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
===== stdout end =====
===== stderr start =====
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [3 lines of output]
      Looking in links: .
      ERROR: Could not find a version that satisfies the requirement setuptools>=40.8.0 (from versions: none)
      ERROR: No matching distribution found for setuptools>=40.8.0
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~python~python_3_9_x86_64-apple-darwin/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~python~python_3_9_x86_64-apple-darwin/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~/python/private/pypi/whl_installer/wheel_installer.py", line 205, in <module>
    main()
  File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~/python/private/pypi/whl_installer/wheel_installer.py", line 190, in main
    subprocess.run(pip_args, check=True, env=env)
  File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~python~python_3_9_x86_64-apple-darwin/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~python~python_3_9_host/python', '-m', 'pip', '--isolated', 'wheel
', '--no-deps', '--no-index', '--find-links', '.', '-r', '/var/folders/nq/dvvx_cwj0d1bqv6jjntc63r40000gr/T/tmphctvbdpw']' returned non-zero exit status 1.
===== stderr end =====
ERROR: /private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~pip~pip/pathspec/BUILD.bazel:31:6: @@rules_python~~pip~pip//pathspec:pkg depends on @@rules_python~~
pip~pip_39_pathspec_sdist_56200de4//:pkg in repository @@rules_python~~pip~pip_39_pathspec_sdist_56200de4 which failed to fetch. no such package '@@rules_python~~pip~pip_39_pathspec_sdist_56200de4//':
rules_python:whl_library(@@rules_python~~pip~pip_39_pathspec_sdist_56200de4) FAIL: repo.execute: whl_library.BuildWheelFromSource(rules_python~~pip~pip_39_pathspec_sdist_56200de4, pathspec==0.10.3): end: f
ailure:
  command: /private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~python~python_3_9_host/python -m python.private.pypi.whl_installer.wheel_installer --requirement
 pathspec==0.10.3 --isolated --extra_pip_args "{\"arg\":[\"--no-index\",\"--find-links\",\".\"]}" --pip_data_exclude "{\"arg\":[]}" --environment "{\"arg\":{}}"
  return code: 1
  working dir: <default: /private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~pip~pip_39_pathspec_sdist_56200de4>
  timeout: 600
  environment:
PYTHONPATH="/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_pytho
n~~internal_deps~pypi__build:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__click:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d
08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__colorama:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__importlib_me
tadata:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__installer:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb
5025/external/rules_python~~internal_deps~pypi__more_itertools:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__packaging:/private/var/t
mp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__pep517:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_pyth
on~~internal_deps~pypi__pip:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__pip_tools:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5
d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__pyproject_hooks:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__se
tuptools:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__tomli:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb50
25/external/rules_python~~internal_deps~pypi__wheel:/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~internal_deps~pypi__zipp"
CPPFLAGS="-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -isystem /private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~python~python_3_9_host/inc
lude/python3.9"
===== stdout start =====
Looking in links: .
Processing ./pathspec-0.10.3.tar.gz (from -r /var/folders/nq/dvvx_cwj0d1bqv6jjntc63r40000gr/T/tmphctvbdpw (line 1))
  File was already downloaded /private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~pip~pip_39_pathspec_sdist_56200de4/pathspec-0.10.3.tar.gz
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
===== stdout end =====
===== stderr start =====
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [3 lines of output]
      Looking in links: .
      ERROR: Could not find a version that satisfies the requirement setuptools>=40.8.0 (from versions: none)
      ERROR: No matching distribution found for setuptools>=40.8.0
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~python~python_3_9_x86_64-apple-darwin/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~python~python_3_9_x86_64-apple-darwin/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~/python/private/pypi/whl_installer/wheel_installer.py", line 205, in <module>
    main()
  File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~/python/private/pypi/whl_installer/wheel_installer.py", line 190, in main
    subprocess.run(pip_args, check=True, env=env)
  File "/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~python~python_3_9_x86_64-apple-darwin/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~python~python_3_9_host/python', '-m', 'pip', '--isolated', 'wheel
', '--no-deps', '--no-index', '--find-links', '.', '-r', '/var/folders/nq/dvvx_cwj0d1bqv6jjntc63r40000gr/T/tmphctvbdpw']' returned non-zero exit status 1.
===== stderr end =====
ERROR: Analysis of target '//:all_requirements' failed; build aborted: Analysis failed
INFO: Elapsed time: 23.853s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
FAILED:
    Fetching repository @@rules_python~~pip~pip_39_pyyaml_sdist_bfdf460b; starting 19s
    Fetching repository @@rules_python~~pip~pip_39_zipp_sdist_84e64a1c; starting 19s
    Fetching repository @@rules_python~~pip~pip_39_certifi_sdist_539cc1d1; starting 19s
    Fetching repository @@rules_python~~pip~pip_39_colorama_sdist_08695f5c; starting 19s
    Fetching repository @@rules_python~~pip~pip_39_isort_sdist_6db30c5d; starting 19s
    Fetching repository @@rules_python~~pip~pip_39_websockets_sdist_88fc51d9; starting 19s
    Fetching repository @@rules_python~~pip~pip_39_python_dateutil_sdist_0123cacc; starting 19s
    Fetching repository @@rules_python~~pip~pip_39_python_magic_sdist_c1ba14b0; starting 19s ... (46 fetches)

And this branch:

$ bazel build --@rules_python//python/config_settings:pip_whl=no //:all_requirements
WARNING: For repository 'bazel_skylib', the root module requires module version bazel_skylib@1.4.1, but got bazel_skylib@1.6.1 in the resolved dependency graph.
WARNING: For repository 'rules_proto', the root module requires module version rules_proto@5.3.0-21.7, but got rules_proto@6.0.0-rc1 in the resolved dependency graph.
DEBUG: /private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~/python/private/pypi/patch_whl.bzl:109:14: WARNING: the resultant RECORD file of the patch wheel is different

    If you are patching on Windows, you may see this warning because of
    a known issue (bazelbuild/rules_python#1639) with file endings.

    If you would like to silence the warning, you can apply the patch that is stored in
      /private/var/tmp/_bazel_ignas.anikevicius/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~~pip~pip_39_requests_sdist_27973dd4/RECORD.patch. The contents of the file are below:
--- a/requests-2.25.1.dist-info/RECORD
+++ b/requests-2.25.1.dist-info/RECORD
@@ -16,6 +16,7 @@
 requests/status_codes.py,sha256=gT79Pbs_cQjBgp-fvrUgg1dn2DQO32bDj4TInjnMPSc,4188
 requests/structures.py,sha256=msAtr9mq1JxHd-JRyiILfdFlpbJwvvFuP3rfUQT_QxE,3005
 requests/utils.py,sha256=_K9AgkN6efPe-a-zgZurXzds5PBC0CzDkyjAE2oCQFQ,30529
+requests-2.25.1.tar.gz,sha256=J5c91KkEpPE7JjoZyGbBO5KjntHJZGVfAl8_jT11uAQ,102161
 requests-2.25.1.dist-info/LICENSE,sha256=CeipvOyAZxBGUsFoaFqwkx54aPnIKEtm9a5u2uXxEws,10142
 requests-2.25.1.dist-info/METADATA,sha256=fRSAA0u0Bi0heD4zYq91wdNUTJlbzhK6_iDOcRRNDx4,4177
 requests-2.25.1.dist-info/WHEEL,sha256=Z-nyYpwrcSqxfdux5Mbn_DQ525iP7J2DG3JgGvOYyTQ,110
INFO: Analyzed target //:all_requirements (49 packages loaded, 3971 targets configured).
INFO: Found 1 target...
Target //:all_requirements up-to-date:
  bazel-bin/all_requirements.sh
INFO: Elapsed time: 41.071s, Critical Path: 0.22s
INFO: 43 processes: 2 internal, 41 darwin-sandbox.
INFO: Build completed successfully, 43 total actions

So the only remaining thing from my point of view for this PR is to ensure that the CHANGELOG.md is written.

EDIT: I think it would be good to also add a test that transitions all_requirements target in the example to a platform which sets the said flag value to no, so that we also test by default that all of the sdists can be built in the examples/bzlmod example.

…uild of sdist

dougthor42 · 2024-08-23T04:37:23Z

@dougthor42 could you let me know if that breaks/fixes build with 0.35 in your project?

Ack 👍 . I hope to test this by the end of the week.

I tested using commit fb3c56d. Everything looked good for us. 🎉

aignas · 2024-08-24T08:55:33Z

I have made some minor adjustments:

Changelog updates, added comments.
Moved the test to the internal rules_python test codebase so that the example is less noisy - figured it would be a better place in the long term. Thanks for the little transition rule!
Changed the PR description to better explain what is going on and how it is fixed and what drawbacks it has.

I tried adding back the --no-index but the build started failing because some build-time dependencies are not present and we are not tracking them - e.g. flit_core is needed for building jinja2 from source and I think there is a ticket (#1325) for that for addressing this part.

ewianda · 2024-08-30T18:08:12Z

@dougthor42 was curious how this played out in your org in general. It did not end well for us; random packages could not be built on random machines at random times. I endded up uploading all the missing wheels to mitigate the issues.

dougthor42 · 2024-10-16T21:53:32Z

@ewianda sorry for taking so long to reply here. We only updated to rules_python 0.36.0 on Oct 4th.

I'm happy to report that we haven't had any issues!

Fixes #2328. In #2126, we discussed and added `--no-build-isolation` to ensure that sdist builds pulled from rules_python supplied dependencies. An unanticipated/misunderstood side-effect of this is that pip will no longer resolve necessary, declared build time dependencies from a package index. That's a significant point of friction for users, and, it's much harder to opt-out of the behavior than to opt-in given the presence of `extra_pip_args`.

ewianda requested review from aignas and groodt as code owners August 18, 2024 18:13

aignas reviewed Aug 18, 2024

View reviewed changes

python/private/pypi/whl_library.bzl Outdated Show resolved Hide resolved

ewianda mentioned this pull request Aug 20, 2024

experimental_index_url fetching of large dep graphs is slow #2014

Closed

aignas mentioned this pull request Aug 21, 2024

bzlmod example fails to build on pip install error when building whls from sdists #2118

Closed

ewianda force-pushed the fix-remove-no-index branch from a239635 to 9bafcbd Compare August 21, 2024 22:49

aignas mentioned this pull request Aug 22, 2024

When the wheel is patched, the sdist is included into the repackaged wheel. #2143

Open

ewianda force-pushed the fix-remove-no-index branch from 9bafcbd to 4efee77 Compare August 22, 2024 16:57

ewianda requested a review from rickeylev as a code owner August 22, 2024 16:57

fix: Remove --no-index when build sdist

a79c508

ewianda force-pushed the fix-remove-no-index branch 5 times, most recently from 023524e to 893539c Compare August 23, 2024 00:56

Add a transition rule that transitions all_requiremnts file to test b…

fb3c56d

…uild of sdist

ewianda force-pushed the fix-remove-no-index branch from 893539c to fb3c56d Compare August 23, 2024 02:18

aignas added 3 commits August 24, 2024 11:29

Merge branch 'main' into fix-remove-no-index

41432e7

refactor: move the transition test for sdist to the main module

9e11ccf

doc: modify the comment and adjust changelog

cab43a6

aignas changed the title ~~fix: Remove --no-index when build sdist~~ fix(whl_library): remove --no-index and add --no-build-isolation when build sdist Aug 24, 2024

chore: update examples/bzlmod lockfile

23e1987

aignas approved these changes Aug 24, 2024

View reviewed changes

aignas added this pull request to the merge queue Aug 24, 2024

aignas mentioned this pull request Aug 24, 2024

[Regression] Can't upgrade from 0.33.1 to 0.35.0 because of change to how sdists are processed #2152

Closed

aignas removed this pull request from the merge queue due to a manual request Aug 24, 2024

aignas added this pull request to the merge queue Aug 24, 2024

Merged via the queue into bazelbuild:main with commit b99bb61 Aug 24, 2024
4 checks passed

mark-thm mentioned this pull request Oct 22, 2024

Not possible to use bottleneck as a dependency on linux arm64 with rules_python 0.37.0 #2328

Closed

mark-thm mentioned this pull request Oct 30, 2024

fix: Remove --no-build-isolation from pip install command #2360

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(whl_library): remove `--no-index` and add `--no-build-isolation` when build sdist #2126

fix(whl_library): remove `--no-index` and add `--no-build-isolation` when build sdist #2126

ewianda commented Aug 18, 2024 •

edited by aignas

Loading

aignas left a comment

ewianda commented Aug 18, 2024 •

edited

Loading

tkinz27 commented Aug 18, 2024

ewianda commented Aug 19, 2024 •

edited

Loading

dougthor42 commented Aug 20, 2024

aignas commented Aug 22, 2024 •

edited

Loading

dougthor42 commented Aug 23, 2024

aignas commented Aug 24, 2024 •

edited

Loading

ewianda commented Aug 30, 2024 •

edited

Loading

dougthor42 commented Oct 16, 2024

fix(whl_library): remove --no-index and add --no-build-isolation when build sdist #2126

fix(whl_library): remove --no-index and add --no-build-isolation when build sdist #2126

Conversation

ewianda commented Aug 18, 2024 • edited by aignas Loading

aignas left a comment

Choose a reason for hiding this comment

ewianda commented Aug 18, 2024 • edited Loading

tkinz27 commented Aug 18, 2024

ewianda commented Aug 19, 2024 • edited Loading

dougthor42 commented Aug 20, 2024

aignas commented Aug 22, 2024 • edited Loading

dougthor42 commented Aug 23, 2024

aignas commented Aug 24, 2024 • edited Loading

ewianda commented Aug 30, 2024 • edited Loading

dougthor42 commented Oct 16, 2024

fix(whl_library): remove `--no-index` and add `--no-build-isolation` when build sdist #2126

fix(whl_library): remove `--no-index` and add `--no-build-isolation` when build sdist #2126

ewianda commented Aug 18, 2024 •

edited by aignas

Loading

ewianda commented Aug 18, 2024 •

edited

Loading

ewianda commented Aug 19, 2024 •

edited

Loading

aignas commented Aug 22, 2024 •

edited

Loading

aignas commented Aug 24, 2024 •

edited

Loading

ewianda commented Aug 30, 2024 •

edited

Loading