uv pip and python -m pip resolve different versions of tensorflow in pyhf developer environment #2062
Labels
compatibility
Compatibility with a specification or another tool
question
Asking for clarification or support
resolver
Related to the package resolver
Comments
4 tasks
|
Will take a look... My guess is that these are both "valid" resolutions (since uv did pin some packages to more recent versions), and the difference is due to semi-arbitrary choices in how resolution played out in the two systems. But there could be a bug somewhere. Also confusing is that |
Thanks!
Correct, |
|
I'm speculating here, but I would make the guess that this is another example of uv backtracking on the package creating an upper bound requirement, rather than backtracking on the package being upper bounded, like in #1398. In this case tensorflow is upper bounding gast, grpcio, keras, numpy, protobuf, tensorboard, tensorflow-estimator, and wrapt. Skimming over the verbose logs I would say the most likely culprit is numpy. I can check later but I suspect the resolution is going to be better (and faster) with a pinned or lower bound requirement on numpy, or one of the other packages. This kind of issue would came up a lot on the pip issue tracker after pip debuted it's resolver, that a new release of pip would cause a different solution. It's not that the resolver is incorrect, it's a kind of performance issue, if the user has strict lower bounds they want on certain packages they should include it in their requirements or constraints. |
Yeah, for just installing Using pip:$ docker run --rm -ti python:3.8 /bin/bash
root@0aa33f8bda2c:/# python -m venv venv && . venv/bin/activate
(venv) root@0aa33f8bda2c:/# python -m pip --quiet install --upgrade pip setuptools wheel
(venv) root@0aa33f8bda2c:/# python -m pip --verbose --no-cache-dir install --upgrade tensorflow tensorflow-probability &> /tmp/python-pip-install-verbose.txt
(venv) root@0aa33f8bda2c:/# python -m pip list
Package Version
---------------------------- --------
absl-py 2.1.0
astunparse 1.6.3
cachetools 5.3.3
certifi 2024.2.2
charset-normalizer 3.3.2
cloudpickle 3.0.0
decorator 5.1.1
dm-tree 0.1.8
flatbuffers 23.5.26
gast 0.4.0
google-auth 2.28.1
google-auth-oauthlib 1.0.0
google-pasta 0.2.0
grpcio 1.62.0
h5py 3.10.0
idna 3.6
importlib-metadata 7.0.1
keras 2.13.1
libclang 16.0.6
Markdown 3.5.2
MarkupSafe 2.1.5
numpy 1.24.3
oauthlib 3.2.2
opt-einsum 3.3.0
packaging 23.2
pip 24.0
protobuf 4.25.3
pyasn1 0.5.1
pyasn1-modules 0.3.0
requests 2.31.0
requests-oauthlib 1.3.1
rsa 4.9
setuptools 69.1.1
six 1.16.0
tensorboard 2.13.0
tensorboard-data-server 0.7.2
tensorflow 2.13.1
tensorflow-estimator 2.13.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability 0.21.0
termcolor 2.4.0
typing_extensions 4.5.0
urllib3 2.2.1
Werkzeug 3.0.1
wheel 0.42.0
wrapt 1.16.0
zipp 3.17.0
(venv) root@0aa33f8bda2c:/# python -m pip list | grep tensor
tensorboard 2.13.0
tensorboard-data-server 0.7.2
tensorflow 2.13.1
tensorflow-estimator 2.13.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability 0.21.0
(venv) root@0aa33f8bda2c:/#python-pip-install-verbose.txt uv v0.1.13:(venv) root@0aa33f8bda2c:/# deactivate
root@0aa33f8bda2c:/# python -m venv /uv-venv && . /uv-venv/bin/activate
(uv-venv) root@0aa33f8bda2c:/# python -m pip --quiet install --upgrade pip setuptools wheel uv
(uv-venv) root@0aa33f8bda2c:/# uv --version
uv 0.1.13
(uv-venv) root@0aa33f8bda2c:/# uv pip --verbose --no-cache install --upgrade tensorflow tensorflow-probability &> /tmp/uv-pip-install-verbose.txt
(uv-venv) root@0aa33f8bda2c:/# uv pip list
Package Version
---------------------------- --------
absl-py 2.1.0
astunparse 1.6.3
cachetools 5.3.3
certifi 2024.2.2
charset-normalizer 3.3.2
cloudpickle 3.0.0
decorator 5.1.1
dm-tree 0.1.8
flatbuffers 23.5.26
gast 0.4.0
google-auth 2.28.1
google-auth-oauthlib 1.0.0
google-pasta 0.2.0
grpcio 1.62.0
h5py 3.10.0
idna 3.6
importlib-metadata 7.0.1
keras 2.13.1
libclang 16.0.6
markdown 3.5.2
markupsafe 2.1.5
numpy 1.24.3
oauthlib 3.2.2
opt-einsum 3.3.0
packaging 23.2
pip 24.0
protobuf 4.25.3
pyasn1 0.5.1
pyasn1-modules 0.3.0
requests 2.31.0
requests-oauthlib 1.3.1
rsa 4.9
setuptools 69.1.1
six 1.16.0
tensorboard 2.13.0
tensorboard-data-server 0.7.2
tensorflow 2.13.1
tensorflow-estimator 2.13.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability 0.21.0
termcolor 2.4.0
typing-extensions 4.5.0
urllib3 2.2.1
uv 0.1.13
werkzeug 3.0.1
wheel 0.42.0
wrapt 1.16.0
zipp 3.17.0
(uv-venv) root@0aa33f8bda2c:/# uv pip list | grep tensor
tensorboard 2.13.0
tensorboard-data-server 0.7.2
tensorflow 2.13.1
tensorflow-estimator 2.13.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability 0.21.0
(uv-venv) root@0aa33f8bda2c:/# |
zanieb
added
question
|
When trying to add |
|
@ImportTaste please open a new issue, that looks unrelated to this one. |
matthewfeickert
added a commit
to scikit-hep/pyhf
that referenced
this issue
Apr 30, 2024
* Use 'uv pip' for all calls to 'pip install' and 'pip uninstall' in CI workflows. - c.f. https://github.com/astral-sh/uv/ - Still use pip for Python 3.8 until astral-sh/uv#2062 is resolved. * Apply subtle changes to install commands in .github/workflows/dependencies-head.yml. - 'uv pip install --upgrade' will try to upgrade all dependencies of the target package as well, which for the dependencies-head workflow isn't the goal. So remove the '--upgrade' from calls that also install from the scientific-python-nightly-wheels package index when testing only particular packages. - 'up pip' and 'pip' have different behavior with regards to --extra-index-url, as 'uv pip' gives --extra-index-url priority over --index-url, where 'pip' does not give priority to either. Use this with 'uv pip' to give priority to the scientific-python-nightly-wheels package index. * Add uv to the 'develop' extras.
|
Here's a more minimal reproducer of the problem, using $ uv --version
uv 0.1.39At this commit, tensorflow = [
# python == 3.8
"tensorflow>=2.7.0; python_version < '3.9' and platform_machine != 'arm64'", # c.f. PR #1962, #2452
"tensorflow-macos>=2.7.0; python_version < '3.9' and platform_machine == 'arm64' and platform_system == 'Darwin'", # c.f. PR #2119, #2452
"tensorflow-probability>=0.11.0; python_version < '3.9'", # c.f. PR #1657, #2452
# python >= 3.9
"tensorflow-probability[tf]>=0.24.0; python_version >= '3.9'" # c.f. PR #2452
]so on Python 3.8 (Linux) running should be the same as running yet these result in different environments, where I (as the developer) can say that the second environment is the expected solve. uv v0.1.39 with tensorflow extra (failure case):$ docker run --rm -ti -v /tmp:/tmp python:3.8 /bin/bash
root@4935323db0ed:/# curl -LsSf https://astral.sh/uv/install.sh | sh
downloading uv 0.1.39 x86_64-unknown-linux-gnu
installing to /root/.cargo/bin
uv
everything's installed!
To add $HOME/.cargo/bin to your PATH, either restart your shell or run:
source $HOME/.cargo/env
root@4935323db0ed:/# . ~/.cargo/env
root@4935323db0ed:/# uv venv && . .venv/bin/activate
Using Python 3.8.19 interpreter at: usr/local/bin/python3
Creating virtualenv at: .venv
(.venv) root@4935323db0ed:/# git clone --single-branch https://github.com/scikit-hep/pyhf.git && cd pyhf && git reset --hard 08755e45ed134cbe6e5e6e238b7b5011700080b3
(.venv) root@4935323db0ed:/pyhf# uv --version
uv 0.1.39
(.venv) root@4935323db0ed:/pyhf# uv pip --verbose --no-cache install '.[tensorflow]' &> /tmp/uv-pip-install-tensorflow-extra-verbose.txt
(.venv) root@4935323db0ed:/pyhf# uv pip list
Package Version
---------------------------- ------------
absl-py 2.1.0
astunparse 1.6.3
attrs 23.2.0
cachetools 5.3.3
certifi 2024.2.2
charset-normalizer 3.3.2
click 8.1.7
cloudpickle 3.0.0
decorator 5.1.1
dm-tree 0.1.8
flatbuffers 24.3.25
gast 0.4.0
google-auth 2.29.0
google-auth-oauthlib 0.4.6
google-pasta 0.2.0
grpcio 1.62.2
h5py 3.11.0
idna 3.7
importlib-metadata 7.1.0
importlib-resources 6.4.0
jsonpatch 1.33
jsonpointer 2.4
jsonschema 4.21.1
jsonschema-specifications 2023.12.1
keras 2.11.0
libclang 18.1.1
markdown 3.6
markupsafe 2.1.5
numpy 1.24.4
oauthlib 3.2.2
opt-einsum 3.3.0
packaging 24.0
pkgutil-resolve-name 1.3.10
protobuf 3.19.6
pyasn1 0.6.0
pyasn1-modules 0.4.0
pyhf 0.7.1.dev238
pyyaml 6.0.1
referencing 0.35.0
requests 2.31.0
requests-oauthlib 2.0.0
rpds-py 0.18.0
rsa 4.9
scipy 1.10.1
setuptools 69.5.1
six 1.16.0
tensorboard 2.11.2
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.1
tensorflow 2.11.1
tensorflow-estimator 2.11.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability 0.21.0
termcolor 2.4.0
tqdm 4.66.2
typing-extensions 4.5.0
urllib3 2.2.1
werkzeug 3.0.2
wheel 0.43.0
wrapt 1.16.0
zipp 3.18.1
(.venv) root@4935323db0ed:/pyhf# uv pip list | grep tensorflow
tensorflow 2.11.1
tensorflow-estimator 2.11.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability 0.21.0
(.venv) root@4935323db0ed:/pyhf# python -c 'import tensorflow; import tensorflow_probability'
2024-04-30 07:01:07.319640: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-30 07:01:07.399553: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-30 07:01:07.402137: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2024-04-30 07:01:07.402156: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2024-04-30 07:01:09.446572: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory
2024-04-30 07:01:09.446619: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory
2024-04-30 07:01:09.446626: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/.venv/lib/python3.8/site-packages/tensorflow_probability/__init__.py", line 20, in <module>
from tensorflow_probability import substrates
File "/.venv/lib/python3.8/site-packages/tensorflow_probability/substrates/__init__.py", line 17, in <module>
from tensorflow_probability.python.internal import all_util
File "/.venv/lib/python3.8/site-packages/tensorflow_probability/python/__init__.py", line 138, in <module>
dir(globals()[pkg_name]) # Forces loading the package from its lazy loader.
File "/.venv/lib/python3.8/site-packages/tensorflow_probability/python/internal/lazy_loader.py", line 57, in __dir__
module = self._load()
File "/.venv/lib/python3.8/site-packages/tensorflow_probability/python/internal/lazy_loader.py", line 37, in _load
self._on_first_access()
File "/.venv/lib/python3.8/site-packages/tensorflow_probability/python/__init__.py", line 59, in _validate_tf_environment
raise ImportError(
ImportError: This version of TensorFlow Probability requires TensorFlow version >= 2.13; Detected an installation of version 2.11.1. Please upgrade TensorFlow to proceed.
(.venv) root@4935323db0ed:/pyhf#uv-pip-install-tensorflow-extra-verbose.txt uv v0.1.39 with tensorflow,tfp on command line (success case):$ docker run --rm -ti -v /tmp:/tmp python:3.8 /bin/bash
root@edbb84aa61db:/# curl -LsSf https://astral.sh/uv/install.sh | sh
downloading uv 0.1.39 x86_64-unknown-linux-gnu
installing to /root/.cargo/bin
uv
everything's installed!
To add $HOME/.cargo/bin to your PATH, either restart your shell or run:
source $HOME/.cargo/env
root@edbb84aa61db:/# . ~/.cargo/env
root@edbb84aa61db:/# uv venv && . .venv/bin/activate
Using Python 3.8.19 interpreter at: usr/local/bin/python3
Creating virtualenv at: .venv
(.venv) root@edbb84aa61db:/# git clone --single-branch https://github.com/scikit-hep/pyhf.git && cd pyhf && git reset --hard 08755e45ed134cbe6e5e6e238b7b5011700080b3
(.venv) root@edbb84aa61db:/pyhf# uv pip --verbose --no-cache install . 'tensorflow>=2.7.0' 'tensorflow-probability>=0.11.0' &> /tmp/uv-pip-install-tensorflow-command-line-verbose.txt
(.venv) root@edbb84aa61db:/pyhf# uv pip list
Package Version
---------------------------- ------------
absl-py 2.1.0
astunparse 1.6.3
attrs 23.2.0
cachetools 5.3.3
certifi 2024.2.2
charset-normalizer 3.3.2
click 8.1.7
cloudpickle 3.0.0
decorator 5.1.1
dm-tree 0.1.8
flatbuffers 24.3.25
gast 0.4.0
google-auth 2.29.0
google-auth-oauthlib 1.0.0
google-pasta 0.2.0
grpcio 1.62.2
h5py 3.11.0
idna 3.7
importlib-metadata 7.1.0
importlib-resources 6.4.0
jsonpatch 1.33
jsonpointer 2.4
jsonschema 4.21.1
jsonschema-specifications 2023.12.1
keras 2.13.1
libclang 18.1.1
markdown 3.6
markupsafe 2.1.5
numpy 1.24.3
oauthlib 3.2.2
opt-einsum 3.3.0
packaging 24.0
pkgutil-resolve-name 1.3.10
protobuf 4.25.3
pyasn1 0.6.0
pyasn1-modules 0.4.0
pyhf 0.7.1.dev238
pyyaml 6.0.1
referencing 0.35.0
requests 2.31.0
requests-oauthlib 2.0.0
rpds-py 0.18.0
rsa 4.9
scipy 1.10.1
setuptools 69.5.1
six 1.16.0
tensorboard 2.13.0
tensorboard-data-server 0.7.2
tensorflow 2.13.1
tensorflow-estimator 2.13.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability 0.21.0
termcolor 2.4.0
tqdm 4.66.2
typing-extensions 4.5.0
urllib3 2.2.1
werkzeug 3.0.2
wheel 0.43.0
wrapt 1.16.0
zipp 3.18.1
(.venv) root@edbb84aa61db:/pyhf# uv pip list | grep tensorflow
tensorflow 2.13.1
tensorflow-estimator 2.13.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability 0.21.0
(.venv) root@edbb84aa61db:/pyhf# python -c 'import tensorflow; import tensorflow_probability'
2024-04-30 07:08:21.163407: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-30 07:08:21.164774: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-30 07:08:21.192250: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-30 07:08:21.192613: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-30 07:08:23.284477: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
(.venv) root@edbb84aa61db:/pyhf#uv-pip-install-tensorflow-command-line-verbose.txt pip (success case):$ docker run --rm -ti -v /tmp:/tmp python:3.8 /bin/bash
root@0c9ba727a892:/# python -m venv .venv && . .venv/bin/activate
(.venv) root@0c9ba727a892:/# python -m pip --quiet install --upgrade pip setuptools wheel
(.venv) root@0c9ba727a892:/# git clone --single-branch https://github.com/scikit-hep/pyhf.git && cd pyhf && git reset --hard 08755e45ed134cbe6e5e6e238b7b5011700080b3
(.venv) root@0c9ba727a892:/pyhf# python -m pip --verbose --no-cache-dir install --upgrade '.[tensorflow]' &> /tmp/python-pip-install-tensorflow-extra-verbose.txt
(.venv) root@0c9ba727a892:/pyhf# python -m pip list
Package Version
---------------------------- ------------
absl-py 2.1.0
astunparse 1.6.3
attrs 23.2.0
cachetools 5.3.3
certifi 2024.2.2
charset-normalizer 3.3.2
click 8.1.7
cloudpickle 3.0.0
decorator 5.1.1
dm-tree 0.1.8
flatbuffers 24.3.25
gast 0.4.0
google-auth 2.29.0
google-auth-oauthlib 1.0.0
google-pasta 0.2.0
grpcio 1.62.2
h5py 3.11.0
idna 3.7
importlib_metadata 7.1.0
importlib_resources 6.4.0
jsonpatch 1.33
jsonpointer 2.4
jsonschema 4.21.1
jsonschema-specifications 2023.12.1
keras 2.13.1
libclang 18.1.1
Markdown 3.6
MarkupSafe 2.1.5
numpy 1.24.3
oauthlib 3.2.2
opt-einsum 3.3.0
packaging 24.0
pip 24.0
pkgutil_resolve_name 1.3.10
protobuf 4.25.3
pyasn1 0.6.0
pyasn1_modules 0.4.0
pyhf 0.7.1.dev238
PyYAML 6.0.1
referencing 0.35.0
requests 2.31.0
requests-oauthlib 2.0.0
rpds-py 0.18.0
rsa 4.9
scipy 1.10.1
setuptools 69.5.1
six 1.16.0
tensorboard 2.13.0
tensorboard-data-server 0.7.2
tensorflow 2.13.1
tensorflow-estimator 2.13.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability 0.21.0
termcolor 2.4.0
tqdm 4.66.2
typing_extensions 4.5.0
urllib3 2.2.1
Werkzeug 3.0.2
wheel 0.43.0
wrapt 1.16.0
zipp 3.18.1
(.venv) root@0c9ba727a892:/pyhf# python -m pip list | grep tensorflow
tensorflow 2.13.1
tensorflow-estimator 2.13.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability 0.21.0
(.venv) root@0c9ba727a892:/pyhf# python -c 'import tensorflow; import tensorflow_probability'
2024-04-30 07:14:14.122645: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-30 07:14:14.124300: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-30 07:14:14.155920: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-30 07:14:14.156250: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-30 07:14:14.749372: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
(.venv) root@0c9ba727a892:/pyhf# |
|
My presumption is that this is a case of #8157 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi. 👋 While trying to adopt
uvfor use in thepyhfCI system (scikit-hep/pyhf#2444) we've found that we're getting different environment solves betweenpython -m pip installanduv pip install. The result ofuv pip installresults in an unusable environment oftensorflowv2.11.1andtensorflow-probabilityv0.20.1whilepython -m pip installresolves in the (in our view "correct") usable environment oftensorflowv2.13.1andtensorflow-probabilityv0.20.1.To avoid overwhelming wall of text I've broken the reproducible example in a Docker container into two sections that can be expanded below.
Here's the other requested info up front though:
python:3.8Docker container imageuv --version):uv 0.1.12Using
pipUsing pip:
python-pip-install-verbose.txt
and now in a virtual environment with
uv(version0.1.12)uv v0.1.12:
uv-pip-install-verbose.txt
The text was updated successfully, but these errors were encountered: