Tracking issue: PyPI dependencies pixi can't manage yet. #771
Labels
🐍 pypi
Issue related to PyPI dependencies
Comments
|
Mediapipe
|
|
I wanted to use this library that is packaged in pypi: https://github.com/spirali/elsie Here is the error I was seeing when trying to use pixi add: Posting in the discord channel got it working with this fix: The reason for this is:
|
|
@pablovela5620 I am also finding this necessary quite often, probably more often than @ruben-arts would like :) In this pattern the environment definition is fragmented between |
@liquidcarbon This is indeed not what we want as UX but we simply need to develop more to support all PyPI packages. Its a weird bunch of requirements we have to support to be equivalent to |
|
@liquidcarbon the |
|
@liquidcarbon I cannot reproduce this, neither on OSX or on windows. It's strange it does not select the |
|
@tylerjw actually it also fails for the same version in pip on apple silicon, like pip we error out when we cannot build the first source distribution. |
|
@pablovela5620 so it seems mediapipe 10.9 is a package with an invalid RECORD file, I manually checked it and it's incorrect. This is mentioned in the PyPa
Mediapipe has a Which in this case triggers the error, I'm unsure why pip does no do this. But I feel its good to adhere to the standard here. In any case, mediapipe 10.8 does seem to work, you could use that instead. Also see: google-ai-edge/mediapipe#5025 |
tdejager
pushed a commit
to prefix-dev/rip
that referenced
this issue
Feb 15, 2024
These PR address 2 major issues and 1 small issue * zip file stamps issue alexcrichton/tar-rs#349 which we encountered when installing tomli-2.1.0 * changed from_filename signature so it can also produce Stree, which it's needed from pixi side * lowercase Root-Is-Purelib so we can install elsie library ( prefix-dev/pixi#771 (comment))
@tdejager Just tried on Win10, original comment was on Win11 PS C:\Users\a\Desktop\code\duckdb-pip> pixi add python=3.11 I was on Pixi 0.9.1, upped to 0.13.0 - same thing But |
|
The conda package you add with https://prefix.dev/channels/conda-forge/packages/python-duckdb |
|
@ruben-arts noted -- I just wasn't sure which part @tdejager was trying to reproduce |
|
Tensorflow metal on Apple silicon MacOS 14.2.1: pixi init tf-metal && cd tf-metal
pixi add "python>=3.11" "tensorflow>=2.13" pip
pixi add --pypi tensorflow-metal
> × failed to resolve `pypi-dependencies`, due to underlying error
╰─▶ The following packages are incompatible
└─ tensorflow-metal * cannot be installed because there are no viable options:
└─ tensorflow-metal 0.1.0 | 0.1.1 | 0.1.2 | 0.2.0 | 0.3.0 | 0.4.0 | 0.5.0 | 0.5.1 | 0.6.0 | 0.7.0 | 0.7.1
| 0.8.0 | 1.0.0 | 1.0.1 | 1.1.0 is excluded because none of the artifacts are compatible with the Python
interpreter or glibc version and there are no supported sdists
pixi run pip install tensorflow-metal
> Successfully installed tensorflow-metal-1.1.0 |
|
Most likely you are missing a system requirement: https://pixi.sh/latest/configuration/#the-system-requirements-table Most likely macos=12.0 |
|
Nice, that fixed it. EDIT: Nevermind, I found the example demonstrating how to do this. Thank you! |
|
Installing (base) ➜ idtracker pixi add --pypi opencv-python-headless
× failed to resolve `pypi-dependencies`, due to underlying error
╰─▶ No metadata could be extracted for the following available artifacts:
- opencv-python-headless-4.9.0.80.tar.gz
Error: × error while processing source distribution 'opencv-python-headless-4.9.0.80.tar.gz':
│ could not build wheel: Traceback (most recent call last):
│ File "/var/folders/rh/y9ws4bgj7x50twypcwtk2ypr0000gp/T/.tmpvjLm8l/build_frontend.py", line 124, in <module>
│ get_requires_for_build_wheel(backend, work_dir)
│ File "/var/folders/rh/y9ws4bgj7x50twypcwtk2ypr0000gp/T/.tmpvjLm8l/build_frontend.py", line 58, in get_requires_for_build_wheel
│ result = f()
│ ^^^
│ File "/var/folders/rh/y9ws4bgj7x50twypcwtk2ypr0000gp/T/.tmpvjLm8l/venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in
│ get_requires_for_build_wheel
│ return self._get_build_requires(config_settings, requirements=['wheel'])
│ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
│ File "/var/folders/rh/y9ws4bgj7x50twypcwtk2ypr0000gp/T/.tmpvjLm8l/venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
│ self.run_setup()
│ File "/var/folders/rh/y9ws4bgj7x50twypcwtk2ypr0000gp/T/.tmpvjLm8l/venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 487, in run_setup
│ super().run_setup(setup_script=setup_script)
│ File "/var/folders/rh/y9ws4bgj7x50twypcwtk2ypr0000gp/T/.tmpvjLm8l/venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
│ exec(code, locals())
│ File "<string>", line 10, in <module>
│ ModuleNotFoundError: No module named 'skbuild'
│
help: Probably an error during processing of source distributions. Please check the error message above. |
|
@roaldarbol The error message is absolutely terrible but if you add: [system-requirements]
macos = "11.0"It should work. |
|
It does indeed! Thanks! |
|
Trying to install
|
It does indeed not work. Testing the repro gives me this error: In both the Thanks for the info @jacobbieker |
Would be kind-of useful if we can keep the build environments for uv as well, so it's easier to debug these things. |
pixi add --pypi -vv "trimesh[all]"
Log
Yes.
The above log shows that the problem seems to be with |
This happens on the osx-64 and osx-arm64 platforms (it works on win-64 and linux-64). I expect this to be some mismatching in the name/metadata of/in the .whl files for the macos specific ones.
Not yet, it would help if the resolver can be more verbose ( Encountered in Deltares/Ribasim#1137 |
|
@evetion that error often means that the Unfortunately, setting that in my |
|
Sorry, my bad. We should make the error more actionable and figure out a better default behavior. |
|
@liblaf I wonder if for you it's a similar issue and adding glibc 2.28 or higher would fix it. e.g. |
|
|
|
|
|
@tdejager Unfortunately it won't work. pyproject.toml[build-system]
build-backend = "hatchling.build"
requires = ["hatchling"]
[project]
authors = [
{ email = "30631553+liblaf@users.noreply.github.com", name = "liblaf" },
]
dependencies = []
description = "Add a short description here"
name = "hello"
requires-python = ">= 3.11"
version = "0.1.0"
[tool.pixi.project]
channels = ["conda-forge"]
platforms = ["linux-64"]
[tool.pixi.pypi-dependencies]
hello = { editable = true, path = "." }
[tool.pixi.system-requirements]
libc = { family = "glibc", version = "2.35" }
linux = "6.5"
[tool.pixi.tasks]pixi add --pypi gmsh |
|
@tdejager I wrote a simple script to test which PyPI packages cannot be managed by pixi. I have only tested a few packages I use. Here is the results from workflow run: Results
I also noticed that pixi tends to build from source distributions rather than using built distributions. If the PyPI package does not provide source distributions, it cannot be installed (e.g. gmsh, jaxlib). I guess this is because pixi fails to resolve built distributions correctly. |
nvidia-cuda-nvrtc-cu11==11.8.89 works with glibc 2.35 with pip and uv |
2 tasks
|
Not sure if this is the right issue, but in a git checkout, while #subdirectory is parsed, it can't be cloned: It seems the url fragment should be stripped when cloning. Also note that the url fragment is duplicated on the line starting with |
|
I had issues with two In both cases, on pixi add equinoxerrors with: Fixed it with Project config: [project]
authors = [{name = "...", email = "..."}]
dependencies = []
description = "Add a short description here"
name = "lru"
requires-python = ">= 3.12"
version = "0.1.0"
[tool.pixi.project]
channels = ["conda-forge"]
platforms = ["osx-arm64"]
[tool.pixi.pypi-dependencies]
lru = { path = ".", editable = true }
[tool.pixi.dependencies]
jax = ">=0.4.30"
pytorch = ">=2.4.0"
ruff = "*"
ipython = "*"
# Feature Definitions
[tool.pixi.feature.cuda]
platforms = ["linux-64"]
system-requirements = {cuda = "12"}
[tool.pixi.feature.cuda.dependencies]
jaxlib = { version = "*", build = "*cuda12*" }
# Environments
[tool.pixi.environments]
cuda = ["cuda"] |
|
@floringogianu I'm unable to reproduce your issue. Is it a recurring issue? Or did you only see it one time? |
|
@ruben-arts got around to test this again and now I'm not seeing this error anymore 🤷. |
|
On @ruben-arts do you prefer this as a new issue or here? Thanks! fyi @xela-95 |
@ruben-arts no hurry on our side, but just to understand, do you prefer this bug report as a comment here or on another issue, thanks! |
|
@traversaro That problem has to do with the system requirements. I really want to get rid of this issue through #346 Adding the following to the Edit: sometimes it helps to look at https://pypi.org/simple/drake/ available wheels. |
|
Thanks @ruben-arts, this fixed our problem! Cross-linking #346 that seems to be the actual root issue here. |
|
Hello, uploading my Error message❯ pixi install
WARN The feature 'notebooks' is defined but not used in any environment
WARN The feature 'matrepr' is defined but not used in any environment
× failed to solve the pypi requirements of 'mlir-dev' 'linux-64'
├─▶ failed to resolve pypi dependencies
╰─▶ Because only finch-mlir<=0.0.2 is available and finch-mlir==0.0.2 has no wheels with a matching Python implementation tag, we can
conclude that finch-mlir>=0.0.2 cannot be used.
And because you require finch-mlir>=0.0.2, we can conclude that your requirements are unsatisfiable.pixi.toml[project]
authors = ["Hameer Abbasi <2190658+hameerabbasi@users.noreply.github.com>"]
channels = ["conda-forge"]
name = "sparse"
platforms = ["osx-arm64", "osx-64", "linux-64", "win-64"]
[pypi-dependencies]
sparse = { path = ".", editable = true }
[system-requirements]
libc = "2.35"
[dependencies]
python = ">=3.10"
numpy = ">=1.17"
numba = ">=0.49"
[feature.extras.dependencies]
dask = ">=2024"
scipy = ">=0.19"
scikit-learn = "*"
[feature.docs.pypi-dependencies]
mkdocs-material = "*"
mkdocstrings = { version = "*", extras = ["python"] }
mkdocs-gen-files = "*"
mkdocs-literate-nav = "*"
mkdocs-section-index = "*"
mkdocs-jupyter = "*"
[feature.tests.tasks]
test = "pytest --pyargs sparse -n auto"
test-mlir = { cmd = "pytest --pyargs sparse.mlir_backend -v" }
test-finch = { cmd = "pytest --pyargs sparse/tests -n auto -v", depends-on = ["precompile"] }
[feature.tests.dependencies]
pytest = ">=3.5"
pytest-cov = "*"
pytest-xdist = "*"
pre-commit = "*"
pytest-codspeed = "*"
[feature.notebooks.dependencies]
nbmake = "*"
matplotlib = "*"
[feature.matrepr.dependencies]
matrepr = "*"
[feature.finch.tasks]
precompile = "python -c 'import finch'"
[feature.finch.pypi-dependencies]
scipy = ">=0.19"
finch-tensor = ">=0.1.31"
[feature.finch.activation.env]
SPARSE_BACKEND = "Finch"
[feature.finch.target.osx-arm64.activation.env]
SPARSE_BACKEND = "Finch"
PYTHONFAULTHANDLER = "${HOME}/faulthandler.log"
[feature.mlir.dependencies]
scipy = ">=0.19"
[feature.mlir.pypi-dependencies]
finch-mlir = ">=0.0.2"
[feature.mlir.activation.env]
SPARSE_BACKEND = "MLIR"
[environments]
tests = ["tests", "extras"]
docs = ["docs", "extras"]
mlir-dev = {features = ["tests", "mlir"], no-default-feature = true}
finch-dev = {features = ["tests", "finch"], no-default-feature = true}To reproduce simply run |
|
The pypi dependency |
|
Hey @enrico5,
For my machine that looks like this: ❯ pixi info | grep glibc
: __glibc=2.35=0Adding the given version to the [tool.pixi.system-requirements]
libc = "2.35" |
|
That works great. Thank you! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
There are a few PyPI packages
pixican't install yet, wherepipcan.Please paste your examples in this issue so we have a list of known packages we can track, test and benchmark with along the way.
Information we would like:
What did you run and what was the outcome?
e.g.
pixi add --pypi packagexIf it doesn't recreate in an empty environment please share your
pixi.tomlthat recreates the issue.What error did
pixireturn?e.g.
Can pip install the package?
Does
pip install packagexwork?What platform are you on?
e.g.
linux-64Did you find a workaround, if so please explain.
e.g. build it into a conda package, using a custom fork, etc.
Your input would greatly help us improve the pixi's experience!
Thanks in advance! ❤️
The text was updated successfully, but these errors were encountered: