Pixi lock file is always out-of-date after introducing `pypi-dependencies` #1217

Hofer-Julian · 2024-04-18T06:58:58Z

Checks

I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

git clone -b pixi-install --single-branch https://github.com/Deltares/Ribasim.git
cd Ribasim
pixi install
pixi install --locked

The last command will fail with × lock-file not up-to-date with the project
I can reproduce this behavior on Linux and Windows machines.

Issue description

Something seems to be fishy with our pixi setup after @Jingru923 moved to using pypi-dependencies: Deltares/Ribasim#1400
Pixi always seem to consider the lock file out-of-date. I would love to make a smaller reproducer, but many other projects use pypi-dependencies just fine, and I don't see what makes the Ribasim setup special.

Expected behavior

Pixi should be able to update the lockfile in a way, that it afterwards is able to recognice that the lockfile is up-to-date.

The text was updated successfully, but these errors were encountered:

tdejager · 2024-04-18T08:00:21Z

So I found out why this is. Note if you run with -v, this gives you the reason why the dependencies are out of date, in this case because it did not expect ribasim to be editable, which at first glance seems strange, because it is defined as editable in the toml.

I think because this project is using a nested structure to include multiple projects this problem occurs.

The pixi.toml specifies ribasim as editable.
The pyproject.toml of ribasim_testmodels specifies ribasim as a regular pypi dependency.

Editables and requirements are kind of disjunct in pypi world. So this may cause issues. I'll need to investigate as bit further.

niemiaszek · 2024-04-18T23:51:39Z

I'm getting similar issue when trying to use my env:

[dependencies]
python = "~=3.11.0"
pandas = "*"
pyyaml = "*"
tqdm = "*"
ffmpeg = "*"
ruff = "*"
pre-commit = "*"
srt = "*"

[pypi-dependencies]
ffmpeg-python = {git= "https://github.com/charlienewey/ffmpeg-python.git", branch = "bugfix/782", rev="37dd98ca210fb8132575bedd719c6f8f9bfd1021"}
tensorflow ="~=2.15.0"
tensorflow-addons = "*"
tensorflow-io = "*"
tensorflow-datasets = "*"
keras-cv = "*"
pytest = "*"
pytest-cov = "*"
clearml = "*"

[feature.train.system-requirements]
cuda = "12.2"

[feature.train.tasks]
setup_precommit = "pre-commit install"
setup_jupyter = "python -m ipykernel install --user --name 'speaker-recognition-pixi'"
test_gpu = {cmd = ["python3", "-c", "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'));"], depends_on=["setup_precommit", "setup_jupyter"]}

[feature.train.dependencies]
ipykernel = "*"
matplotlib = "*"
pandas = "*"
umap-learn = "*"
bokeh = "*"
pre-commit = "*"
dask = ">=2024.4.1,<2024.5"
ffmpeg = "*"

[feature.train.pypi-dependencies]
tensorflow = {version="~=2.15.0", extras=["and-cuda"]}
ray = {version = "*", extras=["data","train","tune","serve"]}
audiomentations = "*"
joblib = "~=1.4.0"

This basically ends changing lockfile every time when I use it. It can also be a bit inconvenient when sharing .lock in git repository. I think that using --frozen option would be a way to go, but still this is a bit concerning.

INFO pixi::lock_file::outdated: the pypi dependencies of environment 'train' for platform linux-64 are out of date because there are more pypi packages in the lock-file than are used by the environment: cachetools, flatbuffers, audioread, nvidia-cuda-cupti-cu12, nvidia-cuda-nvrtc-cu12, coverage, markdown, yarl, googleapis-common-protos, requests-oauthlib, astunparse, namex, clearml, opencensus-context, google-api-core

I guess tensorflow is not doing easy work for this env

ruben-arts · 2024-04-19T06:44:01Z

@niemiaszek Thanks for you example. We've been looking for a smaller example this helps! If you find any smaller reproducible examples please let us know!

niemiaszek · 2024-04-19T09:25:53Z

Sure, I will try to come with some examples. As I started to use Pixi daily, I will have more chance to experiment with "minimal not working examples". I will give it a chance over the weekend

baszalmstra · 2024-04-22T15:38:34Z

@Hofer-Julian I can confirm that your issue is fixed by #1251

@niemiaszek I am pretty sure your issue is solved by #1253 (not entirely sure because your example doesnt include a lock).

akhmerov · 2024-07-04T16:07:12Z

I confirm that the lockfile is declared as up to date, but an issue still remains. Specifically, it seems to always get overwritten with no changes despite pixi declaring that it is up to date. This, for example, makes direnv trigger on every action.

tdejager · 2024-07-04T16:32:52Z

Can you provide the pixi.toml?

akhmerov · 2024-07-04T16:54:23Z

My OS is Ubuntu 24.04 if that's important. I have the latest pixi (0.24.2). The toml is what I got after

pixi init
pixi add python
pixi add "black @ https://github.com/psf/black.git@24.4.2

[project]
name = "pixi_test"
version = "0.1.0"
description = "Add a short description here"
authors = ["Anton Akhmerov <anton.akhmerov@gmail.com>"]
channels = ["conda-forge"]
platforms = ["linux-64"]

[tasks]

[dependencies]
python = ">=3.12.4,<3.13"

[pypi-dependencies]
black = { git = "https://github.com/psf/black.git", rev = "24.4.2" }

I added .envrc as recommended by the docs and enabled direnv in fish shell.

watch_file pixi.lock 
eval "$(pixi shell-hook)"

I then observe that direnv runs pixi initialization on every action in the terminal.

pavelzw · 2024-07-04T17:01:13Z

You are using sdists instead of wheels. I experienced something similar in #1046

tdejager · 2024-07-05T07:49:49Z

My OS is Ubuntu 24.04 if that's important. I have the latest pixi (0.24.2). The toml is what I got after
pixi init
pixi add python
pixi add "black @ https://github.com/psf/black.git@24.4.2
[project]
name = "pixi_test"
version = "0.1.0"
description = "Add a short description here"
authors = ["Anton Akhmerov <anton.akhmerov@gmail.com>"]
channels = ["conda-forge"]
platforms = ["linux-64"]

[tasks]

[dependencies]
python = ">=3.12.4,<3.13"

[pypi-dependencies]
black = { git = "https://github.com/psf/black.git", rev = "24.4.2" }
I added .envrc as recommended by the docs and enabled direnv in fish shell.
watch_file pixi.lock 
eval "$(pixi shell-hook)"
I then observe that direnv runs pixi initialization on every action in the terminal.

Actually, that does look like a 🐛!

 INFO pixi::environment: verifying prefix location is unchanged, with prefix file: /private/tmp/black/.pixi/envs/default/conda-meta/pixi_env_prefix
 INFO pixi::lock_file::outdated: the pypi dependencies of environment 'default' for platform osx-arm64 are out of date because the requirement 'black @ git+https://github.com/psf/black.git@24.4.2' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::resolve::uv_resolution_context: uv keyring provider is disabled
 INFO pixi::environment: Creating prefix file at: /private/tmp/black/.pixi/envs/default/conda-meta/pixi_env_prefix
 INFO pixi::environment: No update needed for the prefix file.
 INFO pixi::environment: Checking if history file exists: /private/tmp/black/.pixi/envs/default/conda-meta/history
 INFO pixi::lock_file::update: updated conda packages in the 'default' prefix in 30ms 992us 792ns
 INFO resolve_pypi{group=default platform=osx-arm64}: pixi::lock_file::resolve::pypi: there are no python packages installed by conda
 INFO solve:solve_tracked: pubgrub::internal::partial_solution: add_decision: root @ 0a0.dev0    
 INFO solve:solve_tracked: pubgrub::internal::partial_solution: add_decision: black @ 24.4.2    
 INFO solve:solve_tracked: pubgrub::internal::partial_solution: add_decision: click @ 8.1.7    
 INFO solve:solve_tracked: pubgrub::internal::partial_solution: add_decision: mypy-extensions @ 1.0.0    
 INFO solve:solve_tracked: pubgrub::internal::partial_solution: add_decision: packaging @ 24.1    
 INFO solve:solve_tracked: pubgrub::internal::partial_solution: add_decision: pathspec @ 0.12.1    
 INFO solve:solve_tracked: pubgrub::internal::partial_solution: add_decision: platformdirs @ 4.2.2    
 INFO pixi::lock_file::update: resolved pypi packages for environment 'default' 'osx-arm64' in 741ms 828us 709ns
 INFO pixi::install_pypi: Nothing to do - Audited 6 distributions in 34ms
✔ The default environment has been installed.

This environment is considered unsatified while it shouldnt be.

tdejager · 2024-07-08T09:21:55Z

Ok @akhmerov I've just noticed, I think you are trying to use a tag as a revision, we don't support tags (yet). I'm thinking that's the problem.

akhmerov · 2024-07-08T12:10:38Z

Thank you for the explanation. I confirm that switching to a revision instead of a tag works.

niemiaszek · 2024-07-08T13:24:58Z

It seems there is WIP PR #1245 . I was also confused when I used branch option and in fact it didn't use desired branch and caused similar issues. My issue #1206

luizirber · 2024-08-21T02:53:45Z

I'm also seeing this one in https://github.com/sourmash-bio/sourmash

To reproduce on a specific commit with the issue (current latest):

$ git clone https://github.com/sourmash-bio/sourmash && cd sourmash && git checkout d3ae5daffb66a794d859e59f774a54383bceb0ac

Edit .envrc to add -v in the last line:

eval "$(pixi -v shell-hook)"

And allow direnv to run:

direnv run

$ direnv allow
direnv: loading ~/work/sourmash/latest/.envrc                                                                                                                             
 INFO pixi::project::manifest::pyproject: Overriding the requires-python with the one defined in pixi dependencies
 WARN pixi::project::manifest::validation: The feature 'test' is defined but not used in any environment
 WARN pixi::project::manifest::validation: The feature 'demo' is defined but not used in any environment
 WARN pixi::project::manifest::validation: The feature 'doc' is defined but not used in any environment
 WARN pixi::project::manifest::validation: The feature 'storage' is defined but not used in any environment
 WARN pixi::project::manifest::validation: The feature 'all' is defined but not used in any environment
 INFO pixi::environment: verifying prefix location is unchanged, with prefix file: /home/luizirber/work/sourmash/latest/.pixi/envs/default/conda-meta/pixi_env_prefix
 INFO pixi::lock_file::outdated: environment 'default' is out of date because the indexes used to previously solve to lock file are missing
 INFO pixi::lock_file::resolve::uv_resolution_context: uv keyring provider is disabled
 INFO resolve_conda{group=default platform=osx-arm64}: pixi::lock_file::update: fetched 7146 records in 2.665596967s
 INFO resolve_conda{group=default platform=linux-aarch64}: pixi::lock_file::update: fetched 14316 records in 3.826917714s
⠓ updating lock-file   [00:00:04] [────────────────────────────────────────]    0/16                                                                                      
  ⠄ default:linux-64     [00:00:04] loading repodata
 INFO resolve_conda{group=default platform=win-64}: pixi::lock_file::update: fetched 14870 records in 5.283371607s
 INFO resolve_conda{group=default platform=osx-64}: pixi::lock_file::update: fetched 15487 records in 6.65827434s
 INFO pixi::lock_file::update: resolved conda environment for environment 'default' 'osx-arm64' in 7s 347ms 197us 493ns
 INFO pixi::lock_file::update: resolved conda environment for environment 'default' 'linux-aarch64' in 7s 354ms 901us 180ns
 INFO resolve_conda{group=default platform=linux-64}: pixi::lock_file::update: fetched 20834 records in 8.427731937s
 INFO pixi::lock_file::update: resolved conda environment for environment 'default' 'win-64' in 8s 435ms 247us 579ns
 INFO pixi::lock_file::update: resolved conda environment for environment 'default' 'osx-64' in 8s 937ms 141us 626ns
 INFO pixi::lock_file::update: resolved conda environment for environment 'default' 'linux-64' in 9s 317ms 445us 952ns
 INFO pixi::environment: Creating prefix file at: /home/luizirber/work/sourmash/latest/.pixi/envs/default/conda-meta/pixi_env_prefix
 INFO pixi::environment: No update needed for the prefix file.
 INFO pixi::environment: Checking if history file exists: /home/luizirber/work/sourmash/latest/.pixi/envs/default/conda-meta/history
 INFO pixi::lock_file::update: updated conda packages in the 'default' prefix in 169ms 859us 452ns
 INFO resolve_pypi{group=default platform=osx-64}: pixi::lock_file::resolve::pypi: the following python packages are assumed to be installed by conda: pyparsing 3.1.2, cffi 1.16.0, tornado 6.4.1, bitstring 4.2.3, pillow 10.4.0, deprecation 2.1.0, python-dateutil 2.9.0, numpy 2.0.1, munkres 1.1.4, filelock 3.15.4, fonttools 4.53.1, cycler 0.12.1, unicodedata2 15.1.0, pycparser 2.22, chardet 5.2.0, tox 4.16.0, colorama 0.4.6, tomli 2.0.1, kiwisolver 1.4.5, contourpy 1.2.1, pyproject-api 1.7.1, distlib 0.3.8, certifi 2024.7.4, packaging 24.1, screed 1.1.3, virtualenv 20.26.3, setuptools 71.0.4, cachetools 5.4.0, pluggy 1.5.0, six 1.16.0, scipy 1.14.0, matplotlib 3.9.1, platformdirs 4.2.2, bitarray 2.9.2
 INFO resolve_pypi{group=default platform=osx-arm64}: pixi::lock_file::resolve::pypi: the following python packages are assumed to be installed by conda: packaging 24.1, colorama 0.4.6, pycparser 2.22, deprecation 2.1.0, tornado 6.4.1, cffi 1.16.0, kiwisolver 1.4.5, python-dateutil 2.9.0, setuptools 71.0.4, contourpy 1.2.1, platformdirs 4.2.2, virtualenv 20.26.3, pyparsing 3.1.2, munkres 1.1.4, scipy 1.14.0, six 1.16.0, unicodedata2 15.1.0, matplotlib 3.9.1, chardet 5.2.0, tox 4.16.0, pillow 10.4.0, bitstring 4.2.3, cycler 0.12.1, numpy 2.0.1, fonttools 4.53.1, bitarray 2.9.2, pluggy 1.5.0, certifi 2024.7.4, filelock 3.15.4, tomli 2.0.1, distlib 0.3.8, pyproject-api 1.7.1, screed 1.1.3, cachetools 5.4.0
 INFO resolve_pypi{group=default platform=linux-aarch64}: pixi::lock_file::resolve::pypi: the following python packages are assumed to be installed by conda: deprecation 2.1.0, pillow 10.4.0, packaging 24.1, unicodedata2 15.1.0, tornado 6.4.1, screed 1.1.3, bitstring 4.2.3, certifi 2024.7.4, setuptools 71.0.4, filelock 3.15.4, munkres 1.1.4, chardet 5.2.0, bitarray 2.9.2, contourpy 1.2.1, tomli 2.0.1, colorama 0.4.6, pycparser 2.22, pyparsing 3.1.2, cffi 1.16.0, scipy 1.14.0, matplotlib 3.9.1, virtualenv 20.26.3, distlib 0.3.8, platformdirs 4.2.2, pyproject-api 1.7.1, cycler 0.12.1, six 1.16.0, pluggy 1.5.0, fonttools 4.53.1, numpy 2.0.1, python-dateutil 2.9.0, cachetools 5.4.0, kiwisolver 1.4.5, tox 4.16.0
 INFO resolve_pypi{group=default platform=win-64}: pixi::lock_file::resolve::pypi: the following python packages are assumed to be installed by conda: bitstring 4.2.3, pyqt5-sip 12.12.2, scipy 1.14.0, deprecation 2.1.0, cachetools 5.4.0, bitarray 2.9.2, pluggy 1.5.0, toml 0.10.2, contourpy 1.2.1, fonttools 4.53.1, pycparser 2.22, colorama 0.4.6, pyqt5 5.15.9, filelock 3.15.4, screed 1.1.3, matplotlib 3.9.1, virtualenv 20.26.3, six 1.16.0, munkres 1.1.4, sip 6.7.12, chardet 5.2.0, platformdirs 4.2.2, packaging 24.1, cffi 1.16.0, cycler 0.12.1, ply 3.11, pillow 10.4.0, tox 4.16.0, tornado 6.4.1, kiwisolver 1.4.5, pyproject-api 1.7.1, distlib 0.3.8, tomli 2.0.1, setuptools 71.0.4, numpy 2.0.1, certifi 2024.7.4, pyparsing 3.1.2, unicodedata2 15.1.0, python-dateutil 2.9.0
 INFO pixi::lock_file::update: resolved pypi packages for environment 'default' 'osx-64' in 91ms 967us 872ns
 INFO pixi::lock_file::update: resolved pypi packages for environment 'default' 'osx-arm64' in 60ms 402us 400ns
 INFO resolve_pypi{group=default platform=linux-64}: pixi::lock_file::resolve::pypi: the following python packages are assumed to be installed by conda: cachetools 5.4.0, contourpy 1.2.1, pyparsing 3.1.2, deprecation 2.1.0, distlib 0.3.8, filelock 3.15.4, sip 6.7.12, pyqt5 5.15.9, bitarray 2.9.2, six 1.16.0, cffi 1.16.0, python-dateutil 2.9.0, setuptools 71.0.4, virtualenv 20.26.3, packaging 24.1, tornado 6.4.1, platformdirs 4.2.2, colorama 0.4.6, ply 3.11, pycparser 2.22, toml 0.10.2, pyqt5-sip 12.12.2, scipy 1.14.0, fonttools 4.53.1, munkres 1.1.4, kiwisolver 1.4.5, certifi 2024.7.4, matplotlib 3.9.1, screed 1.1.3, pluggy 1.5.0, tomli 2.0.1, tox 4.16.0, pillow 10.0.1, chardet 5.2.0, numpy 2.0.1, unicodedata2 15.1.0, bitstring 4.2.3, pyproject-api 1.7.1, cycler 0.12.1
 INFO pixi::lock_file::update: resolved pypi packages for environment 'default' 'linux-aarch64' in 71ms 846us 103ns
 INFO pixi::lock_file::update: resolved pypi packages for environment 'default' 'linux-64' in 7ms 536us 725ns
 INFO pixi::lock_file::update: resolved pypi packages for environment 'default' 'win-64' in 45ms 16us 535ns
direnv: export +ADDR2LINE +AR +AS +BUILD +CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER +CC +CC_FOR_BUILD +CFLAGS +CMAKE_ARGS +CMAKE_PREFIX_PATH +CONDA_BUILD_SYSROOT +CONDA_DEFAULT_ENV +CONDA_PREFIX +CONDA_TOOLCHAIN_BUILD +CONDA_TOOLCHAIN_HOST +CPP +CPPFLAGS +CXX +CXXFILT +CXXFLAGS +CXX_FOR_BUILD +DEBUG_CFLAGS +DEBUG_CPPFLAGS +DEBUG_CXXFLAGS +DWP +ELFEDIT +GCC +GCC_AR +GCC_NM +GCC_RANLIB +GPROF +GSETTINGS_SCHEMA_DIR +GSETTINGS_SCHEMA_DIR_CONDA_BACKUP +GXX +HOST +LD +LDFLAGS +LD_GOLD +MESON_ARGS +NM +OBJCOPY +OBJDUMP +PIXI_ENVIRONMENT_NAME +PIXI_ENVIRONMENT_PLATFORMS +PIXI_EXE +PIXI_IN_SHELL +PIXI_PROJECT_MANIFEST +PIXI_PROJECT_NAME +PIXI_PROJECT_ROOT +PIXI_PROJECT_VERSION +PIXI_PROMPT +RANLIB +READELF +SIZE +STRINGS +STRIP +XML_CATALOG_FILES +_CONDA_PYTHON_SYSCONFIGDATA_NAME +build_alias +host_alias ~PATH

What caught my attention was the line:

 INFO pixi::lock_file::outdated: environment 'default' is out of date because the indexes used to previously solve to lock file are missing

what is the index missing?

pixi info:

$ pixi info
      Pixi version: 0.26.1
          Platform: linux-64
  Virtual packages: __unix=0=0
                  : __linux=6.8.12=0
                  : __glibc=2.39=0
                  : __archspec=1=skylake
         Cache dir: /home/luizirber/.cache/rattler/cache
      Auth storage: /home/luizirber/.rattler/credentials.json

ruben-arts · 2024-08-21T06:35:49Z

@luizirber You've got an interesting situation. You perfectly mimicked the [project.dependencies] with the [tool.pixi.dependencies] so our logic is testing the pypi logic without having to install anything. This seems like a bug, will look into it some more.

ruben-arts · 2024-08-21T09:09:10Z

@luizirber That will be fixed in: #1862

baszalmstra · 2024-08-22T15:03:52Z

Fixed with #1862

Hofer-Julian added the 🐞 bug Something isn't working label Apr 18, 2024

ruben-arts mentioned this issue Aug 21, 2024

fix: satisfy when there is not pypi packages in the lockfile #1862

Merged

baszalmstra closed this as completed Aug 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pixi lock file is always out-of-date after introducing `pypi-dependencies` #1217

Pixi lock file is always out-of-date after introducing `pypi-dependencies` #1217

Hofer-Julian commented Apr 18, 2024

tdejager commented Apr 18, 2024

niemiaszek commented Apr 18, 2024

ruben-arts commented Apr 19, 2024

niemiaszek commented Apr 19, 2024

baszalmstra commented Apr 22, 2024

akhmerov commented Jul 4, 2024

tdejager commented Jul 4, 2024

akhmerov commented Jul 4, 2024

pavelzw commented Jul 4, 2024

tdejager commented Jul 5, 2024 •

edited

Loading

tdejager commented Jul 8, 2024

akhmerov commented Jul 8, 2024

niemiaszek commented Jul 8, 2024

luizirber commented Aug 21, 2024 •

edited

Loading

ruben-arts commented Aug 21, 2024

ruben-arts commented Aug 21, 2024

baszalmstra commented Aug 22, 2024

Pixi lock file is always out-of-date after introducing pypi-dependencies #1217

Pixi lock file is always out-of-date after introducing pypi-dependencies #1217

Comments

Hofer-Julian commented Apr 18, 2024

Checks

Reproducible example

Issue description

Expected behavior

tdejager commented Apr 18, 2024

niemiaszek commented Apr 18, 2024

ruben-arts commented Apr 19, 2024

niemiaszek commented Apr 19, 2024

baszalmstra commented Apr 22, 2024

akhmerov commented Jul 4, 2024

tdejager commented Jul 4, 2024

akhmerov commented Jul 4, 2024

pavelzw commented Jul 4, 2024

tdejager commented Jul 5, 2024 • edited Loading

tdejager commented Jul 8, 2024

akhmerov commented Jul 8, 2024

niemiaszek commented Jul 8, 2024

luizirber commented Aug 21, 2024 • edited Loading

ruben-arts commented Aug 21, 2024

ruben-arts commented Aug 21, 2024

baszalmstra commented Aug 22, 2024

Pixi lock file is always out-of-date after introducing `pypi-dependencies` #1217

Pixi lock file is always out-of-date after introducing `pypi-dependencies` #1217

tdejager commented Jul 5, 2024 •

edited

Loading

luizirber commented Aug 21, 2024 •

edited

Loading