Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pixi as an alternative to conda-lock #571

Open
scottyhq opened this issue Aug 6, 2024 · 3 comments
Open

Pixi as an alternative to conda-lock #571

scottyhq opened this issue Aug 6, 2024 · 3 comments
Labels
enhancement New feature or request

Comments

@scottyhq
Copy link
Member

scottyhq commented Aug 6, 2024

Is your feature request related to a problem? Please describe.
conda-lock has worked really well for this project! Looking forward it might be worth also supporting or switching to pixi for environment management https://pixi.sh

Describe the solution you'd like
Try using pixi instead of conda-lock and supporting pixi.lock as another file that can be installed

&& mamba install conda-lock -y \

; if test -f "conda-lock.yml" ; then echo "Using conda-lock.yml" & \
conda-lock install --name ${CONDA_ENV} \
; elif test -f "environment.yml" ; then echo "Using environment.yml" & \
mamba env create --name ${CONDA_ENV} -f environment.yml \
; else echo "No conda-lock.yml or environment.yml! *creating default env*" ; \
mamba create --name ${CONDA_ENV} pangeo-notebook \

Describe alternatives you've considered
Stick with what's been working fine

Additional context
Discussion of pixi on conda-lock repo conda/conda-lock#615

Also https://github.com/prefix-dev/pixi-docker

@scottyhq scottyhq added the enhancement New feature or request label Aug 6, 2024
@weiji14
Copy link
Member

weiji14 commented Aug 6, 2024

Been thinking about this too. I'd seriously consider making the jump once repo2docker supports pixi lock files (jupyterhub/repo2docker#1339), though we can test things out using the Dockerfile path too. We'll need to document this change though as some people might be depending on the conda-lock files and changing to pixi.lock will break their workflows.

@weiji14
Copy link
Member

weiji14 commented Nov 28, 2024

Was wondering how fast pixi would generate a lock file, so I tried out the pre-v3 release of conda-lock that allows generating a pixi.toml file from a conda environment.yml file (added in conda/conda-lock#664). Following the draft docs at conda/conda-lock#708, install pixi and conda-lock from main branch and then run:

cd pangeo-notebook/
conda-lock render-lock-spec --kind=pixi.toml  --stdout --file environment.yml --file ../base-notebook/environment.yml --pixi-project-name=pangeo-notebook --platform linux-64 --platform linux-aarch64 --platform osx-64 --platform osx-arm64 > pixi.toml

outputs the following pixi.toml file (this is from the branch at #592):

# This file was generated by conda-lock for the pixi environment manager.
# For more information, see <https://github.com/conda/conda-lock> and <https://pixi.sh>.
# Source files:
# - environment.yml
# - ../base-notebook/environment.yml

[project]
name = "pangeo-notebook"
platforms = ["linux-64", "linux-aarch64", "osx-64", "osx-arm64"]
channels = ["conda-forge"]

[dependencies]
adlfs = "*"
argopy = "<0.1.16"
awscli = "*"
black = "*"
boto3 = "*"
bottleneck = "*"
cartopy = "*"
cdsapi = "*"
cf_xarray = "*"
cfgrib = "*"
ciso = "*"
cmocean = "*"
contextily = "*"
dask-geopandas = "*"
dask-ml = "*"
datashader = "*"
descartes = "*"
duckdb-cli = "*"
earthaccess = "*"
eofs = "*"
erddapy = "*"
esmpy = "*"
fastjmd95 = "*"
flox = "*"
fsspec = "*"
gcm_filters = "*"
gcsfs = "*"
gdal = "*"
geocube = "*"
geopandas = "*"
geopy = "*"
geoviews-core = "*"
gh = "*"
gh-scoped-creds = "*"
git-lfs = "*"
gsw = "*"
h5netcdf = "*"
h5py = "*"
holoviews = "*"
hvplot = "*"
intake = "*"
intake-esm = "*"
intake-geopandas = "*"
intake-stac = "*"
intake-xarray = "*"
ipdb = "*"
ipykernel = "*"
ipyleaflet = "*"
ipympl = "*"
ipytree = "*"
ipywidgets = "*"
jupyter-panel-proxy = "*"
jupyter-resource-usage = "*"
jupyter-sshd-proxy = "*"
jupyterlab-git = "*"
jupyterlab-lsp = "*"
jupyterlab-myst = "*"
jupyterlab_code_formatter = "*"
kerchunk = "*"
line_profiler = "*"
lonboard = "*"
lxml = "*"
lz4 = "*"
matplotlib-base = "*"
memory_profiler = "*"
metpy = "*"
nb_conda_kernels = "*"
nbstripout = "*"
nc-time-axis = "*"
netcdf4 = "*"
numbagg = "*"
numcodecs = "*"
numpy = "*"
numpy_groupies = "*"
odc-stac = "*"
openssh = "*"
pandas = "*"
panel = "*"
pangeo-notebook = "2024.11.15.*"
param = "*"
parcels = "*"
pip = "*"
planetary-computer = "*"
pop-tools = "*"
pyarrow = "*"
pycamhd = "*"
pydap = "*"
pystac = "*"
pystac-client = "*"
python = "3.12.*"
python-blosc = "*"
python-duckdb = "*"
python-gist = "*"
python-graphviz = "*"
python-lsp-ruff = "*"
python-xxhash = "*"
rasterio = "*"
rechunker = "*"
rio-cogeo = "*"
rioxarray = "*"
ruff = "*"
s3fs = "*"
satpy = "*"
scikit-image = "*"
scikit-learn = "*"
scipy = "*"
seaborn = "*"
snakeviz = "*"
sparse = "*"
stackstac = "*"
tiledb-py = "*"
timezonefinder = "*"
watermark = "*"
xarray = "*"
xarray-spatial = "*"
xarray_leaflet = "*"
xarrayutils = "*"
xbatcher = "*"
xclim = "*"
xesmf = "*"
xgboost = "*"
xgcm = "*"
xhistogram = "*"
xmip = "*"
xmitgcm = "*"
xpublish = "*"
xrft = "*"
xskillscore = "*"
xvec = "*"
xxhash = "*"
zarr = "*"

Then, I tried to generate a pixi.lock file from the pixi.toml file. This was ran using pixi=0.38.0:

pixi update

and... it took 42 seconds 🤯, compared to 23m using conda-lock + mamba=1.5.9.

That said, pixi environments are typically project-centric (one environment per project), rather than a global-centric (one environment, many projects) - https://pixi.sh/latest/switching_from/conda/#environment-vs-project. There is a way to create 'global' detached-environments, but it doesn't look recommended.

We'll need to think through this a bit for pangeo-docker-images, ideally we could get the benefits of the fast resolvo solver used by pixi, but still have dependency specification files compatible with tools from the rest of the conda ecosystem.

@weiji14
Copy link
Member

weiji14 commented Nov 28, 2024

Oo, it does look like pixi allows exporting conda explicit specification files at least, according to https://pixi.sh/latest/reference/cli/#project-export-conda-explicit-spec:

pixi project export conda-explicit-spec .

produces several default_{platform}_conda_spec.txt files. Sample from linux-64:

# Generated by `pixi project export`
# platform: linux-64
@EXPLICIT
https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2#d7c89558ba9fa0495403155b64376d81
https://conda.anaconda.org/conda-forge/linux-64/libgomp-14.2.0-h77fa898_1.conda#cc3573974587f12dda90d96e3e55a702
https://conda.anaconda.org/conda-forge/linux-64/_openmp_mutex-4.5-2_gnu.tar.bz2#73aaf86a425cc6e73fcf236a5a46396d
https://conda.anaconda.org/conda-forge/linux-64/libgcc-14.2.0-h77fa898_1.conda#3cb76c3f10d3bc7f1105b2fc9db984df
https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-14.2.0-h69a702a_1.conda#e39480b9ca41323497b05492a63bc35b
...

Now if we can find a way for it to export to a unified conda-lock.yml file, that would be sweet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants