Set up reproducible Python environments with conda-lock #2901

zaneselvans · 2023-09-28T01:44:18Z

PR Overview

This is the conda-lock specific stuff from #2900. See #2896 for some more general discussion of using lockfiles to make our environments reproducible.

Draft `conda-lock` setup for conversation...

Added configuration of conda-lock in pyproject.toml -- it still gets the newest versions of recordlinkage and sphinx-autoapi from PyPI.
See conda-lock.yml as an example of the lockfiles it generates.

To create a new lockfile:

conda-lock --mamba -f pyproject.toml

To install all the packages specified in that lockfile in an environment named pudl-dev:

conda-lock install --mamba -n pudl-dev --dev -E doc -E datasette

The --dev installs the somewhat standardized test and dev package extras.
Note that this requires having conda-lock installed in your base conda environment.

To get into the environment and work with PUDL:

mamba activate pudl-dev
pip install -e ./

You should see that absolutely every dependency has already been satisfied, and the only new package that gets installed is catalystcoop.pudl itself.

PR Checklist

Merge the most recent version of the branch you are merging into (probably dev).
All CI checks are passing. Run tests locally to debug failures
Make sure you've included good docstrings.
For major data coverage & analysis changes, run data validation tests
Include unit tests for new functions and classes.
Defensive data quality/sanity checks in analyses & data processing functions.
Update the release notes and reference reference the PR and related issues.
Do your own explanatory review of the PR to help the reviewer understand what's going on and identify issues preemptively.

zaneselvans · 2023-09-28T01:47:53Z

pyproject.toml

+[tool.conda-lock]
+channels = [
+    "conda-forge",
+    "defaults",
+]
+platforms = [
+    "osx-arm64",
+    "linux-64",
+]
+channel-priority = "strict"
+name = "pudl-dev"
+
+[tool.conda-lock.dependencies]
+recordlinkage = {source = "pypi"}
+sphinx-autoapi = {source = "pypi"}
+pip = ">=23,<24"
+python = ">=3.11,<3.12"
+python-snappy = ">=0.6,<1"
+sqlite = ">=3.36,<4"


This is all the configuration for conda-lock

With this pyproject.toml all of our dependencies except for recordlinkage and sphinx-autoapi are installed from conda-forge (and those two packages alone are singled out for installation from PyPI).

Our programmatic non-python dependencies (snappy, sqlite..) also get installed from conda-forge.

We could add jupyterlab and the other interactive dependencies from devtools/environment.yml into the dev optional dependencies above, and pandoc / nodejs here if we wanted, and eliminate devtools/environment.yml and probably our other *-environment.yml files.

Would need to add some infrastructure for updating the lockfile on a regular basis and get used to using it to install our environment, but it does seem to be working.

The reason these two get pulled from pypi is because the latest versions haven't hit conda yet, basically? And because sphinx-autoapi==2.1.0 doesn't specify astroid<3, so it's actually just broken?

Consolidating our various environment.ymls into one place seems convenient!

Yes, that's right -- we need a version of the package that can only be sourced from PyPII. I've requested to be made a maintainer on one, and someone from conda-forge has commented on the potential abandonment of sphinx-autoapi in the most recent bot PR, so hopefully they'll both be available from conda. But at the same time it's good to know that in the event there's some package we can really only get from PyPI, there's a way to do it without breakage.

And yes it would be really great to have pyproject.toml be the ONLY source file, with dependencies broken out into groups / extras that can be selectively installed from the lockfile. I didn't figure it out but I suspect there's a way to categorize the conda-only dependencies so that they can also be associated with a particular grouping, so we could have jupyterlab, pandoc, nodejsetc. be part of thedevor ajupytergroup -- they'd be invisible topip(which couldn't satisfy many of them) but available in the lockfile for installation withmamba`

jdangerx

This seems really promising!

In terms of getting it to play nicely with other tools, we can use the conda-lock render subcommand to make normal environment.ymls for our conda-using tools, which should handle most cases. Except for tox, which seems like it might need the tox-conda plugin to play nice with conda.

That leaves the "update conda.lock whenever there's a dependency change in pyproject.toml" flow - seems like a job for a custom pre-commit hook.

jdangerx · 2023-09-28T16:20:17Z

pyproject.toml

+[tool.conda-lock]
+channels = [
+    "conda-forge",
+    "defaults",
+]
+platforms = [
+    "osx-arm64",
+    "linux-64",
+]
+channel-priority = "strict"
+name = "pudl-dev"
+
+[tool.conda-lock.dependencies]
+recordlinkage = {source = "pypi"}
+sphinx-autoapi = {source = "pypi"}
+pip = ">=23,<24"
+python = ">=3.11,<3.12"
+python-snappy = ">=0.6,<1"
+sqlite = ">=3.36,<4"


The reason these two get pulled from pypi is because the latest versions haven't hit conda yet, basically? And because sphinx-autoapi==2.1.0 doesn't specify astroid<3, so it's actually just broken?

Consolidating our various environment.ymls into one place seems convenient!

jdangerx · 2023-09-28T16:21:55Z

pyproject.toml

+sphinx-autoapi = {source = "pypi"}
+pip = ">=23,<24"
+python = ">=3.11,<3.12"
+python-snappy = ">=0.6,<1"


Why do we need to explicitly include python-snappy here instead of in our normal dependencies?

Hmm. Here I just pulled over the packages that we needed to have in environment.yml because they had non-python package components that we needed to satisfy. But I think python-snappy could go into the main dependencies here. I think it would mean that libsnappydev would need to be installed on the system if the package were getting installed with pip rather than conda via the lockfile, but that's already true.

zaneselvans · 2023-09-28T18:06:04Z

Oh I hadn't seen conda-lock render that's great to know!

For Tox we could use tox-conda and let it use the environment generated by the lockfile / rendered environment files... or I think we could just let it keep using pip to install the package based on pyproject.toml couldn't we? If it's running inside a conda environment that's satisfying the binary dependencies like libsnappydev then in theory the pip installed virtual environment should work fine, and would test whether it can actually be solved / installed using pip. But it does seem like it would be less duplicative / more reproducible to use the locked conda versions for tox.

zaneselvans · 2023-09-28T18:09:59Z

I do think we should think a bit about what kind of installation scenarios and environmental isolation we care about.

If we're treating PUDL as an application, and only expect it to be reliably installable in a conda environment corresponding to the lockfile, then what do we think about publishing tagged releases to PyPI and conda-forge? And what do we think about using Tox to create an isolated environment into which a built package is installed, separate from the code that's stored in the repository?

One could imagine a world in which we assume PUDL will only ever be run from a checked-out repository, and only ever run in a conda environment built from a lockfile, which would imply a different set of things to test.

zaneselvans · 2023-09-28T18:33:11Z

@jdangerx Would it really make sense to update the lockfile with a pre-commit hook? Like on every commit? Or maybe just once a week with a GitHub action? With pre-commit I imagine it would be changing every day. Probably multiple times a day. And might be annoying to merge.

…ython

…a-forge

[pre-commit.ci] auto fixes from pre-commit.com hooks For more information, see https://pre-commit.ci

Remove sphinx-autoapi from pypi deps, and no longer required snappy-python Update conda-lock file Update conda-lock.yml now that all dependencies are available on conda-forge Consolidate conda env files under environments/ dir Quote the pip install command Remove pip install of pudl from environment.yml Move all remaining dev-environment.yml deps to pyproject.toml Update lockfile Render platform-specific environment files in github action Update lockfile Add some comment notes to workflow Render environment for Read The Docs. Use environment not explicit rendered lockfile Add readthedocs specific sphinx extension Don't render explicit conda env for RTD since it can't read it. Comment out non-working linux-aarch64 platform for now. Switch to using rendered lockfiles. Remove deprecated environment files Update lockfile Install git into the docker image. Use micromamba and unrendered multi-platform lockfile. Add main category to micromamba environment creation. Use conda-lock not base as env name Update lockfile

For more information, see https://pre-commit.ci

Update Lockfile

* Add a conda-lock setup for discussion. * Move python-snappy into project.dependencies in pyproject.toml * Remove sphinx-autoapi from pypi deps, and no longer required snappy-python * Switch to using conda-forge version of recordlinkage v0.16 * Update conda-lock.yml now that all dependencies are available on conda-forge * Consolidate conda env files under environments/ dir * Add a GitHub action to relock dependencies * Quote the pip install command * Remove pip install of pudl from environment.yml * Rename workflow * Only build lockfile from pyproject.toml, don't install extras. * Just install conda-lock, not pudl, before running conda-lock. * install conda-lock with pip * Move all remaining dev-environment.yml deps to pyproject.toml * Add other platforms; make draft PR against dev. * Comment out dev base branch for now. * Remove pandas extras and recordlinkage deps from pyproject.toml * Use conda-lock --micromamba rather than --mamba * Don't specify grpcio, or specific recordlinkage version * Render platform-specific environment files in github action * Fix paths relative to environments directory * Add some comment notes to workflow * Render environment for Read The Docs. * Use environment not explicit rendered lockfile * Add readthedocs specific sphinx extension * Don't render explicit conda env for RTD since it can't read it. * Build linux-aarch64 lockfile. Use conda-lock.yml in workflows. * Comment out non-working linux-aarch64 platform for now. * Switch to using rendered lockfiles. * Remove deprecated environment files * Switch to using a micromamba docker image * Install git into the docker image. * Use micromamba and unrendered multi-platform lockfile. * Add main category to micromamba environment creation. * Use conda-lock not base as env name * Add a conda-lock setup for discussion. * Move python-snappy into project.dependencies in pyproject.toml * Remove sphinx-autoapi from pypi deps, and no longer required snappy-python

* Add a conda-lock setup for discussion. * Move python-snappy into project.dependencies in pyproject.toml * Remove sphinx-autoapi from pypi deps, and no longer required snappy-python * Switch to using conda-forge version of recordlinkage v0.16 * Update conda-lock.yml now that all dependencies are available on conda-forge * Consolidate conda env files under environments/ dir * Add a GitHub action to relock dependencies * Quote the pip install command * Remove pip install of pudl from environment.yml * Rename workflow * Only build lockfile from pyproject.toml, don't install extras. * Just install conda-lock, not pudl, before running conda-lock. * install conda-lock with pip * Move all remaining dev-environment.yml deps to pyproject.toml * Add other platforms; make draft PR against dev. * Comment out dev base branch for now. * Remove pandas extras and recordlinkage deps from pyproject.toml * Use conda-lock --micromamba rather than --mamba * Don't specify grpcio, or specific recordlinkage version * Render platform-specific environment files in github action * Fix paths relative to environments directory * Add some comment notes to workflow * Render environment for Read The Docs. * Use environment not explicit rendered lockfile * Add readthedocs specific sphinx extension * Don't render explicit conda env for RTD since it can't read it. * Build linux-aarch64 lockfile. Use conda-lock.yml in workflows. * Comment out non-working linux-aarch64 platform for now. * Switch to using rendered lockfiles. * Remove deprecated environment files * Switch to using a micromamba docker image * Install git into the docker image. * Use micromamba and unrendered multi-platform lockfile. * Add main category to micromamba environment creation. * Use conda-lock not base as env name * Add a conda-lock setup for discussion. * Move python-snappy into project.dependencies in pyproject.toml * Remove sphinx-autoapi from pypi deps, and no longer required snappy-python * Add linux-aarch64 platform back into conda-lock settings.

# Set up reproducible Python environments with conda-lock (#2901) * Add a conda-lock setup for discussion. * Move python-snappy into project.dependencies in pyproject.toml * Remove sphinx-autoapi from pypi deps, and no longer required snappy-python * Switch to using conda-forge version of recordlinkage v0.16 * Update conda-lock.yml now that all dependencies are available on conda-forge * Consolidate conda env files under environments/ dir * Add a GitHub action to relock dependencies * Quote the pip install command * Remove pip install of pudl from environment.yml * Rename workflow * Only build lockfile from pyproject.toml, don't install extras. * Just install conda-lock, not pudl, before running conda-lock. * install conda-lock with pip * Move all remaining dev-environment.yml deps to pyproject.toml * Add other platforms; make draft PR against dev. * Comment out dev base branch for now. * Remove pandas extras and recordlinkage deps from pyproject.toml * Use conda-lock --micromamba rather than --mamba * Don't specify grpcio, or specific recordlinkage version * Render platform-specific environment files in github action * Fix paths relative to environments directory * Add some comment notes to workflow * Render environment for Read The Docs. * Use environment not explicit rendered lockfile * Add readthedocs specific sphinx extension * Don't render explicit conda env for RTD since it can't read it. * Build linux-aarch64 lockfile. Use conda-lock.yml in workflows. * Comment out non-working linux-aarch64 platform for now. * Switch to using rendered lockfiles. * Remove deprecated environment files * Switch to using a micromamba docker image * Install git into the docker image. * Use micromamba and unrendered multi-platform lockfile. * Add main category to micromamba environment creation. * Use conda-lock not base as env name * Add a conda-lock setup for discussion. * Move python-snappy into project.dependencies in pyproject.toml * Remove sphinx-autoapi from pypi deps, and no longer required snappy-python * Add linux-aarch64 platform back into conda-lock settings. ## Move tox.ini and update-lockfile logic to Makefile - Remove deprecated API_KEY_EIA envvar - Add .gitignore in new environments directory - Create Makefile and migrat tox.ini logic into it - Replace spaces in Makefile with tabs - Install pudl via pip install --no-deps to avoid contaminating the locked environment - Move pip install and coverage XML logic from GHA workflow into the Makefile - Increase the minimum versions of some dependencies. - Move update-lockfile GHA workflow logic into Makefile - Attempt to run slow tests in parallel using "wait" to prevent premature exit of the shell ## Adjust Dockerfile & deployment use micromamba * Use micromamba not conda in Dockerfile CMD, also use pip install --no-deps * Use micromamba not conda in command passed to build container * Use default mambauser rather than catalyst in docker container * Remove --no-capture-output which isn't supported by micromamba. Is this a problem? * Remove uninterpolated vars in .env and more --no-capture-output * Separate ETL and pytest commands. * Stop trying to run tests in parallel. Sigh. * Add google cloud sdk to conda environment. * Install Google Cloud SDK from conda-forge. * Add back in the making of required directories. Oops. * Attempt to have micromamba run pass through output * Use prettier to standardize formatting of lockfiles. * Add dagster (server startup) target to Makefile * Update conda lockfile and rerender environment files * Attempt to trigger update-lockfile when pyproject.toml is changed. * Remove non-required upper bounds on dependency versions. * Use correct branch name for update-lockfile trigger. * Fix incorrect nbformat minimum version * Update Makefile in response to PR feedback ## Set up triggers to automate updating of conda lockfiles * Remove dagster-postgres version to avoid PyPI conda-forge conflict. * Break up and distribute the nuke target. * Resolve issues with pandera dependency. Reorganize makefile. * Add triggers and commit/PR for workflow_dispatch, pull_request, schedule and set appropriate GITHUB_REF values for each case. * Use push instead of pull_request to trigger on path. This avoids re-locking the dependencies every single time you push to a PR that had a change to pyproject.toml *somewhere* in it. * Also trigger based on path if .github/workflows/update-lockfile.yml changes. * Update conda-lock.yml and rendered conda environment files. ## Consolidate optional dependencies; test nuke; parallelize tests * Move previous dev, test, and datasette optional dependencies into the required dependencies to simplify application installation. * Test make nuke; parallelize --live-dbs tests * Move prettier into conda-only dependencies * Update conda-lock.yml and rendered conda environment files. * Remove action test file hashlog * Remove merge markers. * Remove transitive astroid dependency that's now correctly included in solve. * Use the real immature library version of dagster-postgres (0.21.6) rather than the accidentally packaged 1.5.6 version found in conda. We'll need to keep an eye out for when dagster-postgres graduates to the stable versioning and update it. This is a bit of a mess because of some broken automation in the conda packaging for dagster which has now been fixed. * Update "make pudl" to remove the old PUDL DB and reinitialize with alembic, rather than writing to the DB that already exists. * Fixed some groupby.agg() deprecation warnings. * Fix dagster-postgres version (again). * Update username in path to settings file * Avoid bugs in ferc_to_sqlite --clobber; don't use cache_dir for pip install. * Make FERC extraction output removal more specific. * Bump numpy and numba minimum versions. * Bump black version in pre-commit * Bump ruff pre-commit hook version ## Update docs to reflect swapping Make in for Tox, application packaging. * Rename tox-pytest and update-lockfile workflows. * Make scheduled PRs against dev rather than conda-lockfile branch. * Update to pyarrow 14 and grpcio 1.59 b/c security * Update release notes * Add CI targets in Makefile. Docs cleanup. * Update conda-lock.yml and rendered conda environment files. --------- Co-authored-by: zaneselvans <zaneselvans@users.noreply.github.com>

zaneselvans added the dependencies Pull requests that update a dependency file label Sep 28, 2023

zaneselvans linked an issue Sep 28, 2023 that may be closed by this pull request

Use lockfile to specify a reproducible python environment #2896

Closed

zaneselvans requested a review from jdangerx September 28, 2023 01:45

zaneselvans commented Sep 28, 2023

View reviewed changes

zaneselvans self-assigned this Sep 28, 2023

jdangerx reviewed Sep 28, 2023

View reviewed changes

zaneselvans changed the title ~~Add a conda-lock setup for discussion.~~ Set up reproducible Python environments with conda-lock Oct 21, 2023

zaneselvans and others added 20 commits October 20, 2023 18:45

Add a conda-lock setup for discussion.

3fc5951

Move python-snappy into project.dependencies in pyproject.toml

5e82997

Remove sphinx-autoapi from pypi deps, and no longer required snappy-p…

da594f1

…ython

Update conda-lock file

a9bde5a

Switch to using conda-forge version of recordlinkage v0.16

ce62f2d

Update conda-lock.yml now that all dependencies are available on cond…

beb0f35

…a-forge

Consolidate conda env files under environments/ dir

08a58c0

Add a GitHub action to relock dependencies

cdde302

Quote the pip install command

468ded1

Remove pip install of pudl from environment.yml

fbe5f0f

Rename workflow

06b45ed

Only build lockfile from pyproject.toml, don't install extras.

1adb606

Just install conda-lock, not pudl, before running conda-lock.

2fd2dad

install conda-lock with pip

2ce49cf

Move all remaining dev-environment.yml deps to pyproject.toml

fb77b54

Add other platforms; make draft PR against dev.

41d9f3d

Comment out dev base branch for now.

54ad813

Remove pandas extras and recordlinkage deps from pyproject.toml

b81e69a

Use conda-lock --micromamba rather than --mamba

846e905

Update lockfile

4bbfc41

[pre-commit.ci] auto fixes from pre-commit.com hooks For more information, see https://pre-commit.ci

zaneselvans added 3 commits October 21, 2023 09:51

Add a conda-lock setup for discussion.

1c981af

Re-render all conda lockfiles

9ae6973

zaneselvans force-pushed the conda-lock branch from 0c08c9a to 9ae6973 Compare October 21, 2023 18:06

zaneselvans and others added 3 commits October 21, 2023 18:12

Update lockfile

eebc98c

[pre-commit.ci] auto fixes from pre-commit.com hooks

dd3d2e3

For more information, see https://pre-commit.ci

Merge pull request #2967 from catalyst-cooperative/update-lockfile

f954409

Update Lockfile

zaneselvans changed the base branch from dev to squash-test October 21, 2023 20:12

zaneselvans marked this pull request as ready for review October 21, 2023 20:12

zaneselvans merged commit aa3d15a into squash-test Oct 21, 2023
11 checks passed

zaneselvans deleted the conda-lock branch October 21, 2023 20:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set up reproducible Python environments with conda-lock #2901

Set up reproducible Python environments with conda-lock #2901

zaneselvans commented Sep 28, 2023 •

edited

Loading

zaneselvans Sep 28, 2023

jdangerx Sep 28, 2023

zaneselvans Sep 28, 2023

zaneselvans Sep 28, 2023

jdangerx left a comment

jdangerx Sep 28, 2023

jdangerx Sep 28, 2023

zaneselvans Sep 28, 2023

zaneselvans commented Sep 28, 2023

zaneselvans commented Sep 28, 2023

zaneselvans commented Sep 28, 2023

Set up reproducible Python environments with conda-lock #2901

Set up reproducible Python environments with conda-lock #2901

Conversation

zaneselvans commented Sep 28, 2023 • edited Loading

PR Overview

Draft conda-lock setup for conversation...

PR Checklist

zaneselvans Sep 28, 2023

Choose a reason for hiding this comment

jdangerx Sep 28, 2023

Choose a reason for hiding this comment

zaneselvans Sep 28, 2023

Choose a reason for hiding this comment

zaneselvans Sep 28, 2023

Choose a reason for hiding this comment

jdangerx left a comment

Choose a reason for hiding this comment

jdangerx Sep 28, 2023

Choose a reason for hiding this comment

jdangerx Sep 28, 2023

Choose a reason for hiding this comment

zaneselvans Sep 28, 2023

Choose a reason for hiding this comment

zaneselvans commented Sep 28, 2023

zaneselvans commented Sep 28, 2023

zaneselvans commented Sep 28, 2023

zaneselvans commented Sep 28, 2023 •

edited

Loading

Draft `conda-lock` setup for conversation...