-
-
Notifications
You must be signed in to change notification settings - Fork 118
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Set up reproducible Python environments with conda-lock #2968
Changes from all commits
95a855c
ad464a2
c9784d4
d8dd9e3
e86a9e7
ff20c66
08f318f
240b99e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,84 @@ | ||
--- | ||
name: update-conda-lockfile | ||
|
||
on: | ||
workflow_dispatch: | ||
schedule: | ||
- cron: "0 9 * * 1-5" # Weekdays at 9AM UTC | ||
push: | ||
paths: | ||
- "pyproject.toml" | ||
- "environments/*" | ||
- ".github/workflows/update-conda-lockfile.yml" | ||
|
||
# What branch does this action run on? | ||
# - workflow_dispatch: Whatever branch it was run against. | ||
# - schedule: Always the same branch (will be dev or main) | ||
# - push: Base branch of the PR. | ||
|
||
jobs: | ||
update-conda-lockfile: | ||
runs-on: ubuntu-latest | ||
if: ${{ (github.event_name == 'push' && github.actor != 'pudlbot') || (github.event_name == 'schedule' && github.repository == 'catalyst-cooperative/pudl') || (github.event_name == 'workflow_dispatch') }} | ||
defaults: | ||
run: | ||
shell: bash -l {0} | ||
steps: | ||
- name: Set GITHUB_REF for use with workflow_dispatch | ||
if: ${{ (github.event_name == 'workflow_dispatch') }} | ||
run: | | ||
echo "GITHUB_REF="${{ github.ref_name }} >> $GITHUB_ENV | ||
- name: Set GITHUB_REF for use with schedule | ||
if: ${{ (github.event_name == 'schedule') }} | ||
run: | | ||
echo "GITHUB_REF=dev" >> $GITHUB_ENV | ||
- name: Set GITHUB_REF for use with push | ||
if: ${{ (github.event_name == 'push') }} | ||
run: | | ||
echo "GITHUB_REF="${{ github.ref_name }} >> $GITHUB_ENV | ||
Comment on lines
+36
to
+38
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It looks like you're setting There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. They were different at some point while I was developing the setup, and it felt more legible to enumerate the 3 cases we're trying to cover. But I could go either way. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ok! I'm down to keep it as is if it feels more legible. |
||
- name: Log final value of GITHUB_REF | ||
run: | | ||
echo "Final GITHUB_REF:" ${{ env.GITHUB_REF }} | ||
- uses: actions/checkout@v4 | ||
with: | ||
token: ${{ secrets.PUDL_BOT_PAT }} | ||
ref: ${{ env.GITHUB_REF }} | ||
- name: Install Micromamba | ||
uses: mamba-org/setup-micromamba@v1 | ||
with: | ||
environment-name: conda-lock | ||
create-args: >- | ||
python=3.11 | ||
conda-lock | ||
prettier | ||
- name: Run conda-lock to recreate lockfile from scratch | ||
run: | | ||
make conda-clean | ||
make conda-lock.yml | ||
- name: Commit updated conda lockfiles to branch | ||
# If running on push due to dependency changes, commit directly to the base | ||
# branch of the existing PR. Don't trigger the workflow again if we're already | ||
# running it as pudlbot (to avoid infinite recursion). | ||
if: ${{ (github.event_name == 'push' && github.actor != 'pudlbot') }} | ||
uses: stefanzweifel/git-auto-commit-action@v5 | ||
with: | ||
file_pattern: "environments/*" | ||
commit_message: "Update conda-lock.yml and rendered conda environment files." | ||
Comment on lines
+59
to
+66
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What's stopping the action from running again on a push? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The second time, the action is being run by There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I looked at the docs for the Would it make sense test this condition and abort earlier in the workflow? |
||
- name: Make a PR to merge updated conda lockfiles | ||
# If we are relocking dependencies on a schedule or workflow_dispatch, we need | ||
# to make our own PR to check whether the updated environment actually solves | ||
# and the tests pass. | ||
if: ${{ (github.event_name == 'schedule' && github.repository == 'catalyst-cooperative/pudl') || (github.event_name == 'workflow_dispatch') }} | ||
uses: peter-evans/create-pull-request@v5 | ||
with: | ||
commit-message: "Update conda-lock.yml and rendered conda environment files." | ||
title: Update Lockfile | ||
body: > | ||
This pull request relocks the dependencies with conda-lock. | ||
It is triggered by [update-conda-lockfile](https://github.com/catalyst-cooperative/pudl/blob/main/.github/workflows/update-conda-lockfile.yml). | ||
labels: dependencies, conda-lock | ||
reviewers: zaneselvans | ||
zaneselvans marked this conversation as resolved.
Show resolved
Hide resolved
|
||
branch: update-conda-lockfile | ||
base: ${{ env.GITHUB_REF }} | ||
draft: true | ||
delete-branch: true |
This file was deleted.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Conda swallows all of the logs if you don't include
--no-capture-output
but it sounds like--attach ""
should work:Maybe we run part of a full build to make sure the logs are coming through before we merge into
dev
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did run one full build, I think it should be in the build outputs if you want to take a look at the logs and see if they look like you'd expect them to. I think using
pytest-xdist
does obscure some of the test logging, but also shaves a couple of hours off the nightly builds.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the output from the parallelized tests if you want to take a look:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These logs look good to me!