Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up workflows #62

Merged
merged 7 commits into from
Sep 11, 2023
Merged

Clean up workflows #62

merged 7 commits into from
Sep 11, 2023

Conversation

joverlee521
Copy link
Contributor

@joverlee521 joverlee521 commented Sep 7, 2023

Description of proposed changes

General clean up of workflows that includes:

  • updates to use centralized ingest scripts that were vendored in Seq counts workflow #60
  • removal of unused scripts
  • Update model workflow to use a top level s3_dst config param
  • Refactor run-model GH Action to use pathogen-repo-build for model workflow

See commits for details

Checklist

Trim back CI jobs for reasons stated in
nextstrain/cli@fab709a
bin/trigger - should have been removed in b5fc2b9
ingest/bin/download-from-s3 - should have been removed in bd00d70
ingest/bin/notify-on-new-location - should have been removed in 2635981
Remove the copy in this repo and update references.
The supporting scripts `sha256sum` and `cloudfront-invalidate` can also
be removed.
Remove the copies in this repo and update references.
No longer need to create an envdir with ingest/bin/write-envdir to use
with  --exec env + `envdir env.d snakemake`.

Moves the env variables in the GitHub Action workflow to the step that
needs them to limit scope of their availability.

Note the `PAT_GITHUB_DISPATCH` envvar was removed because we no longer
trigger any downstream actions since
b5fc2b9
The model results are expected to be uploaded to the same AWS S3 bucket,
so remove the unnecessary complexity caused by the nested `s3_dst`
config param. This will make it easier to transition to using the
pathogen-repo-build workflow for the run-model GitHub Action.
Since the workflow config has been updated in the previous commit to use
a top level `s3_dst` param, we can now easily override it in the
GH Action workflow with input values for trial runs. This allows us to
directly use the reusable pathogen-repo-build workflow.

Note that `DATA_PROVENANCES` and `S3_DST` are passed as envvars to the
reusable workflow so they can be interpolated in the build command, but
they do _not_ need to be passed to the build runtime.

Removed `bin/set-trial-optional-config` since we are no longer
using a full config file to override the workflow configs.
@joverlee521
Copy link
Contributor Author

Merging since test runs worked fine, will monitor automated runs tomorrow.

@joverlee521 joverlee521 merged commit 0915c38 into main Sep 11, 2023
5 checks passed
@joverlee521 joverlee521 deleted the clean-up branch September 11, 2023 23:50
@joverlee521
Copy link
Contributor Author

Following up here that all automated workflows ran as expected last night and this morning.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

1 participant