Pathogen repo build #6

Workflow file for this run

.github/workflows/pathogen-repo-build.yaml at 3e5cbf5

	# DO NOT EDIT - GENERATED

	# This workflow is intended to be called by workflows in our various pathogen
	# build repos. See workflow-templates/pathogen-repo-builds.yaml (a "starter"
	# workflow) in this repo for an example of what the caller workflow looks like.

	name: Pathogen repo build
	defaults:
	run:
	# This is the same as GitHub Action's `bash` keyword as of 20 June 2023:
	# https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsshell
	#
	# Completely spelling it out here so that GitHub can't change it out from under us
	# and we don't have to refer to the docs to know the expected behavior.
	shell: bash --noprofile --norc -eo pipefail {0}
	on:
	workflow_call:
	inputs:
	repo:
	description: >-
	Repository name with owner (e.g. nextstrain/zika). Defaults to the repository of the caller workflow.
	type: string
	default: ${{ github.repository }}
	required: false
	runtime:
	description: >-
	Nextstrain runtime under which to run the build. Currently only supports docker, conda, and aws-batch. Defaults to "docker".

	The aws-batch runtime requires AWS credentials. These may indirectly come from assuming a role via GitHub Actions' OIDC provider or directly from secrets.

	The AWS_ROLE_ARN environment variable or GitHub Actions configuration variable is used if present.

	The following secrets are used if present and no role is to be assumed:

	- AWS_ACCESS_KEY_ID - AWS_SECRET_ACCESS_KEY

	They must be defined in the repo's Actions secrets and passed to this workflow with `secrets: inherit`.
	type: string
	default: docker
	required: false
	run:
	description: >-
	The full `nextstrain build` command to run for the build. Defaults to `nextstrain build .`

	Use the runtime input to select the runtime for the build instead of the runtime selection options to ensure that the runtime is properly set up within the GitHub Action job.

	The pathogen repo is cloned to the top level of the working directory of the GitHub Action, so use `.` to point to the pathogen repo directory.

	If your build runs longer than the 6 hour limit for GitHub Action jobs, consider using the `--detach` flag for the aws-batch runtime.

	All environment variables provided via the env input and all secrets provided via `secrets: inherit` can be passed to the build runtime via the `--env` option.
	type: string
	default: nextstrain build .
	required: false
	env:
	description: >-
	Environment variables to set for this reusable workflow since environment variables in the caller workflow are not propagated to reusable workflows. This is expected to be a string containing YAML.

	This is easily produced, for example, by pretending you're writing normal nested YAML within a literal multi-line block scalar (introduced by "\|"):


	with:
	env: \|
	FOO: bar
	I_CANT_BELIEVE: "it's not YAML"
	would_you_believe: \|
	it's
	not
	yaml

	Do not use for secrets! Instead, pass them via GitHub Action's dedicated secrets mechanism.
	type: string
	default: ""
	required: false
	artifact-name:
	description: >-
	Name to use for the build output artifact uploaded at end of the workflow.

	If you're invoking this workflow multiple times from the same calling workflow, you should set this. Otherwise, the default "build-outputs" is probably fine.
	type: string
	default: build-outputs
	required: false
	artifact-paths:
	description: >-
	List of paths to include in the build output artifact uploaded at the end of the workflow, as a string following the format of the `paths` input of the `actions/upload-artifact` action. For example:


	with:
	artifact-paths: \|
	results/
	auspice/
	logs/

	The default paths included in the artifact are:


	build.log
	auspice/
	results/
	benchmarks/
	logs/
	.snakemake/log/

	The "build.log" contains log messages from the `nextstrain build` command. The other paths are common output paths for Nextstrain builds. If a path does not exist in your build, then the action will still succeed and will print out a warning for the non-existent file(s). Use an exclude pattern for any of the default paths that you would like to exclude from the artifact (e.g. !build.log).

	This is not supported for builds on AWS Batch because the workflow detaches from the build. Please use the `nextstrain build` command locally to reattach to AWS Batch builds to download outputs.
	type: string
	required: false
	workflow_dispatch:
	inputs:
	runtime:
	description: >-
	Nextstrain runtime under which to run the build. Currently only supports docker, conda, and aws-batch. Defaults to "docker".

	The aws-batch runtime requires AWS credentials. These may indirectly come from assuming a role via GitHub Actions' OIDC provider or directly from secrets.

	The AWS_ROLE_ARN environment variable or GitHub Actions configuration variable is used if present.

	The following secrets are used if present and no role is to be assumed:

	- AWS_ACCESS_KEY_ID - AWS_SECRET_ACCESS_KEY

	They must be defined in the repo's Actions secrets and passed to this workflow with `secrets: inherit`.
	type: string
	default: docker
	required: false
	run:
	description: >-
	The full `nextstrain build` command to run for the build. Defaults to `nextstrain build .`

	Use the runtime input to select the runtime for the build instead of the runtime selection options to ensure that the runtime is properly set up within the GitHub Action job.

	The pathogen repo is cloned to the top level of the working directory of the GitHub Action, so use `.` to point to the pathogen repo directory.

	If your build runs longer than the 6 hour limit for GitHub Action jobs, consider using the `--detach` flag for the aws-batch runtime.

	All environment variables provided via the env input and all secrets provided via `secrets: inherit` can be passed to the build runtime via the `--env` option.
	type: string
	default: nextstrain build .
	required: false
	env:
	description: >-
	Environment variables to set for this reusable workflow since environment variables in the caller workflow are not propagated to reusable workflows. This is expected to be a string containing YAML.

	This is easily produced, for example, by pretending you're writing normal nested YAML within a literal multi-line block scalar (introduced by "\|"):


	with:
	env: \|
	FOO: bar
	I_CANT_BELIEVE: "it's not YAML"
	would_you_believe: \|
	it's
	not
	yaml

	Do not use for secrets! Instead, pass them via GitHub Action's dedicated secrets mechanism.
	type: string
	default: ""
	required: false
	artifact-name:
	description: >-
	Name to use for the build output artifact uploaded at end of the workflow.

	If you're invoking this workflow multiple times from the same calling workflow, you should set this. Otherwise, the default "build-outputs" is probably fine.
	type: string
	default: build-outputs
	required: false
	artifact-paths:
	description: >-
	List of paths to include in the build output artifact uploaded at the end of the workflow, as a string following the format of the `paths` input of the `actions/upload-artifact` action. For example:


	with:
	artifact-paths: \|
	results/
	auspice/
	logs/

	The default paths included in the artifact are:


	build.log
	auspice/
	results/
	benchmarks/
	logs/
	.snakemake/log/

	The "build.log" contains log messages from the `nextstrain build` command. The other paths are common output paths for Nextstrain builds. If a path does not exist in your build, then the action will still succeed and will print out a warning for the non-existent file(s). Use an exclude pattern for any of the default paths that you would like to exclude from the artifact (e.g. !build.log).

	This is not supported for builds on AWS Batch because the workflow detaches from the build. Please use the `nextstrain build` command locally to reattach to AWS Batch builds to download outputs.
	type: string
	required: false
	repo:
	description: >-
	Repository name with owner (e.g. nextstrain/zika).
	type: string
	default: ""
	required: true
	env:
	NEXTSTRAIN_GITHUB_DIR: .git/nextstrain/.github
	permissions:
	id-token: write
	jobs:
	workflow-context:
	runs-on: ubuntu-latest
	steps:
	- id: workflow-context
	uses: nextstrain/.github/actions/workflow-context@master
	outputs:
	repository: ${{ steps.workflow-context.outputs.repository }}
	sha: ${{ steps.workflow-context.outputs.sha }}
	run-build:
	needs: workflow-context
	runs-on: ubuntu-latest
	steps:
	- name: Checkout build repository
	uses: actions/checkout@v3
	with:
	repository: ${{ inputs.repo }}
	- # Need to run this after the build repo is cloned so that cloning the
	# build repo does not overwrite the .git dir and remove the extra support files
	# that we need from nextstrain/.github repo
	name: Checkout ${{ needs.workflow-context.outputs.repository }} (sha ${{ needs.workflow-context.outputs.sha }})
	uses: actions/checkout@v3
	with:
	repository: ${{ needs.workflow-context.outputs.repository }}
	ref: ${{ needs.workflow-context.outputs.sha }}
	path: ${{ env.NEXTSTRAIN_GITHUB_DIR }}
	- if: inputs.env
	name: Set environment variables
	env:
	env: ${{ inputs.env }}
	run: >
	# shellcheck disable=SC2154

	echo "$env" \| "$NEXTSTRAIN_GITHUB_DIR"/bin/yaml-to-envvars \| tee -a "$GITHUB_ENV"

	- name: Set secrets as environment variables
	env:
	secrets: ${{ toJson(secrets) }}
	run: >
	# shellcheck disable=SC2154

	echo "$secrets" \| jq 'del(.github_token)' \| "$NEXTSTRAIN_GITHUB_DIR"/bin/json-to-envvars \| tee -a "$GITHUB_ENV"

	- if: inputs.runtime == 'aws-batch'
	uses: aws-actions/configure-aws-credentials@v2
	with:
	# XXX FIXME
	aws-region: us-east-1
	aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
	aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
	role-to-assume: ${{ env.AWS_ROLE_ARN \|\| vars.AWS_ROLE_ARN }}
	- name: Setup runtime ${{ inputs.runtime }}
	uses: ./.git/nextstrain/.github/actions/setup-nextstrain-cli
	with:
	cli-version: ">=7.1.0"
	runtime: ${{ inputs.runtime }}
	- name: Run build via ${{ inputs.runtime }}
	env:
	NEXTSTRAIN_BUILD_COMMAND: ${{ inputs.run }}
	run: \|
	# shellcheck disable=SC2154
	set -x

	eval "$NEXTSTRAIN_BUILD_COMMAND" \|& tee build.log
	- if: ${{ inputs.runtime == 'aws-batch' }}
	name: Get AWS Batch job id
	id: aws-batch
	run: \|
	echo "AWS_BATCH_JOB_ID=$(tail -n1 build.log \| sed -nE 's/.+attach ([-a-f0-9]+).+/\1/p')" >> "$GITHUB_ENV"
	- if: env.AWS_BATCH_JOB_ID
	name: Generate AWS Batch summary
	run: \|
	"$NEXTSTRAIN_GITHUB_DIR"/bin/interpolate-env < "$NEXTSTRAIN_GITHUB_DIR"/text-templates/attach-aws-batch.md \
	> "$GITHUB_STEP_SUMMARY"
	- if: always()
	uses: actions/upload-artifact@v3
	with:
	if-no-files-found: warn
	name: ${{ inputs.artifact-name }}
	path: \|
	build.log
	auspice/
	results/
	benchmarks/
	logs/
	.snakemake/log/
	${{ inputs.artifact-paths }}
	outputs:
	AWS_BATCH_JOB_ID: ${{ env.AWS_BATCH_JOB_ID }}
	# Wait for up to 6 hours (the GitHub Actions job timeout) for the AWS Batch
	# job to finish.
	wait-1:
	# XXX FIXME: drop this
	timeout-minutes: 2
	needs: [run-build, workflow-context]
	if: needs.run-build.outputs.AWS_BATCH_JOB_ID
	# XXX FIXME: We should use a self-hosted runner to avoid eating up our
	# org-wide concurrency quota on GitHub's runners. For example, for about
	# $100/yr (committed to, but not paid upfront) we could self-host a single,
	# always-on runner on a t4g.small EC2 instance with 2 CPUs and 2GB of RAM.
	# We could also use other auto-scaling solutions and likely reduce the cost
	# further while increasing capacity. Then we'd select for that with
	# runs-on: labels and/or groups, e.g. with the default labels:
	#
	# runs-on: [self-hosted, linux, ARM64]
	#
	# -trs, 4 August 2023
	runs-on: ubuntu-latest
	steps:
	# Uses needs.workflow-context.outputs
	- # Need to run this after the build repo is cloned so that cloning the
	# build repo does not overwrite the .git dir and remove the extra support files
	# that we need from nextstrain/.github repo
	name: Checkout ${{ needs.workflow-context.outputs.repository }} (sha ${{ needs.workflow-context.outputs.sha }})
	uses: actions/checkout@v3
	with:
	repository: ${{ needs.workflow-context.outputs.repository }}
	ref: ${{ needs.workflow-context.outputs.sha }}
	path: ${{ env.NEXTSTRAIN_GITHUB_DIR }}
	- if: inputs.runtime == 'aws-batch'
	uses: aws-actions/configure-aws-credentials@v2
	with:
	# XXX FIXME
	aws-region: us-east-1
	aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
	aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
	role-to-assume: ${{ env.AWS_ROLE_ARN \|\| vars.AWS_ROLE_ARN }}
	- name: Setup runtime ${{ inputs.runtime }}
	uses: ./.git/nextstrain/.github/actions/setup-nextstrain-cli
	with:
	cli-version: ">=7.1.0"
	runtime: ${{ inputs.runtime }}
	# XXX TODO: This will reiterate the full logs emitted so far on every
	# wait-N job, which is annoying, but ah well. Avoiding that is more work
	# than I care to spend right now for too little gain. It's maybe more
	# desirable this way anyhow: the full logs of the Batch job will be
	# captured in GitHub Actions logs by the last successful wait-N job.
	# -trs, 4 August 2023
	- name: Attach to AWS Batch job
	env:
	AWS_BATCH_JOB_ID: ${{ needs.run-build.outputs.AWS_BATCH_JOB_ID }}
	run: \|
	nextstrain build \
	--aws-batch \
	--attach "$AWS_BATCH_JOB_ID" \
	--no-download
	# Wait for up to another 6 hours (hours 6–12) if the preceding wait-N job
	# timed out. Our heuristic for job timeout is when the job is cancelled but
	# the workflow isn't.
	wait-2:
	# XXX FIXME: drop this
	timeout-minutes: 2
	# XXX FIXME: We should use a self-hosted runner to avoid eating up our
	# org-wide concurrency quota on GitHub's runners. For example, for about
	# $100/yr (committed to, but not paid upfront) we could self-host a single,
	# always-on runner on a t4g.small EC2 instance with 2 CPUs and 2GB of RAM.
	# We could also use other auto-scaling solutions and likely reduce the cost
	# further while increasing capacity. Then we'd select for that with
	# runs-on: labels and/or groups, e.g. with the default labels:
	#
	# runs-on: [self-hosted, linux, ARM64]
	#
	# -trs, 4 August 2023
	runs-on: ubuntu-latest
	steps:
	# Uses needs.workflow-context.outputs
	- # Need to run this after the build repo is cloned so that cloning the
	# build repo does not overwrite the .git dir and remove the extra support files
	# that we need from nextstrain/.github repo
	name: Checkout ${{ needs.workflow-context.outputs.repository }} (sha ${{ needs.workflow-context.outputs.sha }})
	uses: actions/checkout@v3
	with:
	repository: ${{ needs.workflow-context.outputs.repository }}
	ref: ${{ needs.workflow-context.outputs.sha }}
	path: ${{ env.NEXTSTRAIN_GITHUB_DIR }}
	- if: inputs.runtime == 'aws-batch'
	uses: aws-actions/configure-aws-credentials@v2
	with:
	# XXX FIXME
	aws-region: us-east-1
	aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
	aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
	role-to-assume: ${{ env.AWS_ROLE_ARN \|\| vars.AWS_ROLE_ARN }}
	- name: Setup runtime ${{ inputs.runtime }}
	uses: ./.git/nextstrain/.github/actions/setup-nextstrain-cli
	with:
	cli-version: ">=7.1.0"
	runtime: ${{ inputs.runtime }}
	# XXX TODO: This will reiterate the full logs emitted so far on every
	# wait-N job, which is annoying, but ah well. Avoiding that is more work
	# than I care to spend right now for too little gain. It's maybe more
	# desirable this way anyhow: the full logs of the Batch job will be
	# captured in GitHub Actions logs by the last successful wait-N job.
	# -trs, 4 August 2023
	- name: Attach to AWS Batch job
	env:
	AWS_BATCH_JOB_ID: ${{ needs.run-build.outputs.AWS_BATCH_JOB_ID }}
	run: \|
	nextstrain build \
	--aws-batch \
	--attach "$AWS_BATCH_JOB_ID" \
	--no-download
	needs: [wait-1, run-build, workflow-context]
	if: needs.wait-1.result == 'cancelled' && !cancelled()
	# 12–18 hours
	wait-3:
	# XXX FIXME: drop this
	timeout-minutes: 2
	# XXX FIXME: We should use a self-hosted runner to avoid eating up our
	# org-wide concurrency quota on GitHub's runners. For example, for about
	# $100/yr (committed to, but not paid upfront) we could self-host a single,
	# always-on runner on a t4g.small EC2 instance with 2 CPUs and 2GB of RAM.
	# We could also use other auto-scaling solutions and likely reduce the cost
	# further while increasing capacity. Then we'd select for that with
	# runs-on: labels and/or groups, e.g. with the default labels:
	#
	# runs-on: [self-hosted, linux, ARM64]
	#
	# -trs, 4 August 2023
	runs-on: ubuntu-latest
	steps:
	# Uses needs.workflow-context.outputs
	- # Need to run this after the build repo is cloned so that cloning the
	# build repo does not overwrite the .git dir and remove the extra support files
	# that we need from nextstrain/.github repo
	name: Checkout ${{ needs.workflow-context.outputs.repository }} (sha ${{ needs.workflow-context.outputs.sha }})
	uses: actions/checkout@v3
	with:
	repository: ${{ needs.workflow-context.outputs.repository }}
	ref: ${{ needs.workflow-context.outputs.sha }}
	path: ${{ env.NEXTSTRAIN_GITHUB_DIR }}
	- if: inputs.runtime == 'aws-batch'
	uses: aws-actions/configure-aws-credentials@v2
	with:
	# XXX FIXME
	aws-region: us-east-1
	aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
	aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
	role-to-assume: ${{ env.AWS_ROLE_ARN \|\| vars.AWS_ROLE_ARN }}
	- name: Setup runtime ${{ inputs.runtime }}
	uses: ./.git/nextstrain/.github/actions/setup-nextstrain-cli
	with:
	cli-version: ">=7.1.0"
	runtime: ${{ inputs.runtime }}
	# XXX TODO: This will reiterate the full logs emitted so far on every
	# wait-N job, which is annoying, but ah well. Avoiding that is more work
	# than I care to spend right now for too little gain. It's maybe more
	# desirable this way anyhow: the full logs of the Batch job will be
	# captured in GitHub Actions logs by the last successful wait-N job.
	# -trs, 4 August 2023
	- name: Attach to AWS Batch job
	env:
	AWS_BATCH_JOB_ID: ${{ needs.run-build.outputs.AWS_BATCH_JOB_ID }}
	run: \|
	nextstrain build \
	--aws-batch \
	--attach "$AWS_BATCH_JOB_ID" \
	--no-download
	needs: [wait-2, run-build, workflow-context]
	if: needs.wait-2.result == 'cancelled' && !cancelled()
	# 18–24 hours
	wait-4:
	# XXX FIXME: drop this
	timeout-minutes: 2
	# XXX FIXME: We should use a self-hosted runner to avoid eating up our
	# org-wide concurrency quota on GitHub's runners. For example, for about
	# $100/yr (committed to, but not paid upfront) we could self-host a single,
	# always-on runner on a t4g.small EC2 instance with 2 CPUs and 2GB of RAM.
	# We could also use other auto-scaling solutions and likely reduce the cost
	# further while increasing capacity. Then we'd select for that with
	# runs-on: labels and/or groups, e.g. with the default labels:
	#
	# runs-on: [self-hosted, linux, ARM64]
	#
	# -trs, 4 August 2023
	runs-on: ubuntu-latest
	steps:
	# Uses needs.workflow-context.outputs
	- # Need to run this after the build repo is cloned so that cloning the
	# build repo does not overwrite the .git dir and remove the extra support files
	# that we need from nextstrain/.github repo
	name: Checkout ${{ needs.workflow-context.outputs.repository }} (sha ${{ needs.workflow-context.outputs.sha }})
	uses: actions/checkout@v3
	with:
	repository: ${{ needs.workflow-context.outputs.repository }}
	ref: ${{ needs.workflow-context.outputs.sha }}
	path: ${{ env.NEXTSTRAIN_GITHUB_DIR }}
	- if: inputs.runtime == 'aws-batch'
	uses: aws-actions/configure-aws-credentials@v2
	with:
	# XXX FIXME
	aws-region: us-east-1
	aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
	aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
	role-to-assume: ${{ env.AWS_ROLE_ARN \|\| vars.AWS_ROLE_ARN }}
	- name: Setup runtime ${{ inputs.runtime }}
	uses: ./.git/nextstrain/.github/actions/setup-nextstrain-cli
	with:
	cli-version: ">=7.1.0"
	runtime: ${{ inputs.runtime }}
	# XXX TODO: This will reiterate the full logs emitted so far on every
	# wait-N job, which is annoying, but ah well. Avoiding that is more work
	# than I care to spend right now for too little gain. It's maybe more
	# desirable this way anyhow: the full logs of the Batch job will be
	# captured in GitHub Actions logs by the last successful wait-N job.
	# -trs, 4 August 2023
	- name: Attach to AWS Batch job
	env:
	AWS_BATCH_JOB_ID: ${{ needs.run-build.outputs.AWS_BATCH_JOB_ID }}
	run: \|
	nextstrain build \
	--aws-batch \
	--attach "$AWS_BATCH_JOB_ID" \
	--no-download
	needs: [wait-3, run-build, workflow-context]
	if: needs.wait-3.result == 'cancelled' && !cancelled()

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pathogen repo build #6

Workflow file

Pathogen repo build #6

Jobs

Run details

Workflow file for this run