Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

split up PR and nightly/release CI #169

Merged
merged 5 commits into from
Oct 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions .github/workflows/build.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
name: build

concurrency:
group: ci-on-${{ github.event_name }}-from-${{ github.ref_name }}
cancel-in-progress: true

on:
# run on pushes to certain branches
push:
branches:
- "main"
- "release/v[0-9][0-9].[0-9][0-9].[0-9][0-9]"
# run on pushes of new release tags
tags:
- v[0-9][0-9].[0-9][0-9].[0-9][0-9]
# run by clicking buttons in the GitHub Actions UI
workflow_dispatch:

jobs:
conda-python-build:
uses: ./.github/workflows/conda-python-build.yaml
with:
script: "ci/build_python.sh"
secrets: inherit
docs-build:
needs:
- conda-python-build
uses: ./.github/workflows/docs-build.yaml
with:
script: "ci/build_docs.sh"
deploy: true
secrets: inherit
62 changes: 62 additions & 0 deletions .github/workflows/conda-python-build.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
name: conda-python-build

concurrency:
group: conda-python-build-on-${{ github.event_name }}-from-${{ github.ref_name }}
cancel-in-progress: true

on:
# run only when called by other workflows
workflow_call:
inputs:
script:
required: true
type: string
description: "relative path to a script that builds conda packages"

env:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

everything below here is identical to what was in github-actions.yml, except that the literal ci/build_python.sh has been replaced by ${{ input.script }}, so that we have the option to use different scripts in the future.

# CUDA architectures to build for
CUDAARCHS: "all-major"
# where conda-build puts files it creates
RAPIDS_CONDA_BLD_OUTPUT_DIR: /tmp/conda-bld-output
# where jobs that download conda packages store the local channel
RAPIDS_LOCAL_CONDA_CHANNEL: /tmp/local-conda-packages

jobs:

build:
strategy:
fail-fast: false
# available legate and cunumeric packages:
#
# * https://anaconda.org/legate/legate
# * https://anaconda.org/legate/cunumeric
#
# Valid set of RAPIDS ci-conda image tags:
#
# * https://hub.docker.com/r/rapidsai/ci-conda/tags
matrix:
ARCH:
- "amd64"
CUDA_VER:
- "12.5.1"
PY_VER:
- "3.10"
- "3.11"
- "3.12"
runs-on: linux-${{ matrix.ARCH }}-cpu4
container:
image: "rapidsai/ci-conda:cuda${{ matrix.CUDA_VER }}-ubuntu22.04-py${{ matrix.PY_VER }}"
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
# the notebooks and PNG files stored in git LFS aren't necessary for package-building
lfs: false
- name: build
run: "${{ inputs.script }}"
- name: upload
uses: actions/upload-artifact@v4
with:
name: legate-boost-conda-cuda${{ matrix.CUDA_VER }}-${{ matrix.ARCH }}-py${{ matrix.PY_VER }}
path: ${{ env.RAPIDS_CONDA_BLD_OUTPUT_DIR }}
if-no-files-found: error
80 changes: 80 additions & 0 deletions .github/workflows/docs-build.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
name: docs-build

concurrency:
group: docs-build-on-${{ github.event_name }}-from-${{ github.ref_name }}
cancel-in-progress: true

on:
# run only when called by other workflows
workflow_call:
inputs:
deploy:
required: true
type: boolean
default: false
description: "set to true to publish docs"
script:
required: true
type: string
description: "relative path to a script that builds conda packages"

env:
# where jobs that download conda packages store the local channel
RAPIDS_LOCAL_CONDA_CHANNEL: /tmp/local-conda-packages

jobs:

build:
runs-on: linux-amd64-cpu4
strategy:
matrix:
include:
- ARCH: amd64
CUDA_VER: "12.5.1"
PY_VER: "3.12"
container:
image: "rapidsai/ci-conda:cuda${{ matrix.CUDA_VER }}-ubuntu22.04-py${{ matrix.PY_VER }}"
steps:
- name: install git-lfs
run: |
conda install --yes \
-c conda-forge \
git-lfs
- uses: actions/checkout@v4
with:
fetch-depth: 0
lfs: true
- name: download conda packages
uses: actions/download-artifact@v4
with:
name: legate-boost-conda-cuda${{ matrix.CUDA_VER }}-${{ matrix.ARCH }}-py${{ matrix.PY_VER }}
path: ${{ env.RAPIDS_LOCAL_CONDA_CHANNEL }}
github-token: ${{ secrets.GITHUB_TOKEN }}
repository: ${{ github.repository }}
run-id: ${{ github.run_id }}
- name: build docs
run: "${{ inputs.script }}"
- uses: actions/upload-pages-artifact@v3
with:
path: docs/build/html

deploy:
needs:
- build
if: inputs.deploy

# Grant GITHUB_TOKEN the permissions required to make a Pages deployment
permissions:
pages: write # to deploy to Pages
id-token: write # to verify the deployment originates from an appropriate source

# Deploy to the github-pages environment
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}

runs-on: ubuntu-latest
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@v4
115 changes: 10 additions & 105 deletions .github/workflows/github-actions.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,10 @@ concurrency:
cancel-in-progress: true

on:
# run on pushes to certain branches (or PRs targeting those branches)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realized that the "or PRs targeting those branches" part of this was wrong. That'd only be true with this very-similar-looking construct:

on:
  pull_request:
    branches:
      - main

(example from numpy)

But here in the rapidsai or, we avoid that by having the copy-pr-bot copy PR branches to branches named pull-request/, and only using on.push: types of conditions.

# run on pushes to certain branches
push:
branches:
- "main"
- "pull-request/[0-9]+"
- "release/v[0-9][0-9].[0-9][0-9].[0-9][0-9]"
# run on pushes of new release tags
tags:
- v[0-9][0-9].[0-9][0-9].[0-9][0-9]
# run by clicking buttons in the GitHub Actions UI
workflow_dispatch:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With these removals, this workflow will now only run on pull-request/ branches, and therefore only on PRs. The new file build.yaml (matching naming conventions from the rest of RAPIDS) will be responsible for running on merges to main or pushes of new tags.


env:
# CUDA architectures to build for
Expand Down Expand Up @@ -49,43 +42,10 @@ jobs:
conda-python-build:
needs:
- pre-commit
strategy:
fail-fast: false
# available legate and cunumeric packages:
#
# * https://anaconda.org/legate/legate
# * https://anaconda.org/legate/cunumeric
#
# Valid set of RAPIDS ci-conda image tags:
#
# * https://hub.docker.com/r/rapidsai/ci-conda/tags
matrix:
ARCH:
- "amd64"
CUDA_VER:
- "12.5.1"
PY_VER:
- "3.10"
- "3.11"
- "3.12"
runs-on: linux-${{ matrix.ARCH }}-cpu4
container:
image: "rapidsai/ci-conda:cuda${{ matrix.CUDA_VER }}-ubuntu22.04-py${{ matrix.PY_VER }}"
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
# the notebooks and PNG files stored in git LFS aren't necessary for package-building
lfs: false
- name: build
run: |
ci/build_python.sh
- name: upload
uses: actions/upload-artifact@v4
with:
name: legate-boost-conda-cuda${{ matrix.CUDA_VER }}-${{ matrix.ARCH }}-py${{ matrix.PY_VER }}
path: ${{ env.RAPIDS_CONDA_BLD_OUTPUT_DIR }}
if-no-files-found: error
uses: ./.github/workflows/conda-python-build.yaml
with:
script: "ci/build_python.sh"
secrets: inherit

conda-python-tests-cpu:
needs:
Expand Down Expand Up @@ -171,63 +131,8 @@ jobs:
docs-build:
needs:
- conda-python-build
runs-on: linux-amd64-cpu4
strategy:
matrix:
include:
- ARCH: amd64
CUDA_VER: "12.5.1"
PY_VER: "3.12"
container:
image: "rapidsai/ci-conda:cuda${{ matrix.CUDA_VER }}-ubuntu22.04-py${{ matrix.PY_VER }}"
steps:
- name: install git-lfs
run: |
conda install --yes \
-c conda-forge \
git-lfs
- uses: actions/checkout@v4
with:
fetch-depth: 0
lfs: true
- name: download conda packages
uses: actions/download-artifact@v4
with:
name: legate-boost-conda-cuda${{ matrix.CUDA_VER }}-${{ matrix.ARCH }}-py${{ matrix.PY_VER }}
path: ${{ env.RAPIDS_LOCAL_CONDA_CHANNEL }}
github-token: ${{ secrets.GITHUB_TOKEN }}
repository: ${{ github.repository }}
run-id: ${{ github.run_id }}
- name: build docs
run: |
ci/build_docs.sh
- uses: actions/upload-pages-artifact@v3
with:
path: docs/build/html

deploy:
needs:
- conda-python-build
- conda-python-tests-cpu
- conda-python-tests-gpu
- docs-build

# only main branch uploads docs
if: github.ref == 'refs/heads/main' && github.event_name == 'push'

# Grant GITHUB_TOKEN the permissions required to make a Pages deployment
permissions:
pages: write # to deploy to Pages
id-token: write # to verify the deployment originates from an appropriate source

# Deploy to the github-pages environment
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}

# Specify runner + deployment step
runs-on: ubuntu-latest
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@v4
uses: ./.github/workflows/docs-build.yaml
with:
script: "ci/build_docs.sh"
deploy: false
secrets: inherit