Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

install diff-filter-buildkite-plugin to only build when relevant files changed #400

Open
wants to merge 20 commits into
base: main
Choose a base branch
from
Open

install diff-filter-buildkite-plugin to only build when relevant files changed #400

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
6d9e107
install diff-filter-build
fatteneder Dec 2, 2024
e535e47
add diff-filter-doc
fatteneder Nov 2, 2024
fa2823c
update *upload_jobs.yml
fatteneder Dec 2, 2024
55bfc6a
rm diff-filter-doc, but build at least x86_64-linux
fatteneder Dec 2, 2024
b4e235a
add option to force build using a PR label
fatteneder Dec 2, 2024
1a2ac8f
rename trigger variable
fatteneder Dec 2, 2024
bdf3c1a
update ignore list
fatteneder Nov 6, 2024
e46f441
add missing BUILD exports
fatteneder Nov 6, 2024
4f5f000
provide bash script as string for docker plugin command
fatteneder Nov 6, 2024
c10892f
fix an if condition
fatteneder Nov 6, 2024
b954379
can we use the step's command instead of docker's?
fatteneder Nov 7, 2024
16f7be9
try without multiline strings
fatteneder Nov 9, 2024
4ab79d4
try with cmd.exe if
fatteneder Nov 9, 2024
0d9c02a
set diff-filter's target_branch option conditionally using environmen…
fatteneder Nov 9, 2024
398dce6
use grep -E everywhere for compatability reasons
fatteneder Nov 10, 2024
5ca0b2c
add an env var for the target branch
fatteneder Nov 11, 2024
f300c57
forward env vars to upload script
fatteneder Dec 2, 2024
5456f8e
fix varname
fatteneder Dec 2, 2024
809798b
also trigger build if PR gets merged
fatteneder Dec 2, 2024
7627e90
Revert "forward env vars to upload script"
fatteneder Dec 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .buildkite/hooks/post-checkout
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,3 +72,8 @@ fi
# Export S3_BUCKET and S3_BUCKET_PREFIX, to force our CI to upload to a different S3 target than the actual CI pipeline would.
export S3_BUCKET="julialang-ephemeral"
export S3_BUCKET_PREFIX="julia-buildkite-uploads/bin"

# for PRs from a julia-buildkite repo we have to set the target branch manually
if [[ -n "$(echo ${BUILDKITE_PULL_REQUEST_REPO} | grep -E "julia-buildkite\.git$")" ]]; then
export DIFF_FILTER_TARGET_BRANCH="master"
fi
188 changes: 121 additions & 67 deletions pipelines/main/launch_unsigned_jobs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,24 @@
# and only need to touch the webui configuration when we need to alter
# something about the privileged steps.

env:
Copy link
Member Author

@fatteneder fatteneder Nov 11, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only needed as this repo's main branch is called "main" whereas the one of the Julia repo is called "master".

Using a default of DIFF_FILTER_TARGET_BRANCH="" tells the diff-filter-buildkite-plugin to use a PR's merge base as the reference against which changes should be compared (https://github.com/fatteneder/diff-filter-buildkite-plugin/blob/7155804f7ba73a9798b4c4225613ca6a5d7d8c12/hooks/diff-filter-impl#L13-L38).

So if the pipeline run's on forks of Julia's repo, it should do the right thing (although untested) even when PRing against a branch other than master. For PRs against julia-buildkite we always compare against Julia's master by manually setting it in .buildkite/hooks/post-checkout.

DIFF_FILTER_TARGET_BRANCH: ""

common:
- diff-filter-build_plugin: &diff-filter-build
https://github.com/fatteneder/diff-filter-buildkite-plugin#main:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this should be moved to this organisation before merge.

name: "TRIGGER_BUILD"
ignore:
- "*.md"
- ".*"
- "julia.spdx.json"
- "CITATION.*"
- "typos.toml"
target_branch: "${DIFF_FILTER_TARGET_BRANCH?}"
- pr-labels_plugin: &pr-labels
sv-oss/github-pr-labels#v0.0.2:
publish-env-var: PULL_REQUEST_LABELS

steps:
- group: "Build"
notify:
Expand All @@ -33,35 +51,45 @@ steps:
# that this script doesn't suddenly break when a new patch release of Julia
# is released.
version: '1.10.6'
- *diff-filter-build
- *pr-labels
commands: |
### Launch Linux build jobs.
FORCE=$(test -n "$(echo $${PULL_REQUEST_LABELS} | grep -E "\bci-force-build\b")" && echo 1 || echo 0)
MERGED=$(test "$${BUILDKITE_PULL_REQUEST}" = "false" && echo 1 || echo 0)
BUILD=$(($${TRIGGER_BUILD} || $${FORCE} || $${MERGED}))
### Launch Linux build jobs
# Regular:
GROUP="Build" \
ALLOW_FAIL="false" \
BUILD="$${BUILD}" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/build_linux.arches \
.buildkite/pipelines/main/platforms/build_linux.yml
# PowerPC (only for Julia prior to 1.12):
GROUP="Build" \
ALLOW_FAIL="false" \
BUILD="$${BUILD}" \
julia .buildkite/pipelines/main/platforms/launch_powerpc.jl \
.buildkite/pipelines/main/platforms/build_linux.powerpc.arches \
.buildkite/pipelines/main/platforms/build_linux.yml
### Launch macOS build jobs:
### Launch macOS packaging jobs
GROUP="Build" \
ALLOW_FAIL="false" \
BUILD="$${BUILD}" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/build_macos.arches \
.buildkite/pipelines/main/platforms/build_macos.yml
### Launch FreeBSD build jobs:
GROUP="Build" \
ALLOW_FAIL="false" \
BUILD="$${BUILD}" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/build_freebsd.arches \
.buildkite/pipelines/main/platforms/build_freebsd.yml
### Launch Windows build jobs:
GROUP="Build" \
ALLOW_FAIL="false" \
BUILD="$${BUILD}" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/build_windows.arches \
.buildkite/pipelines/main/platforms/build_windows.yml
Expand All @@ -79,17 +107,24 @@ steps:
- JuliaCI/external-buildkite#v1:
version: "./.buildkite-external-version"
repo_url: "https://github.com/JuliaCI/julia-buildkite"
- *diff-filter-build
- *pr-labels
commands: |
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/analyzegc.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/doctest.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/pdf_docs/build_pdf_docs.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/embedding.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/trimming.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/llvmpasses.yml
# buildkite-agent pipeline upload .buildkite/pipelines/main/misc/whitespace.yml # Currently runs in GitHub Actions instead of Buildkite
FORCE=$(test -n "$(echo $${PULL_REQUEST_LABELS} | grep -E "\bci-force-build\b")" && echo 1 || echo 0)
MERGED=$(test "$${BUILDKITE_PULL_REQUEST}" = "false" && echo 1 || echo 0)
BUILD=$(($${TRIGGER_BUILD} || $${FORCE} || $${MERGED}))
if [[ $${BUILD} == 1 ]]; then
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/analyzegc.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/embedding.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/trimming.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/llvmpasses.yml
# buildkite-agent pipeline upload .buildkite/pipelines/main/misc/whitespace.yml # Currently runs in GitHub Actions instead of Buildkite

buildkite-agent pipeline upload .buildkite/pipelines/main/misc/sanitizers/asan.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/sanitizers/tsan.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/sanitizers/asan.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/sanitizers/tsan.yml
fi
agents:
queue: "julia"
os: "linux"
Expand All @@ -103,37 +138,43 @@ steps:
- JuliaCI/external-buildkite#v1:
version: "./.buildkite-external-version"
repo_url: "https://github.com/JuliaCI/julia-buildkite"
- *diff-filter-build
- *pr-labels
commands: |
export ALLOW_FAIL="false"
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/gcext.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/test_revise.yml

### Launch Linux test jobs.
# Regular:
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_linux.arches \
.buildkite/pipelines/main/platforms/test_linux.yml
# i686-linux-gnu:
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_linux.i686.arches \
.buildkite/pipelines/main/platforms/test_linux.i686.yml
### Launch macOS test jobs:
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_macos.arches \
.buildkite/pipelines/main/platforms/test_macos.yml
### Launch FreeBSD test jobs:
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_freebsd.arches \
.buildkite/pipelines/main/platforms/test_freebsd.yml
### Launch Windows test jobs:
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_windows.arches \
.buildkite/pipelines/main/platforms/test_windows.yml
FORCE=$(test -n "$(echo $${PULL_REQUEST_LABELS} | grep -E "\bci-force-build\b")" && echo 1 || echo 0)
MERGED=$(test "$${BUILDKITE_PULL_REQUEST}" = "false" && echo 1 || echo 0)
BUILD=$(($${TRIGGER_BUILD} || $${FORCE} || $${MERGED}))
if [[ $${BUILD} == 1 ]]; then
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/gcext.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/test_revise.yml
### Launch Linux test jobs.
# Regular:
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_linux.arches \
.buildkite/pipelines/main/platforms/test_linux.yml
# i686-linux-gnu:
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_linux.i686.arches \
.buildkite/pipelines/main/platforms/test_linux.i686.yml
### Launch macOS test jobs
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_macos.arches \
.buildkite/pipelines/main/platforms/test_macos.yml
### Launch FreeBSD test jobs
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_freebsd.arches \
.buildkite/pipelines/main/platforms/test_freebsd.yml
### Launch windows test jobs
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_windows.arches \
.buildkite/pipelines/main/platforms/test_windows.yml
fi
agents:
queue: "julia"
os: "linux"
Expand All @@ -144,15 +185,22 @@ steps:
- JuliaCI/external-buildkite#v1:
version: "./.buildkite-external-version"
repo_url: "https://github.com/JuliaCI/julia-buildkite"
- *diff-filter-build
- *pr-labels
commands: |
export ALLOW_FAIL="true"
# Launch Linux allowed-to-fail build jobs:
FORCE=$(test -n "$(echo $${PULL_REQUEST_LABELS} | grep -E "\bci-force-build\b")" && echo 1 || echo 0)
MERGED=$(test "$${BUILDKITE_PULL_REQUEST}" = "false" && echo 1 || echo 0)
BUILD=$(($${TRIGGER_BUILD} || $${FORCE} || $${MERGED}))
# Launch Linux allowed-to-fail build jobs
GROUP="Allow Fail" \
BUILD="$${BUILD}" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/build_linux.soft_fail.arches \
.buildkite/pipelines/main/platforms/build_linux.yml
# Launch macOS allowed-to-fail build jobs:
# Launch macOS allowed-to-fail build jobs
GROUP="Allow Fail" \
BUILD="$${BUILD}" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/build_macos.soft_fail.arches \
.buildkite/pipelines/main/platforms/build_macos.yml
Expand All @@ -174,35 +222,41 @@ steps:
# that this script doesn't suddenly break when a new patch release of Julia
# is released.
version: '1.10.6'
- *diff-filter-build
- *pr-labels
commands: |
export ALLOW_FAIL="true"

### Launch Linux allowed-to-fail test jobs.
# Regular:
GROUP="Allow Fail" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_linux.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_linux.yml
# PowerPC (only for Julia prior to 1.12):
GROUP="Test" \
julia .buildkite/pipelines/main/platforms/launch_powerpc.jl \
.buildkite/pipelines/main/platforms/test_linux.powerpc.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_linux.yml
### Launch macOS allowed-to-fail test jobs:
GROUP="Allow Fail" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_macos.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_macos.yml
### Launch FreeBSD allowed-to-fail jobs:
GROUP="Allow Fail" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_freebsd.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_freebsd.yml
### Launch Windows allowed-to-fail test jobs:
GROUP="Allow Fail" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_windows.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_windows.yml
FORCE=$(test -n "$(echo $${PULL_REQUEST_LABELS} | grep -E "\bci-force-build\b")" && echo 1 || echo 0)
MERGED=$(test "$${BUILDKITE_PULL_REQUEST}" = "false" && echo 1 || echo 0)
BUILD=$(($${TRIGGER_BUILD} || $${FORCE} || $${MERGED}))
if [[ $${BUILD} == 1 ]]; then
### Launch Linux allowed-to-fail test jobs.
# Regular:
GROUP="Allow Fail" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_linux.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_linux.yml
# PowerPC (only for Julia prior to 1.12):
GROUP="Test" \
julia .buildkite/pipelines/main/platforms/launch_powerpc.jl \
.buildkite/pipelines/main/platforms/test_linux.powerpc.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_linux.yml
### Launch macOS allowed-to-fail test jobs
GROUP="Allow Fail" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_macos.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_macos.yml
### Launch FreeBSD allowed-to-fail jobs
GROUP="Allow Fail" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_freebsd.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_freebsd.yml
### Launch windows allowed-to-fail test jobs
GROUP="Allow Fail" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_windows.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_windows.yml
fi
agents:
queue: "julia"
os: "linux"
Loading