Skip to content

Commit

Permalink
sagemathgh-36686: .ci/merge-fixes.sh: Obtain patches via URL, make …
Browse files Browse the repository at this point in the history
…customizable by repository variable

    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->
Instead of using `gh pr checkout`, we obtain the CI fixes via their
patch URLs.

- This is faster because typically we do not have to unshallow the repo
to apply the patches (seconds instead of ~2 minutes)
- Fewer surprises when applied to a PR based on an older release
- Conjecturally uses fewer API queries, helping avoid sagemath#36685

When a repository variable `SAGE_CI_FIXES_FROM_REPOSITORIES` is set in a
fork, it is used instead of the hardcoded sagemath/sage as the source(s)
of the CI fixes; this gives better control in decentralized development.
When set to "none", this also makes it possible to see the "ground
truth", addressing a concern raised in
sagemath#36349 (comment). See
comments at the top of `.ci/merge-fixes.sh` for details.

Also improving the display of details of the merged PRs, as requested by
@tornaria in
sagemath#36696 (comment)

Example runs:
- https://github.com/sagemath/sage/actions/runs/6854931832/job/186389018
59?pr=36686 (here it recovers gracefully from failed merges due to the
10.2.rc1/10.2.rc2 tagging confusions)
- https://github.com/mkoeppe/sage/actions/runs/6855410957/job/1864038039
8#step:3:8 (I have set `SAGE_CI_FIXES_FROM_REPOSITORIES=none` in my fork
-- for testing this PR)

<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#36686
Reported by: Matthias Köppe
Reviewer(s): Gonzalo Tornaría, Kwankyu Lee, Matthias Köppe
  • Loading branch information
Release Manager committed Dec 6, 2023
2 parents 8278562 + 49b7f74 commit a706971
Show file tree
Hide file tree
Showing 8 changed files with 87 additions and 33 deletions.
112 changes: 79 additions & 33 deletions .ci/merge-fixes.sh
Original file line number Diff line number Diff line change
@@ -1,36 +1,82 @@
#!/bin/sh
# Merge open PRs from sagemath/sage labeled "blocker".
REPO="sagemath/sage"
GH="gh -R $REPO"
PRs="$($GH pr list --label "p: blocker / 1" --json number --jq '.[].number')"
if [ -z "$PRs" ]; then
echo 'Nothing to do: Found no open PRs with "blocker" status.'
else
echo "Found PRs: " $PRs
export GIT_AUTHOR_NAME="ci-sage workflow"
export GIT_AUTHOR_EMAIL="ci-sage@example.com"
export GIT_COMMITTER_NAME="$GIT_AUTHOR_NAME"
export GIT_COMMITTER_EMAIL="$GIT_AUTHOR_EMAIL"
git tag -f test_base
git commit -q -m "Uncommitted changes" --no-allow-empty -a
for a in $PRs; do
git fetch --unshallow --all > /dev/null 2>&1 && echo "Unshallowed."
echo "::group::Merging PR https://github.com/$REPO/pull/$a"
git tag -f test_head
$GH pr checkout -b pr-$a $a
git checkout -q test_head
if git merge --no-edit --squash -q pr-$a; then
echo "::endgroup::"
if git commit -q -m "Merge https://github.com/$REPO/pull/$a" -a --no-allow-empty; then
echo "Merged #$a"
# Apply open PRs labeled "blocker" from sagemath/sage as patches.
# This script is invoked by various workflows in .github/workflows
#
# The repository variable SAGE_CI_FIXES_FROM_REPOS can be set
# to customize this for CI runs in forks:
#
# - If set to a whitespace-separated list of repositories, use them instead of sagemath/sage.
# - If set to "none", do not apply any PRs.
#
# https://docs.github.com/en/actions/learn-github-actions/variables#creating-configuration-variables-for-a-repository
export GIT_AUTHOR_NAME="ci-sage workflow"
export GIT_AUTHOR_EMAIL="ci-sage@example.com"
export GIT_COMMITTER_NAME="$GIT_AUTHOR_NAME"
export GIT_COMMITTER_EMAIL="$GIT_AUTHOR_EMAIL"
mkdir -p upstream
for REPO in ${SAGE_CI_FIXES_FROM_REPOSITORIES:-sagemath/sage}; do
case $REPO in
none)
echo "Nothing to do for 'none' in SAGE_CI_FIXES_FROM_REPOSITORIES"
;;
*/*)
echo "Getting open PRs with 'blocker' status from https://github.com/$REPO/pulls?q=is%3Aopen+label%3A%22p%3A+blocker+%2F+1%22"
GH="gh -R $REPO"
REPO_FILE="upstream/ci-fixes-${REPO%%/*}-${REPO##*/}"
PRs="$($GH pr list --label "p: blocker / 1" --json number --jq '.[].number' | tee $REPO_FILE)"
date -u +"%Y-%m-%dT%H:%M:%SZ" > $REPO_FILE.date # Record the date, for future reference
if [ -z "$PRs" ]; then
echo "Nothing to do: Found no open PRs with 'blocker' status in $REPO."
else
echo "Empty, skipped"
echo "Found open PRs with 'blocker' status in $REPO: $(echo $PRs)"
git tag -f test_base
git commit -q -m "Uncommitted changes" --no-allow-empty -a
for a in $PRs; do
# We used to pull the branch and merge it (after unshallowing), but when run on PRs that are
# based on older releases, it has the side effect of updating to this release,
# which may be confusing.
#
# Here, we obtain the "git am"-able patch of the PR branch.
# This also makes it unnecessary to unshallow the repository.
#
# Considered alternative: Use https://github.com/$REPO/pull/$a.diff,
# which squashes everything into one diff without commit metadata.
PULL_URL="https://github.com/$REPO/pull/$a"
PULL_FILE="$REPO_FILE-$a"
PATH=build/bin:$PATH build/bin/sage-download-file --quiet "$PULL_URL.patch" $PULL_FILE.patch
date -u +"%Y-%m-%dT%H:%M:%SZ" > $PULL_FILE.date # Record the date, for future reference
LAST_SHA=$(sed -n -E '/^From [0-9a-f]{40}/s/^From ([0-9a-f]{40}).*/\1/p' $PULL_FILE.patch | tail -n 1)
COMMITS_URL="https://github.com/$REPO/commits/$LAST_SHA"
echo "::group::Applying PR $PULL_URL @ $COMMITS_URL as a patch"
export GIT_COMMITTER_NAME="$GIT_AUTHOR_NAME applying $PULL_URL @ $COMMITS_URL"
if git am --signoff --empty=keep < $PULL_FILE.patch; then
echo "---- Applied patch --------------------------------------------------------------------------------"
cat $PULL_FILE.patch
echo "--------------------------------------------------------------------8<-----------------------------"
echo "::endgroup::"
elif git am --abort \
&& if git fetch --unshallow --all > /dev/null 2>&1; then echo "Unshallowed"; fi \
&& echo "Retrying with 3-way merge" \
&& git am --empty=keep --3way < $PULL_FILE.patch; then
echo "---- Applied patch --------------------------------------------------------------------------------"
cat $PULL_FILE.patch
echo "--------------------------------------------------------------------8<-----------------------------"
echo "::endgroup::"
else
echo "---- Failure applying patch -----------------------------------------------------------------------"
git am --signoff --show-current-patch=diff
echo "--------------------------------------------------------------------8<-----------------------------"
echo "::endgroup::"
echo "Failure applying $PULL_URL as a patch, resetting"
git am --signoff --abort
fi
done
fi
else
echo "::endgroup::"
echo "Failure merging #$a, resetting"
git reset --hard
fi
done
git log test_base..HEAD
fi
;;
*)
echo "Repository variable SAGE_CI_FIXES_FROM_REPOSITORIES, if set, should be a whitespace-separated list of repositories or 'none'"
echo "Got: $SAGE_CI_FIXES_FROM_REPOSITORIES"
exit 1
;;
esac
done
1 change: 1 addition & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ jobs:
.ci/merge-fixes.sh
env:
GH_TOKEN: ${{ github.token }}
SAGE_CI_FIXES_FROM_REPOSITORIES: ${{ vars.SAGE_CI_FIXES_FROM_REPOSITORIES }}
- name: Store CI fixes in upstream artifact
run: |
mkdir -p upstream
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/ci-conda.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ jobs:
.ci/merge-fixes.sh
env:
GH_TOKEN: ${{ github.token }}
SAGE_CI_FIXES_FROM_REPOSITORIES: ${{ vars.SAGE_CI_FIXES_FROM_REPOSITORIES }}

- name: Create conda environment files
run: ./bootstrap-conda
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/doc-build-pdf.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ jobs:
.ci/merge-fixes.sh
env:
GH_TOKEN: ${{ github.token }}
SAGE_CI_FIXES_FROM_REPOSITORIES: ${{ vars.SAGE_CI_FIXES_FROM_REPOSITORIES }}
- name: Store CI fixes in upstream artifact
run: |
mkdir -p upstream
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/doc-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ jobs:
.ci/merge-fixes.sh
env:
GH_TOKEN: ${{ github.token }}
SAGE_CI_FIXES_FROM_REPOSITORIES: ${{ vars.SAGE_CI_FIXES_FROM_REPOSITORIES }}
- name: Store CI fixes in upstream artifact
run: |
mkdir -p upstream
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,8 @@ jobs:
.ci/merge-fixes.sh
env:
GH_TOKEN: ${{ github.token }}
SAGE_CI_FIXES_FROM_REPOSITORIES: ${{ vars.SAGE_CI_FIXES_FROM_REPOSITORIES }}

- name: Show disk space
run: |
df -h
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ jobs:
.ci/merge-fixes.sh
env:
GH_TOKEN: ${{ github.token }}
SAGE_CI_FIXES_FROM_REPOSITORIES: ${{ vars.SAGE_CI_FIXES_FROM_REPOSITORIES }}

- name: Set up Python
uses: actions/setup-python@v4
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/macos.yml
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ jobs:
.ci/merge-fixes.sh
env:
GH_TOKEN: ${{ github.token }}
SAGE_CI_FIXES_FROM_REPOSITORIES: ${{ vars.SAGE_CI_FIXES_FROM_REPOSITORIES }}

- uses: actions/download-artifact@v3
with:
Expand Down

0 comments on commit a706971

Please sign in to comment.