Skip to content

Commit

Permalink
sagemathgh-37237: CI macOS: Fix failure with macos-13-homebrew, add t…
Browse files Browse the repository at this point in the history 
…ests on M1 runners, add timeouts

    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

- Fixing the failure with `macos-13-homebrew`, as seen in
https://github.com/sagemath/sage/actions/runs/7761817710/job/21171032513
and upstream uses of macos.yml
dimpase/primecountpy#11 (comment),
sagemath/cypari2#152
- Generalizing the `ld_classic` workaround from sagemath#36599 for the case of
full XCode.
  - This is to fix the failure seen on `homebrew-macos-13-usrlocal-
standard-xcode_15.0`, e.g. in https://github.com/scipopt/SCIP-
SDP/actions/runs/7793548242/job/21253452882?pr=9#step:10:2766
- Deactivating the `ld_classic` workaround when `LD` is set, as is in a
conda environment; as seen e.g. in https://groups.google.com/g/sage-
devel/c/1viBzw-ZaoQ/m/S_u9K20xAAAJ
- [Apple Silicon runners recently became
available](https://github.blog/changelog/2024-01-30-github-actions-
introducing-the-new-m1-macos-runner-available-to-open-source/); we add
some tests (example run:
https://github.com/mkoeppe/sage/actions/runs/7772086965/job/21194110510)
- On Apple Silicon, `tox -e local-conda-forge` now uses the correct
installer
- Intel runners were recently upgraded too, increase parallelism.
- Add self-destruct sequence to cancel workflows before the 6h limit
(catches linbox-related build hang)
- Actually handle the input `extra_sage_packages` (for
scipopt/PySCIPOpt#630)

<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

Follow-ups, not addressed here:
- conda-forge-macos-latest-minimal tachyon build failure - needs sagemath#36969
- adding a config that tests arm64 conda-forge-minimal - would need
config.guess updates in numerous packages
- arm64 conda-forge-standard - matplotlib build failure (https://github.
com/mkoeppe/sage/actions/runs/7810111886/job/21313997438#step:10:2749)
- hangs or excessive build time that lead to timeouts, probably from
linbox

### :memo: Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### :hourglass: Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#37237
Reported by: Matthias Köppe
Reviewer(s): Matthias Köppe, Tobias Diez
  • Loading branch information
Release Manager committed Feb 25, 2024
2 parents 7b3701e + 2165bef commit 2ca4b2d
Show file tree
Hide file tree
Showing 5 changed files with 46 additions and 17 deletions.
10 changes: 6 additions & 4 deletions .github/workflows/ci-macos.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ jobs:
uses: ./.github/workflows/macos.yml
with:
stage: "1"
timeout: 14400

stage-2:
uses: ./.github/workflows/macos.yml
Expand Down Expand Up @@ -90,7 +91,7 @@ jobs:
run: |
git config --global user.email "nobody@example.com"
git config --global user.name "Sage GitHub CI"
SAGE_ROOT=. SAGE_SRC=./src src/bin/sage-update-version $(git describe --tags) || echo "(ignoring error)"
SAGE_ROOT=. SAGE_SRC=./src src/bin/sage-update-version $(cat src/VERSION.txt).dev0 || echo "(ignoring error)"
- name: make dist
run: |
./configure --enable-download-from-upstream-url && make dist
Expand All @@ -108,7 +109,7 @@ jobs:
fail-fast: false
max-parallel: 4
matrix:
os: [ macos-11, macos-12 ]
os: [ macos-11, macos-12, macos-14 ]
tox_system_factor: [macos-nobootstrap]
tox_packages_factor: [minimal]
xcode_version_factor: [default]
Expand All @@ -129,7 +130,7 @@ jobs:
if: contains(matrix.tox_system_factor, 'nobootstrap')
- name: Move homebrew away
run: |
(cd /usr/local && for a in bin etc include lib opt sbin share; do sudo mv $a $a-moved; done)
(cd $(brew --prefix) && for a in bin etc include lib opt sbin share; do sudo mv $a $a-moved; done)
- name: Select Xcode version
run: |
if [ ${{ matrix.xcode_version_factor }} != default ]; then sudo xcode-select -s /Applications/Xcode_${{ matrix.xcode_version_factor }}.app; fi
Expand All @@ -140,7 +141,8 @@ jobs:
# We use a high parallelization on purpose in order to catch possible parallelization bugs in the build scripts.
# For doctesting, we use a lower parallelization to avoid timeouts.
run: |
MAKE="make -j12" tox -e $TOX_ENV -- SAGE_NUM_THREADS=4 $TARGETS
(sleep 20000; pkill make) &
MAKE="make -j12" tox -e $TOX_ENV -- SAGE_NUM_THREADS=6 $TARGETS
- name: Prepare logs artifact
run: |
mkdir -p "artifacts/$LOGS_ARTIFACT_NAME"; cp -r .tox/*/log "artifacts/$LOGS_ARTIFACT_NAME"
Expand Down
33 changes: 24 additions & 9 deletions .github/workflows/macos.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,17 +19,21 @@ on:
type: string
# System configuration
osversion_xcodeversion_toxenv_tuples:
# As of 2024-02, "runs-on: macos-latest" is macos-12.
# and "runs-on: macos-14" selects the new M1 runners.
# https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories
description: 'Stringified JSON object'
default: >-
[["latest", "", "homebrew-macos-usrlocal-minimal"],
["latest", "", "homebrew-macos-usrlocal-standard"],
["11", "xcode_13.2.1", "homebrew-macos-usrlocal-minimal"],
[["11", "xcode_13.2.1", "homebrew-macos-usrlocal-minimal"],
["12", "", "homebrew-macos-usrlocal-minimal"],
["12", "", "homebrew-macos-usrlocal-standard"],
["12", "", "homebrew-macos-usrlocal-python3_xcode-standard"],
["12", "", "homebrew-macos-usrlocal-maximal"],
["13", "xcode_15.0", "homebrew-macos-usrlocal-standard"],
["latest", "", "homebrew-macos-usrlocal-maximal"],
["latest", "", "homebrew-macos-usrlocal-python3_xcode-standard"],
["14", "", "homebrew-macos-opthomebrew-standard"],
["latest", "", "conda-forge-macos-minimal"],
["latest", "", "conda-forge-macos-standard"]]
["latest", "", "conda-forge-macos-standard"],
["14", "", "conda-forge-macos-standard"]]
type: string
extra_sage_packages:
description: 'Extra Sage packages to install as system packages'
Expand All @@ -41,6 +45,10 @@ on:
free_disk_space:
default: false
type: boolean
timeout:
description: 'Elapsed time (seconds) at which to kill the build'
default: 20000
type: number
#
# For use in upstream CIs.
#
Expand Down Expand Up @@ -74,10 +82,16 @@ jobs:
repository: ${{ inputs.sage_repo }}
ref: ${{ inputs.sage_ref }}
fetch-depth: 10000

- uses: actions/setup-python@v5
# As of 2024-02-03, the macOS M1 runners do not have preinstalled python or pipx.
# Installing pipx follows the approach of https://github.com/pypa/cibuildwheel/pull/1743
id: python
with:
python-version: "3.8 - 3.12"
update-environment: false
- name: Install test prerequisites
run: |
brew install tox
"${{ steps.python.outputs.python-path }}" -m pip install pipx
- name: Download upstream artifact
uses: actions/download-artifact@v3
with:
Expand Down Expand Up @@ -129,7 +143,8 @@ jobs:
*) export TARGETS_PRE="${{ inputs.targets_pre }}" TARGETS="${{ inputs.targets }} TARGETS_OPTIONAL="${{ inputs.targets_optional }}
;;
esac
MAKE="make -j12" tox -e $TOX_ENV -- SAGE_NUM_THREADS=4 $TARGETS
(sleep ${{ inputs.timeout }}; pkill make) &
MAKE="make -j12" EXTRA_SAGE_PACKAGES="${{ inputs.extra_sage_packages }}" "${{ steps.python.outputs.python-path }}" -m pipx run tox -e $TOX_ENV -- SAGE_NUM_THREADS=6 $TARGETS
- name: Prepare logs artifact
run: |
mkdir -p "artifacts/$LOGS_ARTIFACT_NAME"; cp -r .tox/*/log "artifacts/$LOGS_ARTIFACT_NAME"
Expand Down
1 change: 1 addition & 0 deletions build/pkgs/dsdp/spkg-install.in
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
cd src
cp ../patches/CMakeLists.txt .
sdh_cmake -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_VERBOSE_MAKEFILE=ON \
-DBUILD_SHARED_LIBS=ON \
-DBLA_VENDOR=OpenBLAS \
-DBLAS_LIBRARIES="$(pkg-config --libs blas)" \
Expand Down
11 changes: 9 additions & 2 deletions src/bin/sage-env
View file
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ fi
# The compilers are set in order of priority by
# 1) environment variables
# 2) compiler installed by sage
# 3) compiler set at configuration time
# 3) compiler set at configuration time
if [ -z "$CC" ]; then
if [ -n "$SAGE_LOCAL" -a -x "$SAGE_LOCAL/bin/gcc" ]; then
CC=gcc
Expand Down Expand Up @@ -373,7 +373,14 @@ if [ -n "$SAGE_LOCAL" ]; then
# On OS X, test whether "ld-classic" is present in the installed
# version of the command-line tools. If so, we add "-ld_classic"
# to LD_FLAGS. See #36599.
if [ "$UNAME" = "Darwin" ] && [ -x "$(xcode-select -p)/usr/bin/ld-classic" ] ; then
# When the full XCode is installed and in use, for example after
# "sudo xcode-select -s /Applications/Xcode.app", then "xcode-select -p"
# gives "/Applications/Xcode.app/Contents/Developer", but "ld-classic"
# is not in the subdirectory "usr/bin/" but rather in the subdirectory
# "Toolchains/XcodeDefault.xctoolchain/usr/bin/". See #37237.
# However, if LD is set explicitly, as it is within conda on macOS,
# do not not do this.
if [ "$UNAME" = "Darwin" ] && [ -z "$LD" ] && [ -x "$(xcode-select -p)/usr/bin/ld-classic" -o -x "$(xcode-select -p)/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld-classic" ] ; then
LDFLAGS="-L$SAGE_LOCAL/lib -Wl,-ld_classic,-rpath,$SAGE_LOCAL/lib $LDFLAGS"
else
LDFLAGS="-L$SAGE_LOCAL/lib -Wl,-rpath,$SAGE_LOCAL/lib $LDFLAGS"
Expand Down
8 changes: 6 additions & 2 deletions tox.ini
View file
Original file line number Diff line number Diff line change
Expand Up @@ -512,6 +512,9 @@ setenv =
# brew caches downloaded files in ${HOME}/Library/Caches. We share it between different toxenvs.
homebrew: SYSTEM=homebrew
local-homebrew: HOMEBREW={envdir}/homebrew
# /opt/homebrew is the default install location on arm64 macOS
local-homebrew-opthomebrew: HOMEBREW=/opt/homebrew
# /usr/local is the default install location on x86_64 macOS
local-{homebrew-usrlocal,nohomebrew}: HOMEBREW=/usr/local
# local-macos-nohomebrew: "best effort" isolation to avoid using a homebrew installation in /usr/local
# We use liblzma from the macOS system - which is available but its headers are not (neither is the xz executable).
Expand All @@ -538,10 +541,11 @@ setenv =
local-conda: PATH={env:CONDA_PREFIX}/bin:/usr/bin:/bin:/usr/sbin:/sbin
local-conda: CONDA_PKGS_DIRS={env:SHARED_CACHE_DIR}/conda_pkgs
local-conda: CONDA_OS=$(uname | sed 's/^Darwin/MacOSX/;')
local-conda: CONDA_ARCH=$(uname -m)
local-conda-forge: CONDA_INSTALLER_URL_BASE=https://github.com/conda-forge/miniforge/releases/latest/download/
local-conda-forge: CONDA_INSTALLER_FILE=Miniforge3-{env:CONDA_OS}-x86_64.sh
local-conda-forge: CONDA_INSTALLER_FILE=Miniforge3-{env:CONDA_OS}-{env:CONDA_ARCH}.sh
local-conda-miniconda: CONDA_INSTALLER_URL_BASE=https://repo.anaconda.com/miniconda/
local-conda-miniconda: CONDA_INSTALLER_FILE=Miniconda3-latest-{env:CONDA_OS}-x86_64.sh
local-conda-miniconda: CONDA_INSTALLER_FILE=Miniconda3-latest-{env:CONDA_OS}-{env:CONDA_ARCH}.sh
local-conda: SETENV=. {env:CONDA_PREFIX}/bin/activate base
local-conda-environment: CONDA_SAGE_ENVIRONMENT=sage-build
local-conda-environment: CONDA_SAGE_ENVIRONMENT_DIR=
Expand Down

0 comments on commit 2ca4b2d

Please sign in to comment.