Move x86 jobs to GHA #169

nulano · 2020-10-30T16:50:30Z

This PR moves the x86 and x86_64 builds to GHA (needs to be manually enabled it seems). Unlike Travis CI, which has 5 parallel jobs, and the .org version has been slow lately, GHA gives 20 parallel jobs (max 5 macOS).

The GHA build of 20 Ubuntu and 10 macOS jobs takes ~30m, barely slower than the remaining 8 Arm64 builds on Travis.org (~26m) if queued immediately. Note that Arm64 seems to have a separate queue and is less affected by the slowdown.

Also adds a Linux 32-bit PyPy wheel, as I do not see a reason it was not added in #153.

The macOS build uses the default XCode, currently 12.0.1. actions/runner-images#1712 shows how to pin to a specific version if necessary. XCode 10.3 is the oldest version currently available: https://github.com/actions/virtual-environments/blob/main/images/macos/macos-10.15-Readme.md#xcode

More details are posted per-line below.

nulano · 2020-10-30T16:51:27Z

.github/workflows/wheels.yml

+      # Uncomment to get SSH access for testing
+      # - name: Setup tmate session
+      #   if: failure()
+      #   uses: mxschmitt/action-tmate@v3


I found this action to be invaluable in debugging the macOS build, as I do not have any other way of using a macOS machine.

nulano · 2020-10-30T16:52:28Z

.github/workflows/wheels.yml

+  build-latest:
+    name: ${{ matrix.python }} ${{ matrix.os-name }} ${{ matrix.platform }} latest
+    runs-on: ${{ matrix.os }}
+    if: "!startsWith(github.ref, 'refs/tags/')"


Latest only runs on non-tagged builds as a second job. This requires duplicating the build matrix.

This is not as necessary on GHA as on Travis due to the speed difference, but I tried to match the Travis jobs as closely as possible for now.

I also put PyPy first to give it a head-start on macOS, it is by far the slowest job at ~15m.

nulano · 2020-10-30T16:58:23Z

.github/workflows/wheels.yml

+      - name: Upload Release
+        uses: fnkr/github-action-ghr@v1.3
+        env:
+          GHR_PATH: .
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}


The "official" actions for this by GHA are no longer actively supported, actions/upload-release-asset#58 (comment). They also break badly if a release already exists, and don't support wildcards.

Testing, I have found this action to work well, the only issue is if a successful build is re-run for whatever reason, with it failing to upload duplicate filenames. The ${{ secrets.GITHUB_TOKEN }} is provided automatically.

nulano · 2020-10-30T16:59:07Z

config.sh

-
-        ORIGINAL_CPPFLAGS=$CPPFLAGS
-        CPPFLAGS=""
-    fi


I have found that GHA no longer seems to require #104.

nulano · 2020-10-30T17:00:16Z

.github/workflows/build.sh

+
+if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
+  # these cause a conflict with built webp and libtiff
+  brew remove --ignore-dependencies webp zstd xz libtiff


This resolves #133 which reappears on GHA.

nulano · 2020-10-30T17:18:12Z

I just noticed that the GHA macOS PyPy test doesn't have ImageTk: https://github.com/nulano/pillow-wheels/runs/1328473365?check_suite_focus=true#step:4:6002
Latest master build on Travis had it: https://travis-ci.org/github/python-pillow/pillow-wheels/jobs/738462521#L5371
Likely related to https://foss.heptapod.net/pypy/pypy/-/issues/3229

Edit: Added brew install tcl-tk as workaround for now.

hugovk · 2020-10-31T21:14:47Z

Thank you for this!

First glance looks good, will take a closer look hopefully this week.

So this reduces build time from ~1h 20m to (~30m + ~26m in parallel => ) ~30m 👍

umarcor · 2020-11-03T09:24:03Z

You might find dbhi/qus useful for migrating ARM jobs to GHA too.

hugovk · 2020-11-05T20:46:40Z

The macOS build uses the default XCode, currently 12.0.1. actions/virtual-environments#1712 shows how to pin to a specific version if necessary. XCode 10.3 is the oldest version currently available: https://github.com/actions/virtual-environments/blob/main/images/macos/macos-10.15-Readme.md#xcode

Travis currently uses xcode9.3 but also has MACOSX_DEPLOYMENT_TARGET=10.10.

I guess it's fine to build GHA on Xcode 12.0.1 as it also sets MACOSX_DEPLOYMENT_TARGET=10.10 like in #104?

Should we also pin to an Xcode version for stability?

nulano · 2020-11-05T22:35:41Z

Travis currently uses xcode9.3 but also has MACOSX_DEPLOYMENT_TARGET=10.10.

I guess it's fine to build GHA on Xcode 12.0.1 as it also sets MACOSX_DEPLOYMENT_TARGET=10.10 like in #104?

It might be a good idea to actually test on an older machine, if anyone has access to one.

Should we also pin to an Xcode version for stability?

GHA periodically remove older versions (e.g. actions/runner-images#1688), so I'm not sure if pinning will cause more issues or less. Also, IIUC the Travis images are different for each XCode version, while on GHA it is always the same image, just a different XCode version. My thinking is that it might be better to leave it at default until a problem comes up.

hugovk · 2020-11-16T18:09:14Z

Oldest I have is 10.13.6 High Sierra, installed fine for Python 3.7.2 (using wheels.zip from https://github.com/nulano/pillow-wheels/actions/runs/350516695) and a quick sanity test was good:

>>> from PIL import Image
>>> im = Image.new("RGB", (300, 30), "#fff")
>>> im.show()

Also good on 10.14.6 Mojave with Python 3.9.0.

Let's merge. Thanks!

nulano added 6 commits October 23, 2020 16:27

move x86 jobs to GHA

0364455

create release

6991e05

move macOS to GHA

6466660

run both tagged and latest

39a3662

create event is redundant

f33661a

put PyPy first

b3f1056

nulano commented Oct 30, 2020

View reviewed changes

install Tcl/Tk for PyPy on macOS

2dc6e41

hugovk mentioned this pull request Nov 2, 2020

The new pricing model for Travis CI python-pillow/Pillow#5028

Closed

5 tasks

hugovk approved these changes Nov 5, 2020

View reviewed changes

radarhere mentioned this pull request Nov 6, 2020

Updated to Python 3.9 nulano/pillow-wheels#1

Merged

Updated to Python 3.9

ff8ba31

hugovk merged commit 4462509 into python-pillow:master Nov 16, 2020

hugovk mentioned this pull request Nov 21, 2020

question about travis ci new pricing plans multi-build/multibuild#364

Closed

riddell-stan mentioned this pull request Dec 1, 2020

Wheel building must be moved off of Travis CI stan-dev/httpstan-wheels#59

Closed

radarhere mentioned this pull request Dec 3, 2020

Added GitHub Actions badge #172

Merged

nulano deleted the gha branch September 7, 2022 23:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move x86 jobs to GHA #169

Move x86 jobs to GHA #169

nulano commented Oct 30, 2020 •

edited

Loading

nulano Oct 30, 2020

nulano Oct 30, 2020

nulano Oct 30, 2020

nulano Oct 30, 2020

nulano Oct 30, 2020

nulano commented Oct 30, 2020 •

edited

Loading

hugovk commented Oct 31, 2020

umarcor commented Nov 3, 2020

hugovk commented Nov 5, 2020

nulano commented Nov 5, 2020

hugovk commented Nov 16, 2020 •

edited

Loading

Move x86 jobs to GHA #169

Move x86 jobs to GHA #169

Conversation

nulano commented Oct 30, 2020 • edited Loading

nulano Oct 30, 2020

Choose a reason for hiding this comment

nulano Oct 30, 2020

Choose a reason for hiding this comment

nulano Oct 30, 2020

Choose a reason for hiding this comment

nulano Oct 30, 2020

Choose a reason for hiding this comment

nulano Oct 30, 2020

Choose a reason for hiding this comment

nulano commented Oct 30, 2020 • edited Loading

hugovk commented Oct 31, 2020

umarcor commented Nov 3, 2020

hugovk commented Nov 5, 2020

nulano commented Nov 5, 2020

hugovk commented Nov 16, 2020 • edited Loading

nulano commented Oct 30, 2020 •

edited

Loading

nulano commented Oct 30, 2020 •

edited

Loading

hugovk commented Nov 16, 2020 •

edited

Loading