Bump workflow actions for node20 #21133

kaos · 2024-07-06T06:13:09Z

We started to run into issues with node16 no longer being functional on the runner, with symptoms like actions/checkout#1809

…-java, actions/setup-node, and actions/setup-python.

This is a potentially breaking change. See https://github.com/actions/upload-artifact/blob/main/docs/MIGRATION.md

kaos · 2024-07-06T06:33:08Z

Hmm.. so, this is still a good change, but it doesn't address the underlying issue, which seems to be node20 being broken on the linux runners we use.

Release run to build wheels using this workflow: https://github.com/pantsbuild/pants/actions/runs/9817274677/job/27108461020

huonw

The actual change here looks reasonable... although I have not checked each action for breaking changes other than the upload-artifact one I already knew about.

huonw · 2024-07-10T10:37:28Z

src/python/pants_release/generate_github_workflows.py

@@ -19,6 +19,23 @@

 from pants.util.strutil import softwrap

+ACTION = {


Nice idea to centralise these versions 👍

huonw · 2024-07-10T10:40:27Z

src/python/pants_release/generate_github_workflows.py

+    "setup-protoc": "arduino/setup-protoc@9b1ee5b22b0a3f1feb8c2ff99b32c89b3c3191e9",
+    "setup-python": "actions/setup-python@v5",
+    "slack-github-action": "slackapi/slack-github-action@v1.24.0",
+    "upload-artifact": "actions/upload-artifact@v4",


At $WORK, we were tripped up by one of the breaking changes in this upgrade: https://github.com/actions/upload-artifact?tab=readme-ov-file#breaking-changes

Uploading to the same named Artifact multiple times.

Due to how Artifacts are created in this new version, it is no longer possible to upload to the same named Artifact multiple times. You must either split the uploads into multiple Artifacts with different names, or only upload once. Otherwise you will encounter an error.

Does Pants encounter this?

Maybe that is this why there's the overwrite: true added in upload_log_artifacts?

yea, that was my attempt at handling that, suspecting that we want the "latest" logs always. Other than that 🤷🏽 so figured I run some workflows manually using this version, once we're passed the other CI issues (the failing build wheels jobs).

huonw · 2024-07-10T10:52:04Z

Hmm.. so, this is still a good change, but it doesn't address the underlying issue, which seems to be node20 being broken on the linux runners we use.

Argh, looks like we might end up somewhat jammed by platform support.

The error message suggests node20 on Linux requires glibc 2.23 or newer.
We run these builds in a manylinux2014 container, presumably for platform support. https://peps.python.org/pep-0599/ suggest manylinux2014 => glibc 2.17.

pants/src/python/pants_release/generate_github_workflows.py

Lines 854 to 865 in 9d77334

    
           # For manylinux compatibility, we build Linux wheels in a container rather than directly 
        
           # on the Ubuntu runner. As a result, we have custom steps here to check out 
        
           # the code, install rustup and expose Pythons. 
        
           # TODO: Apply rust caching here. 
        
           if platform == Platform.LINUX_X86_64: 
        
               container = {"image": "quay.io/pypa/manylinux2014_x86_64:latest"} 
        
           elif platform == Platform.LINUX_ARM64: 
        
               # Unfortunately Equinix do not support the CentOS 7 image on the hardware we've been 
        
               # generously given by the Works on ARM program. So we have to build in this image. 
        
               container = { 
        
                   "image": "ghcr.io/pantsbuild/wheel_build_aarch64:v3-8384c5cf", 
        
               }

Thus, since 2.17 < 2.23, we're stuck...

An idea we might need to separate the "build orchestration" (and all the various actions) and the actual builds:

have the overall build run within a GHA-provided "default" image
run the actual pants commands within the compatibility image, either at the GHA level, or maybe via a docker_environment!

I'm not sure the best way to actually do this.

kaos · 2024-07-10T12:05:47Z

Want to hold off merging until we've tested these workflows properly, as there are breaking changes in at least the upload-artifact action.

benjyw · 2024-07-10T16:30:45Z

Can we run in a new container than manylinux2014 ?

benjyw · 2024-07-10T16:31:23Z

Oh, and why do we care about node versions? Is this our own JS backend, or related to various actions being written in JS?

huonw · 2024-07-10T19:23:07Z

Can we run in a new container than manylinux2014 ?

At the cost of reducing platform support. I believe Pants will currently runs on Linux platforms with glibc 2.17 - 2.22, but may not if we bump the image?

kaos · 2024-07-11T07:58:17Z

Oh, and why do we care about node versions? Is this our own JS backend, or related to various actions being written in JS?

Github Actions: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/

benjyw · 2024-07-11T10:18:16Z

And we're running actions inside the container? I guess we need the action to run natively but build in a container?

kaos · 2024-07-11T13:48:40Z

And we're running actions inside the container? I guess we need the action to run natively but build in a container?

Well, the linux wheel build jobs specify a container image, so I guess the entire job is executed in it, and thus the GH actions to clone the repo sources:

  build_wheels_linux_x86_64:
    container:
      image: quay.io/pypa/manylinux2014_x86_64:latest
    ...
  build_wheels_linux_arm64:
    container:
      image: ghcr.io/pantsbuild/wheel_build_aarch64:v3-8384c5cf
    ...

benjyw · 2024-07-11T15:42:52Z

Right, but we can't continue to do that, so I guess the fix has to be something like "run the build in docker inside a native GHA job"

huonw · 2024-07-12T02:50:24Z

Maybe docker_environment can work for this?

stuhood · 2024-07-12T22:52:07Z

Maybe docker_environment can work for this?

Possibly. The actual "build wheels" step is:

pants/src/python/pants_release/release.py

Lines 475 to 487 in 7b2dcad

    
           def build_all_wheels() -> None: 
        
               build_pants_wheels() 
        
               build_3rdparty_wheels() 
        
               install_and_test_packages( 
        
                   CONSTANTS.pants_stable_version, 
        
                   extra_pip_args=[ 
        
                       "--only-binary=:all:", 
        
                       "-f", 
        
                       str(CONSTANTS.deploy_3rdparty_wheel_dir / CONSTANTS.pants_stable_version), 
        
                       "-f", 
        
                       str(CONSTANTS.deploy_pants_wheel_dir / CONSTANTS.pants_stable_version), 
        
                   ], 
        
               )

... which invokes:

./pants package for pantsbuild.pants and pantsbuild.pants.testutil
pip for thirdparty wheels

The ./pants package step should definitely be possible to run in the docker environment... running pip in there as well would require an extra wrapper script, but might be feasible too.

tobni · 2024-07-13T23:22:45Z

So, python_distribution does not have the wiring to use environments-preview, but that was easy to fix, tried it locally.

But wont also the native_client need to be built in the container? That's not "inside pants" atm, meaning even using docker_environment we would still be linking to glibc 2.23?

huonw · 2024-07-14T11:01:24Z

Ah, good catch. Maybe it's best to do some variant of docker run ... pants package ... (at least for now, to unblock releases).

benjyw · 2024-07-14T13:38:11Z

Yes, the wheel is not the main problem, the engine is

tobni · 2024-07-14T15:19:47Z

env:
  ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true

seems practical if all we're looking for is an unblock?

link: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/

To opt out of this and continue using Node16 while it is still available in the runner, you can choose to set ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true
as an ‘env’ in their workflow or as an environment variable on your runner machine. This will only work until we upgrade the runner removing Node16 later in the spring.

Edit: Doesn't seem to have any effect when applied ontop of the changes herein, maybe newer JS actions do not honor it?

Edit2: Indeed, checkout@v4 unconditionally requires node20.

benjyw · 2024-07-14T18:19:39Z

Oooh yes, I would take an unblock right now.

tobni · 2024-07-14T18:59:26Z

My attempt at the temporary fix: #21172 @benjyw
The run: https://github.com/pantsbuild/pants/actions/runs/9929989447 (still spinning last I checked)

Edit: It passed with some warnings, but none of them seem critical.

tobni · 2024-07-15T12:40:11Z

Merging this and then #21172

Renders github actions and env settings to continue to use node16 inside the containers that match the glibc (2.17) of manylinux_2014 builder images. The new default node version for gha requires a much higher version. I checked if https://github.com/nodejs/unofficial-builds had node binaries we could use. It does not. Based on #21133

We started to run into issues with node16 no longer being functional on the runner, with symptoms like actions/checkout#1809

…sbuild#21172) Renders github actions and env settings to continue to use node16 inside the containers that match the glibc (2.17) of manylinux_2014 builder images. The new default node version for gha requires a much higher version. I checked if https://github.com/nodejs/unofficial-builds had node binaries we could use. It does not. Based on pantsbuild#21133

We started to run into issues with node16 no longer being functional on the runner, with symptoms like actions/checkout#1809

WorkerPants · 2024-07-24T07:51:43Z

I tried to automatically cherry-pick this change back to each relevant milestone, so that it is available in those older releases of Pants.

❌ 2.20.x

I was unable to cherry-pick this PR to 2.20.x, likely due to merge-conflicts.

Steps to Cherry-Pick locally

To resolve:

(Ensure your git working directory is clean)

Run the following script to reproduce the merge-conflicts:

git fetch https://github.com/pantsbuild/pants main \
  && git fetch https://github.com/pantsbuild/pants 2.20.x \
  && git checkout -b cherry-pick-21133-to-2.20.x FETCH_HEAD \
  && git cherry-pick 9a70a83c43b7a5a56b1ca1d7a61a16cd3cb8f860

Fix the merge conflicts and commit the changes
Run build-support/cherry_pick/make_pr.sh "21133" "2.20.x"

Please note that I cannot re-run CI if a job fails. Please work with your PR approver(s) to re-run CI if necessary.

❌ 2.21.x

I was unable to cherry-pick this PR to 2.21.x, likely due to merge-conflicts.

Steps to Cherry-Pick locally

To resolve:

(Ensure your git working directory is clean)

Run the following script to reproduce the merge-conflicts:

git fetch https://github.com/pantsbuild/pants main \
  && git fetch https://github.com/pantsbuild/pants 2.21.x \
  && git checkout -b cherry-pick-21133-to-2.21.x FETCH_HEAD \
  && git cherry-pick 9a70a83c43b7a5a56b1ca1d7a61a16cd3cb8f860

Fix the merge conflicts and commit the changes
Run build-support/cherry_pick/make_pr.sh "21133" "2.21.x"

Please note that I cannot re-run CI if a job fails. Please work with your PR approver(s) to re-run CI if necessary.

✔️ 2.22.x

Successfully opened #21198.

When you're done manually cherry-picking, please remove the needs-cherrypick label on this PR.

Thanks again for your contributions!

🤖 Beep Boop here's my run link

We started to run into issues with node16 no longer being functional on the runner, with symptoms like actions/checkout#1809 Co-authored-by: Andreas Stenius <git@astekk.se>

We started to run into issues with node16 no longer being functional on the runner, with symptoms like actions/checkout#1809

We started to run into issues with node16 no longer being functional on the runner, with symptoms like actions/checkout#1809 Co-authored-by: Andreas Stenius <git@astekk.se>

Renders github actions and env settings to continue to use node16 inside the containers that match the glibc (2.17) of manylinux_2014 builder images. The new default node version for gha requires a much higher version. I checked if https://github.com/nodejs/unofficial-builds had node binaries we could use. It does not. Based on #21133

…ry-pick of #21172) (#21209) Renders github actions and env settings to continue to use node16 inside the containers that match the glibc (2.17) of manylinux_2014 builder images. The new default node version for gha requires a much higher version. I checked if https://github.com/nodejs/unofficial-builds had node binaries we could use. It does not. Based on #21133 Co-authored-by: Tobias Nilsson <tobias.jens.nilsson@gmail.com>

Renders github actions and env settings to continue to use node16 inside the containers that match the glibc (2.17) of manylinux_2014 builder images. The new default node version for gha requires a much higher version. I checked if https://github.com/nodejs/unofficial-builds had node binaries we could use. It does not. Based on #21133

…ry-pick of #21172) (#21211) Renders github actions and env settings to continue to use node16 inside the containers that match the glibc (2.17) of manylinux_2014 builder images. The new default node version for gha requires a much higher version. I checked if https://github.com/nodejs/unofficial-builds had node binaries we could use. It does not. Based on #21133 Co-authored-by: Tobias Nilsson <tobias.jens.nilsson@gmail.com>

…ry-pick of #21172) (#21212) Renders github actions and env settings to continue to use node16 inside the containers that match the glibc (2.17) of manylinux_2014 builder images. The new default node version for gha requires a much higher version. I checked if https://github.com/nodejs/unofficial-builds had node binaries we could use. It does not. Based on #21133 Co-authored-by: Tobias Nilsson <tobias.jens.nilsson@gmail.com>

While trying to land #21528 to was observed that the wheel building jobs were failing during git checkout with wonderful errors like `/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)`. This appears to be the same symptoms as actions/checkout#1809 which pointed the the deprecation notice at <https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/>. If this all sounds familiar that is because we *mostly* cleaned this up in #21133, but kept a fewer uses of the older actions for `manylinux2014` compatibility. However, from the notice: "To opt out of this and continue using Node16 while it is still available in the runner, you can choose to set ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true as an ‘env’ in their workflow or as an environment variable on your runner machine. This will *only work until we upgrade the runner removing Node16 later in the spring*. (emphasis added)" From the wheel job failures during #21528 and my attempts to fiddle with all of `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION`, `ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION`, `ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION` I think the removal time promised for the Spring has finally come, but I'm not that familiar with GitHub Actions and could be missing something. As a consequence the version of both `actions/upload-artifacts` and `actions/checkout` is bumped to `v4`. To make this testable now and in the future, wheels are now build on ci config changes. Why is bumping to `manylinux_2_28` (the next oldest manylinux) not a big deal? The "2.28" refers to glibc [2.28](https://sourceware.org/legacy-ml/libc-alpha/2018-08/msg00003.html) in 2018. That is older than Debian stable and if you need to use software that old you are presumably paying someone for an enterprise distribution. NOTE: I think we should bump the manylinux version in `main` regardless, but if my read if the situation is correct we may need to backport this any active release line. Without this or a more complex change we would also be unable to release after December 5th per #21616. ref #21195 #21616

kaos added 2 commits July 6, 2024 08:08

refactor used actions into table.

a180732

bump actions/cache, actions/checkout, actions/setup-go, actions/setup…

32bf00b

…-java, actions/setup-node, and actions/setup-python.

kaos added category:internal CI, fixes for not-yet-released features, etc. release-notes:not-required PR doesn't require mention in release notes labels Jul 6, 2024

Bump upload-artifacts, and download-artifacts for node20.

9d77334

This is a potentially breaking change. See https://github.com/actions/upload-artifact/blob/main/docs/MIGRATION.md

huonw approved these changes Jul 10, 2024

View reviewed changes

tobni mentioned this pull request Jul 14, 2024

Allow unesecure node temporarily for manylinux container builds #21172

Merged

tobni merged commit 9a70a83 into main Jul 15, 2024
28 of 31 checks passed

tobni deleted the kaos/bump-workflow-actions-for-node20 branch July 15, 2024 12:42

tobni pushed a commit to tobni/pants that referenced this pull request Jul 22, 2024

Bump workflow actions for node20 (pantsbuild#21133)

e88caef

We started to run into issues with node16 no longer being functional on the runner, with symptoms like actions/checkout#1809

huonw mentioned this pull request Jul 24, 2024

CI uses node 16 actions via ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION for Linux x86-64 & arm64 builds #21195

Open

huonw modified the milestones: 2.21.x, 2.20.x Jul 24, 2024

huonw added the needs-cherrypick label Jul 24, 2024

WorkerPants pushed a commit that referenced this pull request Jul 24, 2024

Bump workflow actions for node20 (#21133)

21213b6

We started to run into issues with node16 no longer being functional on the runner, with symptoms like actions/checkout#1809

WorkerPants added the auto-cherry-picking-failed Auto Cherry-Picking Failed label Jul 24, 2024

huonw pushed a commit that referenced this pull request Jul 25, 2024

Bump workflow actions for node20 (#21133)

d1428f9

We started to run into issues with node16 no longer being functional on the runner, with symptoms like actions/checkout#1809

huonw pushed a commit that referenced this pull request Jul 25, 2024

Bump workflow actions for node20 (#21133)

07264e0

We started to run into issues with node16 no longer being functional on the runner, with symptoms like actions/checkout#1809

huonw removed the needs-cherrypick label Jul 25, 2024

WorkerPants mentioned this pull request Jul 25, 2024

Allow unesecure node temporarily for manylinux container builds (Cherry-pick of #21172) #21209

Merged

huonw mentioned this pull request Jul 26, 2024

Allow unesecure node temporarily for manylinux container builds (Cherry-pick of #21172) #21211

Merged

huonw mentioned this pull request Jul 26, 2024

Allow unesecure node temporarily for manylinux container builds (Cherry-pick of #21172) #21212

Merged

huonw mentioned this pull request Nov 5, 2024

actions/upload-artifacts@v3 is being removed (5 Dec 2024), affects Linux build-wheels jobs #21616

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump workflow actions for node20 #21133

Bump workflow actions for node20 #21133

kaos commented Jul 6, 2024

kaos commented Jul 6, 2024 •

edited

Loading

huonw left a comment

huonw Jul 10, 2024

huonw Jul 10, 2024

kaos Jul 10, 2024

huonw commented Jul 10, 2024

kaos commented Jul 10, 2024

benjyw commented Jul 10, 2024

benjyw commented Jul 10, 2024

huonw commented Jul 10, 2024

kaos commented Jul 11, 2024

benjyw commented Jul 11, 2024

kaos commented Jul 11, 2024

benjyw commented Jul 11, 2024

huonw commented Jul 12, 2024

stuhood commented Jul 12, 2024

tobni commented Jul 13, 2024 •

edited

Loading

huonw commented Jul 14, 2024

benjyw commented Jul 14, 2024

tobni commented Jul 14, 2024 •

edited

Loading

benjyw commented Jul 14, 2024

tobni commented Jul 14, 2024 •

edited

Loading

tobni commented Jul 15, 2024

WorkerPants commented Jul 24, 2024

		@@ -19,6 +19,23 @@

		from pants.util.strutil import softwrap

		ACTION = {

Bump workflow actions for node20 #21133

Bump workflow actions for node20 #21133

Conversation

kaos commented Jul 6, 2024

kaos commented Jul 6, 2024 • edited Loading

huonw left a comment

Choose a reason for hiding this comment

huonw Jul 10, 2024

Choose a reason for hiding this comment

huonw Jul 10, 2024

Choose a reason for hiding this comment

kaos Jul 10, 2024

Choose a reason for hiding this comment

huonw commented Jul 10, 2024

kaos commented Jul 10, 2024

benjyw commented Jul 10, 2024

benjyw commented Jul 10, 2024

huonw commented Jul 10, 2024

kaos commented Jul 11, 2024

benjyw commented Jul 11, 2024

kaos commented Jul 11, 2024

benjyw commented Jul 11, 2024

huonw commented Jul 12, 2024

stuhood commented Jul 12, 2024

tobni commented Jul 13, 2024 • edited Loading

huonw commented Jul 14, 2024

benjyw commented Jul 14, 2024

tobni commented Jul 14, 2024 • edited Loading

benjyw commented Jul 14, 2024

tobni commented Jul 14, 2024 • edited Loading

tobni commented Jul 15, 2024

WorkerPants commented Jul 24, 2024

❌ 2.20.x

❌ 2.21.x

✔️ 2.22.x

kaos commented Jul 6, 2024 •

edited

Loading

tobni commented Jul 13, 2024 •

edited

Loading

tobni commented Jul 14, 2024 •

edited

Loading

tobni commented Jul 14, 2024 •

edited

Loading