update the manylinux wheel glibc version to 2.28 cherry-pick of pants…

…build#21650) While trying to land pantsbuild#21528 to was observed that the wheel building jobs were failing during git checkout with wonderful errors like `/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)`. This appears to be the same symptoms as actions/checkout#1809 which pointed the the deprecation notice at <https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/>. If this all sounds familiar that is because we *mostly* cleaned this up in pantsbuild#21133, but kept a fewer uses of the older actions for `manylinux2014` compatibility. However, from the notice: "To opt out of this and continue using Node16 while it is still available in the runner, you can choose to set ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true as an ‘env’ in their workflow or as an environment variable on your runner machine. This will *only work until we upgrade the runner removing Node16 later in the spring*. (emphasis added)" From the wheel job failures during pantsbuild#21528 and my attempts to fiddle with all of `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION`, `ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION`, `ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION` I think the removal time promised for the Spring has finally come, but I'm not that familiar with GitHub Actions and could be missing something. As a consequence the version of both `actions/upload-artifacts` and `actions/checkout` is bumped to `v4`. To make this testable now and in the future, wheels are now build on ci config changes. Why is bumping to `manylinux_2_28` (the next oldest manylinux) not a big deal? The "2.28" refers to glibc [2.28](https://sourceware.org/legacy-ml/libc-alpha/2018-08/msg00003.html) in 2018. That is older than Debian stable and if you need to use software that old you are presumably paying someone for an enterprise distribution. NOTE: I think we should bump the manylinux version in `main` regardless, but if my read if the situation is correct we may need to backport this any active release line. Without this or a more complex change we would also be unable to release after December 5th per pantsbuild#21616. ref pantsbuild#21195 pantsbuild#21616
cburroughs · Dec 12, 2024 · ffd8ec8 · ffd8ec8
1 parent 7b5186f
commit ffd8ec8
Show file tree

Hide file tree

Showing 4 changed files with 88 additions and 66 deletions.
diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml
@@ -6,7 +6,7 @@
 jobs:
   build_wheels_linux_arm64:
     container:
-      image: quay.io/pypa/manylinux2014_aarch64:latest
+      image: quay.io/pypa/manylinux_2_28_aarch64:latest
     env:
       ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
       PANTS_REMOTE_CACHE_READ: 'false'
@@ -23,7 +23,7 @@ jobs:
     - run-id=${{ github.run_id }}
     steps:
     - name: Check out code
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
       with:
         fetch-depth: 10
         ref: ${{ needs.release_info.outputs.build-ref }}
@@ -43,6 +43,14 @@ jobs:
 
         echo "/opt/python/cp39-cp39/bin" >> $GITHUB_PATH
 
+        echo "/opt/python/cp310-cp310/bin" >> $GITHUB_PATH
+
+        echo "/opt/python/cp311-cp311/bin" >> $GITHUB_PATH
+
+        echo "/opt/python/cp312-cp312/bin" >> $GITHUB_PATH
+
+        echo "/opt/python/cp313-cp313/bin" >> $GITHUB_PATH
+
         '
     - name: Install Protoc
       uses: arduino/setup-protoc@9b1ee5b22b0a3f1feb8c2ff99b32c89b3c3191e9
@@ -58,7 +66,7 @@ jobs:
     - continue-on-error: true
       if: always()
       name: Upload pants.log
-      uses: actions/upload-artifact@v3
+      uses: actions/upload-artifact@v4
       with:
         name: logs-wheels-and-pex-Linux-ARM64
         overwrite: 'true'
@@ -83,7 +91,7 @@ jobs:
     timeout-minutes: 90
   build_wheels_linux_x86_64:
     container:
-      image: quay.io/pypa/manylinux2014_x86_64:latest
+      image: quay.io/pypa/manylinux_2_28_x86_64:latest
     env:
       ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
       PANTS_REMOTE_CACHE_READ: 'false'
@@ -96,7 +104,7 @@ jobs:
     - ubuntu-22.04
     steps:
     - name: Check out code
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
       with:
         fetch-depth: 10
         ref: ${{ needs.release_info.outputs.build-ref }}
@@ -116,14 +124,22 @@ jobs:
 
         echo "/opt/python/cp39-cp39/bin" >> $GITHUB_PATH
 
+        echo "/opt/python/cp310-cp310/bin" >> $GITHUB_PATH
+
+        echo "/opt/python/cp311-cp311/bin" >> $GITHUB_PATH
+
+        echo "/opt/python/cp312-cp312/bin" >> $GITHUB_PATH
+
+        echo "/opt/python/cp313-cp313/bin" >> $GITHUB_PATH
+
         '
     - name: Install Protoc
       uses: arduino/setup-protoc@9b1ee5b22b0a3f1feb8c2ff99b32c89b3c3191e9
       with:
         repo-token: ${{ secrets.GITHUB_TOKEN }}
         version: 23.x
     - name: Install Go
-      uses: actions/setup-go@v4
+      uses: actions/setup-go@v5
       with:
         go-version: 1.19.5
     - env:
@@ -137,7 +153,7 @@ jobs:
     - continue-on-error: true
       if: always()
       name: Upload pants.log
-      uses: actions/upload-artifact@v3
+      uses: actions/upload-artifact@v4
       with:
         name: logs-wheels-and-pex-Linux-x86_64
         overwrite: 'true'

diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml
@@ -325,14 +325,14 @@ jobs:
     timeout-minutes: 60
   build_wheels_linux_arm64:
     container:
-      image: quay.io/pypa/manylinux2014_aarch64:latest
+      image: quay.io/pypa/manylinux_2_28_aarch64:latest
     env:
       ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
       MODE: debug
       PANTS_REMOTE_CACHE_READ: 'false'
       PANTS_REMOTE_CACHE_WRITE: 'false'
-    if: ((github.repository_owner == 'pantsbuild') && (needs.classify_changes.outputs.release == 'true')) && (needs.classify_changes.outputs.docs_only
-      != 'true')
+    if: ((github.repository_owner == 'pantsbuild') && (needs.classify_changes.outputs.release == 'true' || needs.classify_changes.outputs.ci_config
+      == 'true')) && (needs.classify_changes.outputs.docs_only != 'true')
     name: Build wheels (Linux-ARM64)
     needs:
     - classify_changes
@@ -344,7 +344,7 @@ jobs:
     - run-id=${{ github.run_id }}
     steps:
     - name: Check out code
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
       with:
         fetch-depth: 10
     - name: Configure Git
@@ -362,6 +362,14 @@ jobs:
 
         echo "/opt/python/cp39-cp39/bin" >> $GITHUB_PATH
 
+        echo "/opt/python/cp310-cp310/bin" >> $GITHUB_PATH
+
+        echo "/opt/python/cp311-cp311/bin" >> $GITHUB_PATH
+
+        echo "/opt/python/cp312-cp312/bin" >> $GITHUB_PATH
+
+        echo "/opt/python/cp313-cp313/bin" >> $GITHUB_PATH
+
         '
     - name: Install Protoc
       uses: arduino/setup-protoc@9b1ee5b22b0a3f1feb8c2ff99b32c89b3c3191e9
@@ -377,30 +385,30 @@ jobs:
     - continue-on-error: true
       if: always()
       name: Upload pants.log
-      uses: actions/upload-artifact@v3
+      uses: actions/upload-artifact@v4
       with:
         name: logs-wheels-and-pex-Linux-ARM64
         overwrite: 'true'
         path: .pants.d/workdir/*.log
     timeout-minutes: 90
   build_wheels_linux_x86_64:
     container:
-      image: quay.io/pypa/manylinux2014_x86_64:latest
+      image: quay.io/pypa/manylinux_2_28_x86_64:latest
     env:
       ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
       MODE: debug
       PANTS_REMOTE_CACHE_READ: 'false'
       PANTS_REMOTE_CACHE_WRITE: 'false'
-    if: ((github.repository_owner == 'pantsbuild') && (needs.classify_changes.outputs.release == 'true')) && (needs.classify_changes.outputs.docs_only
-      != 'true')
+    if: ((github.repository_owner == 'pantsbuild') && (needs.classify_changes.outputs.release == 'true' || needs.classify_changes.outputs.ci_config
+      == 'true')) && (needs.classify_changes.outputs.docs_only != 'true')
     name: Build wheels (Linux-x86_64)
     needs:
     - classify_changes
     runs-on:
     - ubuntu-22.04
     steps:
     - name: Check out code
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
       with:
         fetch-depth: 10
     - name: Configure Git
@@ -418,14 +426,22 @@ jobs:
 
         echo "/opt/python/cp39-cp39/bin" >> $GITHUB_PATH
 
+        echo "/opt/python/cp310-cp310/bin" >> $GITHUB_PATH
+
+        echo "/opt/python/cp311-cp311/bin" >> $GITHUB_PATH
+
+        echo "/opt/python/cp312-cp312/bin" >> $GITHUB_PATH
+
+        echo "/opt/python/cp313-cp313/bin" >> $GITHUB_PATH
+
         '
     - name: Install Protoc
       uses: arduino/setup-protoc@9b1ee5b22b0a3f1feb8c2ff99b32c89b3c3191e9
       with:
         repo-token: ${{ secrets.GITHUB_TOKEN }}
         version: 23.x
     - name: Install Go
-      uses: actions/setup-go@v4
+      uses: actions/setup-go@v5
       with:
         go-version: 1.19.5
     - env:
@@ -439,7 +455,7 @@ jobs:
     - continue-on-error: true
       if: always()
       name: Upload pants.log
-      uses: actions/upload-artifact@v3
+      uses: actions/upload-artifact@v4
       with:
         name: logs-wheels-and-pex-Linux-x86_64
         overwrite: 'true'
@@ -451,8 +467,8 @@ jobs:
       MODE: debug
       PANTS_REMOTE_CACHE_READ: 'false'
       PANTS_REMOTE_CACHE_WRITE: 'false'
-    if: ((github.repository_owner == 'pantsbuild') && (needs.classify_changes.outputs.release == 'true')) && (needs.classify_changes.outputs.docs_only
-      != 'true')
+    if: ((github.repository_owner == 'pantsbuild') && (needs.classify_changes.outputs.release == 'true' || needs.classify_changes.outputs.ci_config
+      == 'true')) && (needs.classify_changes.outputs.docs_only != 'true')
     name: Build wheels (macOS10-15-x86_64)
     needs:
     - classify_changes
@@ -520,8 +536,8 @@ jobs:
       MODE: debug
       PANTS_REMOTE_CACHE_READ: 'false'
       PANTS_REMOTE_CACHE_WRITE: 'false'
-    if: ((github.repository_owner == 'pantsbuild') && (needs.classify_changes.outputs.release == 'true')) && (needs.classify_changes.outputs.docs_only
-      != 'true')
+    if: ((github.repository_owner == 'pantsbuild') && (needs.classify_changes.outputs.release == 'true' || needs.classify_changes.outputs.ci_config
+      == 'true')) && (needs.classify_changes.outputs.docs_only != 'true')
     name: Build wheels (macOS11-ARM64)
     needs:
     - classify_changes

diff --git a/docs/notes/2.24.x.md b/docs/notes/2.24.x.md
@@ -1,3 +1,4 @@
+
 # 2.24.x Release Series
 
 Pants 2 is a fast, scalable, user-friendly build system for codebases of all sizes. It's currently focused on Python, Go, Java, Scala, Kotlin, Shell, and Docker, with support for other languages and frameworks coming soon.
@@ -135,6 +136,8 @@ PyO3, the interface crate between Rust and Python, has been upgraded to v0.22.x.
 
 Previously `SetupKwargs` took `_allow_banned_keys` which would allow one to pass in certain critical setuptools args (ex: `install_requires`) that Pants calculates for you.  If you Really Really know what you are doing you can know also use `_overwrite_banned_keys` to exclusively use your own values and ignore the Pants calculated ones.
 
+The oldest [glibc version](https://www.sourceware.org/glibc/wiki/Glibc%20Timeline) supported by the published Pants wheels is now 2.28.  This should have no effect unless you are running on extremely old Linux distributions.  See <https://github.com/pypa/manylinux> for background context on Python wheels and C libraries.
+
 ## Full Changelog
 
 For the full changelog, see the individual GitHub Releases for this series: <https://github.com/pantsbuild/pants/releases>
diff --git a/src/python/pants_release/generate_github_workflows.py b/src/python/pants_release/generate_github_workflows.py
@@ -18,40 +18,27 @@
 from pants_release.common import die
 
 
-def action(name: str, node16_compat: bool = False) -> str:
-    # Versions of actions compatible with node16 and the `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION` setting.
-    # glibc 2.17 is required to build manylinux_2014 wheels, but node.js does do not ship glibc 2.17 compatible
-    # binaries for node >= v17.
-    if node16_compat:
-        version_map = {
-            "checkout": "actions/checkout@v3",
-            "upload-artifact": "actions/upload-artifact@v3",
-            "setup-go": "actions/setup-go@v4",
-        }
-    else:
-        version_map = {
-            "action-send-mail": "dawidd6/action-send-mail@v3.8.0",
-            "cache": "actions/cache@v4",
-            "checkout": "actions/checkout@v4",
-            "download-artifact": "actions/download-artifact@v4",
-            "github-action-required-labels": "mheap/github-action-required-labels@v4.0.0",
-            "rust-cache": "benjyw/rust-cache@5ed697a6894712d2854c80635bb00a2496ea307a",
-            "setup-go": "actions/setup-go@v5",
-            "setup-java": "actions/setup-java@v4",
-            "setup-node": "actions/setup-node@v4",
-            "setup-protoc": "arduino/setup-protoc@9b1ee5b22b0a3f1feb8c2ff99b32c89b3c3191e9",
-            "setup-python": "actions/setup-python@v5",
-            "slack-github-action": "slackapi/slack-github-action@v1.24.0",
-            "upload-artifact": "actions/upload-artifact@v4",
-        }
+def action(name: str) -> str:
+    version_map = {
+        "action-send-mail": "dawidd6/action-send-mail@v3.8.0",
+        "cache": "actions/cache@v4",
+        "checkout": "actions/checkout@v4",
+        "download-artifact": "actions/download-artifact@v4",
+        "github-action-required-labels": "mheap/github-action-required-labels@v4.0.0",
+        "rust-cache": "benjyw/rust-cache@5ed697a6894712d2854c80635bb00a2496ea307a",
+        "setup-go": "actions/setup-go@v5",
+        "setup-java": "actions/setup-java@v4",
+        "setup-node": "actions/setup-node@v4",
+        "setup-protoc": "arduino/setup-protoc@9b1ee5b22b0a3f1feb8c2ff99b32c89b3c3191e9",
+        "setup-python": "actions/setup-python@v5",
+        "slack-github-action": "slackapi/slack-github-action@v1.24.0",
+        "upload-artifact": "actions/upload-artifact@v4",
+    }
     try:
         return version_map[name]
     except KeyError as e:
-        configured_version = (
-            "Node 16 compatible version configured" if node16_compat else "version configured"
-        )
         raise ValueError(
-            f"Requested github action '{name}' does not have a {configured_version}.\n"
+            f"Requested github action '{name}' does not have a version configured.\n"
             f"Current known versions: {version_map}"
         ) from e
 
@@ -126,7 +113,7 @@ def hash_files(path: str) -> str:
 PYTHON_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.12", "3.13", "3.11"]
 
 DONT_SKIP_RUST = "needs.classify_changes.outputs.rust == 'true'"
-DONT_SKIP_WHEELS = "needs.classify_changes.outputs.release == 'true'"
+DONT_SKIP_WHEELS = "needs.classify_changes.outputs.release == 'true' || needs.classify_changes.outputs.ci_config == 'true'"
 IS_PANTS_OWNER = "github.repository_owner == 'pantsbuild'"
 
 # NB: This overrides `pants.ci.toml`.
@@ -238,7 +225,7 @@ def checkout(
         # We need to fetch a few commits back, to be able to access HEAD^2 in the PR case.
         {
             "name": "Check out code",
-            "uses": action("checkout", node16_compat=containerized),
+            "uses": action("checkout"),
             "with": {
                 **fetch_depth_opt,
                 **({"ref": ref} if ref else {}),
@@ -367,10 +354,10 @@ def install_jdk() -> Step:
     }
 
 
-def install_go(node16_compat: bool = False) -> Step:
+def install_go() -> Step:
     return {
         "name": "Install Go",
-        "uses": action("setup-go", node16_compat=node16_compat),
+        "uses": action("setup-go"),
         "with": {"go-version": "1.19.5"},
     }
 
@@ -584,10 +571,10 @@ def bootstrap_pants(self) -> Sequence[Step]:
             self.native_binaries_upload(),
         ]
 
-    def upload_log_artifacts(self, name: str, node16_compat: bool = False) -> Step:
+    def upload_log_artifacts(self, name: str) -> Step:
         return {
             "name": "Upload pants.log",
-            "uses": action("upload-artifact", node16_compat=node16_compat),
+            "uses": action("upload-artifact"),
             "if": "always()",
             "continue-on-error": True,
             "with": {
@@ -836,9 +823,9 @@ def build_wheels_job(
     # the code, install rustup and expose Pythons.
     # TODO: Apply rust caching here.
     if platform == Platform.LINUX_X86_64:
-        container = {"image": "quay.io/pypa/manylinux2014_x86_64:latest"}
+        container = {"image": "quay.io/pypa/manylinux_2_28_x86_64:latest"}
     elif platform == Platform.LINUX_ARM64:
-        container = {"image": "quay.io/pypa/manylinux2014_aarch64:latest"}
+        container = {"image": "quay.io/pypa/manylinux_2_28_aarch64:latest"}
     else:
         container = None
 
@@ -853,6 +840,10 @@ def build_wheels_job(
                     echo "/opt/python/cp37-cp37m/bin" >> $GITHUB_PATH
                     echo "/opt/python/cp38-cp38/bin" >> $GITHUB_PATH
                     echo "/opt/python/cp39-cp39/bin" >> $GITHUB_PATH
+                    echo "/opt/python/cp310-cp310/bin" >> $GITHUB_PATH
+                    echo "/opt/python/cp311-cp311/bin" >> $GITHUB_PATH
+                    echo "/opt/python/cp312-cp312/bin" >> $GITHUB_PATH
+                    echo "/opt/python/cp313-cp313/bin" >> $GITHUB_PATH
                     """
                 ),
             },
@@ -891,11 +882,7 @@ def build_wheels_job(
             "steps": [
                 *initial_steps,
                 install_protoc(),  # for prost crate
-                *(
-                    []
-                    if platform == Platform.LINUX_ARM64
-                    else [install_go(node16_compat=bool(container))]
-                ),
+                *([] if platform == Platform.LINUX_ARM64 else [install_go()]),
                 {
                     "name": "Build wheels",
                     "run": "./pants run src/python/pants_release/release.py -- build-wheels",
@@ -906,7 +893,7 @@ def build_wheels_job(
                     "run": "./pants package src/python/pants:pants-pex",
                     "env": helper.platform_env(),
                 },
-                helper.upload_log_artifacts(name="wheels-and-pex", node16_compat=bool(container)),
+                helper.upload_log_artifacts(name="wheels-and-pex"),
                 *(
                     [
                         {