-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
checkout@v3 using node20 instead of node16 #1809
Comments
Our team found that setting these before starting the runners get's things going again. Still unsure what's causing this.
|
I am having the same issue while running below versions, on ubuntu 20.04 VM:
Error:
Issue started today, i.e. 3 Jul 10am PDT. |
Hello! Our development team is also experiencing the same issue. We are using AWS instances as self-hosted. AWS AMI :
Git Action Error :
|
Following this blog post: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/
|
I tried setting this inside a workflow running on GH hosted runners and it didn't work.
v4 always used Node 20
See above |
Fixed by renaming the cd $RUNNER_HOME/actions-runner/externals
mv node20 node20.bak
ln -s node16 node20 |
I tried modifying /home/runner/actions-runner/.env and env.sh and it works fine.
and adding their environment variables into 'varCheckList' array in env.sh.
Then restart runner service
|
Looks like this is happening because of: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/ |
Quoting the post paragraph about that workaround: "This will only work until we upgrade the runner removing Node16 later in the spring" We are walking a tightrope. Just wanted to raise some awareness about that. Plan to actually upgrade your github actions in the short term or move away from them. E.g. I think I'll quit my job. 😄 |
My glibc version:
So why do I get this error? I am using checkout@v4.1.7. I do not understand what I have to do to upgrade. |
Here's a critical issue: Firstly, you shouldn't upgrade such fundamental environments arbitrarily. Even if you do upgrade, you must ensure the environment is correct. Upgrading to Node 20 and causing a glib error is absolutely unacceptable, especially in a product from a company like Microsoft. |
I wish there was an alternative version of the checkout action, maybe written in rust so we could just deploy a static binary. Then we would just compile that against a really old version of glibc. This would also simplify running GitHub Actions with nektos/act which does not map node into the container. Update: I found checkout-action in GitHub's market place and added support for private repos. Not sure if that is a permanent solution but at least our CI jobs are passing again... |
This means you now can't run any docker image that has a glibc older than 2.27. That includes:
What a huge f*ckup. I'm baffled by the amount of incompetence. What's wrong with providing a statically linked node binary? Do you need help with redistribution github? |
I'm really confused trying to understand how these workflows work:
it's failing with the glibc not found error:
but the I can't believe that |
Some release jobs have to be executed on a custom runner due to GitHub forcing a newer nodejs version: actions/checkout#1809 This whole issue is a huge rabbit hole where you can get lost. In short: * The actions/checkout@4 versions require the nodejs 20 version * Prebuilt nodejs 20 are linking against a newer GLIBC version * Even though jobs are executed in a docker image, some commands are from the runner machine, (e.g. nodejs is from the ubuntu-latest runner). * Downgrading to actions/checkout@3 worked for some time. However, due to https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/ it seems like we *have* to do something. Thus, we are running jobs for older distro images on a customer runner, provided by maerwald. In particular, when writing this message, the distro images that have be run on the custom runner are: * debian:9 * ubuntu:18.04 * mint:19.3 * fedora:27 * centos:7
Some release jobs have to be executed on a custom runner due to GitHub forcing a newer nodejs version: actions/checkout#1809 This whole issue is a huge rabbit hole where you can get lost. In short: * The actions/checkout@4 versions require the nodejs 20 version * Prebuilt nodejs 20 are linking against a newer GLIBC version * Even though jobs are executed in a docker image, some commands are from the runner machine, (e.g. nodejs is from the ubuntu-latest runner). * Downgrading to actions/checkout@3 worked for some time. However, due to https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/ it seems like we *have* to do something. Thus, we are running jobs for older distro images on a customer runner, provided by maerwald. In particular, when writing this message, the distro images that have be run on the custom runner are: * debian:9 * ubuntu:18.04 * mint:19.3 * fedora:27 * centos:7
e2e pipeline fails on Ubuntu 16.04 and 18.04 due to the recent default switch to node20 in actions/checkout. Node20 requires GLIBC 2.28 or higher, which these environments lack. To resolve this, we’re setting the `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true` environment variable to allow fallback to a compatible Node version, restoring pipeline stability. Useful links: * [GitHub changelog: node20 default](https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/) * [GitHub issue #1809](actions/checkout#1809)
While trying to land #21528 to was observed that the wheel building jobs were failing during git checkout with wonderful errors like `/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)`. This appears to be the same symptoms as actions/checkout#1809 which pointed the the deprecation notice at <https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/>. If this all sounds familiar that is because we *mostly* cleaned this up in #21133, but kept a fewer uses of the older actions for `manylinux2014` compatibility. However, from the notice: "To opt out of this and continue using Node16 while it is still available in the runner, you can choose to set ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true as an ‘env’ in their workflow or as an environment variable on your runner machine. This will *only work until we upgrade the runner removing Node16 later in the spring*. (emphasis added)" From the wheel job failures during #21528 and my attempts to fiddle with all of `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION`, `ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION`, `ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION` I think the removal time promised for the Spring has finally come, but I'm not that familiar with GitHub Actions and could be missing something. As a consequence the version of both `actions/upload-artifacts` and `actions/checkout` is bumped to `v4`. To make this testable now and in the future, wheels are now build on ci config changes. Why is bumping to `manylinux_2_28` (the next oldest manylinux) not a big deal? The "2.28" refers to glibc [2.28](https://sourceware.org/legacy-ml/libc-alpha/2018-08/msg00003.html) in 2018. That is older than Debian stable and if you need to use software that old you are presumably paying someone for an enterprise distribution. NOTE: I think we should bump the manylinux version in `main` regardless, but if my read if the situation is correct we may need to backport this any active release line. Without this or a more complex change we would also be unable to release after December 5th per #21616. ref #21195 #21616
While trying to land pantsbuild#21528 to was observed that the wheel building jobs were failing during git checkout with wonderful errors like `/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)`. This appears to be the same symptoms as actions/checkout#1809 which pointed the the deprecation notice at <https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/>. If this all sounds familiar that is because we *mostly* cleaned this up in pantsbuild#21133, but kept a fewer uses of the older actions for `manylinux2014` compatibility. However, from the notice: "To opt out of this and continue using Node16 while it is still available in the runner, you can choose to set ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true as an ‘env’ in their workflow or as an environment variable on your runner machine. This will *only work until we upgrade the runner removing Node16 later in the spring*. (emphasis added)" From the wheel job failures during pantsbuild#21528 and my attempts to fiddle with all of `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION`, `ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION`, `ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION` I think the removal time promised for the Spring has finally come, but I'm not that familiar with GitHub Actions and could be missing something. As a consequence the version of both `actions/upload-artifacts` and `actions/checkout` is bumped to `v4`. To make this testable now and in the future, wheels are now build on ci config changes. Why is bumping to `manylinux_2_28` (the next oldest manylinux) not a big deal? The "2.28" refers to glibc [2.28](https://sourceware.org/legacy-ml/libc-alpha/2018-08/msg00003.html) in 2018. That is older than Debian stable and if you need to use software that old you are presumably paying someone for an enterprise distribution. NOTE: I think we should bump the manylinux version in `main` regardless, but if my read if the situation is correct we may need to backport this any active release line. Without this or a more complex change we would also be unable to release after December 5th per pantsbuild#21616. ref pantsbuild#21195 pantsbuild#21616
While trying to land #21528 to was observed that the wheel building jobs were failing during git checkout with wonderful errors like ``/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)``. This appears to be the same symptoms as actions/checkout#1809 which pointed the the deprecation notice at <https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/>. If this all sounds familiar that is because we *mostly* cleaned this up in #21133, but kept a fewer uses of the older actions for `manylinux2014` compatibility. However, from the notice: "To opt out of this and continue using Node16 while it is still available in the runner, you can choose to set ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true as an ‘env’ in their workflow or as an environment variable on your runner machine. This will *only work until we upgrade the runner removing Node16 later in the spring*. (emphasis added)" From the wheel job failures during #21528 and my attempts to fiddle with all of `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION`, `ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION`, `ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION` I think the removal time promised for the Spring has finally come, but I'm not that familiar with GitHub Actions and could be missing something. As a consequence the version of both `actions/upload-artifacts` and `actions/checkout` is bumped to `v4`. To make this testable now and in the future, wheels are now build on ci config changes. Why is bumping to `manylinux_2_28` (the next oldest manylinux) not a big deal? The "2.28" refers to glibc [2.28](https://sourceware.org/legacy-ml/libc-alpha/2018-08/msg00003.html) in 2018. That is older than Debian stable and if you need to use software that old you are presumably paying someone for an enterprise distribution. NOTE: I think we should bump the manylinux version in `main` regardless, but if my read if the situation is correct we may need to backport this any active release line. Without this or a more complex change we would also be unable to release after December 5th per #21616. ref #21195 #21616 Co-authored-by: cburroughs <chris.burroughs@gmail.com>
It appears Github has finally fulfilled their promise of removing the older node versions, and now all workflows will run with node20. The For our use case, we still needed to be able to run some CI jobs with node14, but the node 14 docker images' glibc is too old now. The thing that worked for us was to instead install node14 with nvm onto a debian 12 image for these jobs. The debian image has a new-enough glibc for the version of node20 that github loads, and nvm is happy to install ancient nodes onto that. jobs:
foo:
container: debian:12
steps:
- name: Run on old node
run: |
apt update && apt -y install wget
touch "$HOME/.bashrc"
wget -qO- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"
nvm install 14.21.3
# node 14 is now available.
... |
Old workaround for actions/checkout#1809 no longer works
Some release jobs have to be executed on a custom runner due to GitHub forcing a newer nodejs version: actions/checkout#1809 This whole issue is a huge rabbit hole where you can get lost. In short: * The actions/checkout@4 versions require the nodejs 20 version * Prebuilt nodejs 20 are linking against a newer GLIBC version * Even though jobs are executed in a docker image, some commands are from the runner machine, (e.g. nodejs is from the ubuntu-latest runner). * Downgrading to actions/checkout@3 worked for some time. However, due to https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/ it seems like we *have* to do something. Thus, we are running jobs for older distro images on a customer runner, provided by maerwald. In particular, when writing this message, the distro images that have be run on the custom runner are: * debian:9 * ubuntu:18.04 * mint:19.3 * fedora:27 * centos:7
Do the same as pytorch/test-infra#5959 and download an old nodejs to keep Ubuntu Bionic working. Bug: actions/checkout#1809 Bug: actions/runner#3373
…build#21650) While trying to land pantsbuild#21528 to was observed that the wheel building jobs were failing during git checkout with wonderful errors like `/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)`. This appears to be the same symptoms as actions/checkout#1809 which pointed the the deprecation notice at <https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/>. If this all sounds familiar that is because we *mostly* cleaned this up in pantsbuild#21133, but kept a fewer uses of the older actions for `manylinux2014` compatibility. However, from the notice: "To opt out of this and continue using Node16 while it is still available in the runner, you can choose to set ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true as an ‘env’ in their workflow or as an environment variable on your runner machine. This will *only work until we upgrade the runner removing Node16 later in the spring*. (emphasis added)" From the wheel job failures during pantsbuild#21528 and my attempts to fiddle with all of `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION`, `ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION`, `ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION` I think the removal time promised for the Spring has finally come, but I'm not that familiar with GitHub Actions and could be missing something. As a consequence the version of both `actions/upload-artifacts` and `actions/checkout` is bumped to `v4`. To make this testable now and in the future, wheels are now build on ci config changes. Why is bumping to `manylinux_2_28` (the next oldest manylinux) not a big deal? The "2.28" refers to glibc [2.28](https://sourceware.org/legacy-ml/libc-alpha/2018-08/msg00003.html) in 2018. That is older than Debian stable and if you need to use software that old you are presumably paying someone for an enterprise distribution. NOTE: I think we should bump the manylinux version in `main` regardless, but if my read if the situation is correct we may need to backport this any active release line. Without this or a more complex change we would also be unable to release after December 5th per pantsbuild#21616. ref pantsbuild#21195 pantsbuild#21616
…) (#21754) While trying to land #21528 to was observed that the wheel building jobs were failing during git checkout with wonderful errors like `/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)`. This appears to be the same symptoms as actions/checkout#1809 which pointed the the deprecation notice at <https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/>. If this all sounds familiar that is because we *mostly* cleaned this up in #21133, but kept a fewer uses of the older actions for `manylinux2014` compatibility. However, from the notice: "To opt out of this and continue using Node16 while it is still available in the runner, you can choose to set ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true as an ‘env’ in their workflow or as an environment variable on your runner machine. This will *only work until we upgrade the runner removing Node16 later in the spring*. (emphasis added)" From the wheel job failures during #21528 and my attempts to fiddle with all of `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION`, `ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION`, `ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION` I think the removal time promised for the Spring has finally come, but I'm not that familiar with GitHub Actions and could be missing something. As a consequence the version of both `actions/upload-artifacts` and `actions/checkout` is bumped to `v4`. To make this testable now and in the future, wheels are now build on ci config changes. Why is bumping to `manylinux_2_28` (the next oldest manylinux) not a big deal? The "2.28" refers to glibc [2.28](https://sourceware.org/legacy-ml/libc-alpha/2018-08/msg00003.html) in 2018. That is older than Debian stable and if you need to use software that old you are presumably paying someone for an enterprise distribution. NOTE: I think we should bump the manylinux version in `main` regardless, but if my read if the situation is correct we may need to backport this any active release line. Without this or a more complex change we would also be unable to release after December 5th per #21616. ref #21195 #21616
Do the same as pytorch/test-infra#5959 and download an old nodejs to keep Ubuntu Bionic working. Bug: actions/checkout#1809 Bug: actions/runner#3373 (cherry picked from commit 54cab09)
Running on rhel7 with self-hosted runners and starting today we're running into issues where @V3 is using node20 instead of node16.
Happening on runner versions 2.316.1 and 2.317.0.
Oddly some of our Repos don't run into this issue when executing checkout@v3 on the same runners.
Run actions/checkout@v3
/ci_storage_agent_1/actions-runner/externals/node20/bin/node: /lib64/libm.so.6: version
GLIBC_2.27' not found (required by /ci_storage_agent_1/actions-runner/externals/node20/bin/node) /ci_storage_agent_1/actions-runner/externals/node20/bin/node: /lib64/libstdc++.so.6: version
GLIBCXX_3.4.20' not found (required by /ci_storage_agent_1/actions-runner/externals/node20/bin/node)/ci_storage_agent_1/actions-runner/externals/node20/bin/node: /lib64/libstdc++.so.6: version
CXXABI_1.3.9' not found (required by /ci_storage_agent_1/actions-runner/externals/node20/bin/node) /ci_storage_agent_1/actions-runner/externals/node20/bin/node: /lib64/libstdc++.so.6: version
GLIBCXX_3.4.21' not found (required by /ci_storage_agent_1/actions-runner/externals/node20/bin/node)/ci_storage_agent_1/actions-runner/externals/node20/bin/node: /lib64/libc.so.6: version
GLIBC_2.28' not found (required by /ci_storage_agent_1/actions-runner/externals/node20/bin/node) /ci_storage_agent_1/actions-runner/externals/node20/bin/node: /lib64/libc.so.6: version
GLIBC_2.25' not found (required by /ci_storage_agent_1/actions-runner/externals/node20/bin/node)The text was updated successfully, but these errors were encountered: