node-daily-v18.x-staging builds fail on rhel8-s390x since 1 Nov

[richardlau]

Starting on 1 Nov, node-daily-v18.x-staging builds have been consistently failing on rhel8-s390x.
e.g.

• 31 Oct (passed): https://ci.nodejs.org/view/Node.js%20Daily/job/node-daily-v18.x-staging/362/
  • https://ci.nodejs.org/job/node-test-commit-linuxone/46583/nodes=rhel8-s390x/
• 1 Nov (failed): https://ci.nodejs.org/view/Node.js%20Daily/job/node-daily-v18.x-staging/363/
  • https://ci.nodejs.org/job/node-test-commit-linuxone/46599/nodes=rhel8-s390x/

From https://ci.nodejs.org/job/node-test-commit-linuxone/46599/nodes=rhel8-s390x/console

09:06:28 npm warn cli npm v10.7.0 does not support Node.js v18.20.5-pre. This version of npm supports the following node versions: `^18.17.0 || >=20.5.0`. You can find the latest version at https://nodejs.org/.
09:10:05 npm error Exit handler never called!
09:10:05 npm error This is an error with npm itself. Please report this error at:
09:10:05 npm error     <https://github.com/npm/cli/issues>
09:10:05 
09:10:05 
09:10:05 npm error A complete log of this run can be found in: /home/iojs/build/workspace/citgm-smoker/npm_cache/_logs/2024-11-01T09_06_28_772Z-debug-0.log
09:10:05 make[1]: *** [Makefile:754: tools/doc/node_modules] Error 1
09:10:05 make: *** [Makefile:583: run-ci] Error 2

The warning can be ignored (expected for non-releases). I'm slightly concerned that the log references a path in the citgm-smoker workspace (from a node-test-commit-linuxone job!).

Both of the above builds ran on commit nodejs/node@1c4decc so we can rule out any of the subsequent changes landed on v18.x-staging. Since it doesn't appear at first to be due to a code change (and other platforms haven't failed), I'm opening for investigation here in nodejs/build.

[richardlau]

Note to check whether this only affects test-ibm-rhel8-s390x-4 (all four failing daily v18.x-staging builds all ran on that machine).

other platforms haven't failed

This includes, so far, rhel9-s390x.

[richardlau]

Note to check whether this only affects test-ibm-rhel8-s390x-4 (all four failing daily v18.x-staging builds all ran on that machine).

Resumed CI for nodejs/node#55708 (was failing on test-ibm-rhel8-s390x-4 with the same error) -- https://ci.nodejs.org/job/node-test-commit-linuxone/46644/nodes=rhel8-s390x/ is running on test-ibm-rhel8-s390x-1

[aduh95]

FWIW I tried cherry-picking all the npm updates to see if that would improve the outcome, but no, it fails with the same error

[richardlau]

https://ci.nodejs.org/job/node-test-commit-linuxone/46648/nodes=rhel8-s390x/console

10:12:25 npm error Exit handler never called!
10:12:25 npm error This is an error with npm itself. Please report this error at:
10:12:25 npm error     <https://github.com/npm/cli/issues>
10:12:25 
10:12:25 
10:12:25 npm error A complete log of this run can be found in: /home/iojs/build/workspace/citgm-smoker/npm_cache/_logs/2024-11-05T10_06_54_406Z-debug-0.log
10:12:25 make[1]: *** [Makefile:754: tools/doc/node_modules] Error 1

So from the log it looks like npm failed to reach the registry. The fetch error on the audits endpoint would indicate that it's trying (and failing) IPv6. Node.js 18 doesn't have happy eyeballs, which would probably explain why the other release lines haven't been affected on this machine.

[iojs@test-ibm-rhel8-s390x-4 ~]$ cat /home/iojs/build/workspace/citgm-smoker/npm_cache/_logs/2024-11-05T10_06_54_406Z-debug-0.log
...
43 http fetch GET https://registry.npmjs.org/@types/mdast/-/mdast-3.0.11.tgz attempt 1 failed with ENETUNREACH
44 http fetch GET https://registry.npmjs.org/@types/hast/-/hast-2.3.4.tgz attempt 1 failed with ENETUNREACH
45 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-2.0.6.tgz attempt 1 failed with ENETUNREACH
46 silly audit bulk request failed undefined
47 verbose audit error FetchError: request to https://registry.npmjs.org/-/npm/v1/security/audits/quick failed, reason: connect ENETUNREACH 2606:4700::6810:1c22:443
47 verbose audit error     at ClientRequest.<anonymous> (/home/iojs/build/workspace/node-test-commit-linuxone/deps/npm/node_modules/minipass-fetch/lib/index.js:130:14)
47 verbose audit error     at ClientRequest.emit (node:events:517:28)
47 verbose audit error     at _destroy (node:_http_client:882:13)
47 verbose audit error     at onSocketNT (node:_http_client:902:5)
47 verbose audit error     at process.processTicksAndRejections (node:internal/process/task_queues:83:21) {
47 verbose audit error   code: 'ENETUNREACH',
47 verbose audit error   errno: 'ENETUNREACH',
47 verbose audit error   syscall: 'connect',
47 verbose audit error   address: '2606:4700::6810:1c22',
47 verbose audit error   port: 443,
47 verbose audit error   type: 'system'
47 verbose audit error }
48 silly audit error undefined
49 silly audit report null
50 http fetch GET https://registry.npmjs.org/@types/mdast/-/mdast-3.0.11.tgz attempt 2 failed with ENETUNREACH
51 http fetch GET https://registry.npmjs.org/@types/hast/-/hast-2.3.4.tgz attempt 2 failed with ENETUNREACH
52 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-2.0.6.tgz attempt 2 failed with ENETUNREACH
53 http fetch GET https://registry.npmjs.org/unist-util-select/-/unist-util-select-5.0.0.tgz attempt 1 failed with ENETUNREACH
54 http fetch GET https://registry.npmjs.org/rehype-stringify/-/rehype-stringify-9.0.3.tgz attempt 1 failed with ENETUNREACH
55 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-2.0.6.tgz attempt 3 failed with ENETUNREACH
56 http fetch GET https://registry.npmjs.org/@types/hast/-/hast-2.3.4.tgz attempt 3 failed with ENETUNREACH
57 http fetch GET https://registry.npmjs.org/unist-util-select/-/unist-util-select-5.0.0.tgz attempt 2 failed with ENETUNREACH
58 http fetch GET https://registry.npmjs.org/rehype-stringify/-/rehype-stringify-9.0.3.tgz attempt 2 failed with ENETUNREACH
59 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-2.0.6.tgz attempt 1 failed with ETIMEDOUT
60 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-2.0.6.tgz attempt 1 failed with ETIMEDOUT
61 http fetch GET https://registry.npmjs.org/@types/debug/-/debug-4.1.8.tgz attempt 1 failed with ETIMEDOUT
62 http fetch GET https://registry.npmjs.org/@types/ms/-/ms-0.7.31.tgz attempt 1 failed with ETIMEDOUT
63 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-3.0.0.tgz attempt 1 failed with ETIMEDOUT
64 http fetch GET https://registry.npmjs.org/css-selector-parser/-/css-selector-parser-2.3.2.tgz attempt 1 failed with ETIMEDOUT
65 http fetch GET https://registry.npmjs.org/highlight.js/-/highlight.js-11.8.0.tgz attempt 1 failed with ETIMEDOUT
66 http fetch GET https://registry.npmjs.org/property-information/-/property-information-6.2.0.tgz attempt 1 failed with ETIMEDOUT
67 http fetch GET https://registry.npmjs.org/rehype-raw/-/rehype-raw-6.1.1.tgz attempt 1 failed with ETIMEDOUT
68 http fetch GET https://registry.npmjs.org/rehype-stringify/-/rehype-stringify-9.0.3.tgz attempt 3 failed with ENETUNREACH
69 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-2.0.6.tgz attempt 2 failed with ENETUNREACH
70 http fetch GET https://registry.npmjs.org/@types/debug/-/debug-4.1.8.tgz attempt 2 failed with ENETUNREACH
71 http fetch GET https://registry.npmjs.org/@types/ms/-/ms-0.7.31.tgz attempt 2 failed with ENETUNREACH
72 http fetch GET https://registry.npmjs.org/property-information/-/property-information-6.2.0.tgz attempt 2 failed with ENETUNREACH
73 http fetch GET https://registry.npmjs.org/unist-util-select/-/unist-util-select-5.0.0.tgz attempt 3 failed with ENETUNREACH
74 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-2.0.6.tgz attempt 2 failed with ENETUNREACH
75 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-3.0.0.tgz attempt 2 failed with ENETUNREACH
76 http fetch GET https://registry.npmjs.org/css-selector-parser/-/css-selector-parser-2.3.2.tgz attempt 2 failed with ENETUNREACH
77 http fetch GET https://registry.npmjs.org/highlight.js/-/highlight.js-11.8.0.tgz attempt 2 failed with ENETUNREACH
78 http fetch GET https://registry.npmjs.org/@types/mdast/-/mdast-3.0.11.tgz attempt 3 failed with ETIMEDOUT
79 http fetch GET https://registry.npmjs.org/rehype-raw/-/rehype-raw-6.1.1.tgz attempt 2 failed with ENETUNREACH
80 http fetch GET https://registry.npmjs.org/property-information/-/property-information-6.2.0.tgz attempt 3 failed with ENETUNREACH
81 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-3.0.0.tgz attempt 3 failed with ENETUNREACH
82 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-2.0.6.tgz attempt 3 failed with ENETUNREACH
83 http fetch GET https://registry.npmjs.org/css-selector-parser/-/css-selector-parser-2.3.2.tgz attempt 3 failed with ENETUNREACH
84 http fetch GET https://registry.npmjs.org/highlight.js/-/highlight.js-11.8.0.tgz attempt 3 failed with ENETUNREACH
85 http fetch GET https://registry.npmjs.org/@types/ms/-/ms-0.7.31.tgz attempt 3 failed with ETIMEDOUT
86 http fetch GET https://registry.npmjs.org/@types/debug/-/debug-4.1.8.tgz attempt 3 failed with ETIMEDOUT
87 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-2.0.6.tgz attempt 3 failed with ETIMEDOUT
88 http fetch GET https://registry.npmjs.org/rehype-raw/-/rehype-raw-6.1.1.tgz attempt 3 failed with ENETUNREACH
89 info ok
90 error Exit handler never called!
91 error This is an error with npm itself. Please report this error at:
92 error     <https://github.com/npm/cli/issues>
...

[richardlau]

I've opened #3952, which has been deployed on to three of the four rhel8-s390x test machines (I'll update the last one when the citgm job it is currently running completes).

[richardlau]

Note to check whether this only affects test-ibm-rhel8-s390x-4 (all four failing daily v18.x-staging builds all ran on that machine).

other platforms haven't failed

This includes, so far, rhel9-s390x.

Well annoyingly rhel9-s390x is now failing, after I started leave 😞.