Move debug testing over to ARM from X86

[mhdawson]

The Red Hat team spent the day looking at how we could move the CI closer to green. One of the top issue causing failures reported in the reliability reports are the debug builds running out of memory (21 failures in last report). - See nodejs/reliability#146 as an example.

The container machine for x86 where this test runs already has 32G of memory but debug builds take a lot of memory and we have a number of containers that can all be running on that machine.

Our new arm machines form equinix metal have 512G memory which should be more than enough. We only have debug coverage on 1 platform/os so thinking an option might be to be to move our debug builds over to ARM. It looks like we have a similar number of container based agents running on the host, the main difference being that there should be a lot more memory.

I created a temporary job to see how the debug build runs on ARM. There are 2 consistent failures:

#41204
#41208

I'm still thinking it would be worth marking the 2 tests ask flaky (1 for main, 17.x,16,x and the other for 14.x) and move over the debug builds to see if that helps us get closer to green.

Any concerns with that approach ?

@nodejs/build, @Trott

[Trott]

Any concerns with that approach ?

Not from me. Thanks for all the work on these CI issues!

[sxa]

LGTM and while I'm ok with marking as flakey we should definitely look into why 41204 is occurring and whether it's related to core/dump settings of some sort on the ARM host.

[mhdawson]

PR to mark 41204 as flaky -#41253. After that lands I'll see if I can configure the switch to arm just for the main branch so that we can validate we have better CI results before looking at backports

[addaleax]

I don't remember the original concerns around why specifically it was that we added a debug build, but I don't think that it was about platform/compiler-specific issues. We now have two configure flags that build Node.js with the same code behavior as a full debug build, namely --debug-node --v8-with-dchecks -- Consequently, for the debug builds in CI, using those flags is probably good enough. Those are fairly lightweight compared to a full debug build, so we could keep that configuration at least for x86?

(And then either do nothing for arm, or add a --debug-node --v8-with-dchecks builds for arm, or add a --debug build for arm, I don't care much, but if possible it would be nice not to lose debug coverage of the x86)