investigate flaky test_make_callback/test-async-hooks-gcable in CI

[gireeshpunathil]

• Version: master
• Platform: linux-containerized-sharedlibs
• Subsystem: async-hooks

11:13:36 not ok 2592 node-api/test_make_callback/test-async-hooks-gcable
11:13:36   ---
11:13:36   duration_ms: 0.332
11:13:36   severity: crashed
11:13:36   exitcode: -11
11:13:36   stack: |-
11:13:36   ...

ref: https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_zlib_x64/16212/consoleFull

[gireeshpunathil]

again:

11:26:27 not ok 2592 node-api/test_make_callback/test-async-hooks-gcable
11:26:27   ---
11:26:27   duration_ms: 0.264
11:26:27   severity: crashed
11:26:27   exitcode: -11
11:26:27   stack: |-
11:26:27   ...

ref: https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_shared_x64/16214/consoleFull

[Trott]

Yeah, this one is out of control today. Always in the containered builds.

[gireeshpunathil]

more instances: https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_shared_x64/16224/consoleFull

I guess most of the recent CI runs have this affected. Pinning to catch attention.

[gireeshpunathil]

cc @nodejs/v8 @nodejs/async_hooks @nodejs/n-api

[addaleax]

@gireeshpunathil @Trott This is likely impossible to debug without core dumps and without access to the machine in question? I’ll open an access request on nodejs/build.

[addaleax]

(gdb) bt
#0  0x000056076b17e854 in v8::internal::ConcurrentMarkingVisitor::VisitPointers(v8::internal::HeapObject, v8::internal::FullObjectSlot, v8::internal::FullObjectSlot) ()
#1  0x000056076b18894c in v8::internal::ConcurrentMarking::Run(int, v8::internal::ConcurrentMarking::TaskState*) ()
#2  0x000056076b0f3b86 in non-virtual thunk to v8::internal::CancelableTask::Run() ()
#3  0x000056076aec9e15 in node::(anonymous namespace)::PlatformWorkerThread(void*) ()
#4  0x00007f8c559dd6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00007f8c5570688f in clone () from /lib/x86_64-linux-gnu/libc.so.6

This might be related/the same bug as #30498?

[addaleax]

Okay, results from debugging so far:

• This is weirdly related to ESM in some way: It always crashes when a major GC happens during our Module::CreateSyntheticModule() calls during bootstrap. I think 796f3d0 might actually be the commit that started these issues.
• This makes it reproduce somewhat frequently (~ 50 % of the time) locally for me:

diff --git a/deps/v8/src/objects/objects.cc b/deps/v8/src/objects/objects.cc
index 227cff8da47a..c1d344c304a0 100644
--- a/deps/v8/src/objects/objects.cc
+++ b/deps/v8/src/objects/objects.cc
@@ -6458,6 +6458,10 @@ Handle<Derived> HashTable<Derived, Shape>::NewInternal(
   Factory* factory = isolate->factory();
   int length = EntryToIndex(capacity);
   RootIndex map_root_index = Shape::GetMapRootIndex();
+  if (Shape::kEntrySize == 2) {
+    isolate->heap()->CollectAllGarbage(
+        Heap::kNoGCFlags, GarbageCollectionReason::kFullHashtable);
+  }
   Handle<FixedArray> array =
       factory->NewFixedArrayWithMap(map_root_index, length, allocation);
   Handle<Derived> table = Handle<Derived>::cast(array);

That might be a good starting point for debugging in V8 here, and/or bisecting V8. (But that’s for tomorrow rather than today :))

@nodejs/v8

This might be related/the same bug as #30498?

I’m very confident that this is indeed the case now.

[addaleax]

Here’s a V8 CL that resolves this issue both locally and on the container host: https://chromium-review.googlesource.com/c/v8/v8/+/1939752