intermittent CI failure: swingset test virtualObjects - virtualObjectGC

[warner]

I'm seeing intermittent CI failures which like this:

  virtualObjects › virtualObjectGC › VO refcount management 3 faceted

  Difference:

  - undefined
  + {
  +   key: 'vom.rc.o+11/3',
  +   result: '1',
  +   type: 'vatstoreGet',
  + }

  › validate (packages/SwingSet/test/liveslots-helpers.js:263:7)
  › voRefcountManagementTest3 (packages/SwingSet/test/virtualObjects/test-virtualObjectGC.js:1299:3)
  › async packages/SwingSet/test/virtualObjects/test-virtualObjectGC.js:1334:3

  ─

  1 test failed

https://github.com/Agoric/agoric-sdk/runs/5706589231?check_suite_focus=true is one example.

This is probably a modern example of #3240, in which gcAndFinalize failed to GC or call finalizers (or called them too early?) when run under Node.js in certain circumstances (Node-v14 vs v16 made a difference). In that ticket, and PR #4644 to address some parts of it, we found a workaround: run all the tests in a single test-foo.js file with test.serial() instead of test(), to prevent the gcAndFinalize from one kernel+vat interacting with the finalizers from a different test in the same file.

But this test file already has that treatement applied.

I'm usually able to make this go away by forcing a re-run of the CI job, but obviously that's not a solution.

[warner]

I ran test/virtualObjects/test*.js in a loop overnight (Node v16.11.1), and the full test suite for a couple of hours, without reproducing the problem.

CI is using v16.14.0, so my next step will be to switch to that version locally and try some more.

[warner]

I gave it 3 hours on test/virtualObjects/test*.js and v16.14.0 with no failures.

[warner]

I just got a reproduction locally during unrelated work. I can't make it stable, though.

[FUDCo]

If you can make it happen, even if it's stochastic with a relatively low probability, I have a minor code tweak to the virtual objects tests to try out that is what made the problem go away in the stores tests those exhibited the same flakiness. Basically, in liveslots-helpers.js, change the default value of the forceGC parameter to setupTestLiveslots to true.

[warner]

Ok I added that forceGC=true to the batch of tests where I've seen the problem (the 8-ish VO lifecycle N (un)faceted ones). We'll see if it holds.

[FUDCo]

Closing. Seems to be under control for now.