v8 test coverage in jest is incorrect when upgrading to 20.10.0

[TWiStErRob]

Version

20.10.0

Platform

Macbook Pro M1/M2 / Ubuntu 22.04 x64 (GitHub runner)

Subsystem

V8

What steps will reproduce the bug?

See jestjs/jest#14766

How often does it reproduce? Is there a required condition?

Found a consistent repro, but initially it was flaky because of the order jest was running tests.

What is the expected behavior? Why is that the expected behavior?

20.9.0 just works
20.10.0 fails

So this is a regression.

What do you see instead?

Coverage information is wrong in jest, unclear why, but based what we know Node is the place where this will most likely be fixed.

Additional information

Based on https://nodejs.org/en/blog/release/v20.10.0, cc @joyeecheung as you might be interested/know what's wrong.

Based on jestjs/jest#14764, cc @kwmhp as you experienced the same problem as us, and figure out the version upgrade is what triggers it.

[joyeecheung]

Can you provide a repro without third-party modules such as jest? Or maybe someone like @SimenB who's more familiar with Jest knows how to minimize the repro.

[TWiStErRob]

I can't, sorry, we spent a long time minimizing this much, and have no clue how to go further.

[Snafuh]

@kwmhp here (work/private account)
I am trying to reproduce this with my test case, but I can't so far
node --test --experimental-test-coverage
return 100% coverage.
I will do some more experiments, as the problem in jest occurred only when executing tests in certain order/in-line.
The issue only occurs with v20.10.0, but the problem might be further downstream with aggregation of test coverage.

[koshic]

Data returned from session.post('Profiler.takePreciseCoverage') (see https://github.com/SimenB/collect-v8-coverage/blob/2dd5d7aad3a8a1a07e33f9bf331f675cb5b9b480/index.js#L25-L27) a bit different for 20.9.0 and 21.5.0:

You can find that data at line jest/v8-flaky-coverage/node_modules/jest-runtime/build/index.js:1165, in this._v8CoverageResult.

Because c8 jest --runInBand shows 100% coverage despite of Node version (== NODE_V8_COVERAGE produces proper data), it looks like the root cause is somewhere near inspector and/or how it used by jest-runtime / collect-v8-coverage packages.

[ritchieanesco]

I am also getting this issue after updating to 20.10.0 from 20.9.0. Is there any workaround?

Our code coverage failure seem to be related to tests that assert error exceptions which are not being included in the code coverage.

[joyeecheung]

#51251 (comment) mentioned c8 produces correct results, so you could try that if that works. Otherwise it's difficult to tell whether this is a Node.js issue without a repro that only uses Node.js and not third-party modules.

[ritchieanesco]

@joyeecheung

@koshic seems to be right with his diagnosis. I can verify that the following fixes the flaky code coverage jestjs/jest#14766 (comment)

The code changes relate to this repo https://github.com/SimenB/collect-v8-coverage/blob/main/index.js

Are you able to provide any insight why this might be the case?

[joyeecheung]

I don't know much about how the jest package implements coverage collection but if commenting out the stop command makes a difference it might be that the package was stopping coverage collection too early or the ordering of start and stop commands it uses is wrong somehow. C8 spawns child processes with NODE_V8_COVERAGE which is implemented by Node.js to start the collection very early and stop the collection when the process is about to exit, that could be why it is more reliable. Maybe the jest approach could be updated to use NODE_V8_COVERAGE instead of using the inspector API too.

[cenfun]

Seems it can be fixed with await this.postSession('Debugger.enable');
see https://github.com/SimenB/collect-v8-coverage/pull/235/files

[kmccammon]

I have been facing the same issue: my coverage report was 100% with 20.9.0, but as of 20.10.0, it now reports less than 100% coverage and somewhat randomly flags lines not covered.

I noticed that I get the same coverage reporting flakiness with 18.20.2 (I initially saw this issue when my GitHub actions started to fail). Coverage works fine with 18.19.1.

So there seems to be a change between 18.19.1 and 18.20.0 that causes the issue as well as between 20.9.0 to 20.10.0.

This is now causing my GitHub CI/CD actions to fail since GitHub is using 18.20.2.