code coverage reporting is flakey #740
Comments
JohnAlbin
added
bug
|
In nodejs/node#51251, someone mentions that the bug is related to the version of Node being used.
It might also be a bug in the |
|
GitHub CI is currently using node v18.20.1 where the code coverage is flakey and occasionally failing. And it looks like v18 versions that were released before v20.10.0 was released (on 2023-11-22) might work. v18.18.x was released before that. |
|
I posted this originally on: nodejs/node#51251 Reposting here in case it's helpful... I have been facing the same issue: my coverage report was 100% with 20.9.0, but as of 20.10.0, it now reports less than 100% coverage and somewhat randomly flags lines not covered. I noticed that I get the same coverage reporting flakiness with 18.20.2 (I initially saw this issue when my GitHub actions started to fail). Coverage works fine with 18.19.1. So there seems to be a change between 18.19.1 and 18.20.0 that causes the issue as well as between 20.9.0 to 20.10.0. This is now causing my GitHub CI/CD actions to fail since GitHub is using 18.20.2. |
|
I tested v18.19 and it does not appear to be flakey. |
|
@kmccammon Yep! That's exactly what I noticed as well. I've changed this project's .nvmrc to use v18.19 for now. Thanks for confirming! I'm leaving this issue open until we can revert the work-around. |
Apparently, there's a difference in jest's V8 test reporting coverage depending on which version of node is being used. nodejs/node#51251 According to the folks talking about that issue in their repo, here: chapter-three/next-drupal#740 (comment) They tested and found that the coverage discrepancy occurs in node 18 after upgrading from version 18.19.1 to version 18.20.0. It also occurs in node 20 after upgrading from version 20.9.0 to version 20.10.0. Locally, we have been using 18.18.2 where V8 coverage reporting still appears to be working correctly for us, but CI is reporting something else (since it's only pinned to version 18 prior to this commit). Since 18.18.2 seems to work for us, and the folks commenting in the linked issue seem to agree that the issue shouldn't appear for us in that version, we'll keep using node version 18.18.2 locally and explicitly instruct GH actions to also use that version of nodejs.
Package containing the bug
next-drupal (NPM package)
Describe the bug
If multiple jest files are run, the code coverage of previously-run tests can be reset during the Jest test run and erroneously report a smaller code coverage value than is truly covered. This faulty report is not consistently reported; sometimes the true coverage value is reported.
Expected behavior
The code coverage report is always consistent and accurate instead of frequently reporting missing coverage for code that is obviously covered by existing tests.
Steps to reproduce:
yarn testmultiple times making slight code changes each time that shouldn't change the coverage value.Additional context
See jestjs/jest#14766 and nodejs/node#51251
The text was updated successfully, but these errors were encountered: