Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[test visibility] Early flake detection for mocha (parallel mode) #4723

Merged
merged 10 commits into from
Sep 27, 2024

Conversation

juan-fernandez
Copy link
Collaborator

@juan-fernandez juan-fernandez commented Sep 25, 2024

What does this PR do?

Add early flake detection logic when --parallel is passed to mocha.

Additionally, add the faulty threshold logic to mocha, which was missing: if too many tests are considered new, we'll bail out of EFD.

Motivation

Allow users of the parallel mode of mocha to use early flake detection.

Plugin Checklist

  • Unit tests.

Copy link

github-actions bot commented Sep 25, 2024

Overall package size

Self size: 7.19 MB
Deduped: 62.55 MB
No deduping: 62.83 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | @datadog/native-appsec | 8.1.1 | 18.67 MB | 18.68 MB | | @datadog/native-iast-taint-tracking | 3.1.0 | 12.27 MB | 12.28 MB | | @datadog/pprof | 5.3.0 | 9.85 MB | 10.22 MB | | protobufjs | 7.2.5 | 2.77 MB | 5.16 MB | | @datadog/native-iast-rewriter | 2.4.1 | 2.14 MB | 2.23 MB | | @opentelemetry/core | 1.14.0 | 872.87 kB | 1.47 MB | | @datadog/native-metrics | 2.0.0 | 898.77 kB | 1.3 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | jsonpath-plus | 9.0.0 | 580.4 kB | 1.03 MB | | import-in-the-middle | 1.8.1 | 71.67 kB | 785.15 kB | | msgpack-lite | 0.1.26 | 201.16 kB | 281.59 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.0 | 109.9 kB | 109.9 kB | | semver | 7.6.3 | 95.82 kB | 95.82 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | lru-cache | 7.14.0 | 74.95 kB | 74.95 kB | | ignore | 5.3.1 | 51.46 kB | 51.46 kB | | int64-buffer | 0.1.10 | 49.18 kB | 49.18 kB | | shell-quote | 1.8.1 | 44.96 kB | 44.96 kB | | istanbul-lib-coverage | 3.2.0 | 29.34 kB | 29.34 kB | | rfdc | 1.3.1 | 25.21 kB | 25.21 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | dc-polyfill | 0.1.4 | 23.1 kB | 23.1 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | path-to-regexp | 0.1.10 | 6.38 kB | 6.38 kB | | module-details-from-path | 1.0.3 | 4.47 kB | 4.47 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@pr-commenter
Copy link

pr-commenter bot commented Sep 25, 2024

Benchmarks

Benchmark execution time: 2024-09-26 14:58:52

Comparing candidate commit 0d94028 in PR branch juan-fernandez/efd-for-mocha-parallel with baseline commit c03632f in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 260 metrics, 6 unstable metrics.

@juan-fernandez juan-fernandez changed the title [test visibility] EFD for mocha [test visibility] Early flake detection for mocha (parallel mode) Sep 25, 2024
@juan-fernandez juan-fernandez marked this pull request as ready for review September 26, 2024 11:50
@juan-fernandez juan-fernandez requested a review from a team as a code owner September 26, 2024 11:50
@juan-fernandez juan-fernandez requested a review from a team as a code owner September 26, 2024 14:46
@@ -266,7 +266,7 @@
"mocha": [
{
"name": "mocha",
"versions": [">=5.2.0"]
"versions": [">=5.2.0", ">=8.0.0"]
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ not picked up automatically by install_plugin_modules because I've spread the instrumentation of mocha in multiple files. I could probably fix install_plugin_modules.js but this is just simpler

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually being addressed in a separate effort to separate tests and test setup from the versions in the implementation.

@juan-fernandez juan-fernandez merged commit 65d50b4 into master Sep 27, 2024
192 checks passed
@juan-fernandez juan-fernandez deleted the juan-fernandez/efd-for-mocha-parallel branch September 27, 2024 07:53
@juan-fernandez juan-fernandez mentioned this pull request Sep 30, 2024
@juan-fernandez juan-fernandez mentioned this pull request Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants