feat: add option to not fail on failing test suite #4771

ilgonmic · 2021-10-18T13:01:32Z

Requirements

Filling out the template is required. Any pull request that does not include enough information to be reviewed in a timely manner may be closed at the maintainers' discretion.
All new code requires tests to ensure against regressions.

Description of the Change

Add possibility to distinguish 2 types of fails:

Problem with test run
Failing tests

When there are problems with test run, it is definitely problem.
When tests were run, but there are some test fails, it can be considered as ok.
It is useful to use Mocha in integrations with different build tools (for example Gradle), when we extract information from Mocha about tests and pass it to build tool and check exit code of Mocha.

Karma has similar feature

Alternate Designs

It can be done with 2 runs of Mocha:

with dry-run and check if JavaScript code can be run without problems
as usual - to run tests

But it requires 2 runs, which is not very convenient.

Why should this be in core?

When using Mocha and look at exit code, it is impossible to distinguish 2 cases

1 failing test (exit code = 1)
Fail with running test (exit code = 1)

But these 2 cases can be completely different

Benefits

New useful feature for build pipelines integrations

Possible Drawbacks

Applicable issues

#4216

github-actions · 2022-02-16T00:45:48Z

This PR hasn't had any recent activity, and I'm labeling it stale. Remove the label or comment or this PR will be closed in 14 days. Thanks for contributing to Mocha!

ilgonmic · 2022-02-17T08:06:10Z

Ping

github-actions · 2022-06-20T00:48:06Z

This PR hasn't had any recent activity, and I'm labeling it stale. Remove the label or comment or this PR will be closed in 14 days. Thanks for contributing to Mocha!

linux-foundation-easycla · 2023-12-02T21:26:26Z

The committers listed above are authorized under a signed CLA.

✅ login: JoshuaKGoldberg / name: Josh Goldberg ✨ (6a5a1f7, 0e8212f, 3eb4dac)
✅ login: ilgonmic / name: Ilya Goncharov (1cf33bc, 964fa2c, 964df70)

JoshuaKGoldberg

Looking good as a start! The logical flow makes sense and seems reasonable to me. 🔥

Requesting changes mostly on docs/naming and testing. A bit of code shuffling too, but I'm open to getting told I'm being overly nitpicky. 😄

lib/cli/run-helpers.js

lib/cli/run.js

test/integration/fixtures/require-undefined.fixture.js

test/integration/options/failOnFailingTestSuite.spec.js

lib/cli/run-helpers.js

lib/cli/run.js

JoshuaKGoldberg · 2024-07-02T16:54:47Z

👋 ping @ilgonmic, is this still something you have time for?

ilgonmic · 2024-07-02T16:57:02Z

Hi @JoshuaKGoldberg
Sorry, I missed notifications from the review, I'll try to take a look on it on this week

…ere no infrastructure problem

ilgonmic · 2024-07-15T10:28:08Z

@JoshuaKGoldberg Hi, I tried to fix all concers, could you please take a look?

JoshuaKGoldberg

🙌 This looks great, thanks so much @ilgonmic!

I'll merge and release this in a new minor version soon. 🚀

docs/index.md

JoshuaKGoldberg · 2024-07-20T17:31:22Z

lib/cli/run-helpers.js

+  return code => {
+    const clampedCode = passOnFailingTestSuite
+      ? 0
+      : Math.min(code, 255);


[Praise] I like the moving of the Math.min(code, 255) here, nice spot!

lib/cli/run-helpers.js

JoshuaKGoldberg · 2024-07-20T19:53:27Z

Published in mocha@10.7.0. 🚀

ilgonmic changed the title ~~Ilgonmic/fail on failing test suite~~ Not fail on failing test suite Oct 18, 2021

ilgonmic changed the title ~~Not fail on failing test suite~~ Add option to not fail on failing test suite Oct 18, 2021

github-actions bot added the stale this has been inactive for a while... label Feb 16, 2022

github-actions bot removed the stale this has been inactive for a while... label Feb 18, 2022

github-actions bot added the stale this has been inactive for a while... label Jun 20, 2022

github-actions bot closed this Jul 4, 2022

JoshuaKGoldberg reopened this Dec 2, 2023

JoshuaKGoldberg removed the stale this has been inactive for a while... label Dec 2, 2023

JoshuaKGoldberg requested changes Mar 4, 2024

View reviewed changes

JoshuaKGoldberg added status: waiting for author waiting on response from OP - more information needed semver-minor implementation requires increase of "minor" version number; "features" labels Mar 4, 2024

JoshuaKGoldberg changed the title ~~Add option to not fail on failing test suite~~ feat: add option to not fail on failing test suite Mar 4, 2024

JoshuaKGoldberg added the stale this has been inactive for a while... label Jul 2, 2024

ilgonmic added 3 commits July 9, 2024 18:53

Add option fail-on-failing-test-suite to not fail test run if there w…

1cf33bc

…ere no infrastructure problem

Add test on failOnFailingTestSuite

964df70

Use parameters which by default is false

964fa2c

ilgonmic force-pushed the ilgonmic/fail-on-failing-test-suite branch from 5c1be0d to 964fa2c Compare July 15, 2024 10:21

ilgonmic requested a review from JoshuaKGoldberg July 15, 2024 16:37

JoshuaKGoldberg removed status: waiting for author waiting on response from OP - more information needed stale this has been inactive for a while... labels Jul 20, 2024

JoshuaKGoldberg approved these changes Jul 20, 2024

View reviewed changes

JoshuaKGoldberg added 2 commits July 20, 2024 15:15

Merge branch 'main' into ilgonmic/fail-on-failing-test-suite

6a5a1f7

Update docs/index.md

0e8212f

JoshuaKGoldberg merged commit deb8679 into mochajs:main Jul 20, 2024
24 checks passed

dlminvestments mentioned this pull request Aug 10, 2024

[Snyk] Upgrade mocha from 10.6.0 to 10.7.0 dlminvestments/node-js-server#1488

Open

abdulrahman305 mentioned this pull request Aug 10, 2024

[Snyk] Upgrade mocha from 10.4.0 to 10.7.0 abdulrahman305/gauge#4

Merged

This was referenced Aug 11, 2024

[Snyk] Upgrade mocha from 10.6.0 to 10.7.0 nodeshift/kube-service-bindings#204

Closed

[Snyk] Upgrade mocha from 10.6.0 to 10.7.0 nodeshift/mern-workshop#230

Closed

malforsaja mentioned this pull request Aug 11, 2024

[Snyk] Upgrade mocha from 10.6.0 to 10.7.0 malforsaja/weather-service#69

Merged

hoang3402 mentioned this pull request Aug 11, 2024

[Snyk] Upgrade mocha from 10.6.0 to 10.7.0 hoang3402/CodeWars#36

Closed

erossignon mentioned this pull request Aug 11, 2024

[Snyk] Upgrade mocha from 10.6.0 to 10.7.0 node-opcua/node-opcua-crypto#49

Closed

hoang3402 mentioned this pull request Aug 12, 2024

[Snyk] Upgrade mocha from 10.6.0 to 10.7.0 hoang3402/CodeWars#37

Closed

abdulrahman305 mentioned this pull request Aug 13, 2024

[Snyk] Upgrade mocha from 10.2.0 to 10.7.0 abdulrahman305/tesseract.js#16

Open

This was referenced Aug 13, 2024

[Snyk] Upgrade mocha from 10.6.0 to 10.7.0 nodeshift/kube-service-bindings#209

Merged

[Snyk] Upgrade mocha from 10.6.0 to 10.7.0 nodeshift/mern-workshop#233

Merged

hoang3402 mentioned this pull request Aug 13, 2024

[Snyk] Upgrade mocha from 10.6.0 to 10.7.0 hoang3402/CodeWars#38

Merged

MaxMood96 mentioned this pull request Aug 25, 2024

[Snyk] Upgrade mocha from 10.2.0 to 10.7.0 MaxMood96/node-oracledb#102

Closed

malforsaja mentioned this pull request Aug 30, 2024

[Snyk] Upgrade mocha from 10.7.0 to 10.7.3 malforsaja/weather-service#71

Merged

abdulrahman305 mentioned this pull request Aug 31, 2024

[Snyk] Upgrade mocha from 10.7.0 to 10.7.3 abdulrahman305/newman#10

Open

This was referenced Aug 31, 2024

[Snyk] Upgrade mocha from 10.7.0 to 10.7.3 nodeshift/kube-service-bindings#223

Closed

[Snyk] Upgrade mocha from 10.7.0 to 10.7.3 nodeshift/mern-workshop#237

Closed

hoang3402 mentioned this pull request Aug 31, 2024

[Snyk] Upgrade mocha from 10.7.0 to 10.7.3 hoang3402/CodeWars#41

Merged

SapiannykSerhii mentioned this pull request Sep 25, 2024

[Snyk] Upgrade mocha from 10.0.0 to 10.7.3 SapiannykSerhii/-NeWFestivaleWave-express_API_app#128

Open

guram21 mentioned this pull request Oct 4, 2024

[Snyk] Upgrade mocha from 10.6.0 to 10.7.3 guram21/Unit_testing#153

Merged

MaxMood96 mentioned this pull request Oct 6, 2024

[Snyk] Upgrade mocha from 10.2.0 to 10.7.3 MaxMood96/node-oracledb#103

Closed

sminnee mentioned this pull request Oct 19, 2024

[Snyk] Upgrade mocha from 10.6.0 to 10.7.3 sminnee/res-suspenders#32

Merged

This was referenced Oct 23, 2024

[Snyk] Upgrade mocha from 9.2.2 to 10.7.3 nerdy-tech-com-gitub/preact#11

Open

[Snyk] Upgrade mocha from 9.2.2 to 10.7.3 nerdy-tech-com-gitub/node#8

Open

rona-dinihari mentioned this pull request Oct 24, 2024

[Snyk] Upgrade mocha from 10.6.0 to 10.7.3 rona-dinihari/node-tesseract#9

Open

abdulrahman305 mentioned this pull request Oct 27, 2024

[Snyk] Upgrade mocha from 10.4.0 to 10.7.3 abdulrahman305/gauge#7

Open

HaLaGu1L mentioned this pull request Nov 1, 2024

[Snyk] Upgrade mocha from 10.1.0 to 10.7.3 HaLaGu1L/wallet-core#7

Open

SkyeFull90 mentioned this pull request Nov 9, 2024

[Snyk] Upgrade mocha from 10.3.0 to 10.7.3 SkyeFull90/100devs-projects-skye#8

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add option to not fail on failing test suite #4771

feat: add option to not fail on failing test suite #4771

ilgonmic commented Oct 18, 2021

github-actions bot commented Feb 16, 2022

ilgonmic commented Feb 17, 2022

github-actions bot commented Jun 20, 2022

linux-foundation-easycla bot commented Dec 2, 2023 •

edited

Loading

JoshuaKGoldberg left a comment

JoshuaKGoldberg commented Jul 2, 2024

ilgonmic commented Jul 2, 2024

ilgonmic commented Jul 15, 2024

JoshuaKGoldberg left a comment

JoshuaKGoldberg Jul 20, 2024

JoshuaKGoldberg commented Jul 20, 2024

feat: add option to not fail on failing test suite #4771

feat: add option to not fail on failing test suite #4771

Conversation

ilgonmic commented Oct 18, 2021

Requirements

Description of the Change

Alternate Designs

Why should this be in core?

Benefits

Possible Drawbacks

Applicable issues

github-actions bot commented Feb 16, 2022

ilgonmic commented Feb 17, 2022

github-actions bot commented Jun 20, 2022

linux-foundation-easycla bot commented Dec 2, 2023 • edited Loading

JoshuaKGoldberg left a comment

Choose a reason for hiding this comment

JoshuaKGoldberg commented Jul 2, 2024

ilgonmic commented Jul 2, 2024

ilgonmic commented Jul 15, 2024

JoshuaKGoldberg left a comment

Choose a reason for hiding this comment

JoshuaKGoldberg Jul 20, 2024

Choose a reason for hiding this comment

JoshuaKGoldberg commented Jul 20, 2024

linux-foundation-easycla bot commented Dec 2, 2023 •

edited

Loading