Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to fail if no tests are run #4062

Closed
eemeli opened this issue Oct 11, 2019 · 1 comment · Fixed by #4716
Closed

Add option to fail if no tests are run #4062

eemeli opened this issue Oct 11, 2019 · 1 comment · Fixed by #4716
Labels
area: browser browser-specific type: feature enhancement proposal

Comments

@eemeli
Copy link

eemeli commented Oct 11, 2019

Is your feature request related to a problem or a nice-to-have?? Please describe.

At the moment, if mocha.run() is called without any tests having been defined, the result is considered a success as the number of failures is 0. In messageformat/messageformat#261 this is causing a problem when Selenium browser tests are being run in a CI environment using mocha-selenium-bridge, and the tests "pass" (note the IE 11 results):

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Testing Chrome 49.0...
  83 passing (179ms)

    ✓ Chrome 49.0 (8602ms)

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Testing Firefox 66.0...
  83 passing (358ms)

    ✓ Firefox 66.0 (10933ms)

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Testing IE 11.0...
  0 passing (0ms)

    ✓ IE 11.0 (6062ms)

In our case, this went unnoticed for a number of months as the result was marked as a success. The bug itself was caused by some unsupported-by-IE11 syntax in a test file.

Describe the solution you'd like
I would like to have an option to consider "0 passing" tests to be considered an error.

Describe alternatives you've considered
I could probably write a custom reporter that would catch this, but that seems like a rather clumsy solution. I'm pretty sure the proper place to fix this would be somewhere around here, but the details of course depend on the implementation.

Additional context
I'm willing to submit a PR for this, but I'd rather get feedback first on how e.g. the error code should be handled, and/or whether a separate event would be appropriate for reporters to customise.

@eemeli eemeli added the type: feature enhancement proposal label Oct 11, 2019
@boneskull boneskull added the area: browser browser-specific label Oct 14, 2019
@boneskull
Copy link
Contributor

@eemeli There have been issues around this in the past.

From what I understand, requiring a test to be run actually breaks some workflows.

So, the requested behavior, if it were to be added, needs to be opt-in, since it's going to be a breaking change otherwise. This means adding a flag. There are plenty of flags already 😄 so it should be pretty straightforward to figure out where things need to be added.

I'm happy to look at a PR for this; we can sort out the details (e.g., what the flag should be called) later.

If any @mochajs/core object to adding such a flag, please comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: browser browser-specific type: feature enhancement proposal
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants