Warn/Allow Failing Tests

[codiophile]

Is your feature request related to a problem or a nice-to-have?? Please describe.
I have an integration test that requires credentials to an external system. The credentials are available on CI, but a dev that clones the project wouldn't have access to those credentials, so that test would always fail locally. I don't want devs to get used to failing tests and I do want the test to run on CI. So skip doesn't solve my problem.

Describe the solution you'd like
I thought there would be some feature for allowing test failures. A warning when the test fails, would be appropriate, but it shouldn't fail the build. I think it.allowFail seems like a reasonable way of providing this feature. It should probably take an optional parameter for a warning message, to explain under which circumstances the test is expected to fail.

Describe alternatives you've considered
I could check if the credentials are available and not run the test if they aren't. I have no control over the external system and if it is down, that shouldn't necessarily block you from merging your change.

I also found that there is an npm module that adds this functionality (mocha-helpers), but it doesn't print any warning when a test fails and it is not part of mocha, so it adds an extra dependency that may break in the future.

Additional context
It seems like this feature has been requested a few times (#1480 and #2451), but it was never implemented. Someone thought a retry feature would solve the problem (#1773), but that's only a way to deal with flaky tests, not a way to deal with tests that are expected to fail under certain conditions.

[juergba]

@codiophile you can use this.skip() for this purpose.
If your credentials aren't availabe, then skip the test, it will be reported as pending.

[codiophile]

@juergba

I don't want devs to get used to failing tests and I do want the test to run on CI. So skip doesn't solve my problem.

[juergba]

@codiophile well ...
Please explain why it('clever test', function() { if (!credentials) this.skip()}) doesn't solve your problem.

[rvalle]

This is a very basic feature for a TDD team.

You write tests first, they all fail.

You provide implementation later... you see how implementation goes fixing the tests (which act as a spec).

skipping has absolutely nothing to do with this feature.