undefined steps should always fail the test run

[charlierudolph]

Based off discussion in #442

The following was an argument for ambiguous steps to cause the test run to fail

consider a scenario which is passing. One of its steps suddenly becomes ambiguous because of a new step definition. That scenario will be reported as ambiguous, which is not considered a failure (Cucumber's return code is still 0). Let's assume the behaviour described in that scenario was modified and got broken recently. It is now possible that nobody will pay attention to it: the build is still green but in fact that piece of behaviour is not tested anymore and is broken.

I think this should be extended to undefined in the case that a step definition is deleted by accident or during a refactoring when it changes and a step is no longer matched. I also consider undefined and ambiguous steps similar cases as in both, cucumber-js could not find what step definition to run.

This leaves pending as the only type of step that can keep the build green in non-strict mode. That at least requires an explicit syntax and is thus less likely to happen by accident.

[brasmusson]

Or, we should change the default mode to strict, so the user explicitly have to specify that pending/undefined/ambiguous scenarios are allowed without failing the build.

[jbpros]

I also consider undefined and ambiguous steps similar cases as in both, cucumber-js could not find what step definition to run.

It makes complete sense to me.

[charlierudolph]

cucumber-js moved to strict mode by default in 2.0. This is regardless of strict mode though. I like only allowing pending steps through on non-strict since those require something explicit from the user.

[mattwynne]

It seems simplest to explain / understand if strict mode means "you have to have all of your steps passing"

[brasmusson]

And how should we explain / understand the non-strict mode?

• "All steps are match to one (and only one) step definition, but the step definitions may be pending"? or,
• "No executed step definition failed (but some steps might not be matched to a step definitions)?

[mattwynne]

When Charlie said

I like only allowing pending steps through on non-strict

I figured we were agreeing on having pending steps fail a strict build too. Did I misunderstand?

[brasmusson]

I interpreted

I like only allowing pending steps through on non-strict

as that undefined and ambiguous steps should fail the build also in non-strict mode (that is, a difference compared to the current behavior of Cucumber-Ruby and Cucumber-JVM). As far as I understand, no one have in this discussion suggested a change of the behavior of the strict mode (everything other than passed and skipped steps fail the build).

[charlierudolph]

I meant it the way @brasmusson understood it.

Proposal:

• Strict mode - succeeds if and only if every step has status passed/skipped
• Non-strict mode - succeeds if and only if every step has status passed/skipped/pending

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.