Ambiguous steps ruin reporting

[enkessler]

Summary

When ambiguous steps are encountered, Cucumber essentially crashes. No reports are generated and no other tests executed. It just dies.

Expected Behavior

All tests should be run and output generated as normal. Some tests might fail/be skipped/have undefined steps, etc. but they will at least get executed.

Current Behavior

As soon as an ambiguous step is encountered, Cucumber declares the ambiguity and then stops. It does not proceed to the next step or even output the results of the run. Using --guess will, naturally, remove this problem but the idea is to have failures due to ambiguous steps (e.g. when using --strict) be handled in a similar way to pending or undefined steps. That is to say, usefully.

Possible Solution

If I recall correctly, it didn't use to be this way and ambiguous steps were handled similarly to undefined/pending steps. Why did that change?

Steps to Reproduce (for bugs)

1. Have a test with ambiguous steps
2. Run it
3. Watch Cucumber fail to fail gracefully

Here is a gist that will trigger the problem (run it one with --guess and once without in order to see the difference)

https://gist.github.com/enkessler/f351c4fd83f1b621789ad1d22cecb369

Context & Motivation

Ambiguous steps happen. Breaking the build happens. However, even a broken build needs useful output but ambiguous steps completely ruin that possibility by crashing the suite immediately.

Your Environment

• Version used: 2.4.0
• Operating System and version: OSX
• Link to your project: N/A (private)

[mattwynne]

Thanks for reporting @enkessler. Do you have time to add a scenario into our acceptance tests that reproduces this? That would be the first step to fixing it.

[enkessler]

@mattwynne I'm a little short on time but I'll try. I've got some other projects to maintain and I'm trying to get them all ready for when I show up to CukenFest.

[brasmusson]

Cucumber-JS has "ambiguous" as a separate result type, which neither Cucumber-Ruby nor Cucumber-JVM have.

[enkessler]

Step 1. Shamelessly steal from Cucumber-JS.
Step 2. Get it working in Ruby
Step 3. Profit.

:)

[enkessler]

Step 0: Make the build work on Windows.

:(

[mattwynne]

Step 0: Make the build work on Windows.

Can you raise a separate ticket about that? With an error message?

[enkessler]

#1115 opened. Links to the failing builds are included. Mind you, I don't necessarily know how to get them working any more than I know how to make ambiguous steps work. But, you know, I'm trying to do my part by keeping you lot informed. ;)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.