Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

When running with --retry, exit code should be 0 if a failed scenario passes on retry #1044

Closed
rishi-freshbooks opened this issue Oct 4, 2016 · 14 comments · Fixed by cucumber/cucumber-ruby-core#141
Closed

When running with --retry, exit code should be 0 if a failed scenario passes on retry #1044

rishi-freshbooks opened this issue Oct 4, 2016 · 14 comments · Fixed by cucumber/cucumber-ruby-core#141
Milestone
3.0.0

Comments

@rishi-freshbooks
Copy link

Summary

When running cucumber with the --retry flag, the process will exit with a non-zero code if the scenario fails on first try but passes on a retry.

Expected Behavior

Running cucumber with --retry flag should exit with code 0 if scenarios pass on first try or within the specified number of retries .

Context & Motivation

The non-zero exit code causes our build job in JenkinsCI to fail when it should report as success.
@rishi-freshbooks rishi-freshbooks changed the title --retry exit code should 0 if a failed scenario passes on retry When running with --retry, exit code should 0 if a failed scenario passes on retry Oct 4, 2016
@rishi-freshbooks rishi-freshbooks changed the title When running with --retry, exit code should 0 if a failed scenario passes on retry When running with --retry, exit code should be 0 if a failed scenario passes on retry Oct 4, 2016
@rishi-freshbooks
Copy link
Author

@mattwynne does this capture the issue appropriately? Is this something that can be addressed in v3.0.0.pre.1?

@mattwynne
Copy link
Member

That's great thanks @rishi-freshbooks. We'll do our best to fix this in a future pre-release of v3, yes.

@mattwynne mattwynne added this to the 3.0 milestone Oct 18, 2016
@streetlogics
Copy link

@mattwynne any progress on this? Just encountering this issue myself - I love that I finally got the --retry flag working in circleci (note: I had to put the command at the end of the line otherwise a --format option that had an --out param seemed to swallow the retry attempt), however, once the retry started "working", circle still saw the run as a failure. 😞

streetlogics pushed a commit to streetlogics/cucumber-ruby that referenced this issue Feb 10, 2017
[ISSUE-cucumber#1044] Updates to support retry success status code 0
c34040d 
-  Updated configuration to count the base number of cases being run
-  Updated failure? method to use cases.total_passed and total_cases to determine if the run was successful
@streetlogics
Copy link

streetlogics commented Feb 10, 2017 •
edited
Loading

@mattwynne - I took a really ugly pass at this (master...streetlogics:issue/retry-success-exit-code-0-1044) just to get something working. Definitely open to any thoughts you might have on improving this for the actual implementation

@mattwynne
Copy link
Member

Awesome, thanks @streetlogics one of us will take a look ASAP

@garie
Copy link

garie commented May 9, 2017

Any update? My team is also interested in this fix as a non-zero exit code causes our Bamboo build to fail.

@brasmusson brasmusson mentioned this issue Jun 5, 2017
Closed
@garside
Copy link

garside commented Jun 5, 2017

Just ran into the --retry flag and the non-zero exit code for circleci. Just hoping to get a pulse check on the status of this retry-success feature that @streetlogics proofed.

@mattwynne, Is there an ETA for it and/or is there anything we could do to help push this across the finish line?

@brasmusson
Copy link
Contributor

Yes, in non-strict mode the exit code should be 0 is all scenarios passes on retry, but in strict mode (option --strict) the exit code should not be 0 is any scenarios were retried.

I think a that first the summary should be updated with a new flaky category (as in 6 Scenarios (1 Passed, 1 Failed, 1 Undefined, 1 Pending, 1 Skipped, 1 Flaky)), then it is straight forward to implement the exit code logic. In non-strict mode the exit code should be 0 if and only if there no failed scenarios. In strict mode the exit code should be 0 if and only if there are no failed, flaky, pending or undefined scenarios.

@streetlogics patch does not handle the strict mode correctly, and as I just stated, I think that the exit code should be based on the summary (if the summary reports a failed scenario - then the exit code should be non-zero)

gondalez pushed a commit to gondalez/cucumber-ruby that referenced this issue Jun 13, 2017
@gondalez
[ISSUE-cucumber#1044] Updates to support retry success status code 0
23e2b9e 
-  Updated configuration to count the base number of cases being run
-  Updated failure? method to use cases.total_passed and total_cases to determine if the run was successful
@gondalez gondalez mentioned this issue Jun 13, 2017
Closed
gondalez pushed a commit to gondalez/cucumber-ruby that referenced this issue Jul 11, 2017
@gondalez
[ISSUE-cucumber#1044] Updates to support retry success status code 0
7c7390c 
-  Updated configuration to count the base number of cases being run
-  Updated failure? method to use cases.total_passed and total_cases to determine if the run was successful
@brasmusson brasmusson mentioned this issue Jul 14, 2017
Merged
6 tasks
@streetlogics streetlogics mentioned this issue Jul 17, 2017
Closed
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this issue Mar 21, 2018
devel/ruby-cucumber-core: update to 3.1.0
16b2863 
## [3.0.0](cucumber/cucumber-ruby-core@v3.0.0.pre.2...v3.0.0) (2017-09-27)

### Changed

* Step#name renamed to #text ([#137](cucumber/cucumber-ruby-core#137) [@olleolleolle](https://github.com/olleolleolle))
* Use past tense in event names (`xStarting` -> `xStarted`) (see [cucumber/cucumber-ruby#1166](cucumber/cucumber-ruby#1166) @brasmusson).

### Added

* Do not create test cases for scenarios with no steps ([#144](cucumber/cucumber-ruby-core#144) @brasmusson)
* Handle selective strict settings ([#143](cucumber/cucumber-ruby-core#143) @brasmusson)

### Fixed

* Fix DataTable's Location to be aware of all of its lines ([#142](cucumber/cucumber-ruby-core#142) @botandrose)

### Improved

* As per [#251](cucumber/common#251): renamed History.md to CHANGELOG.md, added contributing message at beginning, and misc formatting. ([#145](cucumber/cucumber-ruby-core#145) [jaysonesmith](https://github.com/jaysonesmith))

## [3.0.0.pre.2](cucumber/cucumber-ruby-core@v2.0.0...3.0.0.pre.2) (2017-07-26)

### New Features

* Add a flaky result type to be used for flaky scenarios ([#141](cucumber/cucumber-ruby-core#141), [cucumber/cucumber-ruby#1044](cucumber/cucumber-ruby#1044) @brasmusson)
* Make the Summary report able to say if the total result is ok ([#140](cucumber/cucumber-ruby-core#140) @brasmusson)
* Replay previous events to new subscribers ([#136](cucumber/cucumber-ruby-core#136) @mattwynne)
* Ruby 2.4.0 compatibility ([#120](cucumber/cucumber-ruby-core#120) @junaruga)
* Use tag expressions ([#116](cucumber/cucumber-ruby-core#116) @brasmusson)
* Access example table row data by param name ([#118](cucumber/cucumber-ruby-core#118) @enkessler)

### Bugfixes

N/A

### Removed Features

N/A

### Refactoring

* Travis: jruby-9.1.10.0 ([#130](cucumber/cucumber-ruby-core#130) @olleolleolle)
* Travis: jruby-9.1.12.0 ([#133](cucumber/cucumber-ruby-core#132) @olleolleolle)
@reach2jeyan
Copy link

Flaky tests still fails the scenario, How can i force cucumber to have a pass if the scenario has passed during the re-run. I see this is the simplest thing I can implement comparatively with cucumber_failures.log or re-run

@xtrasimplicity
Copy link
Member

xtrasimplicity commented Apr 17, 2018 •
edited
Loading

@reach2jeyan have you tried using the flags defined in this (merged) PR?

@reach2jeyan
Copy link

@xtrasimplicity Yes! and still I get cucumber has failed with 1 flaky scenario and since the flaky scenario has previously failed

@xtrasimplicity
Copy link
Member

@reach2jeyan Hmm, that's interesting. Could you please post a MCVE?

@prashanth-sams
Copy link

prashanth-sams commented Jul 12, 2018 •
edited
Loading

@xtrasimplicity but re-run failed tests are not going to be updated in the parent report; it will remain as a separate report with left-over failed scenarios

@mattwynne is there a way to get the consolidated report after re-run tests? or any work-around ignoring the failed tests if the tests are passed on --retry

@lock
Copy link

lock bot commented Jul 12, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked as resolved and limited conversation to collaborators Jul 12, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
3.0.0
Development

Successfully merging a pull request may close this issue.

Add a flaky result type to be used for flaky scenarios. cucumber/cucumber-ruby-core
9 participants
@mattwynne @garie @streetlogics @garside @brasmusson @prashanth-sams @rishi-freshbooks @xtrasimplicity @reach2jeyan