Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add decorator for tests that still runs test but not count result #297

Open
andrewhsu opened this issue Mar 29, 2022 · 0 comments
Open

add decorator for tests that still runs test but not count result #297

andrewhsu opened this issue Mar 29, 2022 · 0 comments

Comments

@andrewhsu
Copy link
Contributor

It would be helpful to have a way to run tests that are known to be flakey but not have the test results be counted towards passing or failing in the final result. It is helpful to run flakey tests to see how often they fail in the context of all the other tests because the flakiness could be caused by the order of the test runs.

The --exclude functionality simply skips the tests and the @ignore decorator also does not run tests but at least mentions in the result tests not run which is helpful to have a quick summary of how many flakey tests were ignored.

In the old issue #112 there was a mention of a decorator called @ok_to_fail that seems to imply the functionality desired. The decorator approach helps keep the expected test behaviour close to the definition of the test.

Would need reporter.py to categorize the results of tests decorated as @ok_to_fail. I suggest having 2 additional test status OPASS and OFAIL to distinguish from existing test status PASS, FAIL, and IGNORE.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant