You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It would be helpful to have a way to run tests that are known to be flakey but not have the test results be counted towards passing or failing in the final result. It is helpful to run flakey tests to see how often they fail in the context of all the other tests because the flakiness could be caused by the order of the test runs.
The --exclude functionality simply skips the tests and the @ignore decorator also does not run tests but at least mentions in the result tests not run which is helpful to have a quick summary of how many flakey tests were ignored.
In the old issue #112 there was a mention of a decorator called @ok_to_fail that seems to imply the functionality desired. The decorator approach helps keep the expected test behaviour close to the definition of the test.
Would need reporter.py to categorize the results of tests decorated as @ok_to_fail. I suggest having 2 additional test status OPASS and OFAIL to distinguish from existing test status PASS, FAIL, and IGNORE.
The text was updated successfully, but these errors were encountered:
It would be helpful to have a way to run tests that are known to be flakey but not have the test results be counted towards passing or failing in the final result. It is helpful to run flakey tests to see how often they fail in the context of all the other tests because the flakiness could be caused by the order of the test runs.
The
--exclude
functionality simply skips the tests and the@ignore
decorator also does not run tests but at least mentions in the result tests not run which is helpful to have a quick summary of how many flakey tests were ignored.In the old issue #112 there was a mention of a decorator called
@ok_to_fail
that seems to imply the functionality desired. The decorator approach helps keep the expected test behaviour close to the definition of the test.Would need
reporter.py
to categorize the results of tests decorated as@ok_to_fail
. I suggest having 2 additional test statusOPASS
andOFAIL
to distinguish from existing test statusPASS
,FAIL
, andIGNORE
.The text was updated successfully, but these errors were encountered: