Flaky Tests

Handling flaky tests in iroh

If you encounter a flaky test in your PR's CI run which is not caused by changes in your PR itself you should not simply re-run the CI. You should:

• Mark the test with #[ignore = "flaky"].
  • #[cfg_attr(target_os = "windows", ignore = "flaky")] works for platform-specific ignores.
• Create an issue labelled flaky-test for this specific test.
  • Optionally assign this to a relevant person if you have a reasonable guess at this.

From now on this flaky test will not slow down normal CI runs, and most importantly merge-group tests.

Running flaky tests

It is possible to run the test suite including all flaky tests in several ways:

• Locally use the --run-ignored flag to cargo nextest run command (or --ignored to cargo test).
• There is a scheduled CI job which periodically runs all the tests, including flaky ones.
• On a PR you can apply the flaky-test label and it will run all tests, including flaky tests.

Ideally we work towards having no flaky tests at all.