Skip to content

Flaky Test Management

Jump to bottom
Andrew Stanton-Nurse edited this page Mar 15, 2019 · 1 revision

As much as we try to keep our test runs consistent, we have some situations in which tests become flaky. Rather than disable those tests entirely, or stop everything to fix them, we have a system to "sequester" flaky tests into a separate test pass that does not fail the build if it fails.

This attribute works in aspnet/Extensions and aspnet/AspNetCore, but not in aspnet/EntityFrameworkCore currently (see https://github.com/aspnet/EntityFrameworkCore/issues/15003).

To mark a test as flaky:

  1. Ensure you have a reference to Microsoft.AspNetCore.Testing in your test project.
  2. Apply the [Flaky] attribute to the test, along with [Fact]/[Theory]/[ConditionalFact]/[ConditionalTheory] (all of those are supported)
  3. Fill in the mandatory first parameters with: A GitHub Issue URL linking to an issue in aspnet/AspNetCore-Internal tracking the test failure, which is labelled test-failure
  4. Fill in at least one "filter" which defines where the test is flaky. There is a static class FlakyOn that has these filters:
    • FlakyOn.All - The test is flaky in any and all environments
    • FlakyOn.Helix.All - The test is flaky on all Helix Queues
    • FlakyOn.Helix.[QueueName] - The test is flaky on the specified Helix queue
    • FlakyOn.AzP.All - The test is flaky in all Azure Pipelines environments
    • FlakyOn.AzP.[OSName] - The test is flaky in Azure Pipelines builds on the specified OS
    • Multiple filters can be applied to mark the test as flaky in multiple environments

In the environments in which the test is marked as flaky, the test will not be run during the standard test pass. Instead, it will be run in a separate pass that doesn't fail the build. Test results are still collected by the build (where applicable) and the "Tests" page on Azure Pipelines will show the failures. However, GitHub will show a green check (as long as all non-flaky tests pass). In any other environments, the test will run during the standard test pass and failures will fail the build.

For questions, tag @aspnet/brt in a comment or ask in the ASP.NET-CI channel on the DotNet Team in Microsoft Teams

Clone this wiki locally