Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ci] Automatically retry failed native unit test failures once. #1095

Merged
merged 1 commit into from
Apr 11, 2023

Conversation

jpobst
Copy link
Contributor

@jpobst jpobst commented Apr 10, 2023

Context: #514

There is a known native crash that occasionally occurs when running unit tests on CI: #514.

This crash doesn't happen "in the wild" so it is unlikely to get fixed anytime soon. As an workaround for our CI, we will use AzDO's retryCountOnTaskFailure parameter to automatically re-run a native test suite once if it fails.

This should reduce the need for us to re-run failed pipelines for flaky tests.

Note that AzDO's implementation isn't perfect, the task will succeed if there are no failures on retry, but the original failures will still be listed in the log. 🤷‍♂️

image

(Pipeline Run)

@jpobst jpobst marked this pull request as ready for review April 10, 2023 23:30
@jonpryor jonpryor merged commit 554d819 into main Apr 11, 2023
@jonpryor jonpryor deleted the test-retry branch April 11, 2023 12:18
jonpryor pushed a commit to dotnet/android that referenced this pull request Apr 13, 2023
Changes: dotnet/java-interop@a172402...554d819

  * dotnet/java-interop@554d8193: [ci] Automatically retry failed native unit test failures once (dotnet/java-interop#1095)
  * dotnet/java-interop@71e586b0: [class-parse] Ignore `module-info.class` file. (dotnet/java-interop#1093)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
jpobst added a commit to dotnet/android that referenced this pull request Apr 17, 2023
The `apk-instrumentation` tests are a good candidate for using `retryCountOnTaskFailure`.  This is because they only take a few minutes to run, and have a low enough flaky rate that it is likely that rerunning the entire test suite will succeed.

As noted in dotnet/java-interop#1095, AzDO's support for this isn't perfect.  The initial failure will still show up in the Error Log, however the task will be green and will not show up as a failed build.
@github-actions github-actions bot locked and limited conversation to collaborators Apr 12, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants