Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[e2e] Steps to reduce flakiness in test reporting #7436

Merged
merged 19 commits into from
Jan 30, 2024
Merged

Conversation

unlikelyzero
Copy link
Collaborator

@unlikelyzero unlikelyzero commented Jan 29, 2024

Closes #7441 #6946

Describe your changes:

  • adds flake finder PR trigger
  • updates circle CI to leverage the custom test runner to shard up Tess and enable test retries

All Submissions:

  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?
  • Is this change backwards compatible? For example, developers won't need to change how they are calling the API or how they've extended core plugins such as Tables or Plots.

Author Checklist

  • Changes address original issue?
  • Tests included and/or updated with changes?
  • Has this been smoke tested?
  • Have you associated this PR with a type: label? Note: this is not necessarily the same as the original issue.
  • Have you associated a milestone with this PR? Note: leave blank if unsure.
  • Is this a breaking change to be called out in the release notes?
  • Testing instructions included in associated issue OR is this a dependency/testcase change?

Reviewer Checklist

  • Changes appear to address issue?
  • Reviewer has tested changes by following the provided instructions?
  • Changes appear not to be breaking changes?
  • Appropriate automated tests included?
  • Code style and in-line documentation are appropriate?

@unlikelyzero unlikelyzero added the pr:e2e:flakefinder Runs the tests 10 times label Jan 29, 2024
@unlikelyzero unlikelyzero marked this pull request as draft January 29, 2024 21:26
Copy link

codecov bot commented Jan 29, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (068ac48) 55.61% compared to head (53213e9) 55.58%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7436      +/-   ##
==========================================
- Coverage   55.61%   55.58%   -0.03%     
==========================================
  Files         668      668              
  Lines       26704    26704              
  Branches     2585     2585              
==========================================
- Hits        14851    14843       -8     
- Misses      11142    11148       +6     
- Partials      711      713       +2     
Flag Coverage Δ *Carryforward flag
e2e-full 41.36% <ø> (ø) Carriedforward from bb27aff
e2e-stable 59.34% <ø> (-0.16%) ⬇️
unit 48.65% <ø> (-0.01%) ⬇️

*This pull request uses carry forward flags. Click here to find out more.

see 3 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 068ac48...53213e9. Read the comment docs.

@github-actions github-actions bot removed the pr:e2e:flakefinder Runs the tests 10 times label Jan 29, 2024
@unlikelyzero unlikelyzero added the pr:e2e:flakefinder Runs the tests 10 times label Jan 29, 2024
@github-actions github-actions bot removed the pr:e2e:flakefinder Runs the tests 10 times label Jan 29, 2024
@unlikelyzero unlikelyzero added the pr:e2e:flakefinder Runs the tests 10 times label Jan 29, 2024
@github-actions github-actions bot removed the pr:e2e:flakefinder Runs the tests 10 times label Jan 29, 2024
@unlikelyzero unlikelyzero changed the title flakefinder gha [e2e] Steps to reduce flakiness in test reporting Jan 30, 2024
@unlikelyzero unlikelyzero marked this pull request as ready for review January 30, 2024 17:40
@unlikelyzero unlikelyzero added this to the Target:4.0.0 milestone Jan 30, 2024
@unlikelyzero unlikelyzero added the type:maintenance tests, chores, or project maintenance label Jan 30, 2024
Copy link
Contributor

@ozyx ozyx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@unlikelyzero unlikelyzero added the pr:e2e:flakefinder Runs the tests 10 times label Jan 30, 2024
@github-actions github-actions bot removed the pr:e2e:flakefinder Runs the tests 10 times label Jan 30, 2024
@unlikelyzero unlikelyzero merged commit 69b81c0 into master Jan 30, 2024
10 of 11 checks passed
@unlikelyzero unlikelyzero deleted the flakefinder branch January 30, 2024 22:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:maintenance tests, chores, or project maintenance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a flake finder GitHub action to isolate flake
2 participants