Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add job to test flakiness of added/changed tests #63943

Merged
merged 6 commits into from
Apr 1, 2024

Conversation

ijjk
Copy link
Member

@ijjk ijjk commented Apr 1, 2024

To help detect when newly added/changed assertions are flakey this adds a job to our build and test workflow to re-run them 3 times. If the changed test is an E2E test it will re-run in both development and production mode to ensure it's not flakey specifically in one of those modes.

Test run with changed test can be seen here https://github.com/vercel/next.js/actions/runs/8511797725/job/23312158523?pr=63943

Closes NEXT-2973

@ijjk ijjk requested review from manovotny and leerob and removed request for a team April 1, 2024 18:05
@ijjk ijjk added the created-by: Next.js team PRs by the Next.js team. label Apr 1, 2024
@ijjk ijjk marked this pull request as draft April 1, 2024 18:08
@ijjk ijjk marked this pull request as ready for review April 1, 2024 18:17
@ijjk
Copy link
Member Author

ijjk commented Apr 1, 2024

Stats from current PR

Default Build (Increase detected ⚠️)
General
vercel/next.js canary ijjk/next.js ci/flakey-tests Change
buildDuration 13.7s 13.8s ⚠️ +110ms
buildDurationCached 7.4s 6.1s N/A
nodeModulesSize 199 MB 199 MB
nextStartRea..uration (ms) 399ms 414ms N/A
Client Bundles (main, webpack)
vercel/next.js canary ijjk/next.js ci/flakey-tests Change
2453-HASH.js gzip 31.4 kB 31.4 kB N/A
3304.HASH.js gzip 181 B 181 B
3f784ff6-HASH.js gzip 53.7 kB 53.7 kB
8299-HASH.js gzip 5.04 kB 5.04 kB N/A
framework-HASH.js gzip 45.2 kB 45.2 kB
main-app-HASH.js gzip 241 B 242 B N/A
main-HASH.js gzip 32.2 kB 32.2 kB N/A
webpack-HASH.js gzip 1.68 kB 1.68 kB N/A
Overall change 99 kB 99 kB
Legacy Client Bundles (polyfills)
vercel/next.js canary ijjk/next.js ci/flakey-tests Change
polyfills-HASH.js gzip 31 kB 31 kB
Overall change 31 kB 31 kB
Client Pages
vercel/next.js canary ijjk/next.js ci/flakey-tests Change
_app-HASH.js gzip 196 B 197 B N/A
_error-HASH.js gzip 184 B 184 B
amp-HASH.js gzip 505 B 505 B
css-HASH.js gzip 324 B 325 B N/A
dynamic-HASH.js gzip 2.5 kB 2.5 kB N/A
edge-ssr-HASH.js gzip 258 B 258 B
head-HASH.js gzip 352 B 352 B
hooks-HASH.js gzip 370 B 371 B N/A
image-HASH.js gzip 4.21 kB 4.21 kB
index-HASH.js gzip 259 B 259 B
link-HASH.js gzip 2.67 kB 2.67 kB N/A
routerDirect..HASH.js gzip 314 B 312 B N/A
script-HASH.js gzip 386 B 386 B
withRouter-HASH.js gzip 309 B 309 B
1afbb74e6ecf..834.css gzip 106 B 106 B
Overall change 6.57 kB 6.57 kB
Client Build Manifests
vercel/next.js canary ijjk/next.js ci/flakey-tests Change
_buildManifest.js gzip 481 B 484 B N/A
Overall change 0 B 0 B
Rendered Page Sizes
vercel/next.js canary ijjk/next.js ci/flakey-tests Change
index.html gzip 529 B 527 B N/A
link.html gzip 541 B 540 B N/A
withRouter.html gzip 524 B 522 B N/A
Overall change 0 B 0 B
Edge SSR bundle Size
vercel/next.js canary ijjk/next.js ci/flakey-tests Change
edge-ssr.js gzip 95.3 kB 95.3 kB N/A
page.js gzip 3.04 kB 3.04 kB N/A
Overall change 0 B 0 B
Middleware size
vercel/next.js canary ijjk/next.js ci/flakey-tests Change
middleware-b..fest.js gzip 624 B 622 B N/A
middleware-r..fest.js gzip 151 B 151 B
middleware.js gzip 25.5 kB 25.5 kB N/A
edge-runtime..pack.js gzip 839 B 839 B
Overall change 990 B 990 B
Next Runtimes
vercel/next.js canary ijjk/next.js ci/flakey-tests Change
app-page-exp...dev.js gzip 170 kB 170 kB
app-page-exp..prod.js gzip 97 kB 97 kB
app-page-tur..prod.js gzip 98.8 kB 98.8 kB
app-page-tur..prod.js gzip 93 kB 93 kB
app-page.run...dev.js gzip 144 kB 144 kB
app-page.run..prod.js gzip 91.5 kB 91.5 kB
app-route-ex...dev.js gzip 21.4 kB 21.4 kB
app-route-ex..prod.js gzip 15.2 kB 15.2 kB
app-route-tu..prod.js gzip 15.2 kB 15.2 kB
app-route-tu..prod.js gzip 14.9 kB 14.9 kB
app-route.ru...dev.js gzip 21.1 kB 21.1 kB
app-route.ru..prod.js gzip 14.9 kB 14.9 kB
pages-api-tu..prod.js gzip 9.55 kB 9.55 kB
pages-api.ru...dev.js gzip 9.82 kB 9.82 kB
pages-api.ru..prod.js gzip 9.55 kB 9.55 kB
pages-turbo...prod.js gzip 22.5 kB 22.5 kB
pages.runtim...dev.js gzip 23.1 kB 23.1 kB
pages.runtim..prod.js gzip 22.4 kB 22.4 kB
server.runti..prod.js gzip 51 kB 51 kB
Overall change 945 kB 945 kB
build cache Overall increase ⚠️
vercel/next.js canary ijjk/next.js ci/flakey-tests Change
0.pack gzip 1.58 MB 1.58 MB N/A
index.pack gzip 105 kB 106 kB ⚠️ +586 B
Overall change 105 kB 106 kB ⚠️ +586 B
Diff details
Diff for middleware.js

Diff too large to display

Commit: 656d019

scripts/test-new-tests.mjs Outdated Show resolved Hide resolved
@ijjk ijjk requested a review from ztanner April 1, 2024 18:50
@Ottapettavan Ottapettavan mentioned this pull request Apr 1, 2024
Copy link
Contributor

@Ethan-Arrowood Ethan-Arrowood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed with @WITS during donut call.

scripts/test-new-tests.mjs Outdated Show resolved Hide resolved
scripts/test-new-tests.mjs Outdated Show resolved Hide resolved
scripts/test-new-tests.mjs Show resolved Hide resolved
Co-authored-by: Ethan Arrowood <ethan@arrowood.dev>
@ijjk ijjk requested a review from Ethan-Arrowood April 1, 2024 19:52
@ijjk ijjk enabled auto-merge (squash) April 1, 2024 20:05
@ijjk ijjk merged commit 2359d3d into vercel:canary Apr 1, 2024
72 checks passed
@ijjk ijjk deleted the ci/flakey-tests branch April 1, 2024 20:35
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 16, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
created-by: Next.js team PRs by the Next.js team. locked
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants