ci: flake detection should run in both bundlers

[ztanner]

We run flake detection on PRs to help catch flaky tests before they hit canary, but we currently only run flaky test detection in Webpack and not Turbopack. This is problematic because there's sufficient forking logic in tests for the different bundlers & we've seen a high likelihood of a test to flake more frequently in one bundler vs another due to things like differing compilation speeds and differing implementation of core features.

This will add some additional time to this CI job to account for needing to run the tests 6 times, but will hopefully catch things before they become a problem later. I think we could also consider reducing the number of times per bundler to be 2 if it turns out to be adding too much time.

This updates the flake detection job to run the tests multiple times in Turbopack as well.

CI run: https://github.com/vercel/next.js/actions/runs/11822662639/job/32940264160?pr=72773

[ztanner]

• ci: flake detection should run in both bundlers #72773 👈
• canary

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @ztanner and the rest of your teammates on Graphite

[ijjk]

Do we need to parallelize more to avoid these being slow?

[ztanner]

Do we need to parallelize more to avoid these being slow?

I think we'll need to see how it goes -- I picked a beefier test to modify and it finished in dev/start flake detection before prod tests finished. I think the current parallelization often goes under utilized because we typically only see a few test changes per PR, but if this causes problems I'd probably want to split the Turbopack flake detection into a separate job

[ijjk]

Failing test suites

Commit: 108d027

__NEXT_EXPERIMENTAL_PPR=true pnpm test-dev test/development/app-hmr/hmr.test.ts (PPR)

• app-dir-hmr > filesystem changes > should update server components after navigating to a page with a different runtime

Expand output

● app-dir-hmr › filesystem changes › should update server components after navigating to a page with a different runtime

expect(received).toBe(expected) // Object.is equality

Expected: "ipad"
Received: "mac"

  65 |       await next.patchFile(envFile, 'MY_DEVICE="ipad"', async () => {
  66 |         await retry(async () => {
> 67 |           expect(await browser.elementByCss('p').text()).toBe('ipad')
     |                                                          ^
  68 |         })
  69 |
  70 |         const logs = await browser.log()

  at toBe (development/app-hmr/hmr.test.ts:67:58)
  at retry (lib/next-test-utils.ts:806:14)
  at development/app-hmr/hmr.test.ts:66:9
  at NextDevInstance.patchFile (lib/next-modes/base.ts:522:9)
  at NextDevInstance.patchFile (lib/next-modes/next-dev.ts:195:16)
  at Object.<anonymous> (development/app-hmr/hmr.test.ts:65:7)

Read more about building and testing Next.js in contributing.md.