chore: make some test retries possible to succeed

[dummdidumm]

We do not have a reset function on those, and they sometimes fail for mysterious reason even if the count is updated, but playwright somehow doesn't notice. This makes it possible for retries to succeed in that case.

Alternative would be to have a reset function on all of them, but I'm not sure if this then has the "fetch goes into the abyss" problem.

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

• It's really useful if your PR references an issue where it is discussed ahead of time. In many cases, features are absent for a reason. For large changes, please create an RFC: https://github.com/sveltejs/rfcs
• This message body should clearly illustrate what problems it solves.
• Ideally, include a test that fails without this PR but passes with it.

Tests

• Run the tests with pnpm test and lint the project with pnpm lint and pnpm check

Changesets

• If your PR makes a change that should be noted in one or more packages' changelogs, generate a changeset by running pnpm changeset and following the prompts. Changesets that add features should be minor and those that fix bugs should be patch. Please prefix changeset messages with feat:, fix:, or chore:.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 708130f

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[benmccann]

I'm not sure this works exactly right. It assumes the counter is incremented exactly once for each retry, but depending on where the test fails it could be incremented zero to two times. Maybe you need to check the initial counter value at the beginning of the test

But I'm also not sure we need to do this at all. These tests are being affected by the same thing as the form action tests, so we should probably fix them in the same way (e.g. #8570)

[dummdidumm]

How would a fix look like besides "not run it"? I'm out of ideas at this point. @Rich-Harris brought up the good idea of grouping all flaky tests in a single describe("flaky tests", ..) which we give way more retries than now, and set the retry in general to 0. If we go that route, we need tests to be retriable. So yeah parse the number from the text or add a reset function.

[benmccann]

I think the issue is probably somewhere on Playwright or WebKit side. I filed one issue with PlayWright before and they didn't look at it because the reproduction was too complicated. Assuming I can't more easily reproduce something that they can look at, I think that not running the tests in WebKit may be more preferable to running on a flaky unreliable platform that doesn't add much in terms of confidence. There's a few PRs dealing with this, so I opened an issue so that we can have a single place to discuss: #8578 (or now we have four places instead of three 😆)

[benmccann]

This hopefully shouldn't need retries now that #8585 was merged. That being said, I'll leave this open since it is still an issue that if it did do a retry for some reason it might not work as you've pointed out. I'll leave it to you to decide if it's worth tweaking this to add a reset function or something or if it's not worth pursuing anymore

[Rich-Harris]

i haven't seen any flakiness since the recent change around cross-browser tests, so i'm gonna be optimistic and close this