chore: better validation messages for experimental retries

[MuazOthman]

• Closes CYCLOUD-1765

PR Tasks

• Have tests been added/updated?
• [n/a] Has a PR for user-facing changes been opened in cypress-documentation?
• [n/a] Have API changes been updated in the type definitions?

[emilyrohrbough]

Could you add screenshots of these error combinations?

[mabela416]

Not passing passedRequired for detect-flake-and-pass-on-threshold and stopIfAnyPassed for detect-flake-but-always-fail is NOT erroring but it's supposed to return an error.

[mabela416]

For this legacy config, I'm also NOT getting an error when starting the run but I do after the run is done

  retries: {
    openMode: true,
    runMode: true,
  },

BUT in develop, I do get an error when trying to start my run

[mabela416]

this config should also error out but instead it lets me do a run which ends with this error

  retries: {
    runMode: true,
    openMode: true
  },

[MuazOthman]

@mabela416 I fixed the validation for the last 2 comments. Validating fields within experimentalOptions are beyond the scope of this PR since adding that would be a change of behavior. cc: @ryanpei for feedback on that.

[cypress]

5 flaky tests on run #52558 ↗︎

0

5467

80

0

5

Details:

Merge branch 'develop' into muaz/CYCLOUD-1765-better-retries-validation
Project: cypress	Commit: 899ea4364d
Status: Passed	Duration: 17:06 💡
Started: Nov 28, 2023 6:50 PM	Ended: Nov 28, 2023 7:07 PM

  commands/storage.cy.ts • 1 flaky test • 5x-driver-chrome:beta

View Output

Test		Artifacts
src/cy/commands/storage > #getAllLocalStorage > gets local storage from all origins		Test Replay

  e2e/origin/commands/storage.cy.ts • 1 flaky test • 5x-driver-chrome:beta

View Output

Test		Artifacts
cy.origin storage > .getAllLocalStorage		Test Replay

  cypress/cypress.cy.js • 3 flaky tests • 5x-driver-chrome:beta

View Output

Test		Artifacts
... > correctly returns currentRetry		Test Replay
... > correctly returns currentRetry		Test Replay
... > correctly returns currentRetry		Test Replay

Review all test suite changes for PR #28214 ↗︎

[mabela416]

@mabela416 I fixed the validation for the last 2 comments. Validating fields within experimentalOptions are beyond the scope of this PR since adding that would be a change of behavior. cc: @ryanpei for feedback on that.

@MuazOthman Validating fields within experimentalOptions is already happening in develop and it's being removed in this PR which should not be happening

[mabela416]

This is the error i get in develop when I don't pass passesRequired or stopIfAnyPassed

[mabela416]

One small nitpick:
When I pass a config without the runMode and openMode, I get this message but I think it should say the value passed was undefined

[MuazOthman]

@mabela416 it shows 0 in that error message because this is the default value, we can probably make this less confusing once strategy-based retries becomes GA.

[MuazOthman]

@emilyrohrbough validation error messages can be seen under packages/config/__snapshots__/validation.spec.ts.js

[cypress-bot]

Released in 13.6.1.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v13.6.1, please open a new issue.