fix: the cookie_behavior tests by syncing cookies immediately if … #25855
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…the application is already stable
AtofStryker
changed the title
28 flaky tests on run #44251 ↗︎
Details:
|
|||||||||||||||||||||||||||||||||||||||||||||
| Test | ||
|---|---|---|
| ... > runs generated spec |
Screenshot
|
|
specs_list_latest_runs.cy.ts • 1 flaky test • app-e2e
| Test | ||
|---|---|---|
| App/Cloud Integration - Latest runs and Average duration > when no runs are recorded > shows placeholders for all visible specs |
Screenshot
|
|
cypress-in-cypress-component.cy.ts • 1 flaky test • app-e2e
| Test | ||
|---|---|---|
| Cypress In Cypress CT > default config > redirects to the specs list with error if a spec is not found |
Screenshot
|
|
studio/studio.cy.ts • 1 flaky test • app-e2e
| Test | ||
|---|---|---|
| Cypress Studio > updates an existing test with a click action |
Screenshot
|
|
cypress-origin-communicator.cy.ts • 1 flaky test • app-e2e
| Test | ||
|---|---|---|
| Cypress In Cypress Origin Communicator > cy.origin passivity with app interactions > passes upon test reload mid test execution |
Screenshot
|
|
The first 5 flaky specs are shown, see all 17 specs in Cypress Cloud.
This comment has been generated by cypress-bot as a result of this project's GitHub integration settings.
AtofStryker
force-pushed
the
fix/set-cross-origin-cookies-if-stable
branch
from
3d824ac to
8001640
Compare
AtofStryker
requested review from
chrisbreiding,
emilyrohrbough,
mjhenkes and
mschile
| @@ -6,6 +6,7 @@ _Released 03/1/2023 (PENDING)_ | |||
| **Bugfixes:** | |||
|
|
|||
| - Fixed an issue where cookies were being duplicated with the same hostname, but a prepended dot. Fixed an issue where cookies may not be expiring correctly. Fixes [#25174](https://github.com/cypress-io/cypress/issues/25174), [#25205](https://github.com/cypress-io/cypress/issues/25205) and [#25495](https://github.com/cypress-io/cypress/issues/25495). | |||
| - Fixed an issue where cookies weren't being synced when the application was stable. Fixed in [#25855](https://github.com/cypress-io/cypress/pull/25855). Fixes [#25835](https://github.com/cypress-io/cypress/issues/25835). | |||
There was a problem hiding this comment.
is this a user-facing fix or just a test fix? if its just for tests, this does't need an entry
There was a problem hiding this comment.
It is also user facing since the cookies will now sync immediately if Cypress is stable
There was a problem hiding this comment.
What does that mean to users though?
There was a problem hiding this comment.
in certain cases cookies weren't synced back to the automation client when using cy.origin (or if a cross origin injection occurs) if the application is already stable. This is mainly problematic for setting cookies inside cy.origin after the page has loaded / is stable that the app couldnt otherwise set
chrisbreiding
approved these changes
Feb 17, 2023
…cross-origin-cookies-if-stable
mjhenkes
approved these changes
Feb 20, 2023
mjhenkes
added a commit
that referenced
this pull request
Feb 21, 2023
* fix: update newProject ref when switching between organizations in SelectCloudProjectModal (#25730) * chore: debug page tooltip distance and artifact border (#25727) * misc: debug page tooltip distance and artifact border * add changelog entry * fix CT test * fix: Improve error handling around calls to `this.next` in middleware (#25702) * chore: update changelog validation example (#25742) * misc: improve debug loading text wrap responsiveness (#25703) * misc: Increase max failures in IATR badge to 99 (#25737) * chore: exclude collaborator issues/PRs from triage project (#25769) * feat: add --auto-cancel-after-failures flag (#25237) Co-authored-by: Emily Rohrbough <emilyrohrbough@users.noreply.github.com> Co-authored-by: Matt Schile <mschile@cypress.io> Co-authored-by: Ryan Pei <ryanppei@gmail.com> Co-authored-by: Emily Rohrbough <emilyrohrbough@yahoo.com> * chore: Update v8 snapshot cache (#25592) * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * Update update_v8_snapshot_cache.yml * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache --------- Co-authored-by: cypress-bot[bot] <2f0651858c6e38e0+cypress-bot[bot]@users.noreply.github.com> Co-authored-by: Ryan Manuel <ryanm@cypress.io> Co-authored-by: cypress-bot[bot] <47117332+cypress-bot[bot]@users.noreply.github.com> * fix: implement new graphql fields for spec counts (#25757) Co-authored-by: Stokes Player <stokes@cypress.io> Co-authored-by: Mike Plummer <mike-plummer@users.noreply.github.com> * feat: Bundle cy.origin() dependencies at runtime (#25626) Co-authored-by: cypress-bot[bot] <2f0651858c6e38e0+cypress-bot[bot]@users.noreply.github.com> Co-authored-by: Ryan Manuel <ryanm@cypress.io> * chore: remove zenhub from release process (#25701) Co-authored-by: Matt Schile <mschile@cypress.io> * feat: add Cypress.Commands.overwriteQuery (#25674) * feat: add Cypress.Commands.overwriteQuery Co-authored-by: Emily Rohrbough <emilyrohrbough@users.noreply.github.com> Co-authored-by: Zach Bloomquist <git@chary.us> * fix: spawn child process with process.env in macOS arm64 (#25753) Co-authored-by: Matt Schile <mschile@cypress.io> Co-authored-by: Emily Rohrbough <emilyrohrbough@users.noreply.github.com> Co-authored-by: Zach Bloomquist <github@chary.us> * chore: lint system tests in CI (#25673) * fix: Suppress filesystem errors during glob search (#25774) * chore: issue with ts-loader missing in binary and problematic esbuild norewrite construct (#25797) * chore: update changelog linting (#25809) * docs(guides): add more detail to code-signing (#25794) Co-authored-by: Emily Rohrbough <emilyrohrbough@users.noreply.github.com> * chore: update workflows.yml to include the v8 snapshot update branch (#25784) Co-authored-by: cypress-bot[bot] <+cypress-bot[bot]@users.noreply.github.com> * chore: internal request preflight (#25772) --------- Co-authored-by: Emily Rohrbough <emilyrohrbough@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: cypress-bot[bot] <2f0651858c6e38e0+cypress-bot[bot]@users.noreply.github.com> Co-authored-by: Ryan Manuel <ryanm@cypress.io> Co-authored-by: Matt Henkes <mjhenkes@gmail.com> Co-authored-by: Zach Bloomquist <git@chary.us> * chore: bump for 12.6.0 release (#25812) * chore: release @cypress/webpack-batteries-included-preprocessor-v2.4.0 [skip ci] * chore: release @cypress/webpack-preprocessor-v5.17.0 [skip ci] * test: skip flaky GitDataSource test (#25825) * chore: making our add-to-triage-board workflow reusable within the Cypress-io org (#25820) * chore: Making our add to triage workflow callable from other projects inside the Cypress-io org in Github * chore: updated cypress-example-kitchensink version (#25828) * fix: duplicate and expired cookies (#25761) * chore: add regression tests for duplicate cookies and bad expiry times * avoid prepending domain with dot for cookies that are set with the server side jar. This is to avoid the cookie being duplicated if it is set or overridden in a different context (request that can actually set the cookie or via document.domain) * feat: use cookie.toString() in the cookie patch to more accurately set cookies on the document, which should include other properties besides key=value * fix: add logic to handle expired cookies in the document.cookie patch, as well as in CDP * chore: build binary for cookie fixes for users to test * chore: change name of fixture to something more accurate * chore: comment why we are using the toughCookie toString method in the patch * [run ci] * chore: add changelog entry * [run ci] * fix: revert back to key=value when getting document.cookie as those are the only values are displayed (oversight on my end) * [run ci] * chore: make compatible with cypress.require * fix: add tests for hostOnly/non hostOnly cookies to make sure property gets sent up to automation client correctly. No longer need custom cookie prop to determine destination * [run ci] * fix: stale unit test * chore: adjust comments * [run ci] * fix: bad domain logic * [run ci] * chore: remove irrelevant comment * [run ci] * fix: adjust cookie login text to spec hostOnly cookie within the cookie patch. This should yield the same behavior as we are bound to same origin within the spec bridge * [run ci] * [run ci] * fix: allow for cookies on request of same key to take precedence over cookies in the jar, regardless of how many hierachy cookies exist in the jar * chore: fix cookie misc tests for cy.origin (dont run cy.origin) * [run ci] * chore: skip misc cookie tests in webkit as headless behavior doesn't clear cookies between tests correctly * Revert "fix: allow for cookies on request of same key to take precedence over cookies in the jar, regardless of how many hierachy cookies exist in the jar" This reverts commit 17de188. * [run ci] * chore: split changelog entry into two parts * chore: update logic to remove else statement and add comments * [run ci] * chore: readd windows snapshot branch in workflows * [run ci] * chore: fix workflows from bad merge * [run ci] * Revert "chore: split changelog entry into two parts" This reverts commit 4352ef5. * [run ci] * fix: Fix type definitions for cy.reload() (#25779) Co-authored-by: Emily Rohrbough <emilyrohrbough@users.noreply.github.com> * misc: Debug header updates (#25823) * fix: allow running tests outside Vite project root folder (#25801) * fix: allow running tests outside Vite project root folder * update snapshots * add changelog entry --------- Co-authored-by: Lachlan Miller <lachlan.miller.1990@outlook.com> * fix: mount component in [data-cy-root] (#25807) * fix(angular): mount component in [data-cy-root] * fix e2e test * add changelog entry * changelog [skip ci] * changelog --------- Co-authored-by: Lachlan Miller <lachlan.miller.1990@outlook.com> * chore: updating add to triage baord github action to use org secret (#25868) * chore: updating add to triage board github action to use org secret * chore: release @cypress/angular-v2.0.2 [skip ci] * chore: release @cypress/vite-dev-server-v5.0.3 [skip ci] * chore: Update v8 snapshot cache (#25822) Co-authored-by: cypress-bot[bot] <+cypress-bot[bot]@users.noreply.github.com> Co-authored-by: Ryan Manuel <ryanm@cypress.io> * feat: support host only cookies (#25853) * feat: allow setCookie API to take a hostOnly option * chore: add jsdoc/typescript description to render to users * chore: add changelog entry * [run ci] * chore: fix types * chore: fix cookie login tests * chore: update e2e cookie system tests * [run ci] * chore: fix cookie command tests. localhost cookies are calculated as hostOnly, which is consistent with how cypress works today * chore: fix system tests for cookies. * [run ci] * chore: fix system tests * chore: skip hostOnly assertions in webkit (for now) * [run ci] * chore: add property definitions to setCookieOptions * [run ci] * chore: add comments to hostOnly prop in firefox when setting a cookie * fix(webpack-dev-server): touch component-index during onSpecsChange to avoid writing to app file (#25861) * testing: try disabling uTimesSync and see what happens * build binaries [run ci] * fix: touch component index file instead of browser.js * build binaries [run ci] * update test * fix test * add test for custom HTML file in config * use existing component index in webpack-dev-server unit tests --------- Co-authored-by: Lachlan Miller <lachlan.miller.1990@outlook.com> * chore: release @cypress/webpack-dev-server-v3.2.4 [skip ci] * chore: improve types for server automation cookie client (#25836) * chore: improve types for automation cookies * [run ci] * fix: the cookie_behavior tests by syncing cookies immediately if … (#25855) * fix: fix the cookie_behavior tests by syncing cookies immediately if the application is already stable * chore: add changelog entry * [run ci] * chore: address comments from code review * feat: Public API for CT Framework Definitions (#25780) * chore: rework component onboarding in launchpad (#25713) * chore: refactoring and types * rework source of frameworks * revert rename * fix tests * fix more tests * types * update code * use same public API internally * rename interfaces * rename * work on dev server api * fix types * fix test * attempt to support getDevServerConfig * tests * add function to define framework [skip ci] * rework a lot of types * fix test * update tests and types * refactor * revert changes * lint * fix test * revert * remove * add "community" label [skip ci] * refactor * types * lint * fix bug * update function name * address feedback * improve types with Pick * refactor using type guard * correct label --------- Co-authored-by: Zachary Williams <ZachJW34@gmail.com> * chore: typing error * feat: scan for 3rd party ct plugins (#25749) * chore: refactoring and types * rework source of frameworks * revert rename * fix tests * fix more tests * types * update code * use same public API internally * rename interfaces * rename * work on dev server api * fix types * fix test * attempt to support getDevServerConfig * tests * add function to define framework [skip ci] * rework a lot of types * fix test * update tests and types * refactor * revert changes * lint * fix test * revert * remove * add "community" label [skip ci] * refactor * types * lint * fix bug * update function name * address feedback * feat: scan for 3rd party ct plugins * add e2e test * unit tests [run ci] * tweak resolution * rebase, address comments * fix windows paths * remove .gitignore * fix test --------- Co-authored-by: Lachlan Miller <lachlan.miller.1990@outlook.com> * lint config * spacing * try fix race cond * fix import error * build binary * try update snapshot * try using require * support namespaced definitions (#25804) * remove category * add icon prop * support esm -> cjs compiled typescript * fix test * misc: add CTA footer to launchpad framework dropdown (#25831) * remove test project dependencies * rebase * windows * windows again * add changelog entry * changelog * revert workflow * remove worklfow --------- Co-authored-by: Zachary Williams <ZachJW34@gmail.com> Co-authored-by: Adam Stone-Lord <adams@cypress.io> * chore: release @cypress/webpack-dev-server-v3.3.0 [skip ci] * fix: Add missing error message when `req.continue` is used incorrectly (#25884) --------- Co-authored-by: Adam Stone-Lord <adams@cypress.io> Co-authored-by: Zachary Williams <ZachJW34@gmail.com> Co-authored-by: Mike Plummer <mike-plummer@users.noreply.github.com> Co-authored-by: Matt Schile <mschile@cypress.io> Co-authored-by: Alejandro Estrada <estrada9166@gmail.com> Co-authored-by: Emily Rohrbough <emilyrohrbough@users.noreply.github.com> Co-authored-by: Ryan Pei <ryanppei@gmail.com> Co-authored-by: Emily Rohrbough <emilyrohrbough@yahoo.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: cypress-bot[bot] <2f0651858c6e38e0+cypress-bot[bot]@users.noreply.github.com> Co-authored-by: Ryan Manuel <ryanm@cypress.io> Co-authored-by: cypress-bot[bot] <47117332+cypress-bot[bot]@users.noreply.github.com> Co-authored-by: Mark Noonan <mark@cypress.io> Co-authored-by: Stokes Player <stokes@cypress.io> Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com> Co-authored-by: Zach Bloomquist <git@chary.us> Co-authored-by: willmsC <50909991+willmsC@users.noreply.github.com> Co-authored-by: Zach Bloomquist <github@chary.us> Co-authored-by: cypress-bot[bot] <+cypress-bot[bot]@users.noreply.github.com> Co-authored-by: Tim Griesser <tgriesser10@gmail.com> Co-authored-by: Matt Henkes <mjhenkes@gmail.com> Co-authored-by: semantic-release-bot <semantic-release-bot@martynus.net> Co-authored-by: Ben M <benm@cypress.io> Co-authored-by: Bill Glesias <bglesias@gmail.com> Co-authored-by: Podles <78863563+podlesny-j@users.noreply.github.com> Co-authored-by: Paolo Caleffi <p.caleffi@dreamonkey.com> Co-authored-by: Lachlan Miller <lachlan.miller.1990@outlook.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…the application is already stable
Additional details
Finally was able to look into the extremely flaky
cookie_behaviortestsIt turns out these tests look to be straight up broken (except in firefox, which makes me think why they were detected as flake), and not sure why they are being picked up as flaky tests. I initially wrote comments to revisit these tests after we added the document.cookie patch in #23643, but I forgot 😬 .
It turns out in the cookie events, we were waiting for stability to be achieved to sync cookies even when the application was already stable. This fix checks if we are stable, and if we are, syncs the cookies immediately. Otherwise, we wait for stability like we currently do. The cookie tests have been updated to reflect these changes and looks to be passing 🎉
Steps to test
run the tests in the cookie folder inside the driver.
How has the user experience changed?
PR Tasks
cypress-documentation?type definitions?