chore: skip problematic component tests that fail on contributor PRs #26924
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mschile
approved these changes
Jun 2, 2023
29 flaky tests on run #47014 ↗︎
Details:
|
|||||||||||||||||||||||||||||||||||||||||||||||||||
| Test | Artifacts | |
|---|---|---|
| network stubbing > intercepting request > can delay and throttle a StaticResponse |
Output
|
|
cypress/cypress.cy.js • 3 flaky tests • 5x-driver-firefox
| Test | Artifacts | |
|---|---|---|
| ... > correctly returns currentRetry |
Output
|
|
| ... > correctly returns currentRetry |
Output
|
|
| ... > correctly returns currentRetry |
Output
|
|
scaffold-component-testing.cy.ts • 1 flaky test • launchpad-e2e
| Test | Artifacts | |
|---|---|---|
| scaffolding component testing > react-vite-ts-unconfigured > scaffolds component testing for React and Vite |
Output
Screenshots
Video
|
|
create-from-component.cy.ts • 1 flaky test • app-e2e
| Test | Artifacts | |
|---|---|---|
| ... > runs generated spec |
Output
Screenshots
Video
|
|
runs.cy.ts • 1 flaky test • app-e2e
| Test | Artifacts | |
|---|---|---|
| App: Runs > Runs - No Runs > displays how to record prompt when connected and no runs in E2E |
Output
Screenshots
Video
|
|
The first 5 flaky specs are shown, see all 17 specs in Cypress Cloud.
This comment has been generated by cypress-bot as a result of this project's GitHub integration settings.
chrisbreiding
approved these changes
Jun 5, 2023
AtofStryker
added a commit
that referenced
this pull request
Jun 12, 2023
* docs: revise contributor advice for node.js setup (#26652) * feat: initial release of cypress/vite-plugin-cypress-esm (#26663) * chore: release @cypress/vite-plugin-cypress-esm-v1.0.0 [skip ci] * chore: upgrade Lerna to v5 and use Nx (#26660) * dependency(deps): update dependency engine.io to v6.4.2 [security] (#26664) * dependency(deps): update dependency engine.io to v6.4.2 [security] * updating changelog --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Matt Schile <mschile@cypress.io> * fix: updated CYPRESS_DOWNLOAD_PATH_TEMPLATE regex to allow multiple replacements (#25531) * feat: Component Testing banner (#26625) Co-authored-by: elevatebart <bart@cypress.io> * chore: Update v8 snapshot cache (#26647) Co-authored-by: cypress-bot[bot] <+cypress-bot[bot]@users.noreply.github.com> Co-authored-by: Ryan Manuel <ryanm@cypress.io> * chore: 12.12.0 release updates (#26697) * chore: cypress[26674] updated github workflows to use checklout@v3 and stop using set-output and move to file use (#26696) * fix: move types condition to the front (#26630) * fix: move `types` condition to the front * chore: update changelog * optimize deps for vite * update config * try optimizeDeps.include * missing dep --------- Co-authored-by: Lachlan Miller <lachlan.miller.1990@outlook.com> * fix: revert #26452 (Yarn Plug n Play compat regression) (#26735) * Revert "fix: correctly resolve dependencies for CT onboarding when using Yarn Plug n Play (#26452)" This reverts commit 7a33f5c. * changelog * changelog * chore: fixing PR link in releaseData.json (#26734) * chore: update stalebot config (#26745) * misc: ACI empty state slideshows (#26692) Co-authored-by: Stokes Player <stokes@cypress.io> * fix: do not cache module resolution during launchpad dependency detection (#26726) Co-authored-by: Mike Plummer <mikep@cypress.io> * fix: Vite esm plugin issues (#26714) * chore: Update v8 snapshot cache (#26707) Co-authored-by: cypress-bot[bot] <+cypress-bot[bot]@users.noreply.github.com> Co-authored-by: Ryan Manuel <ryanm@cypress.io> * chore: adding in repo token to explicitly use that while running commands [skip ci] (#26746) * chore(dependency): update dependency @percy/cypress to ^3.1.2 🌟 (#26717) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com> * chore: Remove console.log (#26756) * chore: put types condition first in the syncing script (#26743) Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com> * test: create lists files after folders when in same directory in specs list (#26723) Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com> * chore: update vm2 to 3.9.19 (#26772) * chore: add telemetry to the proxy (#26695) * chore: set up instrumentation and instrument middleware * chore: set up console exporter * chore: add parent span option to telemetry package * chore: set up telemetry verbose mode * chore: instrument the network proxy - part 1 * chore: make sure to terminate spans when request is aborted * fix telemetry, create/end the request middle prior to sending the outbound request * avoid telemetry ts build step, create entrypoint into packages/telemetry using TS conventions * allow env vars to be "true" or "1" * when creating child span, inherit their attributes directly from the parent * create custom honeycomb exporter and span processor to log traces * remove duplicate code that's already called in this.setRootContext * cleanup * more clean up * update honeycomb network:proxy attributes, update console.log message * yarn lock * chore: remove performance API in middleware * chore: end response on correct event * recursively gather parent attributes on close * added key and some clean up * github action detector, move verbose into index, verbose log commands * some tests * clean up honeycomb exporter * some renaming * testing console trace link exporter * Don't lose the top span when running in verbose. * link to the right place for prod/dev * changes to verbose to make sure it is read in the browser * Apply suggestions from code review * pass parent attributes between telemetry instances * default to false * 'fix' build issues * src not dist * add back on start span * once more with feeling * Fix some tests * try this i guess * revert auto build * Apply suggestions from code review Co-authored-by: Bill Glesias <bglesias@gmail.com> * support failed commands * Address PR comments * Address PR Comments * error handling * handle all the errors --------- Co-authored-by: Bill Glesias <bglesias@gmail.com> Co-authored-by: Brian Mann <brian.mann86@gmail.com> * chore: update triage workflow to add comment to contributor prs (#26493) Co-authored-by: Ben M <benm@cypress.io> * chore: telemetry pr cleanup (#26776) * fix: fix UI flash when switching to debug page (#26761) * chore: add Nx Cloud (#26712) * chore: add empty nx.json [run ci] * chore: add nx cloud runner [run ci] * chore: add nx-cloud dep [run ci] * chore: update local nx cloud accessToken to be read-only * feat: update git related messages for runs and debug (#26758) * feat(app): update DebugError copy * feat: set isUsingGit project flag and consume in DebugContainer * feat(app): update not using git icon for DebugError * feat(app): displays alert on runs page when not using git * feat(app): add component for when no runs for current branch * feat(app): add warning for no runs for branch on runs container * chore: add feat to CHANGELOG * chore: remove logged status value * chore: resolve import from merge conflict * test(app): stub branch name for e2e runs spec * chore: add line break in changelog entry * chore: cleanup PR * chore: fix i18n import for DebugBranchError * chore: add utm and update Warning links to inline * chore: capitalize Git in i18n * ref: warning liink * test: add i18n to tests * test(app): change utm_source assertions * chore: cleanup pr * chore: remove unused prop * test(app): remove no git warning when moving to runs page in e2e * chore: change template logic * chore: remove duplicate RUNS_TAB_MEDIUM const * Changed Debug test assertion and reordered new components for Debug --------- Co-authored-by: Stokes Player <stokes.player@gmail.com> * chore: rename video processing events to capture/compress (#26800) * chore: change processing nomenclature to compressing when printing the run. * chore: rename 'capturing of' to 'capturing' * chore: rename upload results to upload screenshots & videos (#26811) * chore: rename upload results to upload screenshots & videos * run ci * chore: capture versions of relevant dependencies with `x-dependencies` header (#26814) * chore: update changlelog script to handle revert pr ref (#26801) * fix: Correct typescript scaffold dependency (#26815) * fix: correct typescript scaffold dependency (#26204) * add changelog * Update change log for PR comment Co-authored-by: Mike Plummer <mike-plummer@users.noreply.github.com> --------- Co-authored-by: Mike Plummer <mike-plummer@users.noreply.github.com> Co-authored-by: Mark Noonan <mark@cypress.io> * chore: 12.13.0 prep (#26833) * chore: 12.13.0 release (#26834) * chore: 12.13.0 changelog fix * remove pending, bump version * fix typo * chore: release @cypress/vite-plugin-cypress-esm-v1.0.1 [skip ci] * chore: Implement runSpec mutation (#26782) * chore: replace gitter badge with discord on readme (#26771) * chore: add GraphQL mutation for sending system notifications via Electron (#26773) * fix: upgrade typescript from 4.7.4 to 4.9.5 (#26826) Snyk has created this PR to upgrade typescript from 4.7.4 to 4.9.5. See this package in npm: See this project in Snyk: https://app.snyk.io/org/cypress-opensource/project/d5b36925-e6ee-455d-9649-6560a9aca413?utm_source=github&utm_medium=referral&page=upgrade-pr * test: fix 2 broken tests for Windows (#26854) * Update stale_issues_and_pr_cleanup.yml Upped the number of operations per run. Have been manually doing that so this job can get through all the issues in the repo with no problems. * chore(dep): [Snyk] Upgrade vite from 2.9.13 to 2.9.15 (#26830) Co-authored-by: Ben M <benm@cypress.io> * Update triage_add_to_project.yml * chore: fix minor background color styling in debug results component (#26887) * Update stale_issues_and_pr_cleanup.yml stalebot was incorrectly configured to run in debug mode. I have updated the default to run in normal mode when running scheduled * chore: Deprecate @cypress/xpath package (#26893) * chore: add telemetry realworld app (#26896) * chore: capture telemetry for realworld app maybe * idk what i was doing * setup record key and telemetry * testing * override project id * some times we just need a little context. * Adding tests * Adding comment * chore(deps): update dependency find-process to v1.4.7 🌟 (#26906) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Jennifer Shehane <jennifer@cypress.io> * chore(deps): update dependency firefox-profile to v4.3.2 🌟 (#26912) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Jennifer Shehane <jennifer@cypress.io> * chore: add browser state action for debug (#26884) * chore: add browser state action for debug (#26763) * Address PR comments - remove unneeded async and test context - genericize openProject function * Revert route change, update spec description * chore: replace gift devDep with simple-git (#26728) * chore: replace arg devDep with minimist; remove unused shx devDep (#26727) * chore: enable caching for lint task (#26791) * chore: remove old performance reporting (#26920) * chore: remove old performance reporting * remove libhoney dep * try this * build and build snapshots if deps are out of date * foiled by a comma * freaking comma * no snapshots maybe ugh * ignore engines instead * don't need this * remove rename support file step * chore: update Snyk to scan all projects (#26867) * SEC-544 chore: [Snyk] Update Snyk flag in Git actn * Update snyk_sca_scan.yaml Removed --debug switch from the test command --------- Co-authored-by: brady-cypressio <90723145+brady-cypressio@users.noreply.github.com> * chore: skip problematic component tests that fail on contributor PRs (#26924) * chore: omit unused circle variables that cause contributor PR issues (#26935) * chore: make git message warnings remain dismissable (#26812) * feat: make git message warnings remain dismissable * chore: update CHANGELOG * chore: update CHANGELOG * chore: remove unneeded code * chore: update BannerId types * chore: fix queryies * chore: clean up PR * chore: move TrackedWarning to frontend-shared * chore: update import * ref: move TrackedWarning to TrackedBanner * chore: udpate CHANGELOG * fix: update TrackedBanner to parse markdown message * chore: set TrackedBanner default message prop * chore: update RunsContainer * chore: add missing tests and update alert type --------- Co-authored-by: Stokes Player <stokes@cypress.io> * chore: replace fast-glob with globby; remove unneeded getenv dep (#26730) * fix: update angular dep min versions (#26908) * fix: update angular dep min versions * chore: update CHANGELOG * chore: add line break to changelog * chore: update changelog pending release date * chore: remove whitespace * fix: upgrade typescript from 4.2.3 to 4.9.5 (#26858) Snyk has created this PR to upgrade typescript from 4.2.3 to 4.9.5. See this package in npm: See this project in Snyk: https://app.snyk.io/org/cypress-opensource/project/5fdaebf8-b115-41d1-a2d9-857261769179?utm_source=github&utm_medium=referral&page=upgrade-pr Co-authored-by: Bill Glesias <bglesias@gmail.com> * chore: Update v8 snapshot cache (#26762) Co-authored-by: cypress-bot[bot] <+cypress-bot[bot]@users.noreply.github.com> Co-authored-by: Ryan Manuel <ryanm@cypress.io> * feat: Implement testing type switch promos (#26894) * feat: Implement testing type switch promos * Add tests, changelog entry * Add tests * Fix button styling * Styling fixes, add framework links * Add missing testId * run ci * Fix spec * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * Fix styling issues * Resolve code review findings * Fix issue with yarn.lock * Fix extra padding at bottom of promo * Add tests for utm params * Add test for switching testing type when both configured * Fix changelog version * Address review comments * Widen promo when no image defined * Prevent flash of promo before query resolves * Reduce top margin * reduce size of text box to match latest figma * update button style to match figma * increase width at which we collapse sidenav * add short versions of the headings * remove skeletons from header * avoid extra height * adjustments for column alignment * fix flaky test * update tests for responsive text changes * update changelog * restore spacing between header items * avoid occasional flash of promo on page load * update text handling * fix types and tests * Update packages/app/src/specs/SpecsList.vue Co-authored-by: Stokes Player <stokes@cypress.io> * updated final e2e bullet * fix question mark icon flashing * text formatting * remove superfluous snapshot [skip ci] --------- Co-authored-by: cypress-bot[bot] <+cypress-bot[bot]@users.noreply.github.com> Co-authored-by: marktnoonan <mark@cypress.io> Co-authored-by: astone123 <adams@cypress.io> Co-authored-by: Stokes Player <stokes@cypress.io> Co-authored-by: Lachlan Miller <lachlan.miller.1990@outlook.com> * chore: release internal-scripts-v1.0.0 [skip ci] * chore: fix changelog links (#26948) * chore: 12.14.0 release (#26950) * chore: Update Chrome (stable) to 114.0.5735.106 and Chrome (beta) to 115.0.5790.13 (#26650) * chore: bump cache version (#26952) Co-authored-by: Ryan Manuel <ryanm@cypress.io> * chore: move release date (#26958) * chore(deps): update dependency @antfu/utils to ^0.7.0 [security] (#26923) * chore: fix base error styling (#26954) * chore: remove low value percy snapshots (#26934) * chore: remove low value percy snapshots * chore: remove more low value percy snapshots [run ci] * chore: remove more low value percy snapshots * remove additional snapshots * reduce snapshots * fix types --------- Co-authored-by: Lachlan Miller <lachlan.miller.1990@outlook.com> * chore: changelog updates (#26964) * chore: stabilize side navigation bar during loading and switching testing types (#26953) * fix: log video path if exists, regardless of compression (#26813) * chore: print the video path whether or not compression is on or fails * chore: fix video replacement regex * chore: add bugfix entry * feat: allow users to pass true to videoCompression config and only a… (#26810) * chore: allow users to pass true to videoCompression config and only allow valudes 1-51 inclusively to be passed in * run ci * chore: allow zero to be option for CRF as we will coerve it to false and skip compression to have a lossless video, which has the same effect * Update cli/CHANGELOG.md * chore: update videoCompression types * chore: update changelog * Update cli/CHANGELOG.md * Update cli/types/cypress.d.ts Co-authored-by: Emily Rohrbough <emilyrohrbough@users.noreply.github.com> * Update packages/config/src/validation.ts Co-authored-by: Emily Rohrbough <emilyrohrbough@users.noreply.github.com> * chore: update config snapshots * Update packages/config/test/validation.spec.ts Co-authored-by: Emily Rohrbough <emilyrohrbough@users.noreply.github.com> * chore: add system test on videoCompression=true coersion * chore: document 0 as being false and not a valid CRF option for cypress video compression and make CRF valid values 1-51 * chore: fix types * chore: fix types * chore: fix change to log as feature and not bugfix * chore: fix server side unit tests --------- Co-authored-by: Emily Rohrbough <emilyrohrbough@users.noreply.github.com> * chore: trigger mac/linux/windows binary builds and v8 snapshot cache update tests * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: fix possible bad merge from mismatched snapshot in CLOUD_AUTO_CANCEL_MISMATCH test * chore: fix possible bad merge from mismatched snapshot in record_spec system test * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache * chore: updating v8 snapshot cache --------- Co-authored-by: Mike McCready <66998419+MikeMcC399@users.noreply.github.com> Co-authored-by: Lachlan Miller <lachlan.miller.1990@outlook.com> Co-authored-by: semantic-release-bot <semantic-release-bot@martynus.net> Co-authored-by: Adam Stone-Lord <adams@cypress.io> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Matt Schile <mschile@cypress.io> Co-authored-by: Mahdi Khashan <58775404+mahdikhashan@users.noreply.github.com> Co-authored-by: Mike Plummer <mike-plummer@users.noreply.github.com> Co-authored-by: elevatebart <bart@cypress.io> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: cypress-bot[bot] <+cypress-bot[bot]@users.noreply.github.com> Co-authored-by: Ryan Manuel <ryanm@cypress.io> Co-authored-by: Ben M <benm@cypress.io> Co-authored-by: Mateusz Burzyński <mateuszburzynski@gmail.com> Co-authored-by: Stokes Player <stokes@cypress.io> Co-authored-by: Mike Plummer <mikep@cypress.io> Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com> Co-authored-by: Luis Furtado <46112985+luis-furtado@users.noreply.github.com> Co-authored-by: Matt Henkes <mjhenkes@gmail.com> Co-authored-by: Brian Mann <brian.mann86@gmail.com> Co-authored-by: Emily Rohrbough <emilyrohrbough@users.noreply.github.com> Co-authored-by: Jordan <jordan@jpdesigning.com> Co-authored-by: Stokes Player <stokes.player@gmail.com> Co-authored-by: Dave Kasper <dave.m.kasper@gmail.com> Co-authored-by: Mark Noonan <mark@cypress.io> Co-authored-by: Ely Lucas <ely@meta-tek.net> Co-authored-by: Snyk bot <snyk-bot@snyk.io> Co-authored-by: Jennifer Shehane <jennifer@cypress.io> Co-authored-by: cypresschris <96069059+cypresschris@users.noreply.github.com> Co-authored-by: brady-cypressio <90723145+brady-cypressio@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Additional details
This PR skips the FileMatch, PromoAction, and SelectorPlayground component tests for contributor PRs inside the
run-app-component-tests-chromeas they are causing failures in CI. The failures are difficult to diagnose as we don't record on contributor PR runs and we are unable to reproduce the problem locally.This PR essentially is a no-op in CI as this PR will not run as a contributor PR and cannot be tested directly with #26897 as the workflow file is from
develop. To replicate the behavior, I created a different branch with commented out code to run like a contributor PR, and the CI run behavior looks as expected.Once this PR merges in, I will rebase #26897 onto develop which should then flow through this workflow.
Steps to test
How has the user experience changed?
PR Tasks
cypress-documentation?type definitions?