Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BREAKING: 13.0 Docs #5315

Merged
merged 30 commits into from
Aug 29, 2023
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
610348c
BREAKING: update documentation by setting default video: false for v1…
AtofStryker Jun 15, 2023
c94c9b3
Merge branch 'main' into v13.0.0
mjhenkes Jun 15, 2023
48c2a95
breaking: update documentation by setting videoCompression: false (#5…
AtofStryker Jun 20, 2023
8f1fc89
BREAKING: remove the videoUploadOnPasses configuration option (#5306)
AtofStryker Jun 20, 2023
674419f
.readFile() is now a query (#5017) (#5316)
mjhenkes Jun 20, 2023
7d78203
Update guides for Cy 13, where assertions are now commands (#5081)
BlueWinds Jul 12, 2023
dcea3ed
Apply suggestions from code review
mjhenkes Jul 12, 2023
a0fcdf8
Remove nodeVersion documentation (#5318)
mjhenkes Jul 19, 2023
dd27a62
docs: v13 video and TR migration info (#5383)
jaffrepaul Jul 24, 2023
9eb430c
Update results for module API and after:spec handler (#5379)
chrisbreiding Aug 16, 2023
f391fef
Merge branch 'main' into v13.0.0
jennifer-shehane Aug 18, 2023
fc9304f
docs: adding documentation for runner-ui cli flags (#5424)
mschile Aug 18, 2023
7829170
docs: test replay docs (#5349)
jaffrepaul Aug 22, 2023
f99749f
Merge branch 'main' into v13.0.0
jennifer-shehane Aug 22, 2023
5051c43
Fix JavaScript caps
jennifer-shehane Aug 22, 2023
939111a
Fix caps of JavaScript
jennifer-shehane Aug 22, 2023
2929a65
Fix typo
jennifer-shehane Aug 22, 2023
2348408
Write a document around Cloud data storage and controls (#5425)
jennifer-shehane Aug 24, 2023
0eb47b4
add changelog, remove node 16 support, and update docker image refere…
AtofStryker Aug 25, 2023
f877204
docs: minor v13 updates (#5438)
jaffrepaul Aug 25, 2023
0b7948d
docs: update v13 content urls (#5428)
jaffrepaul Aug 25, 2023
79fe056
docs: v13 changelog summary (#5441)
jaffrepaul Aug 25, 2023
ff326c4
Merge branch 'main' into v13.0.0
jaffrepaul Aug 25, 2023
2ec3eb4
Apply suggestions from code review
jaffrepaul Aug 25, 2023
009f44b
lint
jaffrepaul Aug 25, 2023
e683686
Update docs/faq/questions/cloud-faq.mdx
jaffrepaul Aug 28, 2023
6e0e27c
include TR note in cypress screenshot api options (#5445)
jaffrepaul Aug 28, 2023
4b850a4
docs: additional test replay content (#5447)
jaffrepaul Aug 28, 2023
cc14939
sync up changelog
chrisbreiding Aug 29, 2023
e09fac2
Update _cloud_free_plan.mdx
jaffrepaul Aug 29, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@ orbs:
executors:
node-executor:
docker:
- image: cimg/node:16.15.1
- image: cimg/node:18.15.0
with-chrome-and-firefox:
resource_class: medium+
docker:
- image: 'cypress/browsers:node16.13.2-chrome97-ff96'
- image: 'cypress/browsers:node18.12.0-chrome106-ff106'

commands:
docs-build:
Expand Down
35 changes: 18 additions & 17 deletions docs/api/commands/screenshot.mdx

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion docs/api/plugins/after-spec-api.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,6 @@ on('after:spec', (spec, results) => {
// {
// title: ['login', 'logs user in'],
// state: 'passed',
// body: 'function () {}',
// // ...more properties...
// }
// ],
Expand Down
16 changes: 14 additions & 2 deletions docs/api/commands/readfile.mdx → docs/api/queries/readfile.mdx
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
title: readFile
slug: /api/commands/readfile
---

Read a file and yield its contents.
Expand Down Expand Up @@ -62,8 +63,8 @@ Pass in an options object to change the default behavior of `cy.readFile()`.
### Yields [<Icon name="question-circle"/>](/guides/core-concepts/introduction-to-cypress#Subject-Management) {#Yields}

- `cy.readFile()` yields the contents of the file.
- The file will not be read from disk again if the results are stored in an
alias.
- The file will be read from disk again if any upcoming command (such as an
assertion) in the chain fails.

## Examples

Expand Down Expand Up @@ -195,6 +196,15 @@ assertions.
cy.readFile('some/nested/path/story.txt').should('eq', 'Once upon a time...')
```

Starting in Cypress 13, `cy.readFile()` is a query, and will continue to read
jaffrepaul marked this conversation as resolved.
Show resolved Hide resolved
the file until all chained commands of any type pass, not just assertions.

```javascript
// will retry until the json file has a `users[123].name` field, and
// the assertion passes
cy.readFile('users.json').its('users.123.name').should('eq', 'John Doe')
```

## Rules

### Requirements [<Icon name="question-circle"/>](/guides/core-concepts/introduction-to-cypress#Chains-of-Commands) {#Requirements}
Expand Down Expand Up @@ -247,6 +257,7 @@ outputs the following:

| Version | Changes |
| --------------------------------------------- | ----------------------------------------- |
| [13.0.0](/guides/references/changelog#13-0-0) | `cy.readFile()` became a query |
| [9.0.0](/guides/references/changelog#9-0-0) | Changed `null` encoding to read as Buffer |
| [0.17.2](/guides/references/changelog#0-17-2) | Improved error messaging |
| [0.17.1](/guides/references/changelog#0-17-1) | `cy.readFile()` command added |
Expand All @@ -255,5 +266,6 @@ outputs the following:

- [`cy.exec()`](/api/commands/exec)
- [`cy.fixture()`](/api/commands/fixture) for a similar command with caching
that does not retry
- [`cy.task()`](/api/commands/task)
- [`cy.writeFile()`](/api/commands/writefile)
119 changes: 104 additions & 15 deletions docs/faq/questions/cloud-faq.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -27,21 +27,34 @@ You can read more [here](/guides/cloud/introduction).

### <Icon name="angle-right" /> What does Cypress record?

We capture the following:
Cypress captures the following:

- Standard Output
- Test Failures
When you run Cypress via `cypress run` passing the `--record` flag the following
data is sent to the Cloud for every run:

- Standard output in the terminal
- Test results
- Test definitions
- Cypress configuration (minus Cypress environment variables)
- Screenshots
- Video
- Videos
- OS environment variables related to CI and git information

You have the option to
[delete videos or screenshots](/guides/guides/screenshots-and-videos#Control-which-videos-to-keep-and-upload-to-Cypress-Cloud)
before being sent to the Cloud so those are not captured.

We have already begun the implementation for capturing even more things from
your run such as:
When [Test Replay](#Test-Replay) capturing is enabled, the additional data outlined
below will be sent to the Cloud. You can
[disable capturing](/guides/cloud/debugging/test-replay#Opt-out-of-Test-Replay) this data.

- Commands
- Network Traffic
- Browser Console Logs
- The rendered DOM and CSS styles for the application under test
- Cypress commands and events represented in the Command Log
- Network traffic within your application under test
- Browser console logs

These will be added in subsequent releases.
Cypress Cloud does not capture anything related to the code of your application
under test or the code within any associated repositories that are associated with your project.

### <Icon name="angle-right" /> How is this different than CI?

Expand Down Expand Up @@ -74,7 +87,7 @@ the number of tests you record each month in your organization.

Tests are recorded when [cypress run](/guides/guides/command-line#cypress-run)
is called with the `--record` flag while supplying the record `--key`. This
means your [test run data](/guides/cloud/recorded-runs#Latest-Runs) is being
means your [test run data](/guides/cloud/debugging/recorded-runs#Latest-Runs) is being
"recorded" to Cypress Cloud.

We consider each time the `it()` function is called to be a single test. So you
Expand Down Expand Up @@ -151,7 +164,7 @@ any test results.
### <Icon name="angle-right" /> What is the projectId for?

The `projectId` is a 6 character string that helps identify your project once
you've [set up your tests to record](/guides/cloud/recorded-runs#Latest-Runs).
you've [set up your tests to record](/guides/cloud/debugging/recorded-runs#Latest-Runs).
It's generated by Cypress and typically is found in your
[Cypress configuration](/guides/references/configuration).

Expand All @@ -172,7 +185,7 @@ section of the [Cypress Cloud](/guides/cloud/introduction) docs.
### <Icon name="angle-right" /> What is a Record Key?

A _Record Key_ is a GUID that's generated automatically by Cypress once you've
[set up your tests to record](/guides/cloud/recorded-runs#Latest-Runs). It helps
[set up your tests to record](/guides/cloud/debugging/recorded-runs#Latest-Runs). It helps
identify your project and authenticate that your project is even _allowed_ to
record tests.

Expand All @@ -191,14 +204,14 @@ section of the [Cypress Cloud](/guides/cloud/introduction) docs.
### <Icon name="angle-right" /> How do I record my tests?

1. First [set up the project to record](/guides/cloud/getting-started#Setup).
2. Then [record your runs](/guides/cloud/recorded-runs#Latest-Runs).
2. Then [record your runs](/guides/cloud/debugging/recorded-runs#Latest-Runs).

After recording your tests, you will see them in
[Cypress Cloud](https://on.cypress.io/cloud) and in the Cypress App [Runs](/guides/core-concepts/cypress-app#Runs) tab.

### <Icon name="angle-right" /> Can I delete a run from Cypress Cloud?

You can [archive a run](/guides/cloud/recorded-runs#Archive-run) so that it does
You can [archive a run](/guides/cloud/debugging/recorded-runs#Archive-run) so that it does
not display in the runs list or in analytics.

**Note:** Archiving the recorded runs has no effect on the amount of tests
Expand Down Expand Up @@ -267,6 +280,7 @@ Cloud.
- `https://assets.cypress.io` - **Asset CDN** (Org logos, icons, videos,
screenshots, etc.)
- `https://authenticate.cypress.io` - **Authentication API**
- `https://capture.cypress.io` - **Cypress Test Replay**
- `https://cloud.cypress.io` - **Cypress Cloud**
- `https://docs.cypress.io` - **Cypress documentation**
- `https://download.cypress.io` - **CDN download of Cypress binary**
Expand Down Expand Up @@ -311,3 +325,78 @@ You can delete your Cypress account from
[your Cypress Cloud profile](https://cloud.cypress.io/profile). Deleting your
account cannot be undone! By deleting your Cypress account, all associated data
in your account will be permanently deleted.

## Test Replay

### <Icon name="angle-right" /> What is Test Replay?

[Test Replay](/guides/cloud/debugging/test-replay) enables you and your team to troubleshoot and debug failed tests faster. After updating to Cypress `v13`, the new feature will automatically record all browser events and allow you to “replay” what your application under test looked like.

Wind back the clock to any point in an application's test execution and directly interact with tests as they happened in CI. Debug complex problems as if you were there when they first happened. You can:

- **Inspect the DOM** at the exact time of a test failure to debug tests quickly
- **View important debugging dimensions** such as network requests, console logs, and more
- **Save time recreating errors in CI locally** - replay tests as they happened in CI

<DocsImage
src="/img/guides/cloud/test-replay/test-replay-ui.png"
alt="Test Replay Demo"
width="80%"
/>

### <Icon name="angle-right" /> What types of testing can I debug with Test Replay?

Anything you can do with Cypress - E2E, Component, API, etc.

### <Icon name="angle-right" /> How is Cypress Test Replay different from other "replay" services?

There is an important distinction between _session_ replay services (LogRocket, FullStory, DataDog, etc) and Cypress Test Replay. In session replay, user actions are captured as the application is used and delivered back in a replay-able format, usually video or stitched DOM snapshots. These are valuable tools for gathering product insight such as user behavior, per session.

Cypress Test Replay captures every detail of your test runs as they happen in CI. Remember, these tests are running in a headless manner on a virtual machine with no UI being rendered. Sometimes tests will error or fail indicating an issue in your application. Since Cypress Cloud is monitoring the health of your CI test suite, each Test Replay offers a chance to step back in time to analyze and leverage [time travel debugging](/guides/core-concepts/cypress-app#Time-traveling), network requests, console logs, JavaScript errors, and element rendering to address problems directly. This is incredibly valuable for developer and team productivity. No more hours wasted recreating CI issues on your local machine!

### <Icon name="angle-right" /> Does the replay reflect the real time it took for the test to run?

Yes! The test is captured in Cypress Cloud as it executed in your CI run. That means the replay will show things like [slow typing](/api/cypress-api/keyboard-api#Slow-down-typing-by-increasing-the-keystroke-delay) or any [`cy.wait()`](/api/commands/wait) included in your test.

### <Icon name="angle-right" /> Are there performance implications when using Test Replay?

Expect an even performance exchange with Test Replay vs. video. There is more activity to capture and replay tests but without the overhead of recording, compressing, and storing video assets. `video` capture is set to `false` by default starting in Cypress `v13`. See the full [v13 changelog](/guides/references/changelog#13-0-0).

### <Icon name="angle-right" /> Is every aspect of my stack included in Test Replay?

Our aim is to create an impactful debugging experience that covers the most ground for users. We will continue to expand and can see a need to support (to name a few):

- Web sockets
- Canvas elements
- Shadow DOM
- Server side events

### <Icon name="angle-right" /> Can I use Test Replay for tests recorded in different browsers?

Test Replay leverages [Chrome DevTools Protocol(CDP)](https://chromedevtools.github.io/devtools-protocol/), so currently supports Chromium-based browsers (Chrome, Edge, and Electron) only.

Test Replay would be disabled, with a message that it's only available on Chromium, for tests run in Firefox or Webkit (Safari). You can still record and capture test [artifacts](/guides/cloud/debugging/recorded-runs#Artifacts) (screenshots, videos and CI logs) via other browsers in separate [run groups](/guides/cloud/smart-orchestration/parallelization#Grouping-test-runs).

### <Icon name="angle-right" /> Can I replay tests from historical Cypress Cloud runs?

Test Replay is available for tests recorded using Cypress `v13` and up. Tests recorded prior to this will not have Test Replay enabled in Cypress Cloud. You will still have any artifacts collected during the pre `v13` test runs.

### <Icon name="angle-right" /> Can I enable or disable Test Replay for specific, individual tests, or is it a global setting? Can I configure my settings so that Test Replay is only enabled for failing test retries?

At this time, users can only opt-out of Test Replay via [project-level settings](/guides/cloud/debugging/test-replay#Opt-out-of-Test-Replay) in Cypress Cloud. There is no local option for opting out at this time.

### <Icon name="angle-right" /> Can I share my test replays?

Yes! You can share Test Replays to any team members who have access to view your Cypress Cloud runs like QA, marketing, design, etc. Open the replay, copy the url from the browser address bar and pass it along.

### <Icon name="angle-right" /> Will Test Replay write any assets to the file system?

Yes, but it is stored in a temporary directory and deleted after the run. There is nothing to `.gitignore`.
jaffrepaul marked this conversation as resolved.
Show resolved Hide resolved

### <Icon name="angle-right" /> Is the network tab feature available exclusively in Test Replay?

All of the data within Test Replay's Developer Tools in Cypress Cloud (including network events) are already available to you when running tests locally in the Cypress app via your browser's built-in developer tools (e.g. Chrome DevTools). When reviewing a test run that occurred in CI, you can review the network events within Test Replay in Cypress Cloud.

### <Icon name="angle-right" /> Can I `console.log()` data in the console view of the Developer Tools panel?

Yes. This displays console logs from within the application under test or your spec files. You may need to use `JSON.stringify(Object)` to display deeply nested data.
Loading