-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add visual regression tests with Playwright #345
Changes from 4 commits
3fb8ab6
f892f6c
2f336c0
220a3d3
faacc39
8876be7
e42d118
d4d8c30
5853b02
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -133,3 +133,4 @@ dmypy.json | |
|
||
# JavaScript | ||
/node_modules | ||
/snapshot_results |
Eric-Arellano marked this conversation as resolved.
Show resolved
Hide resolved
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -47,16 +47,32 @@ Sometimes Sphinx's caching can get in a bad state. First, try running `tox -e do | |
We are in the process of migrating our theme from Pytorch to Furo (see https://github.com/Qiskit/qiskit_sphinx_theme/issues/232). To build the local docs using the Furo theme, use `THEME=_qiskit_furo` in front of the command, e.g. `THEME=_qiskit_furo tox -e docs`. | ||
|
||
------ | ||
## Run JavaScript tests | ||
## Visual regression testing | ||
|
||
We write some tests in JavaScript (Node.js) to have automated checks of the theme, e.g. that certain components render properly. | ||
We use visual regression testing via [Playwright](https://playwright.dev/docs/test-snapshots) to take screenshots of the site and check that every change we make is intentional. If a screenshot has changed, the test will fail. You can inspect the folder `snapshot_results` to compare the before and after. If the change was intentional, then we update the expected screenshot: | ||
|
||
To run these tests, you first need to install Node.js on your machine. If you expect to use JavaScript in other projects, we recommend using [NVM](https://github.com/nvm-sh/nvm). Otherwise, consider using [Homebrew](https://formulae.brew.sh/formula/node) or installing [Node.js directly](https://nodejs.org/en). | ||
1. Find the "actual" snapshot for the failing test, such as `footer-snapshot-has-not-changed-1-actual.png`. | ||
Eric-Arellano marked this conversation as resolved.
Show resolved
Hide resolved
|
||
2. Copy that snapshot into the folder `tests/js/snapshots.test.js-snapshots`. Rename the `-actual.png` file ending to be `-linux.png` and overwrite the prior file. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why do we need to rename the file from actual to linux? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Because This is how Playwright works. Not something we can adjust. |
||
|
||
Then: | ||
We upload `snapshot_results` in CI. So, you can get the changed snapshot from GitHub Actions: | ||
|
||
1. Navigate to the GitHub Actions page for the "Tests" action. | ||
2. Open the "Summary" page with the house icon. | ||
3. Under the "Artifacts" section, there should be a "snapshot_results" entry. Download it. | ||
Eric-Arellano marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
You can also run the tests locally. First, you need to install: | ||
|
||
1. [Node.js](https://nodejs.org/en). If you expect to use JavaScript in other projects, consider using [NVM](https://github.com/nvm-sh/nvm). Otherwise, consider using [Homebrew](https://formulae.brew.sh/formula/node) or installing [Node.js directly](https://nodejs.org/en). | ||
2. [Docker](https://www.docker.com). You must also ensure that it is running. | ||
* If you cannot install Docker Desktop (such as IBM contributors), you can use [Rancher Desktop](https://rancherdesktop.io). When installing, choose Moby/Dockerd as the engine, rather than nerdctl. To ensure it's running, open up the app "Rancher Desktop". | ||
|
||
Then, to run the tests locally: | ||
|
||
1. `npm install` | ||
2. `npm test` | ||
2. Build the docs with Furo, `THEME=_qiskit_furo tox -e docs` | ||
3. `npm test` | ||
|
||
You must rebuild the docs with `THEME=_qiskit_furo tox -e docs` whenever you make changes to the theme or docs folder. The docs will not automatically rebuild. | ||
Eric-Arellano marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
------ | ||
## Updating bundled web components | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note that this isn't the standard
npm test
, which will use Docker. I optimizednpm test
to be the ideal experience for local developers.