Add visual regression tests with Playwright

.github/workflows/main.yml

@@ -10,6 +10,9 @@ jobs:
     name: Sample Docs Build
     runs-on: ubuntu-latest
     if: github.repository_owner == 'Qiskit'
+    container:
+      # Keep in sync with tests/js/Dockerfile's base image.
+      image: mcr.microsoft.com/playwright:v1.34.0-jammy
     steps:
       - uses: actions/checkout@v3
         with:
@@ -20,23 +23,25 @@ jobs:
           node-version: 18
       - name: Install Node.js dependencies
         run: npm ci
-      - name: Run Node.js tests
-        run: npm test
       - name: Set up Python
         uses: actions/setup-python@v4
         with:
           python-version: 3.9
       - name: Install Ubuntu deps
-        run: sudo apt-get install -y pandoc graphviz
+        run: apt-get update && apt-get install -y pandoc graphviz
       - name: Install tox
         run: python -m pip install -U tox
+
       - name: Run lint
         run: tox run -e lint
       - name: Run Pytest
         run: tox run -e py
       # For some reason, the Jupyter Extension fails to build properly if one theme has already
       # been built with a custom BUILD_DIR. So, we eagerly upload the artifacts for each theme
       # build, and then delete the BUILD_DIR before proceeding to the next theme.
+      #
+      # But, we build Furo with the default BUILD_DIR because our JavaScript Snapshot tests start
+      # the server at that standard location.
       - name: Create artifacts/ folder
         run: mkdir artifacts
       - name: Build Legacy theme
@@ -47,13 +52,21 @@ jobs:
           rm -rf example_docs/_build_legacy
       - name: Build Furo theme
         run: |
-          THEME=_qiskit_furo BUILD_DIR=example_docs/_build_furo/html tox run -e docs
-          tar -zcvf furo_html_docs.tar.gz example_docs/_build_furo/html
+          THEME=_qiskit_furo tox run -e docs
+          tar -zcvf furo_html_docs.tar.gz example_docs/docs/_build/html
           mv furo_html_docs.tar.gz artifacts/.
-          rm -rf example_docs/_build_furo
       - name: Upload Sphinx builds
         uses: actions/upload-artifact@v3
         if: always()
         with:
           name: html_docs
-          path: artifacts
+          path: artifacts
+
+      - name: Run JavaScript and snapshot tests
+        run: npm run _run-snapshot-tests
+      - name: Upload snapshot results
+        if: failure()
+        uses: actions/upload-artifact@v3
+        with:
+          name: snapshot_results
+          path: snapshot_results/

.gitignore

@@ -133,3 +133,4 @@ dmypy.json
 
 # JavaScript
 /node_modules
+/snapshot_results

CONTRIBUTING.md

@@ -47,16 +47,50 @@ Sometimes Sphinx's caching can get in a bad state. First, try running `tox -e do
 We are in the process of migrating our theme from Pytorch to Furo (see https://github.com/Qiskit/qiskit_sphinx_theme/issues/232). To build the local docs using the Furo theme, use `THEME=_qiskit_furo` in front of the command, e.g. `THEME=_qiskit_furo tox -e docs`.
 
 ------
-## Run JavaScript tests
+## Visual regression testing
 
-We write some tests in JavaScript (Node.js) to have automated checks of the theme, e.g. that certain components render properly.
+We use visual regression testing via [Playwright](https://playwright.dev/docs/test-snapshots) to take screenshots of the site and check that every change we make is intentional. If a screenshot has changed, the test will fail. 
 
-To run these tests, you first need to install Node.js on your machine. If you expect to use JavaScript in other projects, we recommend using [NVM](https://github.com/nvm-sh/nvm). Otherwise, consider using [Homebrew](https://formulae.brew.sh/formula/node) or installing [Node.js directly](https://nodejs.org/en).
+If the change was intentional, we need to update the screenshot. Otherwise, it means your change unintentionally impacted something, so you need to tweak your change.
 
-Then:
+The test runner creates a folder called `snapshot_results`, which is useful to determine what the difference is. For each failed test, there will be three files:
+
+* `<my-test-name>-actual.png`, what your change resulted in.
+* `<my-test-name>-expected.png`, what we expected.
+* `<my-test-name>-diff.png`, a heat map showing where the differences are.
+
+### How to check snapshot results in CI
+
+We upload `snapshot_results` in CI. So, you can get the changed snapshot from GitHub Actions:
+
+1. Navigate to the GitHub Actions page for the "Tests" action.
+2. Open the "Summary" page with the house icon.
+3. Under the "Artifacts" section, there should be a "snapshot_results" entry. Download it.
+
+### How to check snapshot results locally
+
+You can also run the tests locally for faster iteration, although it requires a little setup. If you don't want to install the below tools, it is 100% okay to rely on CI for snapshot testing.
+
+First, you need to install:
+
+1. [Node.js](https://nodejs.org/en). If you expect to use JavaScript in other projects, consider using [NVM](https://github.com/nvm-sh/nvm). Otherwise, consider using [Homebrew](https://formulae.brew.sh/formula/node) or installing [Node.js directly](https://nodejs.org/en).
+2. [Docker](https://www.docker.com). You must also ensure that it is running.
+   * If you cannot install Docker Desktop (such as IBM contributors), you can use [Rancher Desktop](https://rancherdesktop.io). When installing, choose Moby/Dockerd as the engine, rather than nerdctl. To ensure it's running, open up the app "Rancher Desktop". 
+
+Then, to run the tests locally:
 
 1. `npm install`
-2. `npm test`
+2. Build the docs with Furo, `THEME=_qiskit_furo tox -e docs`
+3. `npm run test-snapshots`
+
+You must rebuild the docs with `THEME=_qiskit_furo tox -e docs` whenever you make changes to the theme or docs folder. The docs will not automatically rebuild.
+
+### How to update the expected snapshot
+
+First, get the `snapshot_results` folder, either by downloading it from CI or by running the tests locally. Then:
+
+1. Find the "actual" snapshot for the failing test, such as `footer-includes-page-analytics-1-actual.png`.
+2. Copy that snapshot into the folder `tests/js/snapshots.test.js-snapshots`. Rename the `-actual.png` file ending to be `-linux.png` and overwrite the prior file.
 
 ------
 ## Updating bundled web components

example_docs/docs/conf.py

@@ -42,7 +42,9 @@
     "qiskit_sphinx_theme",
 ]
 
-html_last_updated_fmt = "%Y/%m/%d"
+# Usually this would be something like "%Y/%m/%d", but we need a deterministic value for our
+# Playwright visual regression tests.
+html_last_updated_fmt = "2020/01/01"
 
 # This allows us to test both the Furo and Pytorch themes. In normal repositories, `html_theme`
 # would be set to one specific theme.