Get Accessibility testing working

[LindsayYoung]

Breakout issue of deployment pipeline issues.

Background

We have been running Pa11y tests but they were just returning "Failed to run" but the checkmark was green

Acceptance Criteria (We'll know we're done when...)

Pa11y tests actually run on:

• home page
• login page

Tasks

• get Pa11y working in GitHub actions

[LindsayYoung]

I have gotten pa11y running locally but the set up is a bit clunky.

• First I got the dashboard going via docker (I changed a port to prevent a clash)
• I ran both containers
• I added the Pa11y-web container to the existing network that the backend app is using docker network connect backend_default docker-pa11y_pa11y-web_1
• I looked up the IP for the container running backend_web_1 using docker network inspect backend_default
• I can hit that IP in the pa11y dashboard and get results

I am going to see if it is easier to get our tests running in the existing container. (I tried that earlier and had different problems, but it may be better than this.)

[LindsayYoung]

I have been trying to just get pa11y-ci to install as packages. This in theory should be easy, but I think I am running into some bugs.

I can install pa11y in the container that GitHub checks out, but it does not run. This is the same error on the main branch, it just terminates with a 0, so Actions thinks that the test pass. (I think it is a connection error but it could also be a chrome issue.)

The container was running the wrong node and npm at first, I got that working locally for me. I will add a PR with more pinning of the versions. I am not sure what caused that issue, but pinning should help make sure we don't have that problem silently.

According to the documentation for pa11y, chromium dependencies should be taken care of in the NPM install. When running pa11y-ci in the container it errors saying that it can't find chromium, Could not find expected browser (chrome) locally. Run npm install to download the correct Chromium revision (869685). I have tried installing chromium in the docker container and then telling it where to look for chromium with, PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium, but I get an error Running as root without --no-sandbox is not supported. See https://crbug.com/638180.

So, I tried creating a new user and running it not as root, and I got the error It is likely you do not have the permissions to access this file as the current user.

It seems like a similar issue to the one described here: pa11y-ci/issues/168 but, this is closed and should have been fixed in the last release. The other remedies, (setting concurrenty to 1 and disabling incognito), also did not work.

At this point I had a feeling that the linux distro and puppeteer were not playing nice. So, I figured I would try a Ubuntu image, since we are running cflinuxfs3 in cloud.gov and that is Ubuntu bionic. I got the new image up and running, but I am getting the same error, our friend, UnhandledPromiseRejectionWarning: Error: Could not find expected browser (chrome) locally. Run npm install to download the correct Chromium revision (869685). And, per usual, npm install does NOT fix this.

[LindsayYoung]

• I did get the Pa11y dashboard going as part of the main docker- compose, based on the image robloach/pa11y:dashboard. But, that did not solve the networking issue. I could reach the app by IP addresses but not through the docker native networking. If I set the IP address for the web app in the compose file, I would have to set all the IPs rather than use the native networking. Since we don't need the dashboard, I am going to see if we can keep this simple. I am going to try a simpler image with just the CI and see if that works

• I tried adding the florenttorregrosa/pa11y-ci:2-alpine container and I am still getting the error: UnhandledPromiseRejectionWarning: Error: Chromium revision is not downloaded. Run "npm install" or "yarn install"

[LindsayYoung]

The Node documentation says it should work on any Node version above 12.

• Upgraded to Node v14.21.1 then tried:
  • pa11y-ci v 3.0.1
  • pa11y-ci v 3.0.0
  • pa11y-ci v 2.4.2

All gave Failed to launch Chrome errors, as they did with Node v12.

When loading, there are relevant warnings:
npm WARN deprecated mkdirp@0.3.0: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.) npm WARN deprecated puppeteer@9.1.1: < 18.1.0 is no longer supported

It just seems like if that was the problem, it shouldn't be working for anyone.

Then I upgraded to Node v16.19.0 and npm install -g pa11y-ci fails, it says it can't find puppeteer. If I try to install puppeteer first, that doesn't work, npm update doesn't help either.

[pkigongo]

Implementing Lighthouse instead (for expediency). We will have accessibility testing - we can add Pa11y later, if we choose to