Running version 3.0.0 #168

dmundra · 2021-11-28T00:03:50Z

After #128, I updated to 3.0.0 but I am getting a puppeteer error. Am I missing something?

npm run pa11y-ci:sitemap 

> pa11y-ci:sitemap
> pa11y-ci --sitemap http://127.0.0.1:4000/sitemap.xml --sitemap-find https://accessibility.civicactions.com --sitemap-replace http://127.0.0.1:4000 --sitemap-exclude "/*.pdf"

Running Pa11y on 107 URLs:
 > http://127.0.0.1:4000/guide/documents - 2 errors
 > http://127.0.0.1:4000/guide/glossary - Failed to run
/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:71
            this._callbacks.set(id, { resolve, reject, error: new Error(), method });
                                                              ^

Error: Protocol error (Target.disposeBrowserContext): Failed to find context with id 1573D003ABD6824A875BA4554C9131FA
    at /Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:71:63
    at new Promise (<anonymous>)
    at Connection.send (/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:70:16)
    at Browser._disposeContext (/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Browser.js:168:32)
    at BrowserContext.close (/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Browser.js:523:29)
    at testRunner (/Users/dmundra/workspaces/accessibility/node_modules/pa11y-ci/lib/pa11y-ci.js:135:21)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

Node.js v17.0.1

The text was updated successfully, but these errors were encountered:

josebolos · 2021-11-28T17:14:28Z

Hi @dmundra,

Could you please provide a public URL that we can test? It's impossible for us to debug the issue otherwise.

I've noticed you're using Node 17, could you please test it again using an LTS version of Node, as those are the only ones that we support?

If I'm reading your example correctly and the URL that you're testing has the same content as https://accessibility.civicactions.com/guide/glossary, this is working correctly with 3.0.0:

Broadly speaking, the error that you show there is a puppeteer error, so there's very little that we can do about it besides updating puppeteer to a latest version. I've seen similar errors in the past with puppeteer choking on servers that are configured in a specific way. If pa11y runs successfully on a public facing URL, but fails to run on a localhost one, this may be the same problem.

dmundra · 2021-11-28T17:34:05Z

Hi @josebolos,

I am testing locally and in GitHub actions for the site you mentioned, CivicActions/accessibility#580 (comment)

I got the same error when trying it against the live sitemap.xml

./node_modules/pa11y-ci/bin/pa11y-ci.js --sitemap https://accessibility.civicactions.com/sitemap.xml
Running Pa11y on 119 URLs:
 > https://accessibility.civicactions.com/guide/events - 2 errors
 > https://accessibility.civicactions.com/guide/glossary - Failed to run
/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:71
            this._callbacks.set(id, { resolve, reject, error: new Error(), method });
                                                              ^

Error: Protocol error (Target.disposeBrowserContext): Failed to find context with id F6EC73A10C23F6F16F75D9DCE662CC99
    at /Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:71:63
    at new Promise (<anonymous>)
    at Connection.send (/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:70:16)
    at Browser._disposeContext (/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Browser.js:168:32)
    at BrowserContext.close (/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Browser.js:523:29)
    at testRunner (/Users/dmundra/workspaces/accessibility/node_modules/pa11y-ci/lib/pa11y-ci.js:135:21)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

Node.js v17.0.1

I am also running with axe but I get the error even when not running with axe. I will try different version of node as well.

I did find this puppeteer issue puppeteer/puppeteer#6258 if it helps. Maybe some of those versions needs to be changed for pa11y-ci.

dmundra · 2021-11-28T19:01:19Z

Same error with node LTS version v16.13.0

joeyciechanowicz · 2021-11-28T20:56:27Z

I couldn't re-create it locally by running

npx pa11y-ci --sitemap  https://accessibility.civicactions.com/sitemap.xml  --sitemap-exclude "/*.pdf"

If possible, could you share your pa11y-ci config file? And also the output of npx pa11y --environment

DaveSkender · 2021-11-28T22:15:04Z

If possible, could you share your pa11y-ci config file? And also the output of npx pa11y --environment

This is @dmundra's config file for that site:

{
  "defaults": {
    "concurrency": 4,
    "standard": "WCAG2AA",
    "runners": ["axe"]
  }
}

He's also using ubuntu-latest in his test action, so it would be similar to my environment:

  System:
    OS: Linux 5.11 Ubuntu 20.04.3 LTS (Focal Fossa)
    CPU: (2) x64 Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
    Memory: 4.13 GB / 6.79 GB
    Shell: 5.0.17 - /bin/bash
  Binaries:
    Node: 16.13.0 - /opt/hostedtoolcache/node/16.13.0/x64/bin/node
    Yarn: 1.22.17 - /usr/local/bin/yarn
    npm: 8.1.0 - /opt/hostedtoolcache/node/16.13.0/x64/bin/npm
  pa11y: 6.1.1

I'm testing on a local GitHub Actions --detached site so I don't blow out my Google Analytics every time this runs; so, please don't run on my production sites to test (can't speak for @dmundra).

For me, this runs fine on my local Windows 10 machine. On a side note, I'm running html-proofer on this same --detached site on ubuntu-latest without problems, so I don't think it's the host.

DaveSkender · 2021-11-28T22:57:38Z

I suspect this has to do with concurrency. I changed my config file to "concurrency": 1 and it ran all the way through with Exit Code 2; however, it's producing false negatives that do not show up when I run locally.

DaveSkender · 2021-11-28T23:17:04Z

Changing config "runners": ["axe"] to "runners": ["htmlcs"] solves my false negative problem.

DaveSkender · 2021-11-28T23:24:37Z

In summary, there are likely two things that don't work in this scenario:

Concurrency > 1
Axe runner

The workaround is to change configurations in .pa11yci to:

{
  "defaults": {
    "concurrency": 1,
    "standard": "WCAG2AA",
    "runners": ["htmlcs"]
  }
}

filippoliverani · 2021-11-29T12:04:09Z

Same error with node LTS version v16.13.0

Same issue on CircleCI's cimg/ruby:3.0-browsers latest image with node v16.13.0

/home/circleci/project/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:71
            this._callbacks.set(id, { resolve, reject, error: new Error(), method });

.pa11yci:

{
  "defaults": {
    "concurrency": 4
  }
}

dmundra · 2021-11-29T14:52:53Z

Thanks @DaveSkender.

I'm testing on a local GitHub Actions --detached site so I don't blow out my Google Analytics every time this runs; so, please don't run on my production sites to test (can't speak for @dmundra).

I am doing the same in GitHub Actions, i.e. running against a local detached Jekyll site. The relevant files:

dmundra · 2021-11-29T14:59:13Z

Confirmed that reducing concurrency to 1 allows the process to run to completion.

aarongoldenthal · 2021-12-10T02:55:36Z

@joeyciechanowicz @josebolos I believe it's the same core issues I came across tonight working on a reporter, and in longer running reporter.afterAll functions (with a couple of reporters, with layers of async) it fails with:

E:\project\node_modules\puppeteer\lib\cjs\puppeteer\common\Connection.js:71
            this._callbacks.set(id, { resolve, reject, error: new Error(), method });

Error: Protocol error (Target.disposeBrowserContext): Target closed.
    at E:\project\node_modules\puppeteer\lib\cjs\puppeteer\common\Connection.js:71:63
    at new Promise (<anonymous>)
    at Connection.send (E:\project\node_modules\puppeteer\lib\cjs\puppeteer\common\Connection.js:70:16)
    at Browser._disposeContext (E:\project\node_modules\puppeteer\lib\cjs\puppeteer\common\Browser.js:168:32)
    at BrowserContext.close (E:\project\node_modules\puppeteer\lib\cjs\puppeteer\common\Browser.js:523:29)
    at testRunner (E:\project\node_modules\pa11y-ci\lib\pa11y-ci.js:135:21)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

It looks to be a missing await at the specified line in the call stack - browserContext.close is an async function. Running with "useIncognitoBrowserContext": false seems to be a workaround since it avoids the line, but adding the await looks like it solved the core issue (at least in my case).

I have a PR if you want it, but it is just the one line.

josebolos · 2022-03-23T17:25:20Z

This should be fixen in v3.0.1 via #170 thanks to @aarongoldenthal.

Feel free to reopen the issue if you're still experiencing this problem.

pa11y/pa11y-ci#168

dmundra mentioned this issue Nov 28, 2021

Updated Pa11y-ci to latest version. Also updated dependencies. CivicActions/accessibility#580

Merged

This was referenced Nov 28, 2021

pa11y-ci Error: Page crashed! GSA/digitalcorps.gsa.gov#18

Closed

chore: update build a11y tests DaveSkender/Stock.Indicators#645

Merged

DaveSkender mentioned this issue Nov 30, 2021

pa11y-ci fixes for concurrency and runner problems cloud-gov/pages-uswds-jekyll#225

Open

aarongoldenthal mentioned this issue Dec 10, 2021

Add missing await for closing incognito browser context. #170

Merged

josebolos closed this as completed Mar 23, 2022

This was referenced Apr 20, 2022

TEST debug build errors uswds/uswds-site#1540

Closed

TEST - resolve circle CI test errors uswds/uswds-site#1541

Closed

LindsayYoung mentioned this issue Dec 30, 2022

Get Accessibility testing working GSA-TTS/FAC#711

Closed

3 tasks

phrdang added a commit to phrdang/berkeley-class-site that referenced this issue Jun 11, 2024

Try to fix pa11y on github actions

5f74229

pa11y/pa11y-ci#168

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Running version 3.0.0 #168

Running version 3.0.0 #168

dmundra commented Nov 28, 2021

josebolos commented Nov 28, 2021

dmundra commented Nov 28, 2021

dmundra commented Nov 28, 2021

joeyciechanowicz commented Nov 28, 2021 •

edited

Loading

DaveSkender commented Nov 28, 2021 •

edited

Loading

DaveSkender commented Nov 28, 2021 •

edited

Loading

DaveSkender commented Nov 28, 2021

DaveSkender commented Nov 28, 2021 •

edited

Loading

filippoliverani commented Nov 29, 2021

dmundra commented Nov 29, 2021

dmundra commented Nov 29, 2021

aarongoldenthal commented Dec 10, 2021 •

edited

Loading

josebolos commented Mar 23, 2022

Running version 3.0.0 #168

Running version 3.0.0 #168

Comments

dmundra commented Nov 28, 2021

josebolos commented Nov 28, 2021

dmundra commented Nov 28, 2021

dmundra commented Nov 28, 2021

joeyciechanowicz commented Nov 28, 2021 • edited Loading

DaveSkender commented Nov 28, 2021 • edited Loading

DaveSkender commented Nov 28, 2021 • edited Loading

DaveSkender commented Nov 28, 2021

DaveSkender commented Nov 28, 2021 • edited Loading

filippoliverani commented Nov 29, 2021

dmundra commented Nov 29, 2021

dmundra commented Nov 29, 2021

aarongoldenthal commented Dec 10, 2021 • edited Loading

josebolos commented Mar 23, 2022

joeyciechanowicz commented Nov 28, 2021 •

edited

Loading

DaveSkender commented Nov 28, 2021 •

edited

Loading

DaveSkender commented Nov 28, 2021 •

edited

Loading

DaveSkender commented Nov 28, 2021 •

edited

Loading

aarongoldenthal commented Dec 10, 2021 •

edited

Loading