Handle accessibility incomplete and violations simultaneously #14354

thibaudcolas · 2022-09-04T20:27:32Z

FAQ

Yes, my issue is not about variability or throttling.
~~[ ] Yes, my issue is not about a specific accessibility audit (file with axe-core instead).~~

URL

https://accessibility-experiments-90d5b5.netlify.app/lighthouse-bypass-audit/index.html

What happened?

The bypass audit based on Axe’s bypass rule always shows as passing when it’s applicable, even when there are issues on the page that Axe itself detects.

Sample page: https://accessibility-experiments-90d5b5.netlify.app/lighthouse-bypass-audit/index.html

Lighthouse results for bypass on this page (JSON export)

"bypass": {
      "id": "bypass",
      "title": "The page contains a heading, skip link, or landmark region",
      "description": "Adding ways to bypass repetitive content lets keyboard users navigate the page more efficiently. [Learn more](https://web.dev/bypass/).",
      "score": 1,
      "scoreDisplayMode": "binary",
      "details": {
        "type": "table",
        "headings": [],
        "items": []
      }
    },

Axe CLI results for the same page

Command:

axe -r bypass https://accessibility-experiments-90d5b5.netlify.app/lighthouse-bypass-audit/index.html --save bypass.json && cat bypass.json

Output:

[
  {
    "testEngine": {
      "name": "axe-core",
      "version": "4.3.5"
    },
    "testRunner": {
      "name": "axe"
    },
    "testEnvironment": {
      "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/104.0.5112.101 Safari/537.36",
      "windowWidth": 800,
      "windowHeight": 600,
      "orientationAngle": 0,
      "orientationType": "landscape-primary"
    },
    "timestamp": "2022-09-03T10:13:28.251Z",
    "url": "https://accessibility-experiments-90d5b5.netlify.app/lighthouse-bypass-audit/index.html",
    "toolOptions": {
      "runOnly": {
        "type": "rule",
        "values": [
          "bypass"
        ]
      },
      "reporter": "v1"
    },
    "inapplicable": [],
    "passes": [],
    "incomplete": [
      {
        "id": "bypass",
        "impact": "serious",
        "tags": [
          "cat.keyboard",
          "wcag2a",
          "wcag241",
          "section508",
          "section508.22.o"
        ],
        "description": "Ensures each page has at least one mechanism for a user to bypass navigation and jump straight to the content",
        "help": "Page must have means to bypass repeated blocks",
        "helpUrl": "https://dequeuniversity.com/rules/axe/4.3/bypass?application=webdriverjs",
        "nodes": [
          {
            "any": [
              {
                "id": "internal-link-present",
                "data": null,
                "relatedNodes": [],
                "impact": "serious",
                "message": "No valid skip link found"
              },
              {
                "id": "header-present",
                "data": null,
                "relatedNodes": [],
                "impact": "serious",
                "message": "Page does not have a heading"
              },
              {
                "id": "landmark",
                "data": null,
                "relatedNodes": [],
                "impact": "serious",
                "message": "Page does not have a landmark region"
              }
            ],
            "all": [],
            "none": [],
            "impact": "serious",
            "html": "<html lang=\"en\" class=\"deque-axe-is-ready\">",
            "target": [
              "html"
            ],
            "failureSummary": "Fix any of the following:\n  No valid skip link found\n  Page does not have a heading\n  Page does not have a landmark region"
          }
        ]
      }
    ],
    "violations": []
  }
]

What did you expect?

Lighthouse’s results for bypass should be consistent with those of Axe. In this instance, Axe reports bypass as "incomplete", so it shouldn’t be a pass with Lighthouse. I would have expected either a "Additional items to manually check", or an error.

What have you tried?

I’ve investigated how the Axe rule works, and suspect the issue is due to Axe changing the output of this rule from violations to incomplete. This apparently happened in Axe v4.2 because although the rule can detect the absence of "bypass" functionality, it can’t detect whether there is actually anything to bypass on a page.

I suspect the issue in Lighthouse is in the Axe audit processing logic, around here, where Axe handles incomplete results differently based on the audit’s display mode:

lighthouse/core/audits/accessibility/axe-audit.js

Lines 47 to 58 in fd304ed

    
           const incomplete = artifacts.Accessibility.incomplete || []; 
        
           const incompleteResult = incomplete.find(result => result.id === this.meta.id); 
        
           if (incompleteResult?.error) { 
        
             return { 
        
               score: null, 
        
               errorMessage: `axe-core Error: ${incompleteResult.error.message || 'Unknown error'}`, 
        
             }; 
        
           } 
        
           const isInformative = this.meta.scoreDisplayMode === Audit.SCORING_MODES.INFORMATIVE; 
        
           const violations = artifacts.Accessibility.violations || []; 
        
           const failureCases = isInformative ? violations.concat(incomplete) : violations;

I’m unclear whether the current logic could work or whether there needs to be additions here.

How were you running Lighthouse?

Chrome DevTools, web.dev

Lighthouse Version

9.6.6

Chrome Version

107

Node Version

No response

OS

Mac

Relevant log output

No response

The text was updated successfully, but these errors were encountered:

adamraine · 2022-09-06T18:24:35Z

Thanks for reporting this and doing an investigation! Not very often that someone reports a page that should be failing an audit but isn't.

Your analysis is correct. The failure is hidden by this line:

lighthouse/core/audits/accessibility/axe-audit.js

Line 58 in fd304ed

    
           const failureCases = isInformative ? violations.concat(incomplete) : violations;

The audit is not classified as informative so we ignore any incomplete reports from axe. I think it's worth discussing making the bypass audit informative to surface these cases.

connorjclark · 2022-09-06T19:27:51Z

we could just make this audit informative
or we could show incomplete items w/o it impacting the score of non-informative axe audits

ref:

#10056 (comment)
#10072
#12536 (comment)

thibaudcolas · 2022-10-07T00:12:16Z

FYI I’ve opened #14438 and #14439, which are the same issue as this as far as I can tell, just with different audits. Feel free to close those if it’s better to handle all three problematic audits at once.

Those are the only three audits where I suspected this issue. I spotted this as part of the Accessibility chapter of the HTTP Archive’s Web Almanac, where we review Lighthouse audit scores across 8M websites. Those 3 audits are the only ones that run on hundreds of thousands / millions of pages, and didn’t report a single failure. Here is the Lighthouse audit data if anyone wants to spend more time looking, and our SQL query (BigQuery).

adamraine · 2022-10-07T20:40:01Z

Thanks for the research @thibaudcolas! I think should merge everything into this audit, but good to know which audits this problem affects.

connorjclark · 2022-10-18T19:59:01Z

(10.0) make these audits informative

or (if possible for incomplete and violations to both occur)

if only incomplete, mark informative
if incomplete and violations, emit them as two separate tables
if just violations, do what we do today

brendankenny · 2022-10-18T20:02:05Z

to rationalize the whole axe informative behavior, start with thread here: #12536 (comment)

thibaudcolas added the bug label Sep 4, 2022

github-actions bot assigned adamraine Sep 4, 2022

devtools-bot added the needs-priority label Sep 4, 2022

adamraine added the needs-discussion label Sep 6, 2022

connorjclark added the P2 label Sep 6, 2022

devtools-bot removed the needs-priority label Sep 6, 2022

This was referenced Oct 6, 2022

th-has-data-cells accessibility audit always shows as passing even in error scenarios #14438

Closed

video-caption accessibility audit always shows as passing even in error scenarios #14439

Closed

connorjclark removed the needs-discussion label Oct 18, 2022

connorjclark added the 10.0 label Oct 18, 2022

adamraine mentioned this issue Oct 18, 2022

core: make bypass, th-has-data-cells, and video-caption informative #14453

Merged

adamraine removed the 10.0 label Oct 19, 2022

connorjclark added the breaking label Oct 24, 2022

adamraine changed the title ~~bypass accessibility audit always shows as passing even in error scenarios~~ Handle accessibility incomplete and violations simultaneously Nov 8, 2022

connorjclark added the 11.0 cranked up to eleven label Nov 30, 2022

adamraine removed the bug label May 15, 2023

adamraine mentioned this issue Jun 26, 2023

11.0 Breaking Changes #14909

Closed

20 tasks

adamraine removed the 11.0 cranked up to eleven label Jul 31, 2023

adamraine added this to the v12.0 milestone Jul 31, 2023

paulirish assigned jazyan and unassigned adamraine Sep 11, 2023

adamraine added P3 and removed P2 labels Sep 11, 2023

adamraine removed this from the v12.0 milestone Apr 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle accessibility incomplete and violations simultaneously #14354

Handle accessibility incomplete and violations simultaneously #14354

thibaudcolas commented Sep 4, 2022 •

edited

Loading

adamraine commented Sep 6, 2022

connorjclark commented Sep 6, 2022

thibaudcolas commented Oct 7, 2022

adamraine commented Oct 7, 2022

connorjclark commented Oct 18, 2022 •

edited by adamraine

Loading

brendankenny commented Oct 18, 2022

Handle accessibility incomplete and violations simultaneously #14354

Handle accessibility incomplete and violations simultaneously #14354

Comments

thibaudcolas commented Sep 4, 2022 • edited Loading

FAQ

URL

What happened?

What did you expect?

What have you tried?

How were you running Lighthouse?

Lighthouse Version

Chrome Version

Node Version

OS

Relevant log output

adamraine commented Sep 6, 2022

connorjclark commented Sep 6, 2022

thibaudcolas commented Oct 7, 2022

adamraine commented Oct 7, 2022

connorjclark commented Oct 18, 2022 • edited by adamraine Loading

brendankenny commented Oct 18, 2022

thibaudcolas commented Sep 4, 2022 •

edited

Loading

connorjclark commented Oct 18, 2022 •

edited by adamraine

Loading