Accessibility Testing Approach

[boazsender]

Starting this issue to discuss and gather input on the various approaches I've heard to accessibility testing in WPT:

1. Use the Accessibility Object Model

AOM is partially implemented in Chrome and Safari, but not Firefox. The partial implementation looks like enough to get started with (ARIA attrs on DOM nodes), but the lack of support in Firefox means we'd need to patch Firefox for this to work. This approach has the benefit of minimal changes to wpt/testdriver, but tests written this way would only test the AOM/Accessibility Tree's accuracy, and not the fully rendered accessible UI. I'm not sure we could trust this level of observation for WPTs purposes.

2. Use WebDriver

This approach would include adding the new webdriver APIs for things like Get Computed Role to testdriver.js. Computed role is implemented in chrome and safari driver, but not yet in gecko. So we'd need to make a gecko driver patch for this to work. I believe this is a higher level of observation than AOM, and would be more trust worthy information, though I may be making an incorrect assumption there. Would appreciate confirmation of that.

3. Use Browser Specific Serialization

This approach would involve expanding Chromium's DumpAccessibilityTree tests to Webkit/Gecko (unless Webkit/Gecko already have this?), integrating that into wpt, and writing ref tests against the dumps. This has the benefit of testing directly against the accessibility tree, and has the downside of introducing unstandardized browser specific testing interfaces. I'm also not sure what this would look like in Webkit/Gecko, would love to know if similar testing serializers exist. If not, we have an additional downside of the scope to add those.

4. Use real screen readers

In this approach we'd use the AT Driver spec currently being developed as part of ARIA-AT to introduce real screen readers to the WPT testing stack. The AT Driver spec is being prototyped in NVDA, with hopefully more implementations coming in other screen readers. This has the benefit of allowing us to write reference tests against the text to be spoken, and working across all browsers out of the box. The downside is that it adds the more complex machinery to the WPT stack.

I'm planning to write an RFC for one of these approaches, but wanted to get input first. What do folks think about each of these approaches?

[zcorpan]

I think option 4 is harder for browser developers to reason about the results as it's testing a representation that the browser doesn't have control over and there's no clear mapping back to the browser accessibility tree.

I'll check which of the other options work for Gecko.

[zcorpan]

For option 1, the relevant thing in AOM to test the accessibility tree I think is "Phase 4: Query computed accessibility tree.", which seems to be implemented ("Yes, out-of-date syntax") in Chrome and Firefox, but not Safari. It's also described as "Speculative long-term goal." and notes that the WebDriver computedRole/computedLabel are likely to be stable sooner.

[boazsender]

I think option 4 is harder for browser developers to reason about the results as it's testing a representation that the browser doesn't have control over and there's no clear mapping back to the browser accessibility tree.

Interesting, I hadn't thought of it this way. Thanks for raising this insight.

I think it might be meaningful for web accessibility writ large to help browser developers reason about the same representation as users experience (e.g. text to be spoken by a screen reader). Browser developers do this for pixel rendering, and it would be great to have browser developers doing this for sonic rendering. That being said, I do understand that there's a layer of indirection with screen readers/OS accessibility APIs that doesn't exist for raster rendering with the rendering tree. Perhaps a combination of accessibility tree tests, and spoken text tests is in order? What do you think?

For option 1, the relevant thing in AOM to test the accessibility tree I think is "Phase 4: Query computed accessibility tree.", which seems to be implemented ("Yes, out-of-date syntax") in Chrome and Firefox, but not Safari. It's also described as "Speculative long-term goal." and notes that the WebDriver computedRole/computedLabel are likely to be stable sooner.

Good point, I glossed over phase 4. I think this approach would be better than option 3, since it's standard.

@nt1m, @gsnedders, and @jensimmons, do you have sense of where AOM phase 4 support falls in Safari's priority list?

[gsnedders]

I think what @cookiecrook said in web-platform-tests/interop#211 (comment) is still mostly what we think, FWIW.

We have a strong preference for doing this via WebDriver (via computedRole and computedLabel to introspect ARIA values); this provides most of the value of the "phase 4" of AOM, especially in combination with child/parent links. But as James said there, there's still significant differences in what that tree is. And while this doesn't cover everything you would eventually get out of the screen reader, it does cover the overwhelming majority of bugs we see.

We can't comment as to future product plans or our prioritisation of potential future features, but I do think it's by far the most viable path forward in the near term—though we may have to accept things in a less than ideal state unless we get further consistency across implementations than we have today (e.g., whether the AX tree is based on the render tree or the DOM), but this would provide an ability to at least check parent/child relationships even if parts of the tree might differ (especially for things like the UA-provided shadow DOM for things like video controls, but also CSS generated content, etc.).

[zcorpan]

Browser developers do this for pixel rendering

Generally, only when necessary. It's not the preferred means of testing things lower in the stack, because it's indirect and it can start failing for reasons unrelated to the thing being tested (e.g. anti-aliasing).

That's not to say that testing with screen readers shouldn't be done. I think it should, as otherwise we can't test OS accessibility APIs or screen readers. But for this investigation area, I think it's out of scope.

[zcorpan]

For option 2, Mozilla is planning to implement in geckodriver in the April-June timeframe per https://bugzilla.mozilla.org/show_bug.cgi?id=1585622#c11

I think we should go ahead with option 2.

[boazsender]

That all makes sense, and great to hear about Mozilla's planned implementation. I agree, let's go forward with option 2. Thanks for thinking through this with me @zcorpan and @gsnedders!

[cookiecrook]

FYI, I'm back in the office this week and working on a partial implementation of option 2.

[clshortfuse]

Hi all. Are there any interest in an outputted tree in addition to the role/label systems?

1. The use of a tree will allow testing a role/label is present at certain point, not a certain node.

Currently, I would like to ensure a Web Component presents a role at a certain point in the layout tree. For example:

<x-button role=none>
  #shadowroot
  <input type=button aria-labelledby=slot>
  <slot id=slot></slot>
</x-button>

And

<x-button role=button>
  #shadowroot
  <slot id=slot></slot>
</x-button>

<x-button> <!-- role set via ElementInternals -->
  #shadowroot
  <slot id=slot></slot>
</x-button>

It would make sense that the next "meaningful" ARIA role starting from <x-button> is button. But there are multiple ways to achieve this, including hiding the attribute via ElementInternals. I think I might be able to iterate the DOM and test each element manually, but I'm worried about closed shadowRoots. I have concerns with just iterating the DOM because I'm not really sure what's being exposed by the browsers to screen-readers. There might be groups or hidden nodes. The DOM and AXTree isn't guaranteed to be 1:1.

There are many web testing suites that just read the DOM, which isn't accurate. Manually testing in the browser is somewhat tedious, but at this point has helped me recognize some browser bugs with computed labels:

• https://bugzilla.mozilla.org/show_bug.cgi?id=1826194
• https://bugs.webkit.org/show_bug.cgi?id=254934

I am also aware of other AXLabel bugs, but haven't had time to file them. But most of them deal in one way or another with complex trees.

---

2. Use of a tree allows expanding testing beyond just label and role.

I recently found a mistake on my part where I had a custom element like above, but using <input type=checkbox>. I used aria-checked=mixed manually, assuming browsers did not apply .indeterminate as aria-checked=mixed. I probably had to do this with IE and it's just something I stuck with. When I used a tester (Playwright) that exposes their own version of an AXTree, I learned of this Chrome bug:

• https://bugs.chromium.org/p/chromium/issues/detail?id=1260371

So, after having checked the aria tree with its properties, I was able to rewrite some code around make sure all the 3 majors browsers properly show aria-checked=mixed

---

Related note (Option 4). I did write something a while ago to emulate a screen reader over a virtual COM port to test JAWS/NVDA, but getting it to work on anything but Windows was too hard of a task for me at the time so I abandoned it. The Webdriver solution seems good enough for me now.

[cookiecrook]

@clshortfuse see

• Investigation: Accessibility Tree Dump API #51

Also this aproach should allow tree walking and cover the other Pros you mentioned above.

• Computing the Accessible Backing Node, or a similar test-only interface WICG/aom#197