Need some way to record accessibility support data

[dd8]

Some rules document accessibility support issues, but don't specify which browser/screen reader combos have support issues. For example:

https://www.w3.org/WAI/standards-guidelines/act/rules/7d6734/

This makes it hard to do regular reviews on the accessibility support section, since the reviewer doesn't know which browser/screen reader combos to retest to confirm if the support issue has been fixed or is still a problem.

At the CG face-to-face in Dublin there was a proposal to use a separate Github repo to record the accessibility support test results as issues, and update these as part of the rule review process.

The type of information that should be recorded for each test result is:

AT version (e.g. NVDA 2022.3)
Browser version (e.g. Chrome 105)
OS version (e.g. macOS 12.6.1)
Interaction mode (next heading key/gesture, tab key, read next item, touch item on screen, etc)

Interaction mode probably needs more discussion, since interaction methods are quite diverse. This is quite important to record since it's not uncommon for some accessibility features to only work in some modes (e.g. iframe titles being read when tabbing, but not when using the read next command in AT).

[dd8]

I'd probably do something like this for each issue in in the accessibility support repo

Rule

https://act-rules.github.io/rules/46ca7f

Accessibility Support

Failed Example 2

https://act-rules.github.io/rules/46ca7f#failed-example-2

• NVDA 2022.2 , Chrome 105, Windows 10, read next image exposed as role=img, aria-label read
• JAWS 2022.2207.25 , Chrome 105, Windows 10, read next image ignored, aria-label not read
• Voiceover 12.6.1, Safari 15.6 macOS 12.6.1, read next image exposed as role=img, aria-label read
• Voiceover 11.5.2, Safari 15.0 macOS 11.5.2, read next image ignored, aria-label not read

[giacomo-petri]

@dd8,

even if the UA/AT version might already clarify it, maybe might be also interesting having the date of the test for an immediate check.

[dd8]

@dd8,

even if the UA/AT version might already clarify it, maybe might be also interesting having the date of the test for an immediate check.

Yes, good idea. I think you want results for different versions of the same UA/AT next to each other in the issue (rather than a batch of results for each test date). That makes it much easier to see changes between versions (e.g. the difference between Safari 15.0 and Safari 15.6 above)

[dd8]

Layout might be clearer with markdown tables:

AT	Browser	OS	Mode	Date	Result
NVDA 2022.2	Chrome 105	Win 10	Read next	2022-09-12	image exposed as role=img, aria-label read
JAWS 2022.2207.25	Chrome 105	Win 10	Read next	2022-09-12	image ignored, aria-label not read
Voiceover 12.6.1	Safari 15.6	macOS 12.6.1	Read next	2022-09-12	image exposed as role=img, aria-label read
Voiceover 11.5.2	Safari 15.0	macOS 11.5.2	Read next	2021-08-30	image ignored, aria-label not read

[WilcoFiers]

This topic might be a good starting point for this work: #1865 (comment)

[mcking65]

The aria-at project has developed a system for writing AT support tests, running tests, building consensus for the tests, and reporting on results. We are very close to the ability to embed summaries of a test plan into the APG. What we are building for APG seems to be what this issue is requesting.

By end of year, we plan to have AT support tables on example pages for at least a few examples. We are well positioned to have most of the APG examples covered by end of next year for JAWS, NVDA, and VoiceOver. By mid year, we hope to have our plan for doing the same on iOS and Android mapped out.

The aria-at project is also building the infra to re-run all test automatically when a new screen reader or browser version is released. That infra will be functional for NVDA in the first half of next year.

If you wanted to leverage aria-at, we would need to discuss:

• How you would contribute the tests and test cases
• How we would separate ACT data from APG data

If this is something you'd like to pursue, please reach out to me directly.

BTW, aria-at is currently focused on testing APG examples, but we plan to expand that scope to include all HTML elements with implicit accessibility semantics. When and how is TBD. AT support tables for APG examples for 3 screen readers is the target we have set as the project's first major milestone.