Test Format V1 Definition

Overview of test files

Each test plan is defined by a set of files in its own directory inside of the "tests" directory. The name of the test plan subdirectory is named to reflect the test case, e.g, "alert" or "horizontal-slider". In the below list of files, the directory for each plan is represented by the string PLAN_DIRECTORY.

The test builder extracts and translates content from the following files:

• tests/PLAN_DIRECTORY/data/commands.csv
• tests/PLAN_DIRECTORY/data/references.csv
• tests/PLAN_DIRECTORY/data/tests.csv
• tests/resources/keys.mjs

commands.csv

The columns in this CSV file contain the following information.

testId

The test id. Multiple rows can have the same testId, e.g. if they vary on mode

task

The task value is a human-readable string. The content of the string is a generic description of an action a user takes in the form of <action> <noun>, such as "operate checkbox" or "navigate to menuitem" or "navigate through checkbox group".

The task string is used as a key to find the set of specific AT commands a user should perform during the execution of the test. These are the sets of the commands that constitute the answer to the question "When will this information be communicated" described in step one of test writing above. It is possible that the same task string will used for more than one tests you write for a single widget (for example, if you are testing navigating to a menuitem in a menubar and navigating to a menuitem in a submenu, the task in both cases will be "navigate to menuitem"). In general, all tests that should be performed using the same set of commands should have the same task string.

mode

The mode value is a list of one of screen reader modes.

at

A key for an assistive technology that is listed in tests/support.json, e.g. voiceover-macos.

commandA through commandF

Keyboard command sequence related to executing the task using key codes in resources/keys.mjs. Each cell is a step in the sequence and only one key code in each cell. The first cell must contain a key code, unused cells should be empty.

references.csv

This file provides values for the references specified in the refs column in the tests.csv file.

The refs column contains ARIA roles, properties and states values associated with the test associated with the row in test.csv.

The columns in this CSV file contain the following information.

refId

The name of reference information. The following is a list of refIds that are often included:

• author
• authorEmail
• designPattern - The value contains the link to the design pattern found in ARIA Practices, like https://w3c.github.io/aria-practices/#checkbox
• example - required - The value contains the link to original example html from aria practices, like https://w3c.github.io/aria-practices/examples/accordion/accordion.html
• reference - required - The value contains a link to the reference html file for the example the user will be using doing the test, like reference/2020-11-23_175030/checkbox-1/checkbox-1.html. A skeleton for this reference can be generated from the example using the update-reference script. The reference should be placed in the reference directory for the test in a subdirectory named with the date of creation format YYYY-MM-DD_HHmmss. Anything this reference needs to change in a way that might effect test results (i.e not just fixing a typo) a whole new reference subdirectory should be created and this file should be updated.
• states, properties, roles - The key should be names of relevant roles, properties, and states for the example used in the tests.csv. The corresponding value should be the url with more information about this reference.

value

The value for the corresponding refId.

tests.csv

The columns in this CSV file contain the following information.

testId

The test id.

title

The title of the test is the name of the test. It should describe the interaction and the AT behavior in straightforward and brief terms. When reviewing support for the widget, a web author should be able to see whether a test with this name passes or fails and be able to get a reasonable idea of the experience of an AT user of this widget.

appliesTo

The name of one or more assistive technologies, that is listed in tests/support.json, e.g. "NVDA,JAWS", or a group, e.g. "Desktop Screen Readers".

mode

Same as in commands.csv documented above.

task

Same as in commands.csv documented above.

setupScript

Is a file name to a file in the js directory.
For example, the file name "focusonfirstlink" corresponds to js\focusonfirstlink.js.

setupScriptDescription

A short description on what the setup script does.

Examples:

• "sets focus on the 'Font' menu item"
• "sets focus on a link just before the menubar"

refs

Space separated list of ARIA roles, properties and states associated with the test associated with the row.

instructions

A short description of the task, starting with a verb. If the description requires more than one statement, the | can be used to separate statements.

Examples:

• "Navigate to the 'Text Formatting' menubar"
• "Open "Font" menu with down arrow. | Navigate to 'Sans-serif' menu item radio."

assertion1 through assertion6

The assertions are human-readable strings. The assertions fully describe the expected behavior of the assistive technology. The list contains the answer to the question "What information should be communicated to the users by the assistive technology as a result of this interaction?" in the preceding section.

Break the assertions down into a list of atomic observable. This is the list of required information a screen reader should communicate in order for the test to pass, and each assertion should be able to fail or pass independently of the others. The assertions should be written in terms of the example widget in order to leave no room for ambiguity as to what the tester should observe after performing the relevant key command. If a user should hear the accessible name when navigating to a checkbox, and the accessible name of the checkbox is "Lettuce", then the test author should write the assertion: "Name 'Lettuce' is conveyed".

The assertions can have priorities, the default is "1", but a assertion that is not as important can be given the priority "2". To give an assertion the priority "2", start the string with 2:.

Example assertions:

• "The role 'menubar' is conveyed"
• "The menubar name 'Text Formatting' is conveyed"
• "The menu item name "Font" is conveyed"
• "Availability of a submenu is conveyed"
• "2: The position of the item and number of items in the menubar are conveyed"

Outdated documentation (TODO)

A link tag that references an APG design pattern

There should be one link tag with href set to the APG example under test. For example, if you are writing tests of the two-state checkbox example, you should include the following link:

<link rel="help" href="https://w3c.github.io/aria-practices/examples/checkbox/checkbox-1/checkbox-1.html">

Multiple link tags that reference the ARIA specification

There should be a link tag for the ARIA specification's definition of all the relevant roles and attributes that come into play during this test.

If you are writing a test for reading a checkbox, you should see the following tags:

<link rel="help" href="https://w3c.github.io/aria/#checkbox">
<link rel="help" href="https://w3c.github.io/aria/#aria-checked">

A script element

The script will contain one test written according to the aria-at-harness test API. verifyATBehavior accepts an object with the following keys. These fully describe how to verify a behavior.

setupTestPage

This key is not mandatory. It is only necessary if the widget must be programmatically put into a state before the user should perform the single AT command. The "setupTestPage" value is a function that takes a document object after page load that represents the test page. The function should put the widget into the appropriate state to perform the test. For example, it might open a submenu or check a checkbox.

specific_user_instructions

The specific_user_instructions value is also a human-readable string. This value should be different for every test file. This string describes the action such that a tester can perform it without ambiguity, and to do so it is specific to the example widget (unlike the task above). For example, "navigate to the first checkbox" or "navigate through the first group in the second submenu".