V2 format test plan example review

[mcking65]

For discussion in August 16, 2023 meeting.

Following is a mockup of a review page for 1 test from the horizontal slider test plan. This mockup is based on the new V2 test format designed in #974.

The top section of the mockup includes a set of radios that enable filtering of tests for each covered AT and the open test page button, neither of which will render correctly in a GitHub comment. You can see those elements of the mockup in this zip of the HTML:
horizontal-slider-TestPlanReviewMockup.zip

Note that every assertion is shown with two different wordings -- an assertion statement and an assertion phrase. This is to improve understandability of content in different contexts:

• Assertion statement is for checkbox labels in the test runner.
• Assertion phrase is a shorter form used in reports.

A report mockup is available in w3c/aria-at-app#744.

Here is the mockup content.

Color Viewer Test plan (9 tests)

Filter tests by covered assistive technology All assistive technologies JAWS NVDA VoiceOver for macOS

Supporting Documentation

• APG Color Viewer Slider Example
• APG Slider Pattern
• Developmen documented in ARIA-AT GitHub issue 403

Test 1: Navigate forwards to a slider

Tested ARIA features:

• slider role specification
• aria-labelledby attribute specification
• aria-valuenow attribute specification
• aria-valuemin attribute specification
• aria-valuemax attribute specification

Open
Test 1 Page

JAWS

Instructions

1. Configure JAWS with default settings. For help, read Configuring Screen Readers for Testing.
2. Activate the "Open test page" button, which opens the example to test in a new window and sets focus on the "Run Test Setup" button.
3. Activate the "Run Test Setup" button, which runs a script that sets focus on a link before the Red slider.
4. Starting at the 'Navigate forwards from here' link, navigate to the 'Red' slider. Do this with each of the following commands or command sequences. If any settings are specified in parenthesies, ensure the settings are active before executing the command or command sequence.
   • Down Arrow then Down Arrow (Virtual Cursor Active)
   • Tab (Virtual Cursor Active)
   • Tab (PC Cursor Active)
5. After performing each command or command sequence listed above, complete the section of the below results form for that command or command sequence.
6. IMPORTANT!Before performing the test with the next command or command sequence:
   • Refresh the test page.
   • Activate the "Run Test Setup" button.
   • Ensure any settings specified for the next command or command sequence are active.

To perform a command with virtual cursor active:

1. Press Alt+Delete to determine which cursor is active.
2. If the PC cursor is active, press Escape to activate the virtual cursor.

To perform a command with PC cursor active:

1. Press Alt+Delete to determine which cursor is active.
2. If the virtual cursor is active, press Insert+z to disable the virtual cursor.

Down Arrow then Down Arrow (virtual cursor active): 3 MUST, 3 SHOULD, AND 0 MAY assertions

Priority	Assertion Phrase	Assertion Statement
MUST	convey role 'slider'	Role 'slider' is conveyed
MUST	convey name 'Red'	Name 'Red' is conveyed
MUST	convey value '128'	Value '128' is conveyed
SHOULD	convey Orientation 'horizontal'	Orientation 'horizontal' is conveyed
SHOULD	convey minimum value '0'	Minimum value '0' is conveyed
SHOULD	convey maximum value '255'	Maximum value '255' is conveyed

Tab (virtual cursor active): 3 MUST, 4 SHOULD, AND 0 MAY assertions

Priority	Assertion Phrase	Assertion Statement
MUST	convey role 'slider'	Role 'slider' is conveyed
MUST	convey name 'Red'	Name 'Red' is conveyed
MUST	convey value '128'	Value '128' is conveyed
SHOULD	convey Orientation 'horizontal'	Orientation 'horizontal' is conveyed
SHOULD	convey minimum value '0'	Minimum value '0' is conveyed
SHOULD	convey maximum value '255'	Maximum value '255' is conveyed
SHOULD	Switch from virtual cursor active to PC cursor active	JAWS switches from virtual cursor active to PC cursor active

Tab (PC cursor active): 3 MUST, 3 SHOULD, AND 0 MAY assertions

Priority	Assertion Phrase	Assertion Statement
MUST	convey role 'slider'	Role 'slider' is conveyed
MUST	convey name 'Red'	Name 'Red' is conveyed
MUST	convey value '128'	Value '128' is conveyed
SHOULD	convey Orientation 'horizontal'	Orientation 'horizontal' is conveyed
SHOULD	convey minimum value '0'	Minimum value '0' is conveyed
SHOULD	convey maximum value '255'	Maximum value '255' is conveyed

NVDA

Instructions

1. Configure NVDA with default settings. For help, read Configuring Screen Readers for Testing.
2. Activate the "Open test page" button, which opens the example to test in a new window and sets focus on the "Run Test Setup" button.
3. Activate the "Run Test Setup" button, which runs a script that sets focus on a link before the Red slider.
4. Starting at the 'Navigate forwards from here' link, navigate to the 'Red' slider. Do this with each of the following commands or command sequences. If any settings are specified in parenthesies, ensure the settings are active before executing the command or command sequence.
   • Down Arrow then Down Arrow (browse mode on)
   • Tab (browse mode on)
   • Tab (focus mode on)
5. After performing each command or command sequence listed above, complete the section of the below results form for that command or command sequence.
6. IMPORTANT!Before performing the test with the next command or command sequence:
   • Refresh the test page.
   • Activate the "Run Test Setup" button.
   • Ensure any settings specified for the next command or command sequence are active.

To perform a command with browse mode on:

1. Press insert+Space.
2. If NVDA made the focus mode sound , press Insert+Space again to turn browse mode back on.

To perform a command with focus mode on:

1. Press insert+Space.
2. If NVDA made the browse mode sound , press Insert+Space again to turn focus mode back on.

Down Arrow then Down Arrow (browse mode on): 3 MUST, 3 SHOULD, AND 0 MAY assertions

Priority	Assertion Phrase	Assertion Statement
MUST	convey role 'slider'	Role 'slider' is conveyed
MUST	convey name 'Red'	Name 'Red' is conveyed
MUST	convey value '128'	Value '128' is conveyed
SHOULD	convey Orientation 'horizontal'	Orientation 'horizontal' is conveyed
SHOULD	convey minimum value '0'	Minimum value '0' is conveyed
SHOULD	convey maximum value '255'	Maximum value '255' is conveyed

Tab (browse mode on): 3 MUST, 4 SHOULD, AND 0 MAY assertions

Priority	Assertion Phrase	Assertion Statement
MUST	convey role 'slider'	Role 'slider' is conveyed
MUST	convey name 'Red'	Name 'Red' is conveyed
MUST	convey value '128'	Value '128' is conveyed
SHOULD	convey Orientation 'horizontal'	Orientation 'horizontal' is conveyed
SHOULD	convey minimum value '0'	Minimum value '0' is conveyed
SHOULD	convey maximum value '255'	Maximum value '255' is conveyed
SHOULD	Switch from browse mode on to focus mode on	NVDA switches from browse mode on to focus mode on

Tab (focus mode on): 3 MUST, 3 SHOULD, AND 0 MAY assertions

Priority	Assertion Phrase	Assertion Statement
MUST	convey role 'slider'	Role 'slider' is conveyed
MUST	convey name 'Red'	Name 'Red' is conveyed
MUST	convey value '128'	Value '128' is conveyed
SHOULD	convey Orientation 'horizontal'	Orientation 'horizontal' is conveyed
SHOULD	convey minimum value '0'	Minimum value '0' is conveyed
SHOULD	convey maximum value '255'	Maximum value '255' is conveyed

VoiceOver for macOS

Instructions

1. Configure VoiceOver for macOS with default settings. For help, read Configuring Screen Readers for Testing.
2. Activate the "Open test page" button, which opens the example to test in a new window and sets focus on the "Run Test Setup" button.
3. Activate the "Run Test Setup" button, which runs a script that sets focus on a link before the Red slider.
4. Starting at the 'Navigate forwards from here' link, navigate to the 'Red' slider. Do this with each of the following commands or command sequences. If any settings are specified in parenthesies, ensure the settings are active before executing the command or command sequence.
   • Control + Option + Right Arrow then Control + Option + Right Arrow
   • Tab
   • J (quick nav on)
5. After performing each command or command sequence listed above, complete the section of the below results form for that command or command sequence.
6. IMPORTANT!Before performing the test with the next command or command sequence:
   • Refresh the test page.
   • Activate the "Run Test Setup" button.
   • Ensure any settings specified for the next command or command sequence are active.

To perform a command with quick nav on:

1. Simultaneously press left and right arrow keys.
2. If VoiceOver said 'quick nav off', press the left and right arrow keys again to turn it back on.

Control + Option + Right Arrow then Control + Option + Right Arrow: 3 MUST, 3 SHOULD, AND 0 MAY assertions

Priority	Assertion Phrase	Assertion Statement
MUST	convey role 'slider'	Role 'slider' is conveyed
MUST	convey name 'Red'	Name 'Red' is conveyed
MUST	convey value '128'	Value '128' is conveyed
SHOULD	convey Orientation 'horizontal'	Orientation 'horizontal' is conveyed
SHOULD	convey minimum value '0'	Minimum value '0' is conveyed
SHOULD	convey maximum value '255'	Maximum value '255' is conveyed

Tab: 3 MUST, 3 SHOULD, AND 0 MAY assertions

Priority	Assertion Phrase	Assertion Statement
MUST	convey role 'slider'	Role 'slider' is conveyed
MUST	convey name 'Red'	Name 'Red' is conveyed
MUST	convey value '128'	Value '128' is conveyed
SHOULD	convey Orientation 'horizontal'	Orientation 'horizontal' is conveyed
SHOULD	convey minimum value '0'	Minimum value '0' is conveyed
SHOULD	convey maximum value '255'	Maximum value '255' is conveyed

J (quick nav on): 3 MUST, 3 SHOULD, AND 0 MAY assertions

Priority	Assertion Phrase	Assertion Statement
MUST	convey role 'slider'	Role 'slider' is conveyed
MUST	convey name 'Red'	Name 'Red' is conveyed
MUST	convey value '128'	Value '128' is conveyed
SHOULD	convey Orientation 'horizontal'	Orientation 'horizontal' is conveyed
SHOULD	convey minimum value '0'	Minimum value '0' is conveyed
SHOULD	convey maximum value '255'	Maximum value '255' is conveyed

[mcking65]

@jscholes

The above mockup and attached zip now reflect your suggestions from today's meeting:

• I added 3 steps to instructions. One before the test task and 2 after. they now include more complete information about initial setup and refreshing the browser when repeating the test for each command.
• The command settings instructions are in details/summary elements.
• The assertion phrases start with lower case.

If this looks good, I will update the spec in #977.

[mcking65]

Test Format Definition V2 is implemented and the new test plan previews are functional.