-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update test builder to support V2 of test format #977
Comments
Note that I have updated the V2 format specification with the following changes. I have not yet revised the requirements in this issue to reflect these updates. Revised testId and assertionId descriptions:
Revised presentationNumber specifications for tests and commands:
Revised requirements related to refId values as described in the assertion and references sections:
Defined new values for reference types:
Revised reference value specification for Added requirements for Added a section describing how link text and href values are calculated for reference links. Updated validation rules. |
Noted, thanks for pointing that out |
Reopening, as merging #988 doesn't actually resolve this. |
Using the preview content from #997and #998, I have Fully tested this against the specs. Since this work is already merged to master in aria-at and does not require a deployment, I am closing the issue as complete. I have raised #1008, #1009, and #1010 to track required follow-up work and address problems I found during test. |
Objective
Revise the aria-at build scripts so they support not only V1 but also the new V2 test format defined in #974.
While versions of the test plans that are written in the V1 format will all eventually be deprecated, for at least a few years, we will continue to need the ability to view plans written in the V1 format.
Note: Consider moving the directory
tests/resources
toscripts/resources
or some other location outside oftests
since the content of that folder is not related to any specific test plan and appears to be related to building the test plans.Changes to build review files
When running the build, a review file for each test plan is generated in the directory
build/review
. When the builder is building a test plan that is written in the V2 format, it should generate an HTML file that has the following differences when compared to the files currently being generated for the V1 format.This zip file contains a mockup of a portion of the slider test plan in V2 format.
horizontal-slider.zip
Note that this description is referring to properties in support.json that are visible in the slider-refactor branch via #975. That branch includes a refactor of the horizontal-slider test plan into the V2 format.
Page title
Change the H1 to have format
TITLE Test Plan (NUMBER_OF_TESTS tests)
where TITLE is the value of thetitle
refId inreferences.csv
.:Example:
Filter radio group
Change the label of the radio group to communicate that the radios represent the AT covered by the plan:
Example:
To ensure there is a radio button for each AT that is covered, use the names of"AT_KEY-command.csv" files as the source of which AT are covered. For each command CSV file, include a radio button labeled using the corresponding
name
property insupport.json
.Supporting documentation section
After the filters, add a level 2 heading with text "Supporting Documentation".
After the heading, list any of the following references that are included in
references.csv
. Since these apply to the entire test plan, do not include these references in the list of relevant specifications under theh2
for each test:For each of the above
refIds
, include a link using link text from thelinkText
column and the URI from thevalue
column.Example:
Changes in list under test title
Make the following changes in the
ul
under the heading with the test title, e.g.,<h2 class="jaws nvda voiceover_macos">Test 1: Navigate forwards to a slider</h2>
.Remove specification of screen reader mode.
Example:
<li>Mode: interaction</li>
Remove "applies to" statement.
Example:
<li>Applies to: jaws, nvda, voiceover_macos</li>
Add a paragraph before the list with text from
ariaSpecsPreface
whereariaSpecsPreface
is the value of the name property of an object in thetestPlanStrings
array insupport.json
and the paragraph text is the value of thestring
property in that object.Change the content of the subsequent lis to:
refIds
specified inassertions.csv
for the assertions covered by this test as specified in theassertions
column oftests.csv
.LinkText
column and the URI from thevalue
column ofreferences.csv
.Example from slider test 1:
Changes under the
h3
for each ATMake the following changes to the content that follows the level 3 heading for each AT, e.g.,
<h3>JAWS</h3>
.Remove the level 4 heading and content covering scripted instructions.
Example:
Make the following changes under
<h4>Tester Instructions</h4>
:Change the heading content to "Instructions".
Example:
<h4>Instructions</h4>
Make the first instructions list item the instruction for configuring the AT with default settings. use the value of the
defaultConfigurationInstructionsHTML
property insupport.json
.Example:
Remove the instructions list item about setting reading or interaction mode.
Example:
<li>Verify the PC Cursor is active by pressing Alt+Delete. If it is not, turn off the Virtual Cursor by pressing Insert+Z.</li>
Make the second instructions list item the open example instruction with form: `openExampleInstruction + " " + setupScriptDescription + "."`` where:
openExampleInstruction
is thename
of astring
in arraytestPlanStrings
insupport.json
.setupScriptDescription
is the value in thesetupScriptDescription
column indata/scripts.csv
on the row where thesetupScript
value matches thesetupScript
value in the row intests.csv
for the test being presented.Example:
Make the third instructions list item the perform task instruction with one of the following two forms:
AT_KEY-commands.csv
have settings specified,testInstructions + " " + commandListPreface + " " + commandListSettingsPreface + ":"
AT_KEY-commands.csv
have settings specified,testInstructions + " " + commandListPreface + ":"
Where:
testInstructions
is pulled fromdata/tests.csv
commandListPreface
is thename
of astring
in arraytestPlanStrings
insupport.json
.commandListSettingsPreface
is thename
of astring
in arraytestPlanStrings
insupport.json
.This list item includes a nested list of all commands assigned to this test in
AT_KEY-commands.csv
where each list item contains a command-sequence string and an optional settings specification.:commands.json
, a string that represents the command. If the command is a sequence, each command in the sequence is separated by the string " then ".screenText
property specified for each setting in thesettings
array for the AT insupport.json
.The pairing of command-sequence string and settings specifications used in these list items is also used in other parts of the test plan display. For ease of reference, It is referred to as a command-specification string.
Example:
If any of the commands in the above list specify a setting, follow the instructions with a paragraph and ordered list for each setting that explains how to activate the setting.
The paragraph contains
settingInstructionsPreface + " " + screenText + ":"
where:settingInstructionsPreface
is thename
of astring
in arraytestPlanStrings
insupport.json
.screenText
is the value of thescreenText
property for that setting in the settings array for that AT insupport.json
.The ordered list contains the content of the
instructions
property for that setting in thesettings
array for that AT insupport.json
. Theinstructions
property is an array, and each element in the array is an item in the ordered list.Example:
Replace the current table listing assertions and their priorities with a series of headings and tables. One table for each command that is preceded by a level 4 heading. The order of the series is determined by the
presentationNumber
values in theAT_KEY-commands.csv
file for the given test.The heading has form:
command-specification string
+ ": X MUST, Y SHOULD, and Z MAY assertions"Where:
command-specification string
is defined above in the description of the instructions section.The table has 3 columns:
Where:
tests.csv
,assertions.csv
, andAT_KEY-commands.csv
files.assertionId
values in theassertions
in the assertions column intests.csv
.Example from the JAWS section of the slider test 1 preview:
CSV validation
Some validation rules to check for errors in the CSV content are documented in the Validation section of the Test Format V2.
The text was updated successfully, but these errors were encountered: