Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support V2 test format when manually running a test #743

Closed
mcking65 opened this issue Aug 15, 2023 · 2 comments
Closed

Support V2 test format when manually running a test #743

mcking65 opened this issue Aug 15, 2023 · 2 comments
Labels
enhancement New feature or request

Comments

@mcking65
Copy link

mcking65 commented Aug 15, 2023

Overview

Change the content shown in the test runner when a test plan written in the V2 format is opened.

Note:

  • Unlike test plan viewing and reporting functions, It is not essential the system be able to support running both V1 and V2 formatted test plans. We could stop running test plans that are not in the V2 format on a planned transition date.
  • Implementing the solution for this issue does not require Change results collection form to support pass/fail assertion verdicts aria-at#969 to be complete. This issue is a higher priority, although it might be more efficient to work on both in the same PR.
  • A few changes specified below are not dependent on the V2 format; they are minor wording improvements that are simple to make while working on this issue.

In general, the changes needed in the runner are exact reflections of a subset of changes specified for the test plan preview generated during test plan build. Those changes are specified in w3c/aria-at#977.

Changes to display of a test in the runner

Change heading 1

Streamline wording of the first level 1 heading. It is currently:

Testing task:1. Navigate forwards to a collapsed disclosure button in reading mode

Replace the string "Testing task:1." with the string "Test 1:"

Test 1: Navigate forwards to a collapsed disclosure button in reading mode

Note that the V2 format contains a column for presentationNumber in tests.csv. That number specifies the sequence of test presentation.

Remove redundant heading 1 and subsequent paragraph

Following the AT and browser version information, there is another h1 followed by a paragraph that describes the task. Remove the h1 and the paragraph. Both are unnecessary.

Example:

Testing task: Navigate forwards to a collapsed disclosure button in reading mode

With JAWS in reading mode, describe how JAWS behaves when performing task "With the reading cursor on the 'Navigate forwards from here' link, navigate to the 'About' disclosure button."

Change instructions

Change the text of the level 2 heading from "Test Instructions" to "Instructions". Implement the instructions format as described in w3c/aria-at#977. Be sure to include the instructions for command settings where appropriate.

Remove success criteria

Remove the level 2 heading with text "Success Criteria" and the paragraph and list that follow it.

Example:

Success Criteria

To pass this test, JAWS needs to meet all the following assertions when each specified command is executed:

  1. Role 'navigation landmark' is conveyed
  2. Name 'Mythical University' is conveyed
  3. Role list is conveyed
  4. Number of list items, '3', is conveyed
  5. Role 'button' is conveyed
  6. Name 'About' is conveyed
  7. State 'collapsed' is conveyed
Command headings

For each command there is a level 3 and level 4 heading. For example:

After 'F'

Assertions After 'F'

The commands in the headings need to be converted to command-specification strings as described in w3c/aria-at#977 with the command followed by settings in parentheses. For example:

After 'F' (virtual cursor active)

Assertions After 'F' (virtual cursor active)

@mcking65 mcking65 converted this from a draft issue Aug 15, 2023
@mcking65 mcking65 added the enhancement New feature or request label Aug 15, 2023
howard-e added a commit that referenced this issue Nov 14, 2023
howard-e added a commit that referenced this issue Nov 14, 2023
howard-e added a commit that referenced this issue Nov 15, 2023
howard-e added a commit that referenced this issue Nov 16, 2023
howard-e added a commit that referenced this issue Nov 16, 2023
howard-e added a commit that referenced this issue Nov 16, 2023
howard-e added a commit that referenced this issue Nov 16, 2023
howard-e added a commit that referenced this issue Nov 16, 2023
howard-e added a commit that referenced this issue Nov 16, 2023
howard-e added a commit that referenced this issue Nov 20, 2023
howard-e added a commit that referenced this issue Nov 27, 2023
howard-e added a commit that referenced this issue Nov 27, 2023
howard-e added a commit that referenced this issue Dec 5, 2023
* Update client/resources

* Update import script to support v2 test format import

* Update import script to include flattened commandsV2.json and add uniquely generated scenario and assertion IDs

* Updating client and server files to support V2 test format

* Remove console.log and add TODO

* Include tokenized assertionStatement in TestPlanVersion.tests[x].assertions

* Add retrieveCommands utility

* Update candidate review and datamanagement page

* Update Reports/queries to also include mayOptionalAssertionResults

* Add support for AtMode and and referencing appropriate screen text in testsResolver

* Remove console log

* Fix edge case issue when saving test result and add priority standardizing utility

* Add assertions checks for may assertions

* Update getMetrics utils

* Update test-import to account for single {at}-focused .collected file

* Address file error

* Fix tests

* Update import branch

* Fix edge case crash when viewing CandidateTestPlanRun

* Update tests and prepare support for testing v2 format test plan versions

* Update tests

* Update workflow

* Clarifying comment

* Update runtest.yml to exclude v2 test format import

* Update server/resolvers/TestPlanVersion/testsResolver.js

Remove unnecessary `at.settings` check

Co-authored-by: Stalgia Grigg <stalgia@bocoup.com>

* Clearer differences between v1 and v2 test format tests being imported

* Include typedef for RenderableContent

* Update thrown error message when missing commands.json (v2)

* Additional check for flattenObject

* Fix bug found after rebase and update jsdoc on 'retrieveAttributes'

* Updating Test Run page to support #743

* Update Test Run page instructions for v2 test format

* Add react-html-parser; update InstructionsRenderer

* Use testPlanVersion.metadata.testFormatVersion

* Stop defaulting testFormatVersion to 1

* Stop using NumberedList

* Update client/components/TestRenderer/utils.js

Co-authored-by: Stalgia Grigg <stalgia@bocoup.com>

* Define instructions variables

* Prepend tests with 'Test {number}' on reports page

* Address CG comments

* Address PR feedback

* Consistency

* Passed/failed for assertion results in TestRenderer

* Update test plan report conflict calculation to support pass/fail assertion results

* Correct assertion legend, margin right for checkbox

* Update inline snapshot for test queue

* Remove failedReason

* Update server/resolvers/TestResultOperations/saveTestResultCommon.js

Co-authored-by: Howard Edwards <howarde.edwards@gmail.com>

* Fix linting issue in saveTestResultCommon

* Undo change to checkAssertionResult

* Consistency

* Do radio group and supporting docs

* Revert "Do radio group and supporting docs"

This reverts commit 579f545.

* Revert aria-at import branch

---------

Co-authored-by: Paul Clue <67766160+Paul-Clue@users.noreply.github.com>
Co-authored-by: Stalgia Grigg <stalgia@bocoup.com>
@ccanash ccanash moved this from In sandbox to In production / Completed in Refactor ARIA-AT test structure and test result presentation project Dec 7, 2023
@ccanash
Copy link
Contributor

ccanash commented Dec 7, 2023

Hi @mcking65 is it ok if we close this?

stalgiag added a commit that referenced this issue Dec 14, 2023
commit fbf8eef
Author: Stalgia Grigg <stalgia@bocoup.com>
Date:   Thu Dec 14 14:46:48 2023 -0800

    Clean up leftover import

commit 15c594a
Merge: 9c99b4f 56205a3
Author: Stalgia Grigg <stalgia@bocoup.com>
Date:   Thu Dec 14 14:29:54 2023 -0800

    Merge branch 'main' into automation-mvp-merged

commit 9c99b4f
Author: Stalgia Grigg <stalgia@bocoup.com>
Date:   Thu Dec 14 14:20:15 2023 -0800

    Remove aria-at git commit cut off date

commit 56205a3
Author: Howard Edwards <howarde.edwards@gmail.com>
Date:   Thu Dec 14 11:44:34 2023 -0500

    Update tests (mock data and FilterButtons.test.jsx) and formatting of frontend components (#867)

commit d63609d
Author: Stalgia Grigg <stalgia@bocoup.com>
Date:   Wed Dec 13 16:16:26 2023 -0800

    Fixes after merge

commit 8ff6e6b
Merge: 2e62661 8215cb9
Author: Stalgia Grigg <stalgia@bocoup.com>
Date:   Wed Dec 13 15:03:10 2023 -0800

    Merge branch 'main' into automation-mvp-rebase-attempt

commit 8215cb9
Author: Stalgia Grigg <stalgia@bocoup.com>
Date:   Tue Dec 12 07:59:44 2023 -0800

    Trap backwards navigation from initial focus on header in focus-trapped modals (#851)

    * Add unit test for reverse navigation from initial focused header, FocusTrapper

    * Update unit tests for FocusTrapper, handle backwards navigation from initially focused header

    * Handle focus leaving non-focusable header element backwards in FocusTrapper

commit 918acdd
Author: Alexander Flenniken <alflennik@users.noreply.github.com>
Date:   Mon Dec 11 17:20:55 2023 -0500

    Allow candidate test plans to advance with override (#832)

    * Allow candidate test plans to advance

    * Implement PR feedback

commit 135e61b
Author: Alexander Flenniken <alflennik@users.noreply.github.com>
Date:   Mon Dec 11 16:59:00 2023 -0500

    Polish test review v2 page (#861)

    * Polish test review v2 page

    * Remove unused variable

commit 234a339
Author: Alexander Flenniken <alflennik@users.noreply.github.com>
Date:   Thu Dec 7 14:57:42 2023 -0500

    Update CONTRIBUTING.md (#859)

commit 460b29c
Author: alflennik <alflennik@users.noreply.github.com>
Date:   Thu Dec 7 13:26:17 2023 -0500

    Update contributing.md

commit 457caea
Author: Howard Edwards <howarde.edwards@gmail.com>
Date:   Tue Dec 5 16:04:23 2023 -0500

    Check for v2 testFormatVersion where applicable and hide link (#855)

commit 6962cef
Author: Howard Edwards <howarde.edwards@gmail.com>
Date:   Tue Dec 5 15:36:41 2023 -0500

    Support v2 Test Format in import script and page changes (#840)

    * Update client/resources

    * Update import script to support v2 test format import

    * Update import script to include flattened commandsV2.json and add uniquely generated scenario and assertion IDs

    * Updating client and server files to support V2 test format

    * Remove console.log and add TODO

    * Include tokenized assertionStatement in TestPlanVersion.tests[x].assertions

    * Add retrieveCommands utility

    * Update candidate review and datamanagement page

    * Update Reports/queries to also include mayOptionalAssertionResults

    * Add support for AtMode and and referencing appropriate screen text in testsResolver

    * Remove console log

    * Fix edge case issue when saving test result and add priority standardizing utility

    * Add assertions checks for may assertions

    * Update getMetrics utils

    * Update test-import to account for single {at}-focused .collected file

    * Address file error

    * Fix tests

    * Update import branch

    * Fix edge case crash when viewing CandidateTestPlanRun

    * Update tests and prepare support for testing v2 format test plan versions

    * Update tests

    * Update workflow

    * Clarifying comment

    * Update runtest.yml to exclude v2 test format import

    * Update server/resolvers/TestPlanVersion/testsResolver.js

    Remove unnecessary `at.settings` check

    Co-authored-by: Stalgia Grigg <stalgia@bocoup.com>

    * Clearer differences between v1 and v2 test format tests being imported

    * Include typedef for RenderableContent

    * Update thrown error message when missing commands.json (v2)

    * Additional check for flattenObject

    * Fix bug found after rebase and update jsdoc on 'retrieveAttributes'

    * Updating Test Run page to support #743

    * Update Test Run page instructions for v2 test format

    * Add react-html-parser; update InstructionsRenderer

    * Use testPlanVersion.metadata.testFormatVersion

    * Stop defaulting testFormatVersion to 1

    * Stop using NumberedList

    * Update client/components/TestRenderer/utils.js

    Co-authored-by: Stalgia Grigg <stalgia@bocoup.com>

    * Define instructions variables

    * Prepend tests with 'Test {number}' on reports page

    * Address CG comments

    * Address PR feedback

    * Consistency

    * Passed/failed for assertion results in TestRenderer

    * Update test plan report conflict calculation to support pass/fail assertion results

    * Correct assertion legend, margin right for checkbox

    * Update inline snapshot for test queue

    * Remove failedReason

    * Update server/resolvers/TestResultOperations/saveTestResultCommon.js

    Co-authored-by: Howard Edwards <howarde.edwards@gmail.com>

    * Fix linting issue in saveTestResultCommon

    * Undo change to checkAssertionResult

    * Consistency

    * Do radio group and supporting docs

    * Revert "Do radio group and supporting docs"

    This reverts commit 579f545.

    * Revert aria-at import branch

    ---------

    Co-authored-by: Paul Clue <67766160+Paul-Clue@users.noreply.github.com>
    Co-authored-by: Stalgia Grigg <stalgia@bocoup.com>

commit 703b01f
Author: jugglinmike <mike@mikepennisi.com>
Date:   Tue Dec 5 12:25:21 2023 -0500

    Do not render zeros in place of omitted content (#852)

    By relying on the "truthiness" of Number values to short-circuit boolean
    expressions, some rendering logic produced the text "0" instead of
    omitting any markup at all.

    Replace the expressions whose value could be coerced to booleans with
    expressions which directly evaluate to "fallback" values that do not
    produce markup (i.e. `false` or an empty array).

commit 7783342
Author: Stalgia Grigg <stalgia@bocoup.com>
Date:   Mon Dec 4 15:34:10 2023 -0800

    Revert "Handle focus leaving non-focusable header element backwards in FocusTrapper"

    This reverts commit 57012cb.

commit 57012cb
Author: Stalgia Grigg <stalgia@bocoup.com>
Date:   Mon Dec 4 14:05:57 2023 -0800

    Handle focus leaving non-focusable header element backwards in FocusTrapper

commit 477abcb
Author: Howard Edwards <howarde.edwards@gmail.com>
Date:   Wed Nov 29 10:10:10 2023 -0500

    Remove additional css meant for now removed 'skipped' view (#848)

commit dba7f40
Author: Alexander Flenniken <alflennik@users.noreply.github.com>
Date:   Tue Nov 28 14:37:24 2023 -0500

    Implement Test Review in React (#846)

    * Implement client-side version of test-review

    * PR polish

    * Add additional specification features

    * Additional minor polish

    * PR feedback

commit b98852b
Author: Paul Clue <67766160+Paul-Clue@users.noreply.github.com>
Date:   Wed Nov 22 15:48:57 2023 -0500

    No longer mark test plan reports with skipped tests as final (#833)

    * Hide mark as final button

    * No mark as final for skipped tests in graphql

    * Remove skipped tests from the UI

    * Add migration for unfinished tests

    * Simplify necessary files

    * Fix failing tests

    * Add code review fixes

    * Remove no-longer-necessary style class

    * Remove reference

commit c29bea9
Merge: c101ecc c1bf56a
Author: Stalgia Grigg <stalgia@bocoup.com>
Date:   Tue Nov 21 09:41:04 2023 -0800

    Merge pull request #844 from w3c/fix-at-selection-dropdown

    Fix Data management page crash when Adding Test Plans to the Test Queue and Opening Assistive Technology select to pick any option

commit c101ecc
Merge: 878fe80 50cd6c8
Author: Stalgia Grigg <stalgia@bocoup.com>
Date:   Tue Nov 21 09:40:43 2023 -0800

    Merge pull request #837 from w3c/accessible-name-test-plan-report-status-dialog

    Fix aria-labeledby for BasicModal/BasicThemedModal

commit 878fe80
Author: Paul Clue <67766160+Paul-Clue@users.noreply.github.com>
Date:   Mon Nov 20 15:08:58 2023 -0500

    Fix empty version strings on Data Management and Test Queue pages (#839)

    * Add versionString to mutation

    * Remove comments

    * Fix similar issue happening on the Test Queue page when assigning and removing testers and also removing test results

    ---------

    Co-authored-by: Howard Edwards <howarde.edwards@gmail.com>

commit a5f0e80
Author: Alexander Flenniken <alflennik@users.noreply.github.com>
Date:   Mon Nov 20 15:01:45 2023 -0500

    Implement target date modal improvements (#835)

    * Implement target date modal improvements

    * Add esc support to target date modal

commit c1bf56a
Author: Stalgia Grigg <stalgia@bocoup.com>
Date:   Thu Nov 16 11:29:56 2023 -0800

    Update DataManagement test mock

commit 9319e37
Author: Stalgia Grigg <stalgia@bocoup.com>
Date:   Thu Nov 16 11:16:37 2023 -0800

    Remove browsers from query and state in TestQueue and DataManagement in favor of at.browsers

commit 7dd31e1
Author: Stalgia Grigg <stalgia@bocoup.com>
Date:   Thu Nov 16 11:07:46 2023 -0800

    Fix bug that causes crash on DataManagement page ManageTestQueue AT dropdown

commit 50cd6c8
Author: Stalgia Grigg <stalgia@bocoup.com>
Date:   Wed Nov 8 16:01:23 2023 -0800

    Unit test for aria-labeledby on BasicModal

commit c1e7f0f
Author: Stalgia Grigg <stalgia@bocoup.com>
Date:   Wed Nov 8 15:42:55 2023 -0800

    Fix aria-labeledby in both BasicModal and BasicThemedModal
Paul-Clue added a commit that referenced this issue Feb 5, 2024
* Update client/resources

* Update import script to support v2 test format import

* Update import script to include flattened commandsV2.json and add uniquely generated scenario and assertion IDs

* Updating client and server files to support V2 test format

* Remove console.log and add TODO

* Include tokenized assertionStatement in TestPlanVersion.tests[x].assertions

* Add retrieveCommands utility

* Update candidate review and datamanagement page

* Update Reports/queries to also include mayOptionalAssertionResults

* Add support for AtMode and and referencing appropriate screen text in testsResolver

* Remove console log

* Fix edge case issue when saving test result and add priority standardizing utility

* Add assertions checks for may assertions

* Update getMetrics utils

* Update test-import to account for single {at}-focused .collected file

* Address file error

* Fix tests

* Update import branch

* Fix edge case crash when viewing CandidateTestPlanRun

* Update tests and prepare support for testing v2 format test plan versions

* Update tests

* Update workflow

* Clarifying comment

* Update runtest.yml to exclude v2 test format import

* Update server/resolvers/TestPlanVersion/testsResolver.js

Remove unnecessary `at.settings` check

Co-authored-by: Stalgia Grigg <stalgia@bocoup.com>

* Clearer differences between v1 and v2 test format tests being imported

* Include typedef for RenderableContent

* Update thrown error message when missing commands.json (v2)

* Additional check for flattenObject

* Fix bug found after rebase and update jsdoc on 'retrieveAttributes'

* Updating Test Run page to support #743

* Update Test Run page instructions for v2 test format

* Add react-html-parser; update InstructionsRenderer

* Use testPlanVersion.metadata.testFormatVersion

* Stop defaulting testFormatVersion to 1

* Stop using NumberedList

* Update client/components/TestRenderer/utils.js

Co-authored-by: Stalgia Grigg <stalgia@bocoup.com>

* Define instructions variables

* Prepend tests with 'Test {number}' on reports page

* Address CG comments

* Address PR feedback

* Consistency

* Passed/failed for assertion results in TestRenderer

* Update test plan report conflict calculation to support pass/fail assertion results

* Correct assertion legend, margin right for checkbox

* Update inline snapshot for test queue

* Remove failedReason

* Update server/resolvers/TestResultOperations/saveTestResultCommon.js

Co-authored-by: Howard Edwards <howarde.edwards@gmail.com>

* Fix linting issue in saveTestResultCommon

* Undo change to checkAssertionResult

* Consistency

* Do radio group and supporting docs

* Revert "Do radio group and supporting docs"

This reverts commit 579f545.

* Re-commit missing update

* Commit to new branch

* Add conditional for test data

* Push to check branch

* Get page to render for v2

* Fix loop

* Fix changes

* Remove TODO items to make test plan review page available after merging in main

* Add support for aria links on review page

* Update single page view grouping under title for v2 format

* Fix styling

* Update InstructionsRenderer to also include settings specific instructions for v2 test plan version

* Include upcoming TODOs

* Include upcoming TODOs

* Add formatting change

* remove comment

* Remove comments

---------

Co-authored-by: Howard Edwards <howarde.edwards@gmail.com>
Co-authored-by: Stalgia Grigg <stalgia@bocoup.com>
alflennik pushed a commit that referenced this issue Feb 22, 2024
…ertion failures (#849)

* Update client/resources

* Update import script to support v2 test format import

* Update import script to include flattened commandsV2.json and add uniquely generated scenario and assertion IDs

* Updating client and server files to support V2 test format

* Remove console.log and add TODO

* Include tokenized assertionStatement in TestPlanVersion.tests[x].assertions

* Add retrieveCommands utility

* Update candidate review and datamanagement page

* Update Reports/queries to also include mayOptionalAssertionResults

* Add support for AtMode and and referencing appropriate screen text in testsResolver

* Remove console log

* Fix edge case issue when saving test result and add priority standardizing utility

* Add assertions checks for may assertions

* Update getMetrics utils

* Update test-import to account for single {at}-focused .collected file

* Address file error

* Fix tests

* Update import branch

* Fix edge case crash when viewing CandidateTestPlanRun

* Update tests and prepare support for testing v2 format test plan versions

* Update tests

* Update workflow

* Clarifying comment

* Update runtest.yml to exclude v2 test format import

* Update server/resolvers/TestPlanVersion/testsResolver.js

Remove unnecessary `at.settings` check

Co-authored-by: Stalgia Grigg <stalgia@bocoup.com>

* Clearer differences between v1 and v2 test format tests being imported

* Include typedef for RenderableContent

* Update thrown error message when missing commands.json (v2)

* Additional check for flattenObject

* Fix bug found after rebase and update jsdoc on 'retrieveAttributes'

* Updating Test Run page to support #743

* Update Test Run page instructions for v2 test format

* Add react-html-parser; update InstructionsRenderer

* Use testPlanVersion.metadata.testFormatVersion

* Stop defaulting testFormatVersion to 1

* Stop using NumberedList

* Update client/components/TestRenderer/utils.js

Co-authored-by: Stalgia Grigg <stalgia@bocoup.com>

* Define instructions variables

* Prepend tests with 'Test {number}' on reports page

* Address CG comments

* Address PR feedback

* Consistency

* Passed/failed for assertion results in TestRenderer

* Update test plan report conflict calculation to support pass/fail assertion results

* Correct assertion legend, margin right for checkbox

* Update inline snapshot for test queue

* Remove failedReason

* Update server/resolvers/TestResultOperations/saveTestResultCommon.js

Co-authored-by: Howard Edwards <howarde.edwards@gmail.com>

* Fix linting issue in saveTestResultCommon

* Undo change to checkAssertionResult

* Consistency

* Update client/resources

* Add support for severity capture and capturing details for all unexpected behaviors

* Update calculations to factor in moderate and high severity impact assertions

* Fix failing tests

* Update collection form to stop using nested fieldsets

* Remove includeImpactCalculation param from calculateAssertionsCount func

* Update aria-at-harness.mjs with PR feedback

* Avoid setting class to the text 'false'

* Update harness resources with severity -> impact variable name change

* Update instances of severity to impact for variable names/usage

* Address PR feedback, where checked unexpected behaviors are being accidentally trapped after initially indicating "Yes" for unexpected behaviors and have now changed to "No"

* Add missing failure view for MAY HAVE behaviors on client

* Ignore unexpectedBehavior.details during conflict comparison

* Update graphql-schema.js description for UnexpectedBehavior.impact

* Update TestRenderer comment

* Remove otherUnexpectedBehaviorText instead of deprecating

* Use lodash's omit

* Update deleteReport test

* Add migration for updating metrics after including unexpected behavior counts

* Update calculation related names

* Remove testsCount update

---------

Co-authored-by: Paul Clue <67766160+Paul-Clue@users.noreply.github.com>
Co-authored-by: Stalgia Grigg <stalgia@bocoup.com>
@mcking65
Copy link
Author

mcking65 commented Mar 6, 2024

This is complete and shipped to prod, closing.

@mcking65 mcking65 closed this as completed Mar 6, 2024
@mcking65 mcking65 moved this from In Progress to In production / Completed in Refactor ARIA-AT test structure and test result presentation project Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Development

No branches or pull requests

2 participants