-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support V2 test format when manually running a test #743
Labels
enhancement
New feature or request
Comments
mcking65
moved this to Ready for sizing
in Refactor ARIA-AT test structure and test result presentation project
Aug 15, 2023
ccanash
moved this from Ready for sizing
to Todo
in Refactor ARIA-AT test structure and test result presentation project
Nov 9, 2023
howard-e
added a commit
that referenced
this issue
Nov 14, 2023
howard-e
added a commit
that referenced
this issue
Nov 14, 2023
ccanash
moved this from Todo
to In Progress
in Refactor ARIA-AT test structure and test result presentation project
Nov 15, 2023
howard-e
added a commit
that referenced
this issue
Nov 15, 2023
howard-e
added a commit
that referenced
this issue
Nov 16, 2023
howard-e
added a commit
that referenced
this issue
Nov 16, 2023
howard-e
added a commit
that referenced
this issue
Nov 16, 2023
howard-e
added a commit
that referenced
this issue
Nov 16, 2023
howard-e
added a commit
that referenced
this issue
Nov 16, 2023
howard-e
added a commit
that referenced
this issue
Nov 16, 2023
howard-e
added a commit
that referenced
this issue
Nov 20, 2023
howard-e
added a commit
that referenced
this issue
Nov 27, 2023
howard-e
added a commit
that referenced
this issue
Nov 27, 2023
ccanash
moved this from In Progress
to In sandbox
in Refactor ARIA-AT test structure and test result presentation project
Dec 5, 2023
howard-e
added a commit
that referenced
this issue
Dec 5, 2023
* Update client/resources * Update import script to support v2 test format import * Update import script to include flattened commandsV2.json and add uniquely generated scenario and assertion IDs * Updating client and server files to support V2 test format * Remove console.log and add TODO * Include tokenized assertionStatement in TestPlanVersion.tests[x].assertions * Add retrieveCommands utility * Update candidate review and datamanagement page * Update Reports/queries to also include mayOptionalAssertionResults * Add support for AtMode and and referencing appropriate screen text in testsResolver * Remove console log * Fix edge case issue when saving test result and add priority standardizing utility * Add assertions checks for may assertions * Update getMetrics utils * Update test-import to account for single {at}-focused .collected file * Address file error * Fix tests * Update import branch * Fix edge case crash when viewing CandidateTestPlanRun * Update tests and prepare support for testing v2 format test plan versions * Update tests * Update workflow * Clarifying comment * Update runtest.yml to exclude v2 test format import * Update server/resolvers/TestPlanVersion/testsResolver.js Remove unnecessary `at.settings` check Co-authored-by: Stalgia Grigg <stalgia@bocoup.com> * Clearer differences between v1 and v2 test format tests being imported * Include typedef for RenderableContent * Update thrown error message when missing commands.json (v2) * Additional check for flattenObject * Fix bug found after rebase and update jsdoc on 'retrieveAttributes' * Updating Test Run page to support #743 * Update Test Run page instructions for v2 test format * Add react-html-parser; update InstructionsRenderer * Use testPlanVersion.metadata.testFormatVersion * Stop defaulting testFormatVersion to 1 * Stop using NumberedList * Update client/components/TestRenderer/utils.js Co-authored-by: Stalgia Grigg <stalgia@bocoup.com> * Define instructions variables * Prepend tests with 'Test {number}' on reports page * Address CG comments * Address PR feedback * Consistency * Passed/failed for assertion results in TestRenderer * Update test plan report conflict calculation to support pass/fail assertion results * Correct assertion legend, margin right for checkbox * Update inline snapshot for test queue * Remove failedReason * Update server/resolvers/TestResultOperations/saveTestResultCommon.js Co-authored-by: Howard Edwards <howarde.edwards@gmail.com> * Fix linting issue in saveTestResultCommon * Undo change to checkAssertionResult * Consistency * Do radio group and supporting docs * Revert "Do radio group and supporting docs" This reverts commit 579f545. * Revert aria-at import branch --------- Co-authored-by: Paul Clue <67766160+Paul-Clue@users.noreply.github.com> Co-authored-by: Stalgia Grigg <stalgia@bocoup.com>
ccanash
moved this from In sandbox
to In production / Completed
in Refactor ARIA-AT test structure and test result presentation project
Dec 7, 2023
Hi @mcking65 is it ok if we close this? |
stalgiag
added a commit
that referenced
this issue
Dec 14, 2023
commit fbf8eef Author: Stalgia Grigg <stalgia@bocoup.com> Date: Thu Dec 14 14:46:48 2023 -0800 Clean up leftover import commit 15c594a Merge: 9c99b4f 56205a3 Author: Stalgia Grigg <stalgia@bocoup.com> Date: Thu Dec 14 14:29:54 2023 -0800 Merge branch 'main' into automation-mvp-merged commit 9c99b4f Author: Stalgia Grigg <stalgia@bocoup.com> Date: Thu Dec 14 14:20:15 2023 -0800 Remove aria-at git commit cut off date commit 56205a3 Author: Howard Edwards <howarde.edwards@gmail.com> Date: Thu Dec 14 11:44:34 2023 -0500 Update tests (mock data and FilterButtons.test.jsx) and formatting of frontend components (#867) commit d63609d Author: Stalgia Grigg <stalgia@bocoup.com> Date: Wed Dec 13 16:16:26 2023 -0800 Fixes after merge commit 8ff6e6b Merge: 2e62661 8215cb9 Author: Stalgia Grigg <stalgia@bocoup.com> Date: Wed Dec 13 15:03:10 2023 -0800 Merge branch 'main' into automation-mvp-rebase-attempt commit 8215cb9 Author: Stalgia Grigg <stalgia@bocoup.com> Date: Tue Dec 12 07:59:44 2023 -0800 Trap backwards navigation from initial focus on header in focus-trapped modals (#851) * Add unit test for reverse navigation from initial focused header, FocusTrapper * Update unit tests for FocusTrapper, handle backwards navigation from initially focused header * Handle focus leaving non-focusable header element backwards in FocusTrapper commit 918acdd Author: Alexander Flenniken <alflennik@users.noreply.github.com> Date: Mon Dec 11 17:20:55 2023 -0500 Allow candidate test plans to advance with override (#832) * Allow candidate test plans to advance * Implement PR feedback commit 135e61b Author: Alexander Flenniken <alflennik@users.noreply.github.com> Date: Mon Dec 11 16:59:00 2023 -0500 Polish test review v2 page (#861) * Polish test review v2 page * Remove unused variable commit 234a339 Author: Alexander Flenniken <alflennik@users.noreply.github.com> Date: Thu Dec 7 14:57:42 2023 -0500 Update CONTRIBUTING.md (#859) commit 460b29c Author: alflennik <alflennik@users.noreply.github.com> Date: Thu Dec 7 13:26:17 2023 -0500 Update contributing.md commit 457caea Author: Howard Edwards <howarde.edwards@gmail.com> Date: Tue Dec 5 16:04:23 2023 -0500 Check for v2 testFormatVersion where applicable and hide link (#855) commit 6962cef Author: Howard Edwards <howarde.edwards@gmail.com> Date: Tue Dec 5 15:36:41 2023 -0500 Support v2 Test Format in import script and page changes (#840) * Update client/resources * Update import script to support v2 test format import * Update import script to include flattened commandsV2.json and add uniquely generated scenario and assertion IDs * Updating client and server files to support V2 test format * Remove console.log and add TODO * Include tokenized assertionStatement in TestPlanVersion.tests[x].assertions * Add retrieveCommands utility * Update candidate review and datamanagement page * Update Reports/queries to also include mayOptionalAssertionResults * Add support for AtMode and and referencing appropriate screen text in testsResolver * Remove console log * Fix edge case issue when saving test result and add priority standardizing utility * Add assertions checks for may assertions * Update getMetrics utils * Update test-import to account for single {at}-focused .collected file * Address file error * Fix tests * Update import branch * Fix edge case crash when viewing CandidateTestPlanRun * Update tests and prepare support for testing v2 format test plan versions * Update tests * Update workflow * Clarifying comment * Update runtest.yml to exclude v2 test format import * Update server/resolvers/TestPlanVersion/testsResolver.js Remove unnecessary `at.settings` check Co-authored-by: Stalgia Grigg <stalgia@bocoup.com> * Clearer differences between v1 and v2 test format tests being imported * Include typedef for RenderableContent * Update thrown error message when missing commands.json (v2) * Additional check for flattenObject * Fix bug found after rebase and update jsdoc on 'retrieveAttributes' * Updating Test Run page to support #743 * Update Test Run page instructions for v2 test format * Add react-html-parser; update InstructionsRenderer * Use testPlanVersion.metadata.testFormatVersion * Stop defaulting testFormatVersion to 1 * Stop using NumberedList * Update client/components/TestRenderer/utils.js Co-authored-by: Stalgia Grigg <stalgia@bocoup.com> * Define instructions variables * Prepend tests with 'Test {number}' on reports page * Address CG comments * Address PR feedback * Consistency * Passed/failed for assertion results in TestRenderer * Update test plan report conflict calculation to support pass/fail assertion results * Correct assertion legend, margin right for checkbox * Update inline snapshot for test queue * Remove failedReason * Update server/resolvers/TestResultOperations/saveTestResultCommon.js Co-authored-by: Howard Edwards <howarde.edwards@gmail.com> * Fix linting issue in saveTestResultCommon * Undo change to checkAssertionResult * Consistency * Do radio group and supporting docs * Revert "Do radio group and supporting docs" This reverts commit 579f545. * Revert aria-at import branch --------- Co-authored-by: Paul Clue <67766160+Paul-Clue@users.noreply.github.com> Co-authored-by: Stalgia Grigg <stalgia@bocoup.com> commit 703b01f Author: jugglinmike <mike@mikepennisi.com> Date: Tue Dec 5 12:25:21 2023 -0500 Do not render zeros in place of omitted content (#852) By relying on the "truthiness" of Number values to short-circuit boolean expressions, some rendering logic produced the text "0" instead of omitting any markup at all. Replace the expressions whose value could be coerced to booleans with expressions which directly evaluate to "fallback" values that do not produce markup (i.e. `false` or an empty array). commit 7783342 Author: Stalgia Grigg <stalgia@bocoup.com> Date: Mon Dec 4 15:34:10 2023 -0800 Revert "Handle focus leaving non-focusable header element backwards in FocusTrapper" This reverts commit 57012cb. commit 57012cb Author: Stalgia Grigg <stalgia@bocoup.com> Date: Mon Dec 4 14:05:57 2023 -0800 Handle focus leaving non-focusable header element backwards in FocusTrapper commit 477abcb Author: Howard Edwards <howarde.edwards@gmail.com> Date: Wed Nov 29 10:10:10 2023 -0500 Remove additional css meant for now removed 'skipped' view (#848) commit dba7f40 Author: Alexander Flenniken <alflennik@users.noreply.github.com> Date: Tue Nov 28 14:37:24 2023 -0500 Implement Test Review in React (#846) * Implement client-side version of test-review * PR polish * Add additional specification features * Additional minor polish * PR feedback commit b98852b Author: Paul Clue <67766160+Paul-Clue@users.noreply.github.com> Date: Wed Nov 22 15:48:57 2023 -0500 No longer mark test plan reports with skipped tests as final (#833) * Hide mark as final button * No mark as final for skipped tests in graphql * Remove skipped tests from the UI * Add migration for unfinished tests * Simplify necessary files * Fix failing tests * Add code review fixes * Remove no-longer-necessary style class * Remove reference commit c29bea9 Merge: c101ecc c1bf56a Author: Stalgia Grigg <stalgia@bocoup.com> Date: Tue Nov 21 09:41:04 2023 -0800 Merge pull request #844 from w3c/fix-at-selection-dropdown Fix Data management page crash when Adding Test Plans to the Test Queue and Opening Assistive Technology select to pick any option commit c101ecc Merge: 878fe80 50cd6c8 Author: Stalgia Grigg <stalgia@bocoup.com> Date: Tue Nov 21 09:40:43 2023 -0800 Merge pull request #837 from w3c/accessible-name-test-plan-report-status-dialog Fix aria-labeledby for BasicModal/BasicThemedModal commit 878fe80 Author: Paul Clue <67766160+Paul-Clue@users.noreply.github.com> Date: Mon Nov 20 15:08:58 2023 -0500 Fix empty version strings on Data Management and Test Queue pages (#839) * Add versionString to mutation * Remove comments * Fix similar issue happening on the Test Queue page when assigning and removing testers and also removing test results --------- Co-authored-by: Howard Edwards <howarde.edwards@gmail.com> commit a5f0e80 Author: Alexander Flenniken <alflennik@users.noreply.github.com> Date: Mon Nov 20 15:01:45 2023 -0500 Implement target date modal improvements (#835) * Implement target date modal improvements * Add esc support to target date modal commit c1bf56a Author: Stalgia Grigg <stalgia@bocoup.com> Date: Thu Nov 16 11:29:56 2023 -0800 Update DataManagement test mock commit 9319e37 Author: Stalgia Grigg <stalgia@bocoup.com> Date: Thu Nov 16 11:16:37 2023 -0800 Remove browsers from query and state in TestQueue and DataManagement in favor of at.browsers commit 7dd31e1 Author: Stalgia Grigg <stalgia@bocoup.com> Date: Thu Nov 16 11:07:46 2023 -0800 Fix bug that causes crash on DataManagement page ManageTestQueue AT dropdown commit 50cd6c8 Author: Stalgia Grigg <stalgia@bocoup.com> Date: Wed Nov 8 16:01:23 2023 -0800 Unit test for aria-labeledby on BasicModal commit c1e7f0f Author: Stalgia Grigg <stalgia@bocoup.com> Date: Wed Nov 8 15:42:55 2023 -0800 Fix aria-labeledby in both BasicModal and BasicThemedModal
Paul-Clue
added a commit
that referenced
this issue
Feb 5, 2024
* Update client/resources * Update import script to support v2 test format import * Update import script to include flattened commandsV2.json and add uniquely generated scenario and assertion IDs * Updating client and server files to support V2 test format * Remove console.log and add TODO * Include tokenized assertionStatement in TestPlanVersion.tests[x].assertions * Add retrieveCommands utility * Update candidate review and datamanagement page * Update Reports/queries to also include mayOptionalAssertionResults * Add support for AtMode and and referencing appropriate screen text in testsResolver * Remove console log * Fix edge case issue when saving test result and add priority standardizing utility * Add assertions checks for may assertions * Update getMetrics utils * Update test-import to account for single {at}-focused .collected file * Address file error * Fix tests * Update import branch * Fix edge case crash when viewing CandidateTestPlanRun * Update tests and prepare support for testing v2 format test plan versions * Update tests * Update workflow * Clarifying comment * Update runtest.yml to exclude v2 test format import * Update server/resolvers/TestPlanVersion/testsResolver.js Remove unnecessary `at.settings` check Co-authored-by: Stalgia Grigg <stalgia@bocoup.com> * Clearer differences between v1 and v2 test format tests being imported * Include typedef for RenderableContent * Update thrown error message when missing commands.json (v2) * Additional check for flattenObject * Fix bug found after rebase and update jsdoc on 'retrieveAttributes' * Updating Test Run page to support #743 * Update Test Run page instructions for v2 test format * Add react-html-parser; update InstructionsRenderer * Use testPlanVersion.metadata.testFormatVersion * Stop defaulting testFormatVersion to 1 * Stop using NumberedList * Update client/components/TestRenderer/utils.js Co-authored-by: Stalgia Grigg <stalgia@bocoup.com> * Define instructions variables * Prepend tests with 'Test {number}' on reports page * Address CG comments * Address PR feedback * Consistency * Passed/failed for assertion results in TestRenderer * Update test plan report conflict calculation to support pass/fail assertion results * Correct assertion legend, margin right for checkbox * Update inline snapshot for test queue * Remove failedReason * Update server/resolvers/TestResultOperations/saveTestResultCommon.js Co-authored-by: Howard Edwards <howarde.edwards@gmail.com> * Fix linting issue in saveTestResultCommon * Undo change to checkAssertionResult * Consistency * Do radio group and supporting docs * Revert "Do radio group and supporting docs" This reverts commit 579f545. * Re-commit missing update * Commit to new branch * Add conditional for test data * Push to check branch * Get page to render for v2 * Fix loop * Fix changes * Remove TODO items to make test plan review page available after merging in main * Add support for aria links on review page * Update single page view grouping under title for v2 format * Fix styling * Update InstructionsRenderer to also include settings specific instructions for v2 test plan version * Include upcoming TODOs * Include upcoming TODOs * Add formatting change * remove comment * Remove comments --------- Co-authored-by: Howard Edwards <howarde.edwards@gmail.com> Co-authored-by: Stalgia Grigg <stalgia@bocoup.com>
alflennik
pushed a commit
that referenced
this issue
Feb 22, 2024
…ertion failures (#849) * Update client/resources * Update import script to support v2 test format import * Update import script to include flattened commandsV2.json and add uniquely generated scenario and assertion IDs * Updating client and server files to support V2 test format * Remove console.log and add TODO * Include tokenized assertionStatement in TestPlanVersion.tests[x].assertions * Add retrieveCommands utility * Update candidate review and datamanagement page * Update Reports/queries to also include mayOptionalAssertionResults * Add support for AtMode and and referencing appropriate screen text in testsResolver * Remove console log * Fix edge case issue when saving test result and add priority standardizing utility * Add assertions checks for may assertions * Update getMetrics utils * Update test-import to account for single {at}-focused .collected file * Address file error * Fix tests * Update import branch * Fix edge case crash when viewing CandidateTestPlanRun * Update tests and prepare support for testing v2 format test plan versions * Update tests * Update workflow * Clarifying comment * Update runtest.yml to exclude v2 test format import * Update server/resolvers/TestPlanVersion/testsResolver.js Remove unnecessary `at.settings` check Co-authored-by: Stalgia Grigg <stalgia@bocoup.com> * Clearer differences between v1 and v2 test format tests being imported * Include typedef for RenderableContent * Update thrown error message when missing commands.json (v2) * Additional check for flattenObject * Fix bug found after rebase and update jsdoc on 'retrieveAttributes' * Updating Test Run page to support #743 * Update Test Run page instructions for v2 test format * Add react-html-parser; update InstructionsRenderer * Use testPlanVersion.metadata.testFormatVersion * Stop defaulting testFormatVersion to 1 * Stop using NumberedList * Update client/components/TestRenderer/utils.js Co-authored-by: Stalgia Grigg <stalgia@bocoup.com> * Define instructions variables * Prepend tests with 'Test {number}' on reports page * Address CG comments * Address PR feedback * Consistency * Passed/failed for assertion results in TestRenderer * Update test plan report conflict calculation to support pass/fail assertion results * Correct assertion legend, margin right for checkbox * Update inline snapshot for test queue * Remove failedReason * Update server/resolvers/TestResultOperations/saveTestResultCommon.js Co-authored-by: Howard Edwards <howarde.edwards@gmail.com> * Fix linting issue in saveTestResultCommon * Undo change to checkAssertionResult * Consistency * Update client/resources * Add support for severity capture and capturing details for all unexpected behaviors * Update calculations to factor in moderate and high severity impact assertions * Fix failing tests * Update collection form to stop using nested fieldsets * Remove includeImpactCalculation param from calculateAssertionsCount func * Update aria-at-harness.mjs with PR feedback * Avoid setting class to the text 'false' * Update harness resources with severity -> impact variable name change * Update instances of severity to impact for variable names/usage * Address PR feedback, where checked unexpected behaviors are being accidentally trapped after initially indicating "Yes" for unexpected behaviors and have now changed to "No" * Add missing failure view for MAY HAVE behaviors on client * Ignore unexpectedBehavior.details during conflict comparison * Update graphql-schema.js description for UnexpectedBehavior.impact * Update TestRenderer comment * Remove otherUnexpectedBehaviorText instead of deprecating * Use lodash's omit * Update deleteReport test * Add migration for updating metrics after including unexpected behavior counts * Update calculation related names * Remove testsCount update --------- Co-authored-by: Paul Clue <67766160+Paul-Clue@users.noreply.github.com> Co-authored-by: Stalgia Grigg <stalgia@bocoup.com>
This is complete and shipped to prod, closing. |
github-project-automation
bot
moved this from In production / Completed
to In Progress
in Refactor ARIA-AT test structure and test result presentation project
Mar 6, 2024
mcking65
moved this from In Progress
to In production / Completed
in Refactor ARIA-AT test structure and test result presentation project
Mar 6, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Overview
Change the content shown in the test runner when a test plan written in the V2 format is opened.
Note:
In general, the changes needed in the runner are exact reflections of a subset of changes specified for the test plan preview generated during test plan build. Those changes are specified in w3c/aria-at#977.
Changes to display of a test in the runner
Change heading 1
Streamline wording of the first level 1 heading. It is currently:
Replace the string "Testing task:1." with the string "Test 1:"
Note that the V2 format contains a column for
presentationNumber
in tests.csv. That number specifies the sequence of test presentation.Remove redundant heading 1 and subsequent paragraph
Following the AT and browser version information, there is another
h1
followed by a paragraph that describes the task. Remove theh1
and the paragraph. Both are unnecessary.Example:
Change instructions
Change the text of the level 2 heading from "Test Instructions" to "Instructions". Implement the instructions format as described in w3c/aria-at#977. Be sure to include the instructions for command settings where appropriate.
Remove success criteria
Remove the level 2 heading with text "Success Criteria" and the paragraph and list that follow it.
Example:
Command headings
For each command there is a level 3 and level 4 heading. For example:
The commands in the headings need to be converted to command-specification strings as described in w3c/aria-at#977 with the command followed by settings in parentheses. For example:
The text was updated successfully, but these errors were encountered: