Update `hashTests` function to support the migration of test results across test plan versions using the v2 test format #880

howard-e · 2023-12-18T14:57:46Z

This PR updates aria.js to account for properties to be ignored during a v2 test format version update so applicable results can be transferred over during phase update process.

This should address the scenario described in #877 and produce the expected results along with the additional requirements:

The procedure that manifest this bug was:

Command button V23.12.06 was in draft review and had three test runs in the test queue.

Results were present for all the runs.

We discovered that a change was needed to the settings specified for two VoiceOver commands.

We made V23.12.13 of the plan with the changes to the VoiceOver commands and merged it to master so it showed on the data management page as R&D Complete.

We advanced V23.12.13 of the command button test plan to draft review.

Expected results:

Since V23.12.06 is deprecated, its runs no longer show in the test queue.

To avoid unnecessary loss of manual test work, equivalent runs of V23.12.13 are added to the test queue, preserving as many test results as possible.

Actual results:

Runs of V23.12.06, which became deprecated, remain in the test queue.

Runs of V23.12.13 that we subsequently added to the test queue were empty, so testers have to start over even for tests that were not changed.

alflennik

I am approving! There's a couple of minor comments but it all feels optional. I verified the tests are passing (big thank you for buffing up the test coverage), and also I rebuilt the app from scratch to verify that it was still able to compare test versions without introducing duplicates.

server/util/aria.test.js

server/util/aria.js

server/util/aria.test.js

…st format version update so applicable results can be transferred over during phase update

…ts and runs

* Rename testWithNoIds -> testWithOmittedAttributes

…ionally, now deleting found duplicates as there is an error when tested against prod data where unnecessary duplicates were created

howard-e · 2024-01-22T18:16:40Z

server/migrations/utils/index.js

+                    );
+                } else {
+                    // Remove any accidentally created TestPlanVersion rows created prior to migration
+                    await queryInterface.sequelize.query(


For reviewers, found an issue during checks against production where several TestPlanVersions were created as duplicates (yet had non-matching hashedTests, because of some failed sync between pushes I suppose).

Also broke this out into a separate utility because several migrations may have to make use this in the future as well. Immediately seeing a need for #863 to use this as well.

Included 6f8dfb9 in the recent staging push, which was required. It displays why this change is needed.

…base

…test format version to also include (an empty) assertionExceptions so hashes will remain in sync; include utility introduced in #880

… page

…t format tests to be imported

…st results process described in #877; update dataManagement.test.js; update sample populate script to support testing v2 test format tests workflow

stalgiag

Thanks for the updates and accompanying expanded test coverage. I was able to confirm that this was functioning as expected by rebuilding the DB and running the unit tests. The new code also has greater clarity than what it replaces. Approving!

alflennik

I'm happy to approve a second time on this one. I like the idea of making it easy to recalculate the hashes, it does seem pretty likely it might become necessary down the line. I took a look at the broader changes to the tests as well and didn't see anything that rang any alarm bells. I see the tests are passing. Nice work!

* Update client/resources * Update graphql-schema.js and testsResolver * Update assertionResultsResolver.js * Update import-tests script to include assertion.assertionExceptions (0 level assertions will be considered internally as 'EXCLUDE') * Update getMetrics and queries to be aware of 'excluded' assertions for commands * Update import branch for testing * Update import branch for testing * Remove 'exclude' property check reliance on frontend * Revert graphql fragment usage * Fix missing renderableContent in anon viewer query * Rename instances of Command.settings to Command.atOperatingMode in graphql-schema.js * Include migration to update existing test plan versions using the v2 test format version to also include (an empty) assertionExceptions so hashes will remain in sync; include utility introduced in #880 * Move atOperatingMode setting to TestsService, getTests * Missing comma * Update comments with reasoning for filtering out assertions that can possibly have no ids * Update convertAssertionPriority usage * Revert test branch being used back to master * Break out in-line filter into function sufficiently support readability purposes --------- Co-authored-by: Stalgia Grigg <stalgia@bocoup.com>

howard-e marked this pull request as ready for review December 20, 2023 16:20

howard-e requested review from alflennik, Paul-Clue and stalgiag December 20, 2023 16:41

howard-e changed the title ~~Update hashTests function to account for v2 test format version~~ Update hashTests function support the migration of test results across test plan versions using the v2 test format Dec 20, 2023

alflennik approved these changes Jan 8, 2024

View reviewed changes

server/util/aria.test.js Outdated Show resolved Hide resolved

server/util/aria.js Outdated Show resolved Hide resolved

server/util/aria.test.js Show resolved Hide resolved

howard-e added 5 commits January 22, 2024 13:12

Update aria.js to account for properties to be ignored during a v2 te…

31f6c41

…st format version update so applicable results can be transferred over during phase update

Recalculate the hashes for already existing v2 items

0ccbdcd

Check testPlanVersions.length before updating related test plan repor…

4cf7143

…ts and runs

* Remove stray eslint-disable

e6511ec

* Rename testWithNoIds -> testWithOmittedAttributes

Created util for regenerating the hashedTests and test results; addit…

e9dcc4c

…ionally, now deleting found duplicates as there is an error when tested against prod data where unnecessary duplicates were created

howard-e commented Jan 22, 2024

View reviewed changes

Update migration testFormatVersion migration to use util following re…

3c6d020

…base

howard-e force-pushed the fix-v2-results-update branch from e832a86 to 3c6d020 Compare January 22, 2024 18:18

howard-e added a commit that referenced this pull request Jan 22, 2024

Include migration to update existing test plan versions using the v2 …

318b7f4

…test format version to also include (an empty) assertionExceptions so hashes will remain in sync; include utility introduced in #880

howard-e changed the title ~~Update hashTests function support the migration of test results across test plan versions using the v2 test format~~ Update hashTests function to support the migration of test results across test plan versions using the v2 test format Jan 23, 2024

howard-e added 6 commits January 23, 2024 15:27

Merge branch 'main' into fix-v2-results-update

747b423

Avoid showing reports for DEPRECATED test plan versions on Test Queue…

022d86e

… page

Update workflow to separate commits for v1-related and v2-related tes…

2e0d090

…t format tests to be imported

Update updatePhaseResolver to better align with the deprecation of te…

fb28a2e

…st results process described in #877; update dataManagement.test.js; update sample populate script to support testing v2 test format tests workflow

Update comments on test

3c03686

Update dataManagement.test.js

20e841e

howard-e requested review from alflennik and evmiguel January 29, 2024 20:26

stalgiag approved these changes Jan 29, 2024

View reviewed changes

alflennik approved these changes Jan 29, 2024

View reviewed changes

alflennik merged commit eaf9109 into main Jan 29, 2024
2 checks passed

alflennik deleted the fix-v2-results-update branch January 29, 2024 21:42

howard-e mentioned this pull request Feb 14, 2024

Consider updating automated code formatting tools' rules #924

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update `hashTests` function to support the migration of test results across test plan versions using the v2 test format #880

Update `hashTests` function to support the migration of test results across test plan versions using the v2 test format #880

howard-e commented Dec 18, 2023 •

edited

Loading

alflennik left a comment

howard-e Jan 22, 2024 •

edited

Loading

stalgiag left a comment

alflennik left a comment

Update hashTests function to support the migration of test results across test plan versions using the v2 test format #880

Update hashTests function to support the migration of test results across test plan versions using the v2 test format #880

Conversation

howard-e commented Dec 18, 2023 • edited Loading

alflennik left a comment

Choose a reason for hiding this comment

howard-e Jan 22, 2024 • edited Loading

Choose a reason for hiding this comment

stalgiag left a comment

Choose a reason for hiding this comment

alflennik left a comment

Choose a reason for hiding this comment

Update `hashTests` function to support the migration of test results across test plan versions using the v2 test format #880

Update `hashTests` function to support the migration of test results across test plan versions using the v2 test format #880

howard-e commented Dec 18, 2023 •

edited

Loading

howard-e Jan 22, 2024 •

edited

Loading