-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Collection Job - Per-test status updates #1000
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I mostly reviewed this from the perspective of GraphQL, the database and the general common practices on the backend. On those accounts, it looks great. I also did try to put on an automation hat and I was able to successfully get "QUEUED", "COMPLETED", "CANCELLED" statuses, but I would appreciate if the other reviewers could give the automation side a good thorough test. I didn't see any unexpected behavior.
It will be great to hugely reduce the number of server requests soon!
as: 'testStatus', | ||
foreignKey: 'collectionJobId', | ||
sourceKey: 'id' | ||
}); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we want the collection job to be deleted when the test plan run is deleted?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I debated this myself - we have some recovery potential via the database if not, but we might want to delete them
type: Sequelize.INTEGER, | ||
allowNull: false, | ||
onDelete: 'CASCADE', | ||
onUpdate: 'CASCADE', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting feature! Definitely going to file this trick away in my brain.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd like to formalize the distinction between "new way" and "old way" a bit. Doing so will further clarify the differences, give us more confidence when removing the deprecated logic in the future, and reduce the number of unnecessary code paths in the interim.
Basically, if we start with something like,
const isEnhanced = 'testRowNumber' in req.params;
Then we could switch on isEnhanced
. (That name could probably be improved, though.)
// When new status is 'COMPLETED' or 'ERROR' or 'CANCELLED' | ||
// update any CollectionJobTestStatus children still 'QUEUED' to be 'CANCELLED' | ||
if ( | ||
status === COLLECTION_JOB_STATUS.COMPLETED || | ||
status === COLLECTION_JOB_STATUS.CANCELLED || | ||
status === COLLECTION_JOB_STATUS.ERROR | ||
) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A utility function like isFinal
(maybe defined as a static method on the new model) would surface the intent and make this more readable.
We've landed the automation-harness patch so it should be safe to remove the deprecated paths entirely, would you prefer I do just that? |
I think that could work. Don't we also have to update the callback URLs? |
I realized while updating the PR to remove the back-compat URLs that some mock work I needed had already been done on the working UI branch so I merged that into this one instead of keeping them separate. I could tease the frontend updates out of this branch if people still prefer. The changes from d87280b adds the frontend for the test queue and manage nvda run dialogs and some ways to test other status. |
Alright - front end tests are now passing as well, so this should be all good - I removed the back-compat URLs as well because the automation harness is properly working with the new style URLs already. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So this is a huge improvement vastly reducing the load on the server and preventing issues we've already seen that took down the server.
I don't see any reason not to get this in, it looks great and I see a lot of small improvements sprinkled here and there which is great.
I verified that it works as intended with both the mock and genuine automation systems.
I feel like in a future update it might be nice to improve a few quirks of the system as it stands now. For example I noticed it doesn't check for status updates on the test run page. This is definitely an improvement we can consider separately and not a blocker.
Secondly, there are still multiple requests happening every second (I see a "TestPlanRunAssertionResultsByTestPlanRunId" query as well as the "TestPlanRunsTestResults" query) and I feel like the architecture of GraphQL should theoretically allow them to all be combined into one. Again, not a blocker.
The main thing which is worth looking at down the line, but not a blocker, is that fact that the system still makes one request for every active collection job in progress. It feels like this will only be an issue if the admins are trying to test every dang test plan that exists, all at once, and while that doesn't seem likely to happen anytime soon, it is pretty conceivable that it could start to happen eventually. That would be a nice enhancement, but this PR solves the much more serious issue of making one request per test.
Again these notes really have nothing to do with the PR itself, which is all about "per test status updates." So I'm definitely happy to approve.
This ended up being a pretty large-scale change, which was not the expectation going into it, so nice job chasing it down through the whole stack and seemingly every system we got 🙏
const BotRunTestStatusList = ({ testPlanReportId, runnableTestsLength }) => { | ||
const pollInterval = 2000; | ||
|
||
const BotRunTestStatusList = ({ testPlanReportId }) => { | ||
const { | ||
data: testPlanRunsQueryResult, | ||
startPolling, | ||
stopPolling |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wow I had no idea these existed, nice use of the tech 😮
transaction: false | ||
} | ||
) | ||
).rejects.toBeDefined(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for covering this less obvious gotcha in GraphQL, I appreciate that!
@@ -12,6 +12,9 @@ const { | |||
} = require('../../middleware/transactionMiddleware'); | |||
const { query } = require('../util/graphql-test-utilities'); | |||
|
|||
// 0 = no chance of test errors, 1 = always errors | |||
const TEST_ERROR_CHANCE = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Way to go, @gnarf!
Issues addressed: * #1105, addresses w3c/aria-at#1070 * #1053, addresses w3c/aria-practices#2971 * #1097, addresses #977 * #1095, addresses #991 * #1093, addresses #934 * #1000, addresses #818 * #1089, addresses #992 * #1067, addresses #993 * #1056, addresses w3c/wai-aria-practices#212 --------- Co-authored-by: alflennik <alflennik@users.noreply.github.com> Co-authored-by: Paul Clue <67766160+Paul-Clue@users.noreply.github.com> Co-authored-by: Mx Corey Frang <corey@bocoup.com> Co-authored-by: Mx. Corey Frang <gnarf37@gmail.com> Co-authored-by: Erika Miguel <erika@bocoup.com> Co-authored-by: Mike Pennisi <mike@mikepennisi.com>
Implementing the updates for #818