[CON-225] Remove manual sync from snapbackSM

[jonaylor89]

Description

This PR moves the manualSyncQueue out of snapbackSM and into the new refactored StateMachineManager.

1. make new queue for manual syncs in stateReconciliationManager/index.js

2. update src/services/stateMachineManager/index.js to consume new queue

3. make serviceRegistry consume new queue as well

4. move over issueSyncRequestsUntilSynced to src/services/stateMachineManager/index.js and call issueSyncRequestsUntilSynced() in middleware.js

5. fix issueSyncRequestsUntilSynced to work with the new queue:

Queue#add (on the new manualSyncQueue object from stateReconciliationManager/index.js)

Tests

The primary way this way tested was by running the protocol and client on my dev box, uploading a new track, and checking the logs to make sure the manual sync was performed.

This PR modifies tests for stateReconciliationQueue so that they work with the new api coming from the stateReconciliationManager but doesn't add any new tests for manualSyncQueue

Supporting Screenshot shots

Monitoring - How will this change be monitored? Are there sufficient logs / alerts?

These changes can be monitoring with Bull health check as well as with new logs.

[theoilie]

just some minor changes and then it should be good to merge assuming it looks fine on the bull dashboard! (/health/bull endpoint)

[theoilie]

couple nits and 🚀

[SidSethi]

i'll take a look async, no need to block on my approval! great work

[SidSethi]

nice!! looks great generally

bit of a deviation from the naming here since there's now stateReconciliationQueue and a manualSyncQueue - will defer to you and @theoilie on standardization here, but smth to think about
i wonder if it would be easier for now to add this as a job type in the stateReconciliationQueue to be consistent with recurring syncs? not sure how i feel about the split states created

per your testing description, want to make sure that it is indeed the new queue you created being used and not the old one

also i don't see any test coverage of this new logic?

for manual testing, would be good to make sure that appropriate configs are in place to never initialize snapback and ensure the correct logic is being selected always

some thoughts from end, will let you and theo sync up

[theoilie]

nice!! looks great generally

bit of a deviation from the naming here since there's now stateReconciliationQueue and a manualSyncQueue - will defer to you and @theoilie on standardization here, but smth to think about i wonder if it would be easier for now to add this as a job type in the stateReconciliationQueue to be consistent with recurring syncs? not sure how i feel about the split states created

per your testing description, want to make sure that it is indeed the new queue you created being used and not the old one

also i don't see any test coverage of this new logic?

for manual testing, would be good to make sure that appropriate configs are in place to never initialize snapback and ensure the correct logic is being selected always

some thoughts from end, will let you and theo sync up

I'd say the disabling snapback side of things is out of scope here as long as this disables snapback's manual sync queue (which it does). I did see Johannes test it manually, and the logs showed that it was processing the sync correctly. @jonaylor89 maybe you could attach a log screenshot showing it process in the correct queue? it would have (queue=manual-sync-queue) as a suffix because you added the JSON queue property to the new manual-sync-queue, but the old manual-sync-queue didn't have that

for state deviation, maybe it would make sense to move toward the one job type per queue model you were thinking of earlier? i.e., breaking up stateReconcliationQueue to recurringSyncQueue and updateReplicaSetQueue

[SidSethi]

🚀 once CI passes :)