Misc state machine improvements / bug fixes #3487

theoilie · 2022-07-16T00:15:17Z

Description

Some changes from QA + digging through the logs a bit:

Fix a rename bug for reconciliation queue history
Consolidate job status logging so we can filter by jobId. Searching for "Queue Job Completed" wasn't ideal
Log error stack traces in monitor-state jobs. I'm trying to debug an error in one of the steps, and it's hard to pin point without the stack trace
Add some default values to fix rare Cannot convert undefined or null to object error in the computeUserSecondarySyncSuccessRatesMap step of some monitor-state jobs (seen on prod)

Tests

Made sure queues are running locally with correct logs. Will be more valuable to test on staging with many users.

Monitoring - How will this change be monitored? Are there sufficient logs / alerts?

We should be able to track any one job more thoroughly now by filtering in Loggly for json.jobId:<JOB_ID>. These logs should include: 'Job waiting', 'Job active', 'Job successfully completed. Parsing result: ' (this one is important for debugging)
Search for Cannot convert undefined or null to object in the logs and see if it's still happening. It was rare, but it should be completely gone now

SidSethi

nice!

SidSethi · 2022-07-18T14:03:15Z

creator-node/src/services/stateMachineManager/stateMonitoring/index.js

-      )
+      const logger = createChildLogger(monitoringQueueLogger, {
+        jobId: job?.id || 'unknown'
+      })


is it possible for job to not have id? only on failed and not other events?

we were treating it as optional before so I kept it consistent with that. maybe it's only optional in the failed event. I searched the docs and found conflicting information, but I'm leaning towards thinking it's possible for a job to not have an ID in the failed event.

this doc says "All jobs in BullMQ need to have a unique job id."
BUT
the actual type that I believe is generated from the code shows Job#id as optional here so I think that takes precedence

also other events just pass the jobId directly instead of the full job object

good point, sounds good

## Changelog - 2022-07-19 [5cd2cc0] Add npmrc credential (#3500) [Joaquin Casares] - 2022-07-19 [db3b7ea] INF-173 Fix upload-dashboards.sh and support library panels (#3439) [Joaquin Casares] - 2022-07-19 [7a9e37d] Fix unread count (#3505) [Michael Piazza] - 2022-07-19 [fbf31bb] fix setauthfn / setuserfn types (#3502) [nicoback2] - 2022-07-19 [b6e3740] Don't dedupe manual syncs to secondaries (#3498) [Dheeraj Manjunath] - 2022-07-19 [e871a35] Add state machine README (#3411) [Theo Ilie] - 2022-07-18 [3517ee3] Fix CN nodejs version (#3501) [Cheran] - 2022-07-18 [c3e886a] Misc state machine improvements / bug fixes (#3487) [Theo Ilie] - 2022-07-18 [7f27e3d] Always require syncMode to be defined (#3497) [Dheeraj Manjunath] - 2022-07-18 [283249d] Run image bake without monitoring (#3499) [Dheeraj Manjunath] - 2022-07-18 [81e43fc] Make new beatfile in each (#3496) [Raymond Jacobson] - 2022-07-18 [c6ad867] [CON-256] /export route can return inconsistent data (#3480) [Johannes Naylor] - 2022-07-18 [ae93f81] Expose trusted notifier data in health check + tests (#3486) [vicky :)] - 2022-07-18 [242370d] throw error for function requiring window (#3493) [nicoback2] - 2022-07-16 [327251f] Cleanup output (#3492) [Cheran] - 2022-07-16 [dfa66d3] Add chain network id for client fixes (#3490) [Cheran] - 2022-07-16 [6f9cc9c] [INF-188] Verify contracts on POA for block explorer (#3491) [Cheran] - 2022-07-15 [9c06d04] Fix migration ordering (#3489) [Raymond Jacobson] - 2022-07-15 [67dc7d8] Increase start.sh timeout (#3483) [Raymond Jacobson] - 2022-07-15 [1dc64bc] [PLAT-254] Remove is_creator usage across protocol (#3475) [Raymond Jacobson] - 2022-07-15 [9231623] INF-86 Automate publishing libs in CircleCI (#3447) [Joaquin Casares] - 2022-07-15 [281da8e] Move trusted notifier init back to await-ed init (#3485) [Dheeraj Manjunath] - 2022-07-15 [cc8cf26] Fix hot reload of libs (#3481) [Cheran] - 2022-07-15 [e3bb6f5] [INF-180] Add SOCKS5 proxy for use with client (#3467) [Cheran] - 2022-07-14 [df69fa6] CON-250 - stateMachine updates to consume primarySyncFromSecondary (Divergent state recovery #3) (#3426) [Sid Sethi]

## Changelog - 2022-07-19 [384b036] INF-86 use new audius-infra ssh key for merging audius-sdk branches (#3508) [Joaquin Casares] - 2022-07-19 [99107b3] [CON-221] Create metric to track users entirely on foundation nodes (#3462) [Johannes Naylor] - 2022-07-19 [5cd2cc0] Add npmrc credential (#3500) [Joaquin Casares] - 2022-07-19 [db3b7ea] INF-173 Fix upload-dashboards.sh and support library panels (#3439) [Joaquin Casares] - 2022-07-19 [7a9e37d] Fix unread count (#3505) [Michael Piazza] - 2022-07-19 [fbf31bb] fix setauthfn / setuserfn types (#3502) [nicoback2] - 2022-07-19 [b6e3740] Don't dedupe manual syncs to secondaries (#3498) [Dheeraj Manjunath] - 2022-07-19 [e871a35] Add state machine README (#3411) [Theo Ilie] - 2022-07-18 [3517ee3] Fix CN nodejs version (#3501) [Cheran] - 2022-07-18 [c3e886a] Misc state machine improvements / bug fixes (#3487) [Theo Ilie] - 2022-07-18 [7f27e3d] Always require syncMode to be defined (#3497) [Dheeraj Manjunath] - 2022-07-18 [283249d] Run image bake without monitoring (#3499) [Dheeraj Manjunath] - 2022-07-18 [81e43fc] Make new beatfile in each (#3496) [Raymond Jacobson] - 2022-07-18 [c6ad867] [CON-256] /export route can return inconsistent data (#3480) [Johannes Naylor] - 2022-07-18 [ae93f81] Expose trusted notifier data in health check + tests (#3486) [vicky :)] - 2022-07-18 [242370d] throw error for function requiring window (#3493) [nicoback2] - 2022-07-16 [327251f] Cleanup output (#3492) [Cheran] - 2022-07-16 [dfa66d3] Add chain network id for client fixes (#3490) [Cheran] - 2022-07-16 [6f9cc9c] [INF-188] Verify contracts on POA for block explorer (#3491) [Cheran] - 2022-07-15 [9c06d04] Fix migration ordering (#3489) [Raymond Jacobson] - 2022-07-15 [67dc7d8] Increase start.sh timeout (#3483) [Raymond Jacobson] - 2022-07-15 [1dc64bc] [PLAT-254] Remove is_creator usage across protocol (#3475) [Raymond Jacobson] - 2022-07-15 [9231623] INF-86 Automate publishing libs in CircleCI (#3447) [Joaquin Casares] - 2022-07-15 [281da8e] Move trusted notifier init back to await-ed init (#3485) [Dheeraj Manjunath] - 2022-07-15 [cc8cf26] Fix hot reload of libs (#3481) [Cheran] - 2022-07-15 [e3bb6f5] [INF-180] Add SOCKS5 proxy for use with client (#3467) [Cheran] - 2022-07-14 [df69fa6] CON-250 - stateMachine updates to consume primarySyncFromSecondary (Divergent state recovery #3) (#3426) [Sid Sethi]

[9a0592f] [PAY-1355] Add space before learn more link in inbox unavailable message (#3507) Reed [989390c] [PAY-1267] Add space before placeholder in chat textinput (#3508) Reed [a1e66ce] [PAY-1377] Compose message icon to TouchableOpacity (#3506) Reed [dc12349] Remove write_metadata_through_chain flag (#3432) Michelle Brier [5523041] Add clientVersion to mobile analytics (#3498) Sebastian Klingler [e2ba8ed] [PAY-1343] First message in new chat shows purple unread indicator (#3503) Reed [52096cf] Fix upload regression on stage (#3502) Andrew Mendelsohn [cdf5663] [PAY-1175] DMs: Autofocus, clear search, elevate placeholder (#3486) Marcus Pasell [d012240] [PAY-1318] Missing Shadow and Space Between Divider and Header (#3500) Michael Piazza [e71845e] [PAY-1336] Updates to mobile chats user search screen (#3489) Reed [9d12c62] [PLAT-1005] Parse for add track to playlist notification from backend (#3439) sabrina-kiam [d9c62b1] [C-2708] Improvements around codepush & force pushing (#3501) Sebastian Klingler [aa26688] [pay-1311] Reset chat state on sign out (#3499) Randy Schott [6079466] [PAY-1337] Fix mobile chat message shadows (#3494) Reed [643d3e0] [PAY-1305] DMs: Add skeletons for chats on web (#3496) Marcus Pasell [5e5c900] [] Add styling fixes from mobile playlist updates QA (#3485) Kyle Shanks [e182e8b] [PAY-1308] Use all messages for tracking latest received (#3484) Randy Schott [1e16ea8] Push props down to SelectPage (#3495) Andrew Mendelsohn [8993e52] [C-2705] Port upload files page to new upload form (#3493) Andrew Mendelsohn [986b621] [PAY-1303] Add toastSagas to mobile (#3488) Marcus Pasell [07c35e2] [PAY-1324][PAY-1326][PAY-1323][PAY-1327][PAY-1331][PAY-1328][PAY-1320] DMs UI fixes 2 (#3491) Michael Piazza [f37c767] [C-2658] Fix favorite Hot & New in sign up (#3492) Sebastian Klingler [404acd6] HOTFIX: left-nav delete-confirmation (#3490) Dylan Jeffers [89ac0a5] [C-2699] Fix collection action button sizes (#3487) Dylan Jeffers [bef1745] [PAY-1330][PAY-1317][PAY-1321] Polish DMs #1 (#3482) Michael Piazza [c7611a8] Rename addToast to registerToast (#3472) Marcus Pasell [614863c] [C-2700][C-2701] upload redesign feature flag and placeholder page (#3480) Andrew Mendelsohn [1f70aa4] [PAY-1316] DMs: Do the thing (Fix mobile chats not populating) (#3483) Marcus Pasell [4cf0c1e] [C-2702] Remove initialSelectedNode from disc-node selector (#3481) Dylan Jeffers [6063b9e] SDK v2.0.3 beta.32 (#3479) Marcus Pasell [c106ed8] [PAY-1283] DMs: Listen for chat messages app wide, and get unread message count on load (#3477) Marcus Pasell [8845ae7] [PAY-1329] Fix marking chats as unread (#3478) Reed [c6347af] Update end of message history copy on mobile (#3476) Reed [2700165] [PAY-1293] Adjust portal and animations to get more consistent behavior on Android (#3475) Randy Schott [afe7deb] [PAY-1295] DMs: Add a toast when chats fail to create (#3468) Marcus Pasell [72d2788] DMs: Rename "Thread" to "Conversation" (#3474) Marcus Pasell [5d69114] DMs: Add end of messages notice (#3466) Marcus Pasell [e35a23f] Change delete thread copy to conversation on mobile (#3473) Reed [31a6a55] [PAY-1301] DMs: Add empty state to search users modal (#3470) Marcus Pasell [032107c] [PAY-1292] DMs: Delete chat style updates (#3469) Marcus Pasell [7441321] [pay-1214] add patch to fix inverted list ANR bug (#3457) Randy Schott [018ff3b] [PAY-1291] Adds new message toast when scrolled up on mobile (#3463) Randy Schott [73b3ee0] Add developer apps page, call new SDK developer apps + grants APIs [C-2624] (#3396) nicoback2 [09f08b3] [PAY-1294] Max length for chat text input on mobile (#3465) Reed [ec305cb] [PAY-1278] Indicator for end of chat history on mobile (#3461) Reed [593914c] Filter out self from create chat search results on mobile (#3462) Reed [a42e359] Fix input with explicit input props (#3464) Dylan Jeffers [5ada3f5] [PAY-1138] Fix headers on Chats and fix old header prod bug (#3442) Michael Piazza [8966c0d] [C-2648] Add redesigned playlist management flow (#3456) Dylan Jeffers [5fa7468] [PAY-1297] DMs: Don't show current user in search results (#3460) Marcus Pasell [4a9824c] [C-2635] Fix offline playback; broken signature fetch (#3459) Andrew Mendelsohn [d93537b] [PAY-1246] Mobile delete chat thread UI (#3458) Reed [12107d4] [PAY-1288] DMs: Show max length in composer (#3454) Marcus Pasell [9f8b112] [PAY-1276] Mobile chats change color on press (#3446) Reed [bb53df1] [C-2646] Add idListArgKey for lists of entities; add fetchArgs types (#3452) Andrew Mendelsohn [a954dce] [INF-415] Upgrade to react-native 0.71.8 (#3429) Sebastian Klingler [1591fb2] Retry CodePush installs if they failed the first time C-2659 (#3451) nicoback2 [23f574b] Feature flag unread messages notif bubble (#3453) Reed

Misc state machine improvements / bug fixes

8d0cec6

theoilie added the content-node Content Node (previously known as Creator Node) label Jul 16, 2022

theoilie requested a review from SidSethi July 16, 2022 00:15

pull-request-size bot added the size/L label Jul 16, 2022

SidSethi approved these changes Jul 18, 2022

View reviewed changes

theoilie merged commit c3e886a into master Jul 19, 2022

theoilie deleted the theo-misc-statemachine branch July 19, 2022 01:35

sliptype pushed a commit that referenced this pull request Sep 10, 2023

[C-2699] Fix collection action button sizes (#3487)

ea919f6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Misc state machine improvements / bug fixes #3487

Misc state machine improvements / bug fixes #3487

theoilie commented Jul 16, 2022

SidSethi left a comment

SidSethi Jul 18, 2022

theoilie Jul 19, 2022

theoilie Jul 19, 2022

SidSethi Jul 19, 2022

Misc state machine improvements / bug fixes #3487

Misc state machine improvements / bug fixes #3487

Conversation

theoilie commented Jul 16, 2022

Description

Tests

Monitoring - How will this change be monitored? Are there sufficient logs / alerts?

SidSethi left a comment

Choose a reason for hiding this comment

SidSethi Jul 18, 2022

Choose a reason for hiding this comment

theoilie Jul 19, 2022

Choose a reason for hiding this comment

theoilie Jul 19, 2022

Choose a reason for hiding this comment

SidSethi Jul 19, 2022

Choose a reason for hiding this comment