Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dont reissue manual syncs + separate manual sync timeout #3572

Merged
merged 1 commit into from
Jul 29, 2022

Conversation

dmanjunath
Copy link
Contributor

@dmanjunath dmanjunath commented Jul 29, 2022

Description

Seeing excessive number of waiting jobs in manual sync queue on bull. Theory is that since write quorum continues to add jobs every few seconds for up to 5 minutes, if there's ever a backlog on manual sync it accelerates a problem and makes it much, much worse. The two fixes here are:

  1. Only issue one manual sync request as part of write quorum (this will be retried up to 2 times)
  2. Have a separate write quorum timeout. Since write quorum has a max timeout of 1 min, i set the monitoring time to 45 seconds

Tests

  • Create a user and upload track to make sure those operations are successful
  • Run mad dog on local and CI successfully
  • Clear state on stage CN7 and let this soak and make sure manual sync queue waiting numbers are reasonable

Monitoring - How will this change be monitored? Are there sufficient logs / alerts?

Number of jobs in manual sync queue

@dmanjunath
Copy link
Contributor Author

dmanjunath commented Jul 29, 2022

@theoilie not fully tested but this is the code I had in mind

Edit - tested now

@dmanjunath dmanjunath marked this pull request as ready for review July 29, 2022 01:10
Copy link
Contributor

@theoilie theoilie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great!

@dmanjunath dmanjunath merged commit e4e64d7 into master Jul 29, 2022
@dmanjunath dmanjunath deleted the dm-write-quorum-prune-manual-syncs branch July 29, 2022 15:03
audius-infra pushed a commit that referenced this pull request Jul 29, 2022
## Changelog

- 2022-07-29 [ba9602c] Revert "Sanity check setIsCreatorToTrue (#3531)" (#3578) [Raymond Jacobson]
- 2022-07-29 [e4e64d7] Dont reissue manual syncs + separate manual sync timeout (#3572) [Dheeraj Manjunath]
- 2022-07-29 [e6fdba5] [CON-289] Bug - cnodeUser.clock inconsistent with clockRecord data (#3560) [Johannes Naylor]
- 2022-07-28 [9da567c] [CON-299] Fix reading clock of undefined (#3571) [Theo Ilie]
- 2022-07-28 [0639488] Unwrap transaction signature from identity response (#3528) [Marcus Pasell]
- 2022-07-28 [0db89c5] [C-324] Migrate solana service to typescript (#3563) [Dylan Jeffers]
- 2022-07-28 [d70bdb9] Add tip email notifs (#3377) [Saliou Diallo]
- 2022-07-28 [42544c7] Fix hedgehog privateKey access (#3569) [Dylan Jeffers]
- 2022-07-28 [6725b54] Fix upload track w prom metrics (#3568) [vicky :)]
- 2022-07-28 [25f99d3] Fixes for Discovery Dockerfile following alpine update (#3566) [Cheran]
- 2022-07-28 [6876cb8] Fix reactions (#3564) [Michael Piazza]
- 2022-07-28 [8cb266e] [CON-295, CON-280] Typescriptify most of state machine & fix bugs (#3565) [Theo Ilie]
- 2022-07-27 [4c30053] [C-726] Consume typed @audius/hedgehog (#3557) [Dylan Jeffers]
sliptype pushed a commit that referenced this pull request Sep 10, 2023
[1957c34] Add remove from playlist overflow action (#3621) Kyle Shanks
[67e0012] [C-2712] Clear favorites on sign-out (#3620) Dylan Jeffers
[97b6b94] [PAY-1421] Fix android reaction offsets (#3611) Michael Piazza
[9867571] [PAY-1494] Change chats copy to beginning of message history (#3617) Reed
[1b7ca71] Add overflow menu options for track list items on mobile (#3608) Kyle Shanks
[810f0df] [PAY-1487] Increase hit slop of mobile dms send button (#3618) Reed
[06dae51]  [PAY-1470] Remove border radius from track/playlist tiles (#3613) Marcus Pasell
[dbdfb84] [PAY-1469] Clear the search after creating a chat (#3616) Marcus Pasell
[3f31554] [PAY-1489] Add more space to reacted to messages (#3615) Marcus Pasell
[ed4f321] [PAY-1473] Clear input on chat change (#3614) Marcus Pasell
[e9683d9] Use emojis instead of 'heart' etc as reactions (#3602) Marcus Pasell
[192fd44] [PAY-1486] DMs: Fix purple border around unfurls (#3612) Marcus Pasell
[b0c2e4a] Lowercase library restricted handle (#3590) Isaac Solo
[a8daa72] Change text to 'Beginning of Conversation' (#3605) Marcus Pasell
[332c0c0] Developer app modal - small fixes (#3609) nicoback2
[0e43db1] [PAY-1476] Inbox unavailable copy update (#3607) Reed
[d862fd5] [PAY-1484] Allow reactions while keyboard is open on mobile (#3606) Reed
[3bd5928] [PAY-944] DMs: Add Mobile Web drawer prompting app download (#3565) Marcus Pasell
[5bea105] Wait for libs to init before initting SDK (#3604) Marcus Pasell
[f4d28e0] Add option to cache bust node_modules (#3603) Dylan Jeffers
[9b165c4] [PAY-1455] Optimize reactions perf (maybe) (#3599) Michael Piazza
[0a1a68b] [PAY-1465] Auto-nav to chat only when permitted (#3598) Reed
[72e9f92] [C-2665 C-2714] Add developer apps settings modal (#3589) Dylan Jeffers
[b787344] [PAY-1456] Fix followee-gated create chat crash (#3593) Reed
[dbae5bd] Reduce MessageLockedButton size on profile screen (#3597) Reed
[097879b] [PAY-1437] Fix drawers sharing state causing crash (#3595) Reed
[818b215] [C-2770] Fix issue where playlist library can disappear (#3596) Dylan Jeffers
[f3ada88] [PAY-1461] Format follower count in chat search screen (#3594) Reed
[eb924e7] [C-2713] Add track metadata fields to new upload form (#3574) Andrew Mendelsohn
[be56d1f] Fix mobile textinput borders + labels (#3592) Reed
[fc7541b] Update collection list id flow to add the create tile (#3591) Kyle Shanks
[dc15518] [PAY-1432] Fix chat playlist artist sometimes undefined (#3585) Reed
[399d0b5] [C-2588] Update share modal and drawer to properly add user twitter handle (#3584) Kyle Shanks
[9f30855] [C-2761] Add mutations to audius-query (#3581) Dylan Jeffers
[597dfe5] [PAY-1448] End the race between SDK and Libs, once and for all (#3588) Marcus Pasell
[56914cf] [C-2768] Update InputV2, add TextField/TextAreaField (#3587) Dylan Jeffers
[c9358b6] [CON-742] Make users and track actions v2 for v2 users/tracks even when feature flags are disabled (#3582) Theo Ilie
[a94f035] audius-query debug flag usage in README (#3586) Reed
[bceb3fe] [INF-417] Generate sourcemaps when running web:prod (#3583) Sebastian Klingler
[f4ad466] [PAY-1454] Redline mobile header to latest figma design (#3576) Michael Piazza
[eec6af0] [PAY-1405] Add padding to bottom of chat user search screen (#3580) Reed
[a6c8ece] [PAY-1435] Navigate to chat after unblocking user on mobile (#3578) Reed
[ee1b6c3] [PLAT-659] Metamask sign in/up on ACDC (#3563) Raymond Jacobson
[fbd4053] [C-2472] Update mobile track list items for playlists to have the overflow menu (#3577) Kyle Shanks
[51f44ac] [PAY-1399] Fix Android chat send button color (#3575) Reed
[7d440de] [PAY-1442] Fix drawer touchables on android (#3572) Reed
[8c90ff6] Allow drag and dropping tracks into chat (#3573) Michael Piazza
[f7489e0] Upgrade dapp store version to v1.5.26 (#3504) Dylan Jeffers
[6be0c64] [PAY-1440][PAY-1441] Mobile inbox settings UI fixes (#3571) Reed
[001007e] [PAY-1434] Dismiss keyboard on create chat screen row press (#3569) Reed
[c70e9fe] [PAY-1446] Set MessageLocked icon in profile screen to 0.4 opacity (#3570) Reed
[2cc8ad0] [C-2716] Improve notifications on fresh account (#3568) Dylan Jeffers
[01f05ba] [C-2474] Generate playlist artwork from track artwork (#3518) Dylan Jeffers
[0355c10] [PAY-1325] Remove userbadges from chat unavailable message in chat screen (#3567) Reed
[da10bbc] [PAY-1389] CTA on create chat when not permitted (#3566) Reed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants