-
Notifications
You must be signed in to change notification settings - Fork 111
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bump random replica select attempt count + filter out existing nodes from healthy replica set #3661
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM but I vaguely remember that @SidSethi might've mentioned something about this code. I don't think we would ever want to reconfig to a node that's already in the set but just want to double check any context here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks for the tag @theoilie
this change makes sense! good catch vicky
realistically this whole flow is not great and could be rewritten for perf + readability, but for now this is a valuable change
## Changelog - 2022-08-11 [8c2deb9] Use correct configuration for full endpoints (#3673) [Sebastian Klingler] - 2022-08-10 [5ab2d47] [CON-222] Fix SequelizeUniqueConstraintError for blockchain track+user IDs (#3660) [Theo Ilie] - 2022-08-10 [1f55962] Bump to version 0.3.64 (#3669) [Cheran] - 2022-08-10 [8e21b0e] Bump random replica select attempt count + filter out existing nodes from healthy replica set (#3661) [vicky :)] - 2022-08-09 [19da757] Fix TypeError for userSecondarySyncMetrics (#3668) [Theo Ilie] - 2022-08-09 [788d482] CON-297 Auth check the requester is the primary of the observed user (#3598) [vicky :)] - 2022-08-09 [7436c17] Add unique constraint to notification (#3657) [Joseph Lee] - 2022-08-09 [4d03415] Add slack-secrets to jobs that use Slack notifications (#3666) [Joaquin Casares] - 2022-08-09 [feed5ab] Revert "Repair missing wallets (#3656)" (#3659) [Raymond Jacobson] - 2022-08-09 [8196748] Change openresty location directive from exact match to regex (#3664) [Dheeraj Manjunath] - 2022-08-08 [1c2f069] Repair missing wallets (#3656) [Raymond Jacobson] - 2022-08-08 [f4d5481] Upgrade service-commands to latest sdk (#3655) [Dheeraj Manjunath] - 2022-08-08 [90a1601] safer kill_running_queries_sql (#3653) [Steve Perkins] - 2022-08-08 [6194260] Rename processImmediateSync to processManualImmediateSync for clarity (#3632) [Dheeraj Manjunath] - 2022-08-05 [f25994c] Fix Prometheus Metric Names for Bull Queue Durations (#3649) [Johannes Naylor] - 2022-08-05 [a2d9937] Add remix cosign (#3650) [Joseph Lee] - 2022-08-05 [bbf5556] [C-749] Support req pagination and ordering in hist/fav/tracks (#3643) [Raymond Jacobson] - 2022-08-05 [d6acc56] INF add alerts for failed gcp bake jobs (#3647) [Joaquin Casares] - 2022-08-05 [a0ee00a] postgres triggers for discovery notifications (#3464) [Joseph Lee] - 2022-08-05 [7a4d9ea] Bump sdk to v0.0.32 [audius-infra]
## Changelog - 2022-08-11 [8c2deb9] Use correct configuration for full endpoints (#3673) [Sebastian Klingler] - 2022-08-10 [5ab2d47] [CON-222] Fix SequelizeUniqueConstraintError for blockchain track+user IDs (#3660) [Theo Ilie] - 2022-08-10 [1f55962] Bump to version 0.3.64 (#3669) [Cheran] - 2022-08-10 [8e21b0e] Bump random replica select attempt count + filter out existing nodes from healthy replica set (#3661) [vicky :)] - 2022-08-09 [19da757] Fix TypeError for userSecondarySyncMetrics (#3668) [Theo Ilie] - 2022-08-09 [788d482] CON-297 Auth check the requester is the primary of the observed user (#3598) [vicky :)] - 2022-08-09 [7436c17] Add unique constraint to notification (#3657) [Joseph Lee] - 2022-08-09 [4d03415] Add slack-secrets to jobs that use Slack notifications (#3666) [Joaquin Casares] - 2022-08-09 [feed5ab] Revert "Repair missing wallets (#3656)" (#3659) [Raymond Jacobson] - 2022-08-09 [8196748] Change openresty location directive from exact match to regex (#3664) [Dheeraj Manjunath] - 2022-08-08 [1c2f069] Repair missing wallets (#3656) [Raymond Jacobson] - 2022-08-08 [f4d5481] Upgrade service-commands to latest sdk (#3655) [Dheeraj Manjunath] - 2022-08-08 [90a1601] safer kill_running_queries_sql (#3653) [Steve Perkins] - 2022-08-08 [6194260] Rename processImmediateSync to processManualImmediateSync for clarity (#3632) [Dheeraj Manjunath] - 2022-08-05 [f25994c] Fix Prometheus Metric Names for Bull Queue Durations (#3649) [Johannes Naylor] - 2022-08-05 [a2d9937] Add remix cosign (#3650) [Joseph Lee] - 2022-08-05 [bbf5556] [C-749] Support req pagination and ordering in hist/fav/tracks (#3643) [Raymond Jacobson] - 2022-08-05 [d6acc56] INF add alerts for failed gcp bake jobs (#3647) [Joaquin Casares] - 2022-08-05 [a0ee00a] postgres triggers for discovery notifications (#3464) [Joseph Lee] - 2022-08-05 [7a4d9ea] Bump sdk to v0.0.32 [audius-infra]
…from healthy replica set (#3661) * bump attempt count + filter out existing nodes * adding named fn
[4a26077] [C-2678] Add Stems and Source Files Modal (#3671) Andrew Mendelsohn [6a3342a] Bump android version to 1.1.391 to fix play-store build (#3678) Dylan Jeffers [2064212] [C-2813] Catch collectibles runtime errors (#3675) Dylan Jeffers [29233ea] [C-2808] Improve storageNodeSelector usage and perf (#3674) Dylan Jeffers [2de2035] Fix mobile image uri (#3670) Dylan Jeffers [55a6089] [C-2807] Add ModalField subforms with cancellation (#3664) Andrew Mendelsohn [e26986f] [C-2812] Use storageNodeSelector in libs, fix mobile images (#3667) Dylan Jeffers [5daea08] Enable playlist updates on prod (#3668) Dylan Jeffers [d90c369] [PAY-1541] Autofocus textinput on mobile search users screen (#3666) Reed [194f894] [PAY-1542][PAY-1539][PAY-1537] Mobile chats QA improvements (#3665) Reed [58b9028] [C-2798] Fix playlist button overlap (#3662) Dylan Jeffers [a935588] [PAY-1538] Fix autocorrect behavior (#3661) Michael Piazza [1a8a6e7] [PAY-1518] DMs: Align the overflow menu bottom center (#3654) Marcus Pasell [c880b2a] [plat-1085] disable fav and repost on hidden tracks in play bar (#3653) sabrina-kiam [d80617e] [PAY-1529] Add initial types for usdc purchases (#3651) Randy Schott [68e87ff] [C-2787] Reregister device-token on app startup (#3660) Dylan Jeffers [ddfa510] [C-2802] Move embed to client monorepo! (#3659) Raymond Jacobson [6a2f008] Add Sentry error logging to Audius query C-2800 (#3656) nicoback2 [79213de] Add Amplitude events to new Write OAuth flows C-2801 (#3657) nicoback2 [6576ae4] Fix android chat message cut off at top (#3658) Reed [160b325] [C-2677] Remix Settings Modal layout complete (#3647) Andrew Mendelsohn [c01c022] [C-2790] Add private DogEar to desktop playlist cards (#3655) Dylan Jeffers [50e8f4e] [PAY-1522][PAY-1451] Fix issues with DMs notification dot (#3649) Michael Piazza [1658810] [PAY-1530] adds usdc feature flag and hooks for fetching it (#3645) Randy Schott
Description
lots of
Not enough healthy nodes found to issue new replica set after 5 attempts
errors goin on in staging. took a lewk at that and it's cus our attempt count is so low and there might be some redunant selection hereTests
The existing tests suite should cover this change as there is no main functionality change, just an optimization.
Monitoring - How will this change be monitored? Are there sufficient logs / alerts?
The existing logs will suffice