Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump random replica select attempt count + filter out existing nodes from healthy replica set #3661

Merged
merged 2 commits into from
Aug 10, 2022

Conversation

vicky-g
Copy link
Contributor

@vicky-g vicky-g commented Aug 9, 2022

Description

lots of Not enough healthy nodes found to issue new replica set after 5 attempts errors goin on in staging. took a lewk at that and it's cus our attempt count is so low and there might be some redunant selection here

Tests

The existing tests suite should cover this change as there is no main functionality change, just an optimization.

Monitoring - How will this change be monitored? Are there sufficient logs / alerts?

The existing logs will suffice

@vicky-g vicky-g requested a review from theoilie August 9, 2022 02:49
@vicky-g vicky-g marked this pull request as ready for review August 9, 2022 04:33
Copy link
Contributor

@theoilie theoilie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but I vaguely remember that @SidSethi might've mentioned something about this code. I don't think we would ever want to reconfig to a node that's already in the set but just want to double check any context here

@theoilie theoilie requested a review from SidSethi August 9, 2022 16:30
Copy link
Contributor

@SidSethi SidSethi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the tag @theoilie

this change makes sense! good catch vicky
realistically this whole flow is not great and could be rewritten for perf + readability, but for now this is a valuable change

@vicky-g vicky-g changed the title bump attempt count + filter out existing nodes Bump random replica select attempt count + filter out existing nodes Aug 9, 2022
@vicky-g vicky-g changed the title Bump random replica select attempt count + filter out existing nodes Bump random replica select attempt count + filter out existing nodes from healthy replica set Aug 9, 2022
@vicky-g vicky-g merged commit 8e21b0e into master Aug 10, 2022
@vicky-g vicky-g deleted the vg-fix-select-random-rset branch August 10, 2022 15:31
audius-infra pushed a commit that referenced this pull request Aug 11, 2022
## Changelog

- 2022-08-11 [8c2deb9] Use correct configuration for full endpoints (#3673) [Sebastian Klingler]
- 2022-08-10 [5ab2d47] [CON-222] Fix SequelizeUniqueConstraintError for blockchain track+user IDs (#3660) [Theo Ilie]
- 2022-08-10 [1f55962] Bump to version 0.3.64 (#3669) [Cheran]
- 2022-08-10 [8e21b0e] Bump random replica select attempt count + filter out existing nodes from healthy replica set (#3661) [vicky :)]
- 2022-08-09 [19da757] Fix TypeError for userSecondarySyncMetrics (#3668) [Theo Ilie]
- 2022-08-09 [788d482] CON-297 Auth check the requester is the primary of the observed user (#3598) [vicky :)]
- 2022-08-09 [7436c17] Add unique constraint to notification (#3657) [Joseph Lee]
- 2022-08-09 [4d03415] Add slack-secrets to jobs that use Slack notifications (#3666) [Joaquin Casares]
- 2022-08-09 [feed5ab] Revert "Repair missing wallets (#3656)" (#3659) [Raymond Jacobson]
- 2022-08-09 [8196748] Change openresty location directive from exact match to regex (#3664) [Dheeraj Manjunath]
- 2022-08-08 [1c2f069] Repair missing wallets (#3656) [Raymond Jacobson]
- 2022-08-08 [f4d5481] Upgrade service-commands to latest sdk (#3655) [Dheeraj Manjunath]
- 2022-08-08 [90a1601] safer kill_running_queries_sql (#3653) [Steve Perkins]
- 2022-08-08 [6194260] Rename processImmediateSync to processManualImmediateSync for clarity (#3632) [Dheeraj Manjunath]
- 2022-08-05 [f25994c] Fix Prometheus Metric Names for Bull Queue Durations (#3649) [Johannes Naylor]
- 2022-08-05 [a2d9937] Add remix cosign (#3650) [Joseph Lee]
- 2022-08-05 [bbf5556] [C-749] Support req pagination and ordering in hist/fav/tracks (#3643) [Raymond Jacobson]
- 2022-08-05 [d6acc56] INF add alerts for failed gcp bake jobs (#3647) [Joaquin Casares]
- 2022-08-05 [a0ee00a] postgres triggers for discovery notifications (#3464) [Joseph Lee]
- 2022-08-05 [7a4d9ea] Bump sdk to v0.0.32 [audius-infra]
audius-infra pushed a commit that referenced this pull request Aug 11, 2022
## Changelog

- 2022-08-11 [8c2deb9] Use correct configuration for full endpoints (#3673) [Sebastian Klingler]
- 2022-08-10 [5ab2d47] [CON-222] Fix SequelizeUniqueConstraintError for blockchain track+user IDs (#3660) [Theo Ilie]
- 2022-08-10 [1f55962] Bump to version 0.3.64 (#3669) [Cheran]
- 2022-08-10 [8e21b0e] Bump random replica select attempt count + filter out existing nodes from healthy replica set (#3661) [vicky :)]
- 2022-08-09 [19da757] Fix TypeError for userSecondarySyncMetrics (#3668) [Theo Ilie]
- 2022-08-09 [788d482] CON-297 Auth check the requester is the primary of the observed user (#3598) [vicky :)]
- 2022-08-09 [7436c17] Add unique constraint to notification (#3657) [Joseph Lee]
- 2022-08-09 [4d03415] Add slack-secrets to jobs that use Slack notifications (#3666) [Joaquin Casares]
- 2022-08-09 [feed5ab] Revert "Repair missing wallets (#3656)" (#3659) [Raymond Jacobson]
- 2022-08-09 [8196748] Change openresty location directive from exact match to regex (#3664) [Dheeraj Manjunath]
- 2022-08-08 [1c2f069] Repair missing wallets (#3656) [Raymond Jacobson]
- 2022-08-08 [f4d5481] Upgrade service-commands to latest sdk (#3655) [Dheeraj Manjunath]
- 2022-08-08 [90a1601] safer kill_running_queries_sql (#3653) [Steve Perkins]
- 2022-08-08 [6194260] Rename processImmediateSync to processManualImmediateSync for clarity (#3632) [Dheeraj Manjunath]
- 2022-08-05 [f25994c] Fix Prometheus Metric Names for Bull Queue Durations (#3649) [Johannes Naylor]
- 2022-08-05 [a2d9937] Add remix cosign (#3650) [Joseph Lee]
- 2022-08-05 [bbf5556] [C-749] Support req pagination and ordering in hist/fav/tracks (#3643) [Raymond Jacobson]
- 2022-08-05 [d6acc56] INF add alerts for failed gcp bake jobs (#3647) [Joaquin Casares]
- 2022-08-05 [a0ee00a] postgres triggers for discovery notifications (#3464) [Joseph Lee]
- 2022-08-05 [7a4d9ea] Bump sdk to v0.0.32 [audius-infra]
dmanjunath pushed a commit that referenced this pull request Aug 16, 2022
…from healthy replica set (#3661)

* bump attempt count + filter out existing nodes

* adding named fn
sliptype pushed a commit that referenced this pull request Sep 10, 2023
sliptype pushed a commit that referenced this pull request Sep 10, 2023
[4a26077] [C-2678] Add Stems and Source Files Modal (#3671) Andrew Mendelsohn
[6a3342a] Bump android version to 1.1.391 to fix play-store build (#3678) Dylan Jeffers
[2064212] [C-2813] Catch collectibles runtime errors (#3675) Dylan Jeffers
[29233ea] [C-2808] Improve storageNodeSelector usage and perf (#3674) Dylan Jeffers
[2de2035] Fix mobile image uri (#3670) Dylan Jeffers
[55a6089] [C-2807] Add ModalField subforms with cancellation (#3664) Andrew Mendelsohn
[e26986f] [C-2812] Use storageNodeSelector in libs, fix mobile images (#3667) Dylan Jeffers
[5daea08] Enable playlist updates on prod (#3668) Dylan Jeffers
[d90c369] [PAY-1541] Autofocus textinput on mobile search users screen (#3666) Reed
[194f894] [PAY-1542][PAY-1539][PAY-1537] Mobile chats QA improvements (#3665) Reed
[58b9028] [C-2798] Fix playlist button overlap (#3662) Dylan Jeffers
[a935588] [PAY-1538] Fix autocorrect behavior (#3661) Michael Piazza
[1a8a6e7] [PAY-1518] DMs: Align the overflow menu bottom center (#3654) Marcus Pasell
[c880b2a] [plat-1085] disable fav and repost on hidden tracks in play bar (#3653) sabrina-kiam
[d80617e] [PAY-1529] Add initial types for usdc purchases (#3651) Randy Schott
[68e87ff] [C-2787] Reregister device-token on app startup (#3660) Dylan Jeffers
[ddfa510] [C-2802] Move embed to client monorepo! (#3659) Raymond Jacobson
[6a2f008] Add Sentry error logging to Audius query C-2800 (#3656) nicoback2
[79213de] Add Amplitude events to new Write OAuth flows C-2801 (#3657) nicoback2
[6576ae4] Fix android chat message cut off at top (#3658) Reed
[160b325] [C-2677] Remix Settings Modal layout complete (#3647) Andrew Mendelsohn
[c01c022] [C-2790] Add private DogEar to desktop playlist cards (#3655) Dylan Jeffers
[50e8f4e] [PAY-1522][PAY-1451] Fix issues with DMs notification dot (#3649) Michael Piazza
[1658810] [PAY-1530] adds usdc feature flag and hooks for fetching it (#3645) Randy Schott
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants