-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Broker change dynamic configuration based on ZK latency to reduce zk-load #342
Comments
|
for now, we have mechanism to control concurrent topic-loading and lookup-request dynamically. So, for now, action could be changing throttling limits for topic-loading and lookup-request.
actually, we don't have any gate at broker which can be enabled dynamically to prioritize zk access. So, not sure what exactly we can do here.
yes, but considering write latency could be enough.? |
@rdhabalia are you guys actively working on this? |
ping @rdhabalia to confirm |
@jiazhai I had some changes checked in local branch and wanted to get thoughts before creating the PR. Will try to look into it again soon. let me know if you guys have any thoughts on it. |
Fixes apache#342 ### Motivation Add support for KeySharedPolicy with AutoSplit or Sticky mode for consumer, which is a useful for some user cases like scalable request-reply pattern. ### Modifications add key shared policy options for consumer, and a helpful constructor for validating hash range list
Closed as stale. Feel free to open a new issue if it's still relevant to the maintained versions. |
Existing behavior
Sometimes broker sees high zk-latency due to gc pauses or high load on zookeeper, and it may cause brokers to lose zk-session which can create cold restart situation and that subsequently creates high pressure on zk as cold restart loads high number of concurrent topics and lookup-requests.
Broker has capability to throttle concurrent lookup-request and topic-loading which helps to reduce back-pressure on zookeeper. with PR #320 Broker will also have capability to monitor zk-latency at real time as well.
Change
So, broker should dynamically control throttling if it sees zk-latency is going higher than configured threshold (eg. threshold=0.75*zkSessionTimeOut) which can reduce zk-load and helps broker to keep zk-session alive and avoid cold-restart. So, we can implement controller which monitors zk-stats and takes appropriate actions (can be enhanced to consider additional variables).
@merlimat @saandrews any thought?
The text was updated successfully, but these errors were encountered: