Leadership pinning: implementation #23691

ztlpn · 2024-10-09T09:56:09Z

Jira ref: https://redpandadata.atlassian.net/browse/CORE-7022

TODO: add it to the list of enterprise features.

Backports Required

Release Notes

Features

Add leadership pinning: ability to set preferred racks for topic partition leaders. To configure, set redpanda.leaders.preference topic config property or default_leaders_preference cluster config property.

Use revision id of the topic creation command, but introduce a distinct type.

src/v/cluster/scheduling/leader_balancer_constraints.h

vbotbuildovich · 2024-10-09T13:26:51Z

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56084#01927120-f364-46d1-b7c1-7498ca75a083
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56115#019271fd-bb95-4ac4-836a-ac4ab4227a9c
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56152#019272b3-cc9d-4c6c-91c5-05288b1f16b8

This way several constraints can use it, not just the topic-aware distribution constraint.

This is a struct that will hold everything needed for the leadership pinning constraint.

No functional changes.

Previously, there were two problems with muted groups: 1) If the transfer was not successful, we didn't mute 2) If the transfer was successful, the group remained muted for a long time, preventing reaching optimum. Make the balancer more aggressive by removing the group from the muted set if we've got a leadership notification after a successful transfer.

Usually this means that the health report hasn't yet been updated, it doesn't make sense to mute those groups for long.

Cleanup the execution model for balancing iterations: 1) Allow the balancing fiber to run to completion (with small mandatory intervals between iterations). 2) Don't ignore timer if it fired while the fiber was active. 3) If we are not throttled or activating after acquiring controller leadership, allow to schedule timer sooner (useful for notifications).

For now, just trigger the balancer soon after topic creation.

vbotbuildovich · 2024-10-09T17:08:57Z

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56115#019271fd-bb8c-4ad1-b1ce-b7b2d3ad0ad9:

"rptest.tests.leadership_transfer_test.LeadershipPinningTest.test_leadership_pinning"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56152#019272b6-d818-4fd6-b67c-2038e07b2821:

"rptest.tests.leadership_transfer_test.LeadershipPinningTest.test_leadership_pinning"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56179#01927362-4df1-449b-ae90-49619d95286b:

"rptest.tests.leadership_transfer_test.LeadershipPinningTest.test_leadership_pinning"
"rptest.tests.leadership_transfer_test.LeadershipPinningTest.test_leadership_pinning"

vbotbuildovich · 2024-10-09T17:09:09Z

Retry command for Build#56115

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/leadership_transfer_test.py::LeadershipPinningTest.test_leadership_pinning

vbotbuildovich · 2024-10-09T20:48:30Z

Retry command for Build#56152

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/leadership_transfer_test.py::LeadershipPinningTest.test_leadership_pinning

ztlpn · 2024-10-09T22:23:02Z

/ci-repeat 1
skip-units
skip-redpanda-build
dt-repeat=10
tests/rptest/tests/leadership_transfer_test.py::LeadershipPinningTest

vbotbuildovich · 2024-10-09T22:55:11Z

Retry command for Build#56179

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/leadership_transfer_test.py::LeadershipPinningTest.test_leadership_pinning
tests/rptest/tests/leadership_transfer_test.py::LeadershipPinningTest.test_leadership_pinning

ztlpn · 2024-10-10T13:48:13Z

/ci-repeat 1
skip-units
skip-redpanda-build
dt-repeat=10
tests/rptest/tests/leadership_transfer_test.py::LeadershipPinningTest

ztlpn · 2024-10-10T14:28:49Z

Ok looks like the test is stable now

ztlpn added 3 commits October 9, 2024 05:15

cluster: fix typo in even_topic_distribution_constraint class name

0efabfb

c/leader_balancer: bind required config properties

d922a37

c/leader_balancer: introduce topic_id_t type

924a8e8

Use revision id of the topic creation command, but introduce a distinct type.

ztlpn requested review from bharathv, bashtanov and mmaslankaprv October 9, 2024 09:56

github-actions bot added the area/redpanda label Oct 9, 2024

ztlpn force-pushed the leadership-pinning-impl branch 2 times, most recently from ad58898 to ea5d50d Compare October 9, 2024 10:16

mmaslankaprv reviewed Oct 9, 2024

View reviewed changes

src/v/cluster/scheduling/leader_balancer_constraints.h Outdated Show resolved Hide resolved

ztlpn added 12 commits October 9, 2024 16:19

c/leader_balancer: move ownership of group-to-topic index to strategy

f437311

This way several constraints can use it, not just the topic-aware distribution constraint.

c/leader_balancer: introduce and build preference_index

ad0a25c

This is a struct that will hold everything needed for the leadership pinning constraint.

c/leader_balancer: implement and use pinning constraint

704cc6a

c/leader_balancer: reorder class fields

11d569c

No functional changes.

c/leader_balancer: use chunked_hash_map instead of btree_map

f588620

c/leader_balancer: mute groups without leader core just for current iter

c5f17b3

Usually this means that the health report hasn't yet been updated, it doesn't make sense to mute those groups for long.

c/leader_balancer: save synced term instead of "need sync" flag

8efc405

c/leader_balancer: add topic-level notifications

42e7a89

For now, just trigger the balancer soon after topic creation.

c/leader_balancer: schedule tick if topic leaders_preference changed

e7b42f1

c/leader_balancer: schedule tick if default leaders_preference changed

b4d30bd

ztlpn force-pushed the leadership-pinning-impl branch from ea5d50d to a91893a Compare October 9, 2024 14:19

ztlpn requested a review from mmaslankaprv October 9, 2024 14:20

ztlpn force-pushed the leadership-pinning-impl branch from a91893a to 26dbd35 Compare October 9, 2024 18:03

tests: add ducktape test for leaders preference

8bd5e66

ztlpn force-pushed the leadership-pinning-impl branch from 26dbd35 to 8bd5e66 Compare October 10, 2024 13:47

mmaslankaprv approved these changes Oct 10, 2024

View reviewed changes

ztlpn merged commit 4c8c4b7 into redpanda-data:dev Oct 10, 2024
16 checks passed

ztlpn deleted the leadership-pinning-impl branch October 10, 2024 15:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Leadership pinning: implementation #23691

Leadership pinning: implementation #23691

ztlpn commented Oct 9, 2024 •

edited

Loading

vbotbuildovich commented Oct 9, 2024 •

edited

Loading

vbotbuildovich commented Oct 9, 2024 •

edited

Loading

vbotbuildovich commented Oct 9, 2024

vbotbuildovich commented Oct 9, 2024

ztlpn commented Oct 9, 2024

vbotbuildovich commented Oct 9, 2024

ztlpn commented Oct 10, 2024

ztlpn commented Oct 10, 2024

Leadership pinning: implementation #23691

Leadership pinning: implementation #23691

Conversation

ztlpn commented Oct 9, 2024 • edited Loading

Backports Required

Release Notes

Features

vbotbuildovich commented Oct 9, 2024 • edited Loading

vbotbuildovich commented Oct 9, 2024 • edited Loading

vbotbuildovich commented Oct 9, 2024

Retry command for Build#56115

vbotbuildovich commented Oct 9, 2024

Retry command for Build#56152

ztlpn commented Oct 9, 2024

vbotbuildovich commented Oct 9, 2024

Retry command for Build#56179

ztlpn commented Oct 10, 2024

ztlpn commented Oct 10, 2024

ztlpn commented Oct 9, 2024 •

edited

Loading

vbotbuildovich commented Oct 9, 2024 •

edited

Loading

vbotbuildovich commented Oct 9, 2024 •

edited

Loading