base: enable Raft CheckQuorum by default #104042
Conversation
|
This change is |
erikgrinaker
force-pushed
the
raft-enable-checkquorum
branch
3 times, most recently
from
0ab119f
to
f5e105d
Compare
|
Hm, there could be mixed-version concerns here. I don't think we can rely on version gates, since we have to configure this during Raft initialization. |
|
I think most of the mixed-version aspects here are fine. Mixing leaders with different Mixing voters with different The one annoying exception is elections following leader replica removals, see #104189 (comment). |
|
Following discussion in #104189 (comment), we'll need to version gate |
erikgrinaker
force-pushed
the
raft-enable-checkquorum
branch
from
f5e105d
to
42b4249
Compare
This won't be necessary, see #104189 (comment). I believe even the follower removal case should be fine in mixed-version clusters. In #104189 (i.e. 23.2), all replicas will campaign when the leader gets removed. If the first remaining replica is a 23.1 node, it will campaign, and can obtain prevotes either from other 23.1 nodes or from 23.2 nodes who themselves transition to pre-candidates. If the first remaining replica is as 23.2 node, it will be able to obtain prevotes both from 23.1 nodes and 23.2 nodes. A similar argument holds e.g. for explicit campaigning during unquiescence. So I don't think there are any mixed-version concerns here, and we won't need a version gate (which is preferable anyway, so that operators can discover any issues before finalizing the upgrade). |
erikgrinaker
force-pushed
the
raft-enable-checkquorum
branch
2 times, most recently
from
ed6e24f
to
ddc968f
Compare
erikgrinaker
force-pushed
the
raft-enable-checkquorum
branch
2 times, most recently
from
56287e9
to
588a97e
Compare
This patch enables Raft CheckQuorum by default. In etcd/raft, this also has the effect of fully enabling PreVote, such that followers won't grant prevotes if they've heard from a leader in the past election timeout interval. This is more robust against partial and asymmetric network partitions. Otherwise, a partitioned node may be able to hold spurious elections and steal leadership away from an established leader. This can cause the leader to become unreachable by the leaseholder, resulting in permanent range unavailability. We are still able to hold immediate elections, e.g. when unquiescing a range to find a dead leader. If a quorum of followers consider the leader dead and forget it (becoming leaderless followers), they will grant prevotes despite having seen the leader recently (i.e. before quiescing), and can hold an election immediately. This is compatibile with 23.1 in mixed-version clusters: * Leaders with mixed `CheckQuorum` settings is fine: they only apply the step-down logic to themselves, and register follower activity regardless of the followers' settings. * Voters with mixed `CheckQuorum` settings if fine: the leader recency criterion is only applied to their own vote, so either they'll enforce it or not. * Campaigning on leader removal is fine-ish: before 23.2 finalization, the first range replica will campaign -- if this replica is 23.2 it will bypass pre-vote and call an immediate election, if it is 23.1 then it will use pre-vote. However, upon receiving the 23.1 pre-vote request, 23.2 nodes will check if the leader is still in the descriptor, and if it isn't they will forget it and grant the pre-vote. A quorum will likely apply the leader removal before receiving pre-vote requests. Otherwise, we will recover after an election timeout. * Campaigning after unquiescing is fine: the logic remains unchanged, and 23.2 nodes will forget the leader and grant prevotes if they find the leader dead according to liveness. * Campaigning during lease acquisitions is fine: this is needed to steal leadership away from an active leader that can't itself acquire an epoch lease because it's failing liveness heartbeats. If a 23.2 node also finds the leader dead in liveness, it will forget it and grant the prevote. Epic: none Release note (bug fix): The Raft PreVote and CheckQuorum mechanisms are now fully enabled. These prevent spurious elections when followers already have an active leader, and cause leaders to step down if they don't hear back from a quorum of followers. This improves reliability under partial and asymmetric network partitions, by avoiding spurious elections and preventing unavailability where a partially partitioned node could steal leadership away from an established leaseholder who would then no longer be able to reach the leader and submit writes.
erikgrinaker
force-pushed
the
raft-enable-checkquorum
branch
from
588a97e
to
de9b2b2
Compare
|
Allright, let's flip the switch. |
|
bors r+ |
|
Build failed: |
|
bors r+ |
|
Build succeeded: |
This patch enables Raft CheckQuorum by default. In etcd/raft, this also has the effect of fully enabling PreVote, such that followers won't grant prevotes if they've heard from a leader in the past election timeout interval.
This is more robust against partial and asymmetric network partitions. Otherwise, a partitioned node may be able to hold spurious elections and steal leadership away from an established leader. This can cause the leader to become unreachable by the leaseholder, resulting in permanent range unavailability.
We are still able to hold immediate elections, e.g. when unquiescing a range to find a dead leader. If a quorum of followers consider the leader dead and forget it (becoming leaderless followers), they will grant prevotes despite having seen the leader recently (i.e. before quiescing), and can hold an election immediately.
This is compatible with 23.1 in mixed-version clusters:
Leaders with mixed
CheckQuorumsettings is fine: they only apply the step-down logic to themselves, and register follower activity regardless of the followers' settings.Voters with mixed
CheckQuorumsettings if fine: the leader recency criterion is only applied to their own vote, so either they'll enforce it or not.Campaigning on leader removal is fine-ish: before 23.2 finalization, the first range replica will campaign -- if this replica is 23.2 it will bypass pre-vote and call an immediate election, if it is 23.1 then it will use pre-vote. However, upon receiving the 23.1 pre-vote request, 23.2 nodes will check if the leader is still in the descriptor, and if it isn't they will forget it and grant the pre-vote. A quorum will likely apply the leader removal before receiving pre-vote requests. Otherwise, we will recover after an election timeout.
Campaigning after unquiescing is fine: the logic remains unchanged, and 23.2 nodes will forget the leader and grant prevotes if they find the leader dead according to liveness.
Campaigning during lease acquisitions is fine: this is needed to steal leadership away from an active leader that can't itself acquire an epoch lease because it's failing liveness heartbeats. If a 23.2 node also finds the leader dead in liveness, it will forget it and grant the prevote.
Resolves #92088.
Touches #49220.
Epic: none.
Release note (bug fix): The Raft PreVote and CheckQuorum mechanisms are now fully enabled. These prevent spurious elections when followers already have an active leader, and cause leaders to step down if they don't hear back from a quorum of followers. This improves reliability under partial and asymmetric network partitions, by avoiding spurious elections and preventing unavailability where a partially partitioned node could steal leadership away from an established leaseholder who would then no longer be able to reach the leader and submit writes.