storage: stuck pre-candidate phase

[irfansharif]

As observed on blue and adriatic this week:

adriatic:

blue:

MsgPreVote{,Resp} counts with later rolling restarts. notably didn’t flatline at a non-zero count (did still end up getting stuck but for a seemingly different reason).

[bdarnell]

Note that in this screenshot with four replicas, the one in the pre-candidate state had been removed but not yet garbage collected. It's expected that a replica in such a state would be in pre-candidate (it would be in the candidate state if pre-vote were disabled), although it's not expected that it would stay this way for long without GC.

On adriatic, we saw many instances of this with only three replicas: one leader, one follower, and one pre-candidate. The pre-candidate node in this case was not far behind the others.

[irfansharif]

+cc #18151.

[irfansharif]

Closing in favor of #18151, +cc etcd-io/etcd#8501. See #18151 (comment):

This sufficiently explains #18129, [...] It aligns with the fact that problem ranges didn't always persist through chaos as with enough leadership changes, replicas stuck in the PreCandidate phase with higher terms were eventually overtaken by healthy replica pairs (with ever increasing terms).