Suggestion: control over leader identity

[shlomi-noach]

I find it convenient that I could have control over the identity of the leader in a stable group. For example, I may be running cross-DC nodes (in an attempt to fight fencing), and would prefer the leader to normally run from a specific, local, "active" DC to where most operations are running.

Or, as another example, I might have a 3-node group, but only 2 are full blown provisioned machines, and the 3rd is only a raft mediator.

Ideally, I would like to be able to point to a specific node and ask "become the leader". I can see various ways to achieve this:

• ask a leader to step down (perhaps re-elections would pick it again, perhaps not)
• configure a/some nodes to never even attempt entering Candidate state (up to quorum - 1 nodes are safe to run this)
• instruct a/some nodes to refrain from entering Candidate state for a bound period of time

I will try some pull-requests to support these ideas. I'm aware that my understanding of Raft is limited, and will work with what makes sense to me following existing use patterns.

Related: #211

[shlomi-noach]

I'm happy to receive any feedback on the idea in general and on the proposed PRs in particular. Thank you!

[shlomi-noach]

#222 seems to be working very well on my environment.

[superfell]

Have you seen the v2 branch which has support for non-voting members? it might be a better fit for the problem you're trying to solve.

[shlomi-noach]

@superfell thank you, I was not aware of v2. Some notes and questions, if you please:

1. AddNonVoter() seems to deprecate the idea behind NeverLeader config: node will never switch to Candidate  #220
2. If I run DemoteVoter() with the leader's ID,

• does the leader step down? (a-la StepDown(): requesting reelections #219)
• Or will it keep leading, and will not participate in next elections?

3. I do not see a BecomeVoter() or similar, the anti-operation of DemoteVoter() -- is it possible to turn a non-voter to be a voter?

My current implementation/use allows for something like: https://myservice/api/raft-yield/1.2.3.4:10008

This is invoked on the leader, and the leader sends a yield message to all nodes (obviously including itself). Any node that isn't 1.2.3.4:10008 temporarily suspends elections for a limited time, and the leader (if not said server) steps down. This makes 1.2.3.4:10008 the only contender for leadership, and it succesfuly becomes the leader. If anything bad happens and it does not assume leadership, the suspension expires and all other nodes are back in the election cycle.
This logic happens with the assistance of the app, since the yield message goes as a raft log message, to be interpreted by the app.

I'm unsure whether v2 will allow similar behavior out of the box; it appears to me it will similarly require app's intervention.

[superfell]

2. If you DemoteVoter() the leader, the leader will step down at the appropriate time, see https://github.com/hashicorp/raft/blob/library-v2-stage-one/raft.go#L509

3. IIRC AddVoter should promote a non-voter to a voter.

[shlomi-noach]

If I read correctly between the lines you will not merge #222 ; if this is the case, please let me know and I will close it.

Having said that I'm running this in production and very easily controlling the identity of the raft cluster's leader.

e.g. I have a cronjob running every 10 minutes checking out my production environment, figuring out which is the "active" (aka "main") data center ; it then looks at my raft cluster:

• is the leader in the active datacenter?
• if not, is there a follower in the active datacenter?
• if so, ask all raft nodes to yield() to that follower:
  • all nodes will suspend their attempts to become leaders
  • the aforementioned follower will recognize it is the one suggested as leader, so will not suspend
  • it will be the only once to suggest itself as leader hence win
  • pizza

This is done with minimal changes to the existing code.

Thank you for your consideration.

[shlomi-noach]

Gentle ping. Please let me know if you at all consider this for inclusion or not.

[preetapan]

@shlomi-noach Sorry for the delayed response.

We've made quite a few changes in library-v2-stage-one for Consul/Nomad including improvements like non voting servers, proper fsyncing of snapshots, the new address provider abstraction, and other improvements. As you can imagine, it is no fun keeping changes in sync across three different branches. We are planning to tag the current master branch as 0.1.0, and then merge library-v2-stage-one into master and tag that as 1.0.0. This will happen sometime next week.

We discussed your proposal internally, and like @superfell points out, AddVoter and DemoteVoter help you achieve the same results. Note that both methods are implemented as configuration change requests, and so calling DemoteVoter on the leader will cause it to step down, and it can only become a candidate (or subsequently a leader) again via AddVoter. Feel free to try out the v2-stage-one branch for this, or by early next week you'll be able to get this in the master branch.

Hope this helps.

[shlomi-noach]

@preetapan thank you for your review!