Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Voting Only Master Node #14340

Closed
pickypg opened this issue Oct 28, 2015 · 5 comments · Fixed by #43410
Closed

Voting Only Master Node #14340

pickypg opened this issue Oct 28, 2015 · 5 comments · Fixed by #43410
Labels
:Distributed Coordination/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. >enhancement high hanging fruit

Comments

@pickypg
Copy link
Member

pickypg commented Oct 28, 2015

The current ideal setup for master nodes is to have 3 master nodes, thus giving high availability (HA) with a quorum of 2 master nodes at any given time.

However, in some cloud regions (and more frequently in internal data centers), this creates a problem because there may only be two available zones to place your nodes. In these scenarios, you do not want to give up HA just to maintain 3 master nodes (to be clear: it's better than not doing it, but it's less than ideal) because it means that you have a lopsided HA environment where one "half" cannot actually survive without the other.

The only real solution to this problem is to have a third zone, even if that third zone is a slightly higher latency (across the globe is not going to work), then plunk a master node into that zone to give a single master node per zone -- thus enabling the actual survivability of any single zone, but not of two zones.

Now, you could do this today with a master node and things would be generally okay except that the master node with the added latency may become the elected master node, which is obviously not ideal.

This is where a voting-only master node improves this setup significantly. If you have a master node that can itself not be elected, but that participates in elections only, then you can have a tiny node (careful with VM size due to automatic network throttling in cloud ecosystems!) that help to survive the loss of any single zone without seriously risking performance.

@pickypg pickypg added >enhancement :Distributed Coordination/Discovery-Plugins Anything related to our integration plugins with EC2, GCP and Azure labels Oct 28, 2015
@bleskes
Copy link
Contributor

bleskes commented Oct 28, 2015

This is definitely on our horizon. We need to do some infra work before we get there though... on it :)

@bertrandfalguiere
Copy link

I'm new to ES, but won't it be better and maybe technically easier to give "points" to the master eligible nodes? Those points would give priority to the best ranked node to become master.
If zones A and B have high availability, but not zone C, you could give:
"node.master_eligibility" : 3 for your master eligible node A,
"node.master_eligibility" : 2 for your master eligible node B,
"node.master_eligibility" : 1 for your laggy master eligible node C.
If you lose A, B is elected if needed, and vice versa.
It would have have the same effect of avoiding C to become master.

And (new case) if you have a fast A and slow B and C, it means that B and C can take over if the cluster lost A, but as soon as the fast A node is back online, it takes its mastership back.

It probably means changing the election process, but no new node status is needed.

Does it make sense? Again i'm new to all this.

@bleskes
Copy link
Contributor

bleskes commented Jun 9, 2016

@bertrandfalguiere no worries and welcome to the repo.

If I read you correctly, you want to have some mechanics to influence master election and indicate you prefer one master over another. Putting all the trickiness of a distributed master election aside (it's not easy to add these things correctly), the main thing to realize here that you need to have a good working master at any given moment. Otherwise your cluster is in a big risk. So have a slow master and a fast master doesn't make sense - you need two fast masters etc.

PS. We try to keep github for concrete issues. For more discussion-y things we ask people to go to https://discuss.elastic.co

@DaveCTurner DaveCTurner added the :Distributed Coordination/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. label Mar 27, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@seang-es
Copy link

In some cloud use cases we might have a two zone deployment with a third zone containing only a tiebreaker master node. By default, this will be configured with 1 GB RAM, and may not be suitable for larger clusters. If this could be configured as a voting-only node, then it could provide quorum without becoming an active master.

ywelsch added a commit that referenced this issue Jun 25, 2019
A voting-only master-eligible node is a node that can participate in master elections but will not act
as a master in the cluster. In particular, a voting-only node can help elect another master-eligible
node as master, and can serve as a tiebreaker in elections. High availability (HA) clusters require at
least three master-eligible nodes, so that if one of the three nodes is down, then the remaining two
can still elect a master amongst them-selves. This only requires one of the two remaining nodes to
have the capability to act as master, but both need to have voting powers. This means that one of
the three master-eligible nodes can be made as voting-only. If this voting-only node is a dedicated
master, a less powerful machine or a smaller heap-size can be chosen for this node. Alternatively, a
voting-only non-dedicated master node can play the role of the third master-eligible node, which
allows running an HA cluster with only two dedicated master nodes.

Closes #14340

Co-authored-by: David Turner <david.turner@elastic.co>
ywelsch added a commit that referenced this issue Jun 26, 2019
A voting-only master-eligible node is a node that can participate in master elections but will not act
as a master in the cluster. In particular, a voting-only node can help elect another master-eligible
node as master, and can serve as a tiebreaker in elections. High availability (HA) clusters require at
least three master-eligible nodes, so that if one of the three nodes is down, then the remaining two
can still elect a master amongst them-selves. This only requires one of the two remaining nodes to
have the capability to act as master, but both need to have voting powers. This means that one of
the three master-eligible nodes can be made as voting-only. If this voting-only node is a dedicated
master, a less powerful machine or a smaller heap-size can be chosen for this node. Alternatively, a
voting-only non-dedicated master node can play the role of the third master-eligible node, which
allows running an HA cluster with only two dedicated master nodes.

Closes #14340

Co-authored-by: David Turner <david.turner@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed Coordination/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. >enhancement high hanging fruit
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants