Master election should demotes nodes which try to join the cluster for the first time #7558
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bleskes
force-pushed
the
master_prefer_non_initial_join
branch
2 times, most recently
from
399117d to
d33b937
Compare
bleskes
force-pushed
the
master_prefer_non_initial_join
branch
from
d33b937 to
32083c5
Compare
|
@kimchy I pushed another update based on our discussion. |
| @@ -139,6 +140,9 @@ | |||
|
|
|||
| private volatile boolean rejoinOnMasterGone; | |||
|
|
|||
| // will be set to true upon the first successful cluster join | |||
| private final AtomicBoolean hasJoinedClusterOnce = new AtomicBoolean(); | |||
There was a problem hiding this comment.
not sure if it will help but maybe make that an integer such that we can log how often it did join it might help when debugging? just an idea
There was a problem hiding this comment.
It might.. won't hurt. I'll add it.
|
I left some comments @bleskes |
…to have extra trace info. Change pingRensponse.target to pingResponse.node, for clarity. Added comments and docs
clintongormley
changed the title
|
LGTM |
|
Re-opening as there is some BWC work to be done for 1.4 |
bleskes
added a commit
to bleskes/elasticsearch
that referenced
this pull request
Sep 12, 2014
…same node elastic#5413 introduced a change where we prefer ping responses containing a master over those who don't. The same change changes the preference of acceptance if both pings have a master indication or if neither do. elastic#7558 added new flag to the PingResponse which changes after a node has joined the cluster for the very first time. Giving preference to older pings cause the wrong value of this flag to be used. This commit restores the preference to the original one.
bleskes
added a commit
to bleskes/elasticsearch
that referenced
this pull request
Sep 16, 2014
…e cluster for the first time With the change in elastic#7493, we introduced a pinging round when a master nodes goes down. That pinging round helps validating the current state of the cluster and takes, by default, 3 seconds. It may be that during that window, a new node tries to join the cluster and starts pinging (this is typical when you quickly restart the current master). If this node gets elected as the new master it will force recovery from the gateway (it has no in memory cluster state), which in turn will cause a full cluster shard synchronisation. While this is not a problem on it's own, it's a shame. This commit demotes "new" nodes during master election so the will only be elected if really needed. Closes elastic#7558
bleskes
added a commit
to bleskes/elasticsearch
that referenced
this pull request
Sep 16, 2014
…he new ping on master gone introduced in elastic#7493 The change in elastic#7558 adds a flag to PingResponse. However, when unicast discovery is used, this extra flag can not be serialized by the very initial pings as they do not know yet what node version they ping (i.e., they have to default to 1.0.0, which excludes changing the serialization format). This commit bypasses this problem by adding a dedicated action which only exist on nodes of version 1.4 or up. Nodes first try to ping this endpoint using 1.4.0 as a serialization version. If that fails they fall back to the pre 1.4.0 action. This is optimal if all nodes are on 1.4.0 or higher, with a small down side if the cluster has mixed versions - but this is a temporary state. Further two bwc protections are added: 1) Disable the preference to nodes who previously joined the cluster if some of the pings are on version < 1.4.0 2) Disable the rejoin on master gone functionality if some nodes in the cluster or version < 1.4.0
bleskes
added a commit
to bleskes/elasticsearch
that referenced
this pull request
Sep 16, 2014
…e cluster for the first time With the change in elastic#7493, we introduced a pinging round when a master nodes goes down. That pinging round helps validating the current state of the cluster and takes, by default, 3 seconds. It may be that during that window, a new node tries to join the cluster and starts pinging (this is typical when you quickly restart the current master). If this node gets elected as the new master it will force recovery from the gateway (it has no in memory cluster state), which in turn will cause a full cluster shard synchronisation. While this is not a problem on it's own, it's a shame. This commit demotes "new" nodes during master election so the will only be elected if really needed. Closes elastic#7558
bleskes
added a commit
to bleskes/elasticsearch
that referenced
this pull request
Sep 16, 2014
…he new ping on master gone introduced in elastic#7493 The change in elastic#7558 adds a flag to PingResponse. However, when unicast discovery is used, this extra flag can not be serialized by the very initial pings as they do not know yet what node version they ping (i.e., they have to default to 1.0.0, which excludes changing the serialization format). This commit bypasses this problem by adding a dedicated action which only exist on nodes of version 1.4 or up. Nodes first try to ping this endpoint using 1.4.0 as a serialization version. If that fails they fall back to the pre 1.4.0 action. This is optimal if all nodes are on 1.4.0 or higher, with a small down side if the cluster has mixed versions - but this is a temporary state. Further two bwc protections are added: 1) Disable the preference to nodes who previously joined the cluster if some of the pings are on version < 1.4.0 2) Disable the rejoin on master gone functionality if some nodes in the cluster or version < 1.4.0 Closes elastic#7694
bleskes
added a commit
that referenced
this pull request
Sep 16, 2014
…e cluster for the first time With the change in #7493, we introduced a pinging round when a master nodes goes down. That pinging round helps validating the current state of the cluster and takes, by default, 3 seconds. It may be that during that window, a new node tries to join the cluster and starts pinging (this is typical when you quickly restart the current master). If this node gets elected as the new master it will force recovery from the gateway (it has no in memory cluster state), which in turn will cause a full cluster shard synchronisation. While this is not a problem on it's own, it's a shame. This commit demotes "new" nodes during master election so the will only be elected if really needed. Closes #7558
bleskes
added a commit
that referenced
this pull request
Sep 16, 2014
…ping on master gone introduced in #7493 The change in #7558 adds a flag to PingResponse. However, when unicast discovery is used, this extra flag can not be serialized by the very initial pings as they do not know yet what node version they ping (i.e., they have to default to 1.0.0, which excludes changing the serialization format). This commit bypasses this problem by adding a dedicated action which only exist on nodes of version 1.4 or up. Nodes first try to ping this endpoint using 1.4.0 as a serialization version. If that fails they fall back to the pre 1.4.0 action. This is optimal if all nodes are on 1.4.0 or higher, with a small down side if the cluster has mixed versions - but this is a temporary state. Further two bwc protections are added: 1) Disable the preference to nodes who previously joined the cluster if some of the pings are on version < 1.4.0 2) Disable the rejoin on master gone functionality if some nodes in the cluster or version < 1.4.0 Closes #7694
|
this is now back ported to 1.x & 1.4 |
clintongormley
changed the title
mute
pushed a commit
to mute/elasticsearch
that referenced
this pull request
Jul 29, 2015
…e cluster for the first time With the change in elastic#7493, we introduced a pinging round when a master nodes goes down. That pinging round helps validating the current state of the cluster and takes, by default, 3 seconds. It may be that during that window, a new node tries to join the cluster and starts pinging (this is typical when you quickly restart the current master). If this node gets elected as the new master it will force recovery from the gateway (it has no in memory cluster state), which in turn will cause a full cluster shard synchronisation. While this is not a problem on it's own, it's a shame. This commit demotes "new" nodes during master election so the will only be elected if really needed. Closes elastic#7558
mute
pushed a commit
to mute/elasticsearch
that referenced
this pull request
Jul 29, 2015
…he new ping on master gone introduced in elastic#7493 The change in elastic#7558 adds a flag to PingResponse. However, when unicast discovery is used, this extra flag can not be serialized by the very initial pings as they do not know yet what node version they ping (i.e., they have to default to 1.0.0, which excludes changing the serialization format). This commit bypasses this problem by adding a dedicated action which only exist on nodes of version 1.4 or up. Nodes first try to ping this endpoint using 1.4.0 as a serialization version. If that fails they fall back to the pre 1.4.0 action. This is optimal if all nodes are on 1.4.0 or higher, with a small down side if the cluster has mixed versions - but this is a temporary state. Further two bwc protections are added: 1) Disable the preference to nodes who previously joined the cluster if some of the pings are on version < 1.4.0 2) Disable the rejoin on master gone functionality if some nodes in the cluster or version < 1.4.0 Closes elastic#7694
clintongormley
added
:Distributed Indexing/Distributed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With the change in #7493 , we introduced a pinging round when a master nodes goes down. That pinging round helps validating the current state of the cluster and takes, by default, 3 seconds. It may be that during that window, a new node tries to join the cluster and starts pinging (this is typical when you quickly restart the current master). If this node gets elected as the new master it will force recovery from the gateway (it has no in memory cluster state), which in turn will cause a full cluster shard synchronisation. While this is not a problem on it's own, it's a shame. This commit demotes "new" nodes during master election so the will only be elected if really needed.