Fix cluster health when closing #61709
Conversation
When master shuts down it's cluster service, a waiting health request would fail rather than fail over to a new master.
henningandersen
added
>bug
:Distributed Indexing/Distributed
|
Pinging @elastic/es-distributed (:Distributed/Distributed) |
elasticmachine
added
the
Team:Distributed (Obsolete)
|
@elasticmachine run elasticsearch-ci/1 |
|
Failure should be fixed by #62061 |
…when_master_closing
| boolean withIndex = randomBoolean(); | ||
| if (withIndex) { | ||
| // create index with many shards to provoke the health request to wait (for green) while master is being shut down. | ||
| createIndex("test", Settings.builder().put(IndexMetadata.SETTING_NUMBER_OF_REPLICAS, randomIntBetween(0, 10)).build()); |
There was a problem hiding this comment.
I'm confused. How can the cluster ever be green with that many replicas? Should this be number_of_shards?
There was a problem hiding this comment.
It ensures that the cluster is yellow when the health request is made, making the health request wait on the observer, triggering the call to onClusterServiceClose when master is shutdown.
The number of replicas is cleared to 0 after having fired all the async restarts and done the master restarts. That ensures that all the requests responds with green status.
There was a problem hiding this comment.
ah ok, can you add a comment to that effect
…when_master_closing
|
@elasticmachine run elasticsearch-ci/packaging-sample-windows |
When master shuts down it's cluster service, a waiting health request would fail rather than fail over to a new master.
When master shuts down it's cluster service, a waiting health request
would fail rather than fail over to a new master.
Fails around once every month in CI:
https://build-stats.elastic.co/goto/49d044995e1b636bfc3cb7e5e2371f3a
For instance:
https://gradle-enterprise.elastic.co/s/r2rocz2i6tah2