Add pre-upgrade check to test cluster routing allocation is enabled #39340
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When following the steps mentioned in upgrade guide https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html if we disable the cluster shard allocation but fail to enable it after upgrading the nodes and plugins, the next step of upgrading internal indices fails. As we did not check the bulk response for reindexing, we delete the old index assuming it has been created. This is fatal as we cannot recover from this state. This commit adds a pre-upgrade check to test the cluster shard allocation setting and fail upgrade if it is disabled. In case there are search or bulk failures then we remove the read only block and fail the upgrade index request. Closes elastic#39339
bizybot
added
>bug
>upgrade
:Data Management/Indices APIs
|
Pinging @elastic/es-core-features |
albertzaharovits
approved these changes
Mar 4, 2019
| @@ -76,32 +80,61 @@ public void upgrade(TaskId task, String index, ClusterState clusterState, Action | |||
| private void innerUpgrade(ParentTaskAssigningClient parentAwareClient, String index, ClusterState clusterState, | |||
| ActionListener<BulkByScrollResponse> listener) { | |||
| String newIndex = index + "-" + version; | |||
| logger.trace("upgrading index {} to new index {}", index, newIndex); | |||
| try { | |||
| checkMasterAndDataNodeVersion(clusterState); | |||
There was a problem hiding this comment.
This is where I would've put the allocation check, but no need to amend the PR for this. The reason is to keep all the checks in a single place.
imotov
approved these changes
Mar 5, 2019
| @@ -106,4 +118,9 @@ public void upgrade(TaskId task, IndexMetaData indexMetaData, ClusterState state | |||
| ActionListener<BulkByScrollResponse> listener) { | |||
| reindexer.upgrade(task, indexMetaData.getIndex().getName(), state, listener); | |||
| } | |||
|
|
|||
| // pkg scope for testing | |||
| InternalIndexReindexer getInternalIndexReindexer() { | |||
There was a problem hiding this comment.
That concerns me a bit. Could you add a comment why this is necessary?
|
@bizybot I think this should be backported to 5.6 as well. We recommend that users run the |
bizybot
added a commit
to bizybot/elasticsearch
that referenced
this pull request
Mar 8, 2019
…lastic#39340) When following the steps mentioned in upgrade guide https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html if we disable the cluster shard allocation but fail to enable it after upgrading the nodes and plugins, the next step of upgrading internal indices fails. As we did not check the bulk request response for reindexing, we delete the old index assuming it has been created. This is fatal as we cannot recover from this state. This commit adds a pre-upgrade check to test the cluster shard allocation setting and fail upgrade if it is disabled. In case there are search or bulk failures then we remove the read-only block and fail the upgrade index request. Closes elastic#39339
bizybot
added a commit
to bizybot/elasticsearch
that referenced
this pull request
Mar 8, 2019
…lastic#39340) When following the steps mentioned in upgrade guide https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html if we disable the cluster shard allocation but fail to enable it after upgrading the nodes and plugins, the next step of upgrading internal indices fails. As we did not check the bulk request response for reindexing, we delete the old index assuming it has been created. This is fatal as we cannot recover from this state. This commit adds a pre-upgrade check to test the cluster shard allocation setting and fail upgrade if it is disabled. In case there are search or bulk failures then we remove the read-only block and fail the upgrade index request. Closes elastic#39339
bizybot
added a commit
to bizybot/elasticsearch
that referenced
this pull request
Mar 8, 2019
…lastic#39340) When following the steps mentioned in upgrade guide https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html if we disable the cluster shard allocation but fail to enable it after upgrading the nodes and plugins, the next step of upgrading internal indices fails. As we did not check the bulk request response for reindexing, we delete the old index assuming it has been created. This is fatal as we cannot recover from this state. This commit adds a pre-upgrade check to test the cluster shard allocation setting and fail upgrade if it is disabled. In case there are search or bulk failures then we remove the read-only block and fail the upgrade index request. Closes elastic#39339
jasontedor
added a commit
to jasontedor/elasticsearch
that referenced
this pull request
Mar 8, 2019
* elastic/master: Add pre-upgrade check to test cluster routing allocation is enabled (elastic#39340) Update logstash-management.json to use typeless template (elastic#38653) Small simplifications to mapping validation. (elastic#39777) Update distribution build instructions to reflect file names with OS/architecture classifiers. (elastic#39762) Give jspawnhelper execute permissions in bundled JDK (elastic#39787) Maintain step order for ILM trace logging (elastic#39522) [ML-DataFrame] fix wire serialization issues in data frame response objects (elastic#39790) fix index refresh in test within 20_mix_typeless_typeful (elastic#39198) Combine overriddenOps and skippedOps in translog (elastic#39771)
bizybot
added a commit
that referenced
this pull request
Mar 12, 2019
…39340) (#39815) When following the steps mentioned in upgrade guide https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html if we disable the cluster shard allocation but fail to enable it after upgrading the nodes and plugins, the next step of upgrading internal indices fails. As we did not check the bulk request response for reindexing, we delete the old index assuming it has been created. This is fatal as we cannot recover from this state. This commit adds a pre-upgrade check to test the cluster shard allocation setting and fail upgrade if it is disabled. In case there are search or bulk failures then we remove the read-only block and fail the upgrade index request. Closes #39339
bizybot
added a commit
that referenced
this pull request
Mar 12, 2019
…39340) (#39816) When following the steps mentioned in upgrade guide https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html if we disable the cluster shard allocation but fail to enable it after upgrading the nodes and plugins, the next step of upgrading internal indices fails. As we did not check the bulk request response for reindexing, we delete the old index assuming it has been created. This is fatal as we cannot recover from this state. This commit adds a pre-upgrade check to test the cluster shard allocation setting and fail upgrade if it is disabled. In case there are search or bulk failures then we remove the read-only block and fail the upgrade index request. Closes #39339
bizybot
added a commit
that referenced
this pull request
Mar 12, 2019
…39340) (#39817) When following the steps mentioned in upgrade guide https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html if we disable the cluster shard allocation but fail to enable it after upgrading the nodes and plugins, the next step of upgrading internal indices fails. As we did not check the bulk request response for reindexing, we delete the old index assuming it has been created. This is fatal as we cannot recover from this state. This commit adds a pre-upgrade check to test the cluster shard allocation setting and fail upgrade if it is disabled. In case there are search or bulk failures then we remove the read-only block and fail the upgrade index request. Closes #39339
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When following the steps mentioned in upgrade guide
https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html
if we disable the cluster shard allocation but fail to enable it after
upgrading the nodes and plugins, the next step of upgrading internal
indices fails. As we did not check the bulk request response for reindexing,
we delete the old index assuming it has been created. This is fatal
as we cannot recover from this state.
This commit adds a pre-upgrade check to test the cluster shard
allocation setting and fail upgrade if it is disabled. In case there
are search or bulk failures then we remove the read-only block and
fail the upgrade index request.
Closes #39339