Perform forced rolling upgrade even if ES is reachable #2022
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There are cases where Elasticsearch is reachable (some Pods are Ready), but cannot respond to any requests. For example, if there is 1/2 master nodes available. See elastic#1847. In such case, the bootlooping/pending 2nd master node will stay stuck forever since we will never reach the force upgrade part of the reconciliation. This commit fixes it by running force upgrades (if required) right after the upscale/spec change phase. This force upgrade phase becomes the new "Step 2". Following steps (downscale and regular upgrade) require the Elasticsearch cluster to be reachable. Due to how this force rolling upgrade deletes some pods and set some expectations, I chose to requeue immediately if it was attempted. This way we don't continue the reconciliation based on a transient state that would require us re-checking expectations. The next reconciliation can be a "regular" one. I think this also tends to simplify a bit the general logic: we first do everything that does not require the ES API (steps 1 and 2), then move on with downscales and standard rolling upgrades if ES is reachable (steps 3 and 4); instead of passing an `esReachable` bool around.
thbkrkr
reviewed
Oct 22, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There are cases where Elasticsearch is reachable
(some Pods are Ready), but cannot respond to any requests.
For example, if there is 1/2 master nodes available. See
#1847. In such case,
the bootlooping/pending 2nd master node will stay stuck forever since we
will never reach the force upgrade part of the reconciliation.
This commit fixes it by running force upgrades (if required) right after
the upscale/spec change phase. This force upgrade phase becomes the new
"Step 2". Following steps (downscale and regular upgrade) require the
Elasticsearch cluster to be reachable.
Due to how this force rolling upgrade deletes some pods and set some
expectations, I chose to requeue immediately if it was attempted. This
way we don't continue the reconciliation based on a transient state
that would require us re-checking expectations. The next reconciliation
can be a "regular" one.
I think this also tends to simplify a bit the general logic: we first do
everything that does not require the ES API (steps 1 and 2), then move
on with downscales and standard rolling upgrades if ES is reachable
(steps 3 and 4); instead of passing an
esReachablebool around.The PR also modifies the existing force upgrade E2E test so that it covers
the case where Elasticsearch can be reached, but responds with 503.
The new version of the test does not pass with the master branch, but
does pass with this branch.
Fixes #1847.