m3db-operator updates multiple statefulsets at the same time #268
Comments
|
@schallert @robskillington looks like another instance of the problem #260 was trying to solve. I filtered out the relevant logs below. The operator first updates statefulset 0 and waits for it to become ready: But it doesn't wait after updating statefulset 1: |
schallert
added a commit
that referenced
this issue
Feb 10, 2021
In debugging #268, I discovered that in all cases where we updated more than 1 statefulset, it was because the observed generation on the set was 1 less than the set's actual generation. This means that the Kubernetes statefulset controller had not yet processed the update, and that the information on `Status` that we check (ready replicas, current replicas, etc.) was not up-to-date. Also out of paranoia, I changed the update behavior to ensure that we keep the old object meta fields around when we send the `Update` to Kubernetes. I think if this isn't the case, we potentially overwrite a set even if it had changed since we last process it. This PR ensures that we stop processing the cluster if any of the set statuses are not fresh, and that we use familiar conflict methods on set updated. Closes #268.
schallert
added a commit
that referenced
this issue
Feb 11, 2021
In debugging #268, I discovered that in all cases where we updated more than 1 statefulset, it was because the observed generation on the set was 1 less than the set's actual generation. This means that the Kubernetes statefulset controller had not yet processed the update, and that the information on `Status` that we check (ready replicas, current replicas, etc.) was not up-to-date. Also out of paranoia, I changed the update behavior to ensure that we keep the old object meta fields around when we send the `Update` to Kubernetes. I think if this isn't the case, we potentially overwrite a set even if it had changed since we last process it. This PR ensures that we stop processing the cluster if any of the set statuses are not fresh, and that we use familiar conflict methods on set updated. Closes #268.
|
Hey @nicklan, sorry for any disruption this caused. Unfortunately it ended up being pretty tricky to debug, but we just landed a fix that should help. We'll be testing this internally over the next few days and should have a release out soon. Please let me know any results if you have a chance to test the latest |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
master, please include the git SHA logged when the operator first starts.v0.12.1
kubectl version.Update m3db as described in: https://operator.m3db.io/getting_started/update_cluster/
I updated our m3dbcluster def then ran:
kubectl -n m3 annotate statefulset -l operator.m3db.io/cluster=m3db operator.m3db.io/update=enabledEach statefulset should be updated one at a time
m3db-rep0 ss updated first, as expected, but then the operator updated both m3db-rep1 and m3db-rep2 at the same time, which means pods across replicas are not available and we have downtime.
Here are the logs from the operator up until the issue occurred:
The text was updated successfully, but these errors were encountered: