Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[1.8] StatefulSet initialized annotation is now ignored. #4426

Merged
merged 1 commit into from
Aug 10, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 7 additions & 40 deletions docs/tasks/debug-application-cluster/debug-stateful-set.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,51 +28,18 @@ This task shows you how to debug a StatefulSet.

## Debugging a StatefulSet

In order to list all the pods which belong to a StatefulSet, which have a label `app=myapp` set on them, you can use the following:
In order to list all the pods which belong to a StatefulSet, which have a label `app=myapp` set on them,
you can use the following:

```shell
kubectl get pods -l app=myapp
```

If you find that any Pods listed are in `Unknown` or `Terminating` state for an extended period of time, refer to the [Deleting StatefulSet Pods](/docs/tasks/manage-stateful-set/delete-pods/) task for instructions on how to deal with them. You can debug individual Pods in a StatefulSet using the [Debugging Pods](/docs/user-guide/debugging-pods-and-replication-controllers/#debugging-pods) guide.

StatefulSets provide a debug mechanism to pause all controller operations on Pods using an annotation. Setting the `pod.alpha.kubernetes.io/initialized` annotation to `"false"` on any StatefulSet Pod will *pause* all operations of the StatefulSet. When paused, the StatefulSet will not perform any scaling operations. Once the debug hook is set, you can execute commands within the containers of StatefulSet pods without interference from scaling operations. You can set the annotation to `"false"` by executing the following:

```shell
kubectl annotate pods <pod-name> pod.alpha.kubernetes.io/initialized="false" --overwrite
```

When the annotation is set to `"false"`, the StatefulSet will not respond to its Pods becoming unhealthy or unavailable. It will not create replacement Pods till the annotation is removed or set to `"true"` on each StatefulSet Pod.

### Step-wise Initialization

You can also use the same annotation to debug race conditions during bootstrapping of the StatefulSet by setting the `pod.alpha.kubernetes.io/initialized` annotation to `"false"` in the `.spec.template.metadata.annotations` field of the StatefulSet prior to creating it.

```yaml
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: my-app
spec:
serviceName: "my-app"
replicas: 3
template:
metadata:
labels:
app: my-app
annotations:
pod.alpha.kubernetes.io/initialized: "false"
...
...
...

```

After setting the annotation, if you create the StatefulSet, you can wait for each Pod to come up and verify that it has initialized correctly. The StatefulSet will not create any subsequent Pods till the debug annotation is set to `"true"` (or removed) on each Pod that has already been created. You can set the annotation to `"true"` by executing the following:

```shell
kubectl annotate pods <pod-name> pod.alpha.kubernetes.io/initialized="true" --overwrite
```
If you find that any Pods listed are in `Unknown` or `Terminating` state for an extended period of time,
refer to the [Deleting StatefulSet Pods](/docs/tasks/manage-stateful-set/delete-pods/) task for
instructions on how to deal with them.
You can debug individual Pods in a StatefulSet using the
[Debugging Pods](/docs/tasks/debug-application-cluster/debug-pod-replication-controller/) guide.

{% endcapture %}

Expand Down
16 changes: 14 additions & 2 deletions docs/tasks/run-application/upgrade-pet-set-to-stateful-set.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,12 @@ Here are some notable changes:

* **StatefulSet is the new PetSet**: PetSet is no longer available in Kubernetes release 1.5 or later. It becomes beta StatefulSet. To understand why the name was changed, see this [discussion thread](https://github.com/kubernetes/kubernetes/issues/27430).
* **StatefulSet guards against split brain**: StatefulSets guarantee at most one Pod for a given ordinal index can be running anywhere in a cluster, to guard against split brain scenarios with distributed applications. *TODO: Link to doc about fencing.*
* **Flipped debug annotation behavior**: The default value of the debug annotation (`pod.alpha.kubernetes.io/initialized`) is now `true`. The absence of this annotation will pause PetSet operations, but will NOT pause StatefulSet operations. In most cases, you no longer need this annotation in your StatefulSet manifests.
* **Flipped debug annotation behavior**:
The default value of the debug annotation (`pod.alpha.kubernetes.io/initialized`) is `true` in 1.5 through 1.7.
The annotation is completely ignored in 1.8 and above, which always behave as if it were `true`.

The absence of this annotation will pause PetSet operations, but will NOT pause StatefulSet operations.
In most cases, you no longer need this annotation in your StatefulSet manifests.


## Upgrading from PetSets to StatefulSets
Expand Down Expand Up @@ -66,7 +71,14 @@ Now, for every PetSet manifest you have, prepare a corresponding StatefulSet man

1. Change `apiVersion` from `apps/v1alpha1` to `apps/v1beta1`.
2. Change `kind` from `PetSet` to `StatefulSet`.
3. If you have the debug hook annotation `pod.alpha.kubernetes.io/initialized` set to `true`, you can remove it because it's redundant. If you don't have this annotation, you should add one, with the value set to `false`, to pause StatefulSets operations.
3. If you have the debug hook annotation `pod.alpha.kubernetes.io/initialized` set to `true`,
you can remove it because it's redundant.
If you don't have this annotation or have it set to `false`,
be aware that StatefulSet operations might resume after the upgrade.

If you are upgrading to 1.6 or 1.7, you can set the annotation explicitly to `false` to maintain
the paused behavior.
If you are upgrading to 1.8 or above, there's no longer any debug annotation to pause StatefulSets.

It's recommended that you keep both PetSet manifests and StatefulSet manifests, so that you can safely roll back and recreate your PetSets,
if you decide not to upgrade your cluster.
Expand Down