Limit Restart rate for pd and tikv in admission webhook

[Yisaer]

What problem does this PR solve?

Currently, the restarter would restart the pod as soon as possible which would have risks for tikv and pd pod.

In this request, the admission webhook would check if there are any former tikv or pd pod which is annotated with tidb.pingcap.com/pod-defer-deleting when being received the pod delete request with tidb.pingcap.com/pod-defer-deleting annotation too. The request would be rejected if existed.

Does this PR introduce a user-facing change?:

Limit the restart rate for pd and tikv, only one instance would be restarted at each time

[aylei]

please add a release note for this enhancement

[aylei]

Does this need to be cherry-picked to 1.1?

[Yisaer]

please add a release note for this enhancement

This is a non user-facing change. Is this necessary to add it in release note? @aylei

[aylei]

LGTM, please add a descriptive release note

[cofyc]

use helper.GetPodOrdinals to get desired ordinals of pods instead

tidb-operator/pkg/manager/member/tidb_member_manager.go

Line 789 in 0bf6259

for id := range helper.GetPodOrdinals(tc.Status.TiDB.StatefulSet.Replicas, set) {

because the ordinals may not be consecutive with AdvancedStatefulset

[cofyc]

maybe we can list all pods of the statefulset? In this case, I think it's simpler because we need to get pod objects from the API server.

[cofyc]

ah, I'm wrong because we need to check the desired pods and return an error if the pod is not created yet, otherwise, pods can be deleted before the previous replica is recreated.

[Yisaer]

I found that the admission pod webhook only considered the pod controlled by apps.StatefulSet for now. Though it would received the deleting request of advancedStatefulset, there are some places still not support AdvancedStatefulset yet.

tidb-operator/pkg/webhook/pod/util.go

Line 126 in 0bf6259

func getOwnerStatefulSetForTiDBComponent(pod *core.Pod, kubeCli kubernetes.Interface) (*apps.StatefulSet, error) {

I think it should be supported in another issue to take it as one whole problem.

[cofyc]

the kind of AdvancedStatefulset is StatefulSet too.

[cofyc]

the only change here is to use helper.GetPodOrdinals to get the list of ordinals instead of for loop.

[cofyc]

the returned set has a method List() to get a sorted slice in asc order, all ordinals <= ordinal can be ignored in the loop

[Yisaer]

I wonder if advancedStatefulSet is enabled, would the following method in pkg/webhook/pod/util.go get the owner advancedstatefulset ?

func getOwnerStatefulSetForTiDBComponent(pod *core.Pod, kubeCli kubernetes.Interface) (*apps.StatefulSet, error) {
	name := pod.Name
	namespace := pod.Namespace
	var ownerStatefulSetName string
	for _, ownerReference := range pod.OwnerReferences {
		if ownerReference.Kind == "StatefulSet" {
			ownerStatefulSetName = ownerReference.Name
			break
		}
	}
	if len(ownerStatefulSetName) == 0 {
		return nil, fmt.Errorf(failToFindTidbComponentOwnerStatefulset, namespace, name)
	}
	return kubeCli.AppsV1().StatefulSets(namespace).Get(ownerStatefulSetName, meta.GetOptions{})
}

[cofyc]

it will because it does not check the APIGroup (the only different between advanced statefulset with k8s statefulset is the APIGroup)
I'm fine to update later because the admission controller does not support --features flag yet.

[aylei]

LGTM

Please do not include ordered list mark at the begining of release note. Refer to https://github.com/pingcap/tidb-operator/blob/master/docs/release-note-guide.md for release note language guide.

[sre-bot]

cherry pick to release-1.1 in PR #1555