tidb-operator fails to delete discovery annotations and labels according to updated CR

[hoyhbx]

Bug Report

What version of Kubernetes are you using?

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:41:01Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.9", GitCommit:"6df4433e288edc9c40c2e344eb336f63fad45cd2", GitTreeState:"clean", BuildDate:"2022-05-19T19:53:08Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

What version of TiDB Operator are you using?

pingcap/tidb-operator:v1.3.2

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

default

What's the status of the TiDB cluster pods?

What did you do?

We first added a label key: value to the discovery service, the label is successfully added to the discovery service. Then later we wanted to remove that label from the discovery service by changing the CR, we found that tidb-operator cannot remove the labels.

What did you expect to see?
labels and annotations in spec.discovery can be added and removed.

What did you see instead?
The tidb-operator fails to remove labels and annotations from discovery service when users specifies the removal in the CR.

[csuzhangxc]

discovery is mainly used for bootstrapping the TidbCluster, so some fields are not well-supported yet. What scenarios do you need to update them?

[hoyhbx]

For example we want to perform some fault injection experiment on Services, and the label is used to select the Services. We later want to remove the label from the discovery and found that the removal cannot be done. Right now we have to manually remove them

[csuzhangxc]

I got it. Could you help to add labels and annotations update for the discovery service?