When the user manually modify tikv/tidb/pd sts casually, it may cause tidb-operator crash

[onlymellb]

Bug Report

What version of Kubernetes are you using?

v1.12.2

What version of TiDB Operator are you using?

This problem exists in all tidb-operator versions

What did you do?

Skip tidb-operator, modify the updateStrategy of any sts in tikv/tidb/pd from RollingUpdate to OnDelete manually.

What did you expect to see?
All running tidb clusters remain normal and do not affect the add, delete, and update operations of other tidb clusters.

What did you see instead?
Tidb-operator crash and affect all other tidb clusters add, delete, and update operations. This is a serious bug because a user modifying the resources own by himself will affect all users.

I0902 09:54:57.340325       1 event.go:221] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"general-release21-ddl-test-exp261-cat0-tidb-cluster", Name:"tidb-cluster", UID:"bdc88cbc-cd14-11e9-b3f4-4cd9
8f4bb97e", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"160724078", FieldPath:""}): type: 'Normal' reason: 'SuccessfulUpdate' update TidbCluster tidb-cluster successful
E0902 09:54:57.377183       1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/home/pingcap/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20181128191346-49ce2735e507/pkg/util/runtime/runtime.go:76
/home/pingcap/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20181128191346-49ce2735e507/pkg/util/runtime/runtime.go:65
/home/pingcap/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20181128191346-49ce2735e507/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/panic.go:522
/usr/local/go/src/runtime/panic.go:82
/usr/local/go/src/runtime/signal_unix.go:390
/home/pingcap/go/src/github.com/pingcap/tidb-operator/pkg/manager/member/tikv_upgrader.go:74
/home/pingcap/go/src/github.com/pingcap/tidb-operator/pkg/manager/member/tikv_member_manager.go:195
/home/pingcap/go/src/github.com/pingcap/tidb-operator/pkg/manager/member/tikv_member_manager.go:116
/home/pingcap/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_control.go:121
/home/pingcap/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_control.go:73
/home/pingcap/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:273
/home/pingcap/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:269
/home/pingcap/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:236
/home/pingcap/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:223
/home/pingcap/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20181128191346-49ce2735e507/pkg/util/wait/wait.go:133
/home/pingcap/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20181128191346-49ce2735e507/pkg/util/wait/wait.go:134
/home/pingcap/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20181128191346-49ce2735e507/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:1337
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x149761d]

goroutine 300 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /home/pingcap/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20181128191346-49ce2735e507/pkg/util/runtime/runtime.go:58 +0x105
panic(0x169dc40, 0x291fb50)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/pingcap/tidb-operator/pkg/manager/member.(*tikvUpgrader).Upgrade(0xc0000f68d0, 0xc0019c8d80, 0xc004658900, 0xc000efe480, 0x0, 0x0)
        /home/pingcap/go/src/github.com/pingcap/tidb-operator/pkg/manager/member/tikv_upgrader.go:74 +0x1cd
github.com/pingcap/tidb-operator/pkg/manager/member.(*tikvMemberManager).syncStatefulSetForTidbCluster(0xc000426160, 0xc0019c8d80, 0x18bc618, 0x4)
        /home/pingcap/go/src/github.com/pingcap/tidb-operator/pkg/manager/member/tikv_member_manager.go:195 +0x402
github.com/pingcap/tidb-operator/pkg/manager/member.(*tikvMemberManager).Sync(0xc000426160, 0xc0019c8d80, 0x0, 0x0)
        /home/pingcap/go/src/github.com/pingcap/tidb-operator/pkg/manager/member/tikv_member_manager.go:116 +0x1c1
github.com/pingcap/tidb-operator/pkg/controller/tidbcluster.(*defaultTidbClusterControl).updateTidbCluster(0xc00017a600, 0xc0019c8d80, 0x0, 0x0)
        /home/pingcap/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_control.go:121 +0xde
github.com/pingcap/tidb-operator/pkg/controller/tidbcluster.(*defaultTidbClusterControl).UpdateTidbCluster(0xc00017a600, 0xc0019c8d80, 0x0, 0x0)
        /home/pingcap/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_control.go:73 +0x8f
github.com/pingcap/tidb-operator/pkg/controller/tidbcluster.(*Controller).syncTidbCluster(...)
        /home/pingcap/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:273
github.com/pingcap/tidb-operator/pkg/controller/tidbcluster.(*Controller).sync(0xc0001c7a40, 0xc00194a1c0, 0xe, 0x0, 0x0)
        /home/pingcap/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:269 +0x1c2
github.com/pingcap/tidb-operator/pkg/controller/tidbcluster.(*Controller).processNextWorkItem(0xc0001c7a40, 0xc0007aa000)
        /home/pingcap/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:236 +0xea
github.com/pingcap/tidb-operator/pkg/controller/tidbcluster.(*Controller).worker(0xc0001c7a40)
        /home/pingcap/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:223 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0017c79b0)
        /home/pingcap/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20181128191346-49ce2735e507/pkg/util/wait/wait.go:133 +0x54
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0017c79b0, 0x3b9aca00, 0x0, 0x1, 0xc00149a060)
        /home/pingcap/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20181128191346-49ce2735e507/pkg/util/wait/wait.go:134 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(0xc0017c79b0, 0x3b9aca00, 0xc00149a060)
        /home/pingcap/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20181128191346-49ce2735e507/pkg/util/wait/wait.go:88 +0x4d
created by github.com/pingcap/tidb-operator/pkg/controller/tidbcluster.(*Controller).Run
        /home/pingcap/go/src/github.com/pingcap/tidb-operator/pkg/controller/tidbcluster/tidb_cluster_controller.go:215 +0x1cc