tikv: fix panic when recycle non-superbatch idle conn (#16299)

[sre-bot]

cherry-pick #16299 to release-4.0

---

What problem does this PR solve?

Problem Summary:

tidb-tikv will use superbatch conn as default, but "cluster table" will establish non-superbatch connection between tidb-tidb.

tidb/store/tikv/client.go

Line 322 in af0a49c

enableBatch := req.StoreTp != kv.TiDB

and superbatch due use streaming, it has recycle idle conn logic to avoid goroutine leak.

but "recycle idle conn logic" doesn't work well when conns map mix superbatch and non-superbatch conn, and will meet segv fault.

[2020/04/10 14:56:42.816 +08:00] [ERROR] [coprocessor.go:657] ["copIteratorWork meet panic"] [r="\"invalid memory address or nil pointer dereference\""] ["stack trace"="github.com/pingcap/tidb/store/tikv.(*copIteratorWorker).handleTask.func1\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:659\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:679\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:199\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:394\ngithub.com/pingcap/tidb/store/tikv.(*rpcClient).recycleDieConnArray\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/client_batch.go:658\ngithub.com/pingcap/tidb/store/tikv.(*rpcClient).recycleDieConnArray\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/client_batch.go:658\ngithub.com/pingcap/tidb/store/tikv.(*rpcClient).SendRequest\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/client.go:319\ngithub.com/pingcap/tidb/store/tikv.(*RegionRequestSender).sendReqToRegion\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/region_request.go:199\ngithub.com/pingcap/tidb/store/tikv.(*RegionRequestSender).SendReqCtx\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/region_request.go:162\ngithub.com/pingcap/tidb/store/tikv.(*clientHelper).SendReqCtx\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:814\ngithub.com/pingcap/tidb/store/tikv.(*copIteratorWorker).handleTaskOnce\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:729\ngithub.com/pingcap/tidb/store/tikv.(*copIteratorWorker).handleTask\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:667\ngithub.com/pingcap/tidb/store/tikv.(*copIteratorWorker).run\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:489"]

What is changed and how it works?

What's Changed:

only recycle superbatch conns

How it Works:

only recycle superbatch conns

Related changes

• Need to cherry-pick to the release branch

Check List

Tests

• Manual test (add detailed scripts or steps below)

setup a tidb cluster with multi-tidb instances, and remove a tikv store, then continue give pressure on tidb to trigger "recycle conn logic"

Side effects

• n/a

Release note

• tikv: fix panic when recycle non-superbatch idle conn between tidb instances

---

This change is 

[sre-bot]

/run-all-tests

[lysu]

/build

[zz-jason]

LGTM

[jackysp]

LGTM

[jackysp]

/merge