kv/client: fix gRPC connection pool, don't close conn when meeting error

[amyangfei]

What problem does this PR solve?

In an internal test, TiCDC can't recover from a TiKV crash or recovery.

We meet endless following errors after killing a TiKV, where store=172.16.4.197:21160 is the killed TiKV server.

[2020/12/10 17:05:01.000 +08:00] [WARN] [client.go:723] ["get grpc stream client failed"] [regionID=11544] [requestID=7814] [storeID=7] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2020/12/10 17:05:01.000 +08:00] [INFO] [region_cache.go:600] ["mark store's regions need be refill"] [store=172.16.4.197:21160]
[2020/12/10 17:05:01.000 +08:00] [INFO] [region_cache.go:414] ["invalidate current region, because others failed on same store"] [region=12557] [store=172.16.4.197:21160]
[2020/12/10 17:05:01.000 +08:00] [INFO] [client.go:656] ["cannot get rpcCtx, retry span"] [regionID=12557] [span="[7480000000000001ff2d5f728000000000ff4bd6de0000000000fa, 7480000000000001ff2d5f728000000000ff52290a0000000000fa)"]

On the other side, recovers a TiKV server meets following error

[2020/12/10 17:23:12.931 +08:00] [INFO] [client.go:363] ["establish stream to store failed, retry later"] [addr=172.16.4.197:21160] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Canceled desc = grpc: the client connection is closing"] [errorVerbose="[CDC:ErrTiKVEventFeed]rpc error: code = Canceled desc = grpc: the client connection is closing\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20201029093017-5a7df2af2ac7/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByCause\n\tgithub.com/pingcap/errors@v0.11.5-0.20201029093017-5a7df2af2ac7/normalize.go:279\ngithub.com/pingcap/ticdc/pkg/errors.WrapError\n\tgithub.com/pingcap/ticdc/pkg/errors/helper.go:28\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\tgithub.com/pingcap/ticdc/cdc/kv/client.go:362\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\tgithub.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\tgithub.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\tgithub.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\tgithub.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\tgithub.com/pingcap/ticdc/cdc/kv/client.go:346\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\tgithub.com/pingcap/ticdc/cdc/kv/client.go:720\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\tgithub.com/pingcap/ticdc/cdc/kv/client.go:477\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1374"]

What is changed and how it works?

We don't close gPRC conn here, let it goes into TransientFailure. If the store recovers, the gPRC conn can be reused.

TODO:

• Add an integration test later
• Refine the gRPC connetion pool

Check List

Tests

• Unit test
• Integration test

Release note

• Fix a bug that TiCDC could fail to continue replicating when a TiKV crashes or recovers from a crash, the bug exists in v4.0.8 only.

[amyangfei]

/run-all-tests

[zier-one]

LGTM

[liuzix]

LGTM

[zier-one]

/merge

[ti-srebot]

/run-all-tests

[ti-srebot]

@amyangfei merge failed.

[amyangfei]

/run-integration-tests

[amyangfei]

/run-integration-tests

[codecov-io]

Codecov Report

Merging #1196 (e70e4ef) into master (b3a87a8) will increase coverage by 0.0675%.
The diff coverage is n/a.

@@               Coverage Diff                @@
##             master      #1196        +/-   ##
================================================
+ Coverage   45.8436%   45.9111%   +0.0675%     
================================================
  Files           112        112                
  Lines         11729      11727         -2     
================================================
+ Hits           5377       5384         +7     
+ Misses         5748       5740         -8     
+ Partials        604        603         -1     

[ti-srebot]

cherry pick to release-4.0 in PR #1198