changefeed restarted and lag >25min when upgrade pd

[fubinzh]

What did you do?

1. ti-operator cluster with 36 tikv, 3 cdc, 3 pd, create changefeed to sync to kafka
2. running workload, throughtput ~50MB/s
3. Upgrade cluester

What did you expect to see?

lag should be less then 10s

What did you see instead?

lag > 25min when uprade pd upgrade, cdc capture restart seen due to etcd ontext cancel

[2023/04/25 13:31:22.129 +00:00] [INFO] [client.go:235] ["WatchWithChan exited"] [role=processor]
[2023/04/25 13:31:22.129 +00:00] [INFO] [capture.go:353] ["processor routine exited"] [captureID=f799bd5d-5cae-448e-b621-7b0ee517ef03] [error="[CDC:ErrPDEtcdAPIError]etcd api call error: context canceled"] [errorVerbose="[CDC:ErrPDEtcdAPIError]etcd api call error: context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:34\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).runEtcdWorker\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:524\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).run.func4\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:352\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1598"]
...
 [2023/04/25 13:31:28.136 +00:00] [INFO] [capture.go:240] ["capture initialized"] [capture="{\"id\":\"b20df9b5-93a9-4746-b705-c5fbc495981d\",\"address\":\"upstream-ticdc-0.upstream-ticdc-peer.cdc-kafka-big-cluster-tps-1686552-1-956.svc:8301\",\"version\":\"v7.1.0-rc.0\"}"]

Versions of the cluster

[root@upstream-ticdc-0 /]# /cdc version
Release Version: v7.1.0-rc.0
Git Commit Hash: 4d633bf
Git Branch: heads/refs/tags/v7.1.0-rc.0
UTC Build Time: 2023-04-21 10:14:01
Go Version: go version go1.20.3 linux/amd64
Failpoint Build: false

[fubinzh]

capture restart is not seen during another testing, however lag is >10s (~40s)

[fubinzh]

/severity major

[asddongmen]

fix by #8884