cdc cli changefeed admin operation blocked

[amyangfei]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

   Execute cdc cli changefeed remove -c xxx blocked.

2. What did you expect to see?

   command runs successfully

3. What did you see instead?

   

   golang stack:
   owner.log

4. Versions of the cluster

   • Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

     v4.0.7
     

   • TiCDC version (execute cdc version):

     v4.0.8
     

[zier-one]

Reproduce the bug:

https://github.com/pingcap/ticdc/pull/1205
Run integration tests in Github Action, we can reproduce this bug.

see goroutine.log here

[zier-one]

fixed in tikv/pd#3283 and #1217

first, the PD client will be blocked in some cases, see tikv/pd#3283.
when a PD client is blocked, the whole StorageWithCurVersionCache will be blocked, and all pullers are frozen, like this:

and then, the owner will try to close the changefeed when a remove-changefeed command is received. however, all the pullers are frozen, the main goroutine of the owner will be blocked in closing the DDL puller, like the following picture: