CDC resolvedTs progress slowly when redo logging is enabled

[fubinzh]

What did you do?

• Create TiDB cluster in GCP
• Create changefeed with redo, storage is GCS

[consistent]
level = "eventual"
storage = "gcs://qa-redo-log/redo-apply-multiple-cf0g92adj5hatumgaddg"` 

• Run tpcc workload （15:12 - ~16:30）
• Pause changefeed for about 1 hours, during with redo apply 2 time
• Workload stop (~16:30)
• Resume changefeed (~16:30)

What did you expect to see?

Changefeed checkpoint should advance normally.

What did you see instead?

• Changefeed doesn't advance for an hour after changefeed resume (puller is pulling)
• After that, sink starts to flush, but throughput is low (about 1024 rows/s)
  

Versions of the cluster

TiCDC version (execute cdc version):

bash-5.1# /cdc version
Release Version: v6.5.0-11-g3808d9732-dirty
Git Commit Hash: 3808d9732201b004147628abf4914fd2c5dd0711
Git Branch: release-6.5
UTC Build Time: 2023-01-10 06:19:02
Go Version: go version go1.19.1 linux/amd64
Failpoint Build: false
bash-5.1#

PD/TiDB/TiKV are v6.5.0

[fubinzh]

cdc log: ticdc.log

[fubinzh]

After the error happens, I did another 2 testing:

1. Update changefeed to local, and resume changefeed => sink flush rows/s still about ~1024
   

2. Enable pull based sink, resume changefeed from now, run workload. => sink flush rows/s increases to ~2000
   

3. Update changefeed and close redo, resume changefeed. => sink flush rows/s increases to ~20K
   

[fubinzh]

CPU & Memory Usage:

[fubinzh]

/severity Major

[fubinzh]

Reproduced the issue as well, when CDC is configured to use SSD.

ticdc0.log
ticdc1.log

monitor backup: https://snapshots.raintank.io/dashboard/snapshot/h74XS86BjNHyHTvu3u6FkYzO4dTN0ArJ