ticdc panic

[fubinzh]

What did you do?

• start kafka open protocal changefeed, enable big table scale
• Use lightning local backend to import 300GB data to upstream TiDB
• Run workload for couple of hours

What did you expect to see?

CDC no panic

What did you see instead?

CDC panic

[root@centos76_vm ticdc]# kubectl logs -p upstream-ticdc-0 -n cdc-kafka-scale-big-table-tps-1560914-1-48 --kubeconfig kubeconfig.yml
[WARN] TiCDC server data-dir is not set. Please use `cdc server --data-dir` to start the cdc server if possible.
panic: send on closed channel
goroutine 5395679 [running]:
[github.com/pingcap/tiflow/cdc/sinkv2/eventsink/mq.(*dmlSink).WriteEvents](http://github.com/pingcap/tiflow/cdc/sinkv2/eventsink/mq.(*dmlSink).WriteEvents)(0xc035acf200, {0xc03a9eec00, 0xb3, 0x2e95b79?})
        [github.com/pingcap/tiflow/cdc/sinkv2/eventsink/mq/mq_dml_sink.go:126](http://github.com/pingcap/tiflow/cdc/sinkv2/eventsink/mq/mq_dml_sink.go:126) +0x173
[github.com/pingcap/tiflow/cdc/sinkv2/tablesink](http://github.com/pingcap/tiflow/cdc/sinkv2/tablesink).(*EventTableSink[...]).UpdateResolvedTs(0xc048a02270?, {0x77878b, 0xc0bb3a0000?, 0x3abcc1e?
})
        [github.com/pingcap/tiflow/cdc/sinkv2/tablesink/table_sink_impl.go:115](http://github.com/pingcap/tiflow/cdc/sinkv2/tablesink/table_sink_impl.go:115) +0x444
[github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*tableSinkWrapper).updateResolvedTs](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*tableSinkWrapper).updateResolvedTs)(0x1?, {0x618d8f50c380004?, 0x1?, 0xc004aca3f0?})
        [github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_wrapper.go:170](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_wrapper.go:170) +0x22
[github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).advanceTableSinkWithBatchID](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).advanceTableSinkWithBatchID)(0xc0007852d0, 0xc01d17b730, 0x618d8f50c38000
4, 0x402a4, 0x1)
        [github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:445](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:445) +0x571
[github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTask.func2(0x0)](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTask.func2(0x0))
        [github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:171](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:171) +0x732
[github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTask.func3](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTask.func3)(0x0, 0x1)
        [github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:214](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:214) +0x45b
[github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTask](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTask)(0xc0007852d0, {0x439ca88, 0xc02ae43500}, 0xc01d17b730)
        [github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:349](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:349) +0x12b8
[github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTasks](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTasks)(0x1?, {0x439ca88, 0xc02ae43500}, 0xc008ae06c0)
        [github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:76](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:76) +0xc5
[github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*SinkManager).startWorkers.func1()](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*SinkManager).startWorkers.func1())
        [github.com/pingcap/tiflow/cdc/processor/sinkmanager/manager.go:197](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager/manager.go:197) +0x90
created by [github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*SinkManager).startWorkers](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*SinkManager).startWorkers)
        [github.com/pingcap/tiflow/cdc/processor/sinkmanager/manager.go:195](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager/manager.go:195) +0x47

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

[root@upstream-tidb-0 /]# /tidb-server -V                
Release Version: v6.6.0-alpha                            
Edition: Community                                       
Git Commit Hash: 93c6492418ae1bfa91479b8b465e61c78f29a995
Git Branch: heads/refs/tags/v6.6.0-alpha                 
UTC Build Time: 2023-02-08 11:51:47                      
GoVersion: go1.19.5                                      
Race Enabled: false                                      
TiKV Min Version: 6.2.0-alpha                            
Check Table Before Drop: false                           
Store: unistore                                          

Upstream TiKV version (execute tikv-server --version):

[root@upstream-tikv-0 /]# /tikv-server -V                                                                                                  
TiKV                                                                                                                                       
Release Version:   6.6.0-alpha                                                                                                             
Edition:           Community                                                                                                               
Git Commit Hash:   d513b789f0709520cdccc20bc4e6560761ad01a5                                                                                
Git Commit Branch: heads/refs/tags/v6.6.0-alpha                                                                                            
UTC Build Time:    2023-02-09 11:50:03                                                                                                     
Rust Version:      rustc 1.67.0-nightly (96ddd32c4 2022-11-14)                                                                             
Enable Features:   pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp clo
ud-azure                                                                                                                                   
Profile:           dist_release                                                                                                            

TiCDC version (execute cdc version):

[2023/02/09 10:46:54.599 +00:00] [INFO] [version.go:47] ["Welcome to Change Data Capture (CDC)"] [release-version=v6.6.0-alpha] [git-hash=0e1322f90b9b72697ca74513c32c0cb64479064c] [git-branch=heads/refs/tags/v6.6.0-alpha] [utc-build-time="2023-02-08 11:41:41"] [go-version="go version go1.19.5 linux/amd64"] [failpoint-build=false]                                                                                                                                                                   

[fubinzh]

/severity critical

[fubinzh]

/found automation

[3AceShowHand]

It's caused by the table sink send resolved ts event to a closed channel.

We can fix this problem by disallowing send events after the sink is closed.