new_ci_collation_with_old_value_test flakes due to etcd worker fails to commit

[overvenus]

Which jobs are flaking?

cdc_integration_test

Which test(s) are flaking?

new_ci_collation_with_old_value_test

Jenkins logs or GitHub Actions link

[2021/12/09 19:54:53.105 +08:00] [DEBUG] [schema_storage.go:397] ["create schema success, schema id"] [name=new_ci_collation_with_old_value_test] [id=57]
[2021/12/09 19:54:53.177 +08:00] [INFO] [mysql.go:337] ["Exec DDL succeeded"] [sql="create database `new_ci_collation_with_old_value_test`"]
[2021/12/09 19:54:53.177 +08:00] [INFO] [async_sink.go:132] ["Execute DDL succeeded"] [changefeed=8b29cece-b1d5-45cc-9c38-4bb29e305220] [ignored=false] [ddl="{\"StartTs\":429667356552462347,\"CommitTs\":429667356552462348,\"TableInfo\":{\"Schema\":\"new_ci_collation_with_
[2021/12/09 19:54:54.105 +08:00] [DEBUG] [etcd_worker.go:406] ["[etcd worker] ==========Update State to ETCD=========="]
[2021/12/09 19:54:54.105 +08:00] [DEBUG] [etcd_worker.go:411] ["[etcd worker] put key"] [key=/tidb/cdc/job/8b29cece-b1d5-45cc-9c38-4bb29e305220] [value="{\"resolved-ts\":429667356578676744,\"checkpoint-ts\":429667356578676744,\"admin-job-type\":0}"]
[2021/12/09 19:54:54.105 +08:00] [DEBUG] [etcd_worker.go:414] ["[etcd worker] ============State Commit============="] [committed=true]
[2021/12/09 19:54:54.505 +08:00] [DEBUG] [etcd_worker.go:406] ["[etcd worker] ==========Update State to ETCD=========="]
[2021/12/09 19:54:54.505 +08:00] [DEBUG] [etcd_worker.go:411] ["[etcd worker] put key"] [key=/tidb/cdc/job/8b29cece-b1d5-45cc-9c38-4bb29e305220] [value="{\"resolved-ts\":429667356578676744,\"checkpoint-ts\":429667356578676744,\"admin-job-type\":0}"]
[2021/12/09 19:54:54.505 +08:00] [DEBUG] [etcd_worker.go:414] ["[etcd worker] ============State Commit============="] [committed=false]
[2021/12/09 19:54:54.706 +08:00] [DEBUG] [etcd_worker.go:406] ["[etcd worker] ==========Update State to ETCD=========="]
[2021/12/09 19:54:54.706 +08:00] [DEBUG] [etcd_worker.go:411] ["[etcd worker] put key"] [key=/tidb/cdc/job/8b29cece-b1d5-45cc-9c38-4bb29e305220] [value="{\"resolved-ts\":429667356578676744,\"checkpoint-ts\":429667356578676744,\"admin-job-type\":0}"]
[2021/12/09 19:54:54.706 +08:00] [DEBUG] [etcd_worker.go:414] ["[etcd worker] ============State Commit============="] [committed=false]
[2021/12/09 19:54:54.905 +08:00] [DEBUG] [etcd_worker.go:406] ["[etcd worker] ==========Update State to ETCD=========="]
[2021/12/09 19:54:54.905 +08:00] [DEBUG] [etcd_worker.go:411] ["[etcd worker] put key"] [key=/tidb/cdc/job/8b29cece-b1d5-45cc-9c38-4bb29e305220] [value="{\"resolved-ts\":429667356578676744,\"checkpoint-ts\":429667356578676744,\"admin-job-type\":0}"]
[2021/12/09 19:54:54.906 +08:00] [DEBUG] [etcd_worker.go:414] ["[etcd worker] ============State Commit============="] [committed=false]
[2021/12/09 19:54:55.106 +08:00] [DEBUG] [etcd_worker.go:406] ["[etcd worker] ==========Update State to ETCD=========="]
[2021/12/09 19:54:55.106 +08:00] [DEBUG] [etcd_worker.go:411] ["[etcd worker] put key"] [key=/tidb/cdc/job/8b29cece-b1d5-45cc-9c38-4bb29e305220] [value="{\"resolved-ts\":429667356578676744,\"checkpoint-ts\":429667356578676744,\"admin-job-type\":0}"]
[2021/12/09 19:54:55.106 +08:00] [DEBUG] [etcd_worker.go:414] ["[etcd worker] ============State Commit============="] [committed=false]
[2021/12/09 19:54:55.305 +08:00] [DEBUG] [etcd_worker.go:406] ["[etcd worker] ==========Update State to ETCD=========="]
[2021/12/09 19:54:55.306 +08:00] [DEBUG] [etcd_worker.go:411] ["[etcd worker] put key"] [key=/tidb/cdc/job/8b29cece-b1d5-45cc-9c38-4bb29e305220] [value="{\"resolved-ts\":429667356578676744,\"checkpoint-ts\":429667356578676744,\"admin-job-type\":0}"]
[2021/12/09 19:54:55.306 +08:00] [DEBUG] [etcd_worker.go:414] ["[etcd worker] ============State Commit============="] [committed=false]
[2021/12/09 19:54:55.505 +08:00] [DEBUG] [etcd_worker.go:406] ["[etcd worker] ==========Update State to ETCD=========="]
[2021/12/09 19:54:55.505 +08:00] [DEBUG] [etcd_worker.go:411] ["[etcd worker] put key"] [key=/tidb/cdc/job/8b29cece-b1d5-45cc-9c38-4bb29e305220] [value="{\"resolved-ts\":429667356578676744,\"checkpoint-ts\":429667356578676744,\"admin-job-type\":0}"]
[2021/12/09 19:54:55.505 +08:00] [DEBUG] [etcd_worker.go:414] ["[etcd worker] ============State Commit============="] [committed=false]
[2021/12/09 19:54:55.705 +08:00] [DEBUG] [etcd_worker.go:406] ["[etcd worker] ==========Update State to ETCD=========="]
[2021/12/09 19:54:55.705 +08:00] [DEBUG] [etcd_worker.go:411] ["[etcd worker] put key"] [key=/tidb/cdc/job/8b29cece-b1d5-45cc-9c38-4bb29e305220] [value="{\"resolved-ts\":429667356578676744,\"checkpoint-ts\":429667356578676744,\"admin-job-type\":0}"]
[2021/12/09 19:54:55.705 +08:00] [DEBUG] [etcd_worker.go:414] ["[etcd worker] ============State Commit============="] [committed=false]

cdc.log.zip
cdc_etcd_check.log.zip
pd1.log.zip
tidb.log.zip

Anything else we need to know

• Does this test exist for other branches as well?

• Has there been a high frequency of failure lately?

[overvenus]

Another failed case: https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_integration_test/detail/cdc_ghpr_integration_test/7242/pipeline