Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

some unstable integration/unit tests #1501

Closed
amyangfei opened this issue Mar 12, 2021 · 13 comments
Closed

some unstable integration/unit tests #1501

amyangfei opened this issue Mar 12, 2021 · 13 comments
Labels
area/ticdc Issues or PRs related to TiCDC. component/test Unit tests and integration tests component. priority/P1 The issue has P1 priority. severity/minor type/bug The issue is confirmed as a bug.

Comments

@amyangfei
Copy link
Contributor

amyangfei commented Mar 12, 2021 •
edited
Loading

Bug Report

Please answer these questions before submitting your issue. Thanks!

  1. What did you do? If possible, provide a recipe for reproducing the error.

    List some unstable integration tests

test lists

  1. pr *: update tidb and br dependency #1490 failure: https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/cdc_ghpr_integration_test/detail/cdc_ghpr_integration_test/4682/pipeline
    cdc state checker failed
[2021/03/12 15:24:28.510 +08:00] [ERROR] [reactor.go:41] ["Verifying startTs failed"] [error="startTs too small, globalCkpt = 423502509874610177, startTs = 423502499768958978, table = 55, capture = bad5ce2c-8075-4bad-b9db-7c13de392090, cfid = 0ecb55a8-b193-440c-bd36-3dd7d8da7f98"] [errorVerbose="startTs too small, globalCkpt = 423502509874610177, startTs = 423502499768958978, table = 55, capture = bad5ce2c-8075-4bad-b9db-7c13de392090, cfid = 0ecb55a8-b193-440c-bd36-3dd7d8da7f98\nmain.(*cdcMonitReactor).verifyStartTs\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/testing_utils/cdc_state_checker/reactor.go:89\nmain.(*cdcMonitReactor).Tick\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/testing_utils/cdc_state_checker/reactor.go:39\ngithub.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:172\nmain.(*cdcMonitor).run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/testing_utils/cdc_state_checker/cdc_monitor.go:95\nmain.main\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/testing_utils/cdc_state_checker/main.go:46\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="main.(*cdcMonitReactor).Tick\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/testing_utils/cdc_state_checker/reactor.go:41\ngithub.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:172\nmain.(*cdcMonitor).run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/testing_utils/cdc_state_checker/cdc_monitor.go:95\nmain.main\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/testing_utils/cdc_state_checker/main.go:46\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"]
[2021/03/12 15:24:28.510 +08:00] [ERROR] [cdc_monitor.go:96] ["etcdWorker exited: test-case-failed"] [error="startTs too small, globalCkpt = 423502509874610177, startTs = 423502499768958978, table = 55, capture = bad5ce2c-8075-4bad-b9db-7c13de392090, cfid = 0ecb55a8-b193-440c-bd36-3dd7d8da7f98"] [errorVerbose="startTs too small, globalCkpt = 423502509874610177, startTs = 423502499768958978, table = 55, capture = bad5ce2c-8075-4bad-b9db-7c13de392090, cfid = 0ecb55a8-b193-440c-bd36-3dd7d8da7f98\nmain.(*cdcMonitReactor).verifyStartTs\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/testing_utils/cdc_state_checker/reactor.go:89\nmain.(*cdcMonitReactor).Tick\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/testing_utils/cdc_state_checker/reactor.go:39\ngithub.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:172\nmain.(*cdcMonitor).run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/testing_utils/cdc_state_checker/cdc_monitor.go:95\nmain.main\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/testing_utils/cdc_state_checker/main.go:46\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="main.(*cdcMonitor).run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/testing_utils/cdc_state_checker/cdc_monitor.go:96\nmain.main\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/testing_utils/cdc_state_checker/main.go:46\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"]
[2021/03/12 15:24:28.510 +08:00] [INFO] [cdc_monitor.go:97] ["CDC state"] [state="{\"Owner\":\"\",\"Captures\":{\"d382a068-6dbc-4a74-8c1f-01a7d5da21a2\":{\"id\":\"d382a068-6dbc-4a74-8c1f-01a7d5da21a2\",\"address\":\"127.0.0.1:8301\",\"version\":\"v4.0.2-372-g93c82ed\"}},\"ChangefeedStatuses\":{\"0ecb55a8-b193-440c-bd36-3dd7d8da7f98\":{\"resolved-ts\":423502510136754177,\"checkpoint-ts\":423502509874610177,\"admin-job-type\":0}},\"TaskPositions\":{\"0ecb55a8-b193-440c-bd36-3dd7d8da7f98\":{\"bad5ce2c-8075-4bad-b9db-7c13de392090\":{\"checkpoint-ts\":423502499768958978,\"resolved-ts\":423502499768958978,\"count\":0,\"error\":null},\"d382a068-6dbc-4a74-8c1f-01a7d5da21a2\":{\"checkpoint-ts\":423502509874610177,\"resolved-ts\":423502510136754177,\"count\":0,\"error\":null}}},\"TaskStatuses\":{\"0ecb55a8-b193-440c-bd36-3dd7d8da7f98\":{\"bad5ce2c-8075-4bad-b9db-7c13de392090\":{\"tables\":{\"53\":{\"start-ts\":423502499742744587,\"mark-table-id\":0},\"55\":{\"start-ts\":423502499768958978,\"mark-table-id\":0}},\"operation\":{\"55\":{\"delete\":false,\"boundary_ts\":423502499768958978,\"done\":false,\"status\":1}},\"admin-job-type\":0},\"d382a068-6dbc-4a74-8c1f-01a7d5da21a2\":{\"tables\":{\"51\":{\"start-ts\":423502499742744587,\"mark-table-id\":0},\"53\":{\"start-ts\":423502499768958978,\"mark-table-id\":0},\"55\":{\"start-ts\":423502499768958978,\"mark-table-id\":0}},\"operation\":null,\"admin-job-type\":0}}}}"]
[2021/03/12 15:24:28.510 +08:00] [PANIC] [main.go:47] ["cdcMonitor exited"] [error="startTs too small, globalCkpt = 423502509874610177, startTs = 423502499768958978, table = 55, capture = bad5ce2c-8075-4bad-b9db-7c13de392090, cfid = 0ecb55a8-b193-440c-bd36-3dd7d8da7f98"] [errorVerbose="startTs too small, globalCkpt = 423502509874610177, startTs = 423502499768958978, table = 55, capture = bad5ce2c-8075-4bad-b9db-7c13de392090, cfid = 0ecb55a8-b193-440c-bd36-3dd7d8da7f98\nmain.(*cdcMonitReactor).verifyStartTs\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/testing_utils/cdc_state_checker/reactor.go:89\nmain.(*cdcMonitReactor).Tick\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/testing_utils/cdc_state_checker/reactor.go:39\ngithub.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:172\nmain.(*cdcMonitor).run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/testing_utils/cdc_state_checker/cdc_monitor.go:95\nmain.main\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/testing_utils/cdc_state_checker/main.go:46\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="main.main\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/testing_utils/cdc_state_checker/main.go:47\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"]
  1. *: update tidb and br dependency #1490, processor_panic case, https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/cdc_ghpr_kafka_integration_test/detail/cdc_ghpr_kafka_integration_test/2362/pipeline
@amyangfei amyangfei added the type/bug The issue is confirmed as a bug. label Mar 12, 2021
@amyangfei
Copy link
Contributor Author

amyangfei commented Mar 26, 2021 •
edited
Loading

unit test

failure: https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/cdc_ghpr_test/detail/cdc_ghpr_test/6726/pipeline

[2021-03-26T08:26:29.106Z] ----------------------------------------------------------------------
[2021-03-26T08:26:29.106Z] FAIL: manager_test.go:138: managerSuite.TestManagerAddRemoveTable
[2021-03-26T08:26:29.106Z] 
[2021-03-26T08:26:29.106Z] manager_test.go:68:
[2021-03-26T08:26:29.106Z]     c.Assert(row.CommitTs, check.Greater, c.lastResolvedTs)
[2021-03-26T08:26:29.106Z] ... compare_one uint64 = 0x371
[2021-03-26T08:26:29.106Z] ... compare_two uint64 = 0x384
[2021-03-26T08:26:29.106Z] 
[2021-03-26T08:26:29.106Z] OOPS: 27 passed, 1 FAILED
[2021-03-26T08:26:29.106Z] --- FAIL: Test (30.96s)
[2021-03-26T08:26:29.106Z] FAIL
[2021-03-26T08:26:29.106Z] coverage: 63.8% of statements
[2021-03-26T08:26:29.106Z] FAIL	github.com/pingcap/ticdc/cdc/sink	31.221s

@amyangfei
Copy link
Contributor Author

unit test

failure: https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/cdc_ghpr_kafka_integration_test/detail/cdc_ghpr_kafka_integration_test/2500/pipeline

[2021-03-26T09:36:09.563Z] [2021/03/26 17:31:15.744 +08:00] [WARN] [merger.go:60] ["Unified Sorter: merger cleaning up timeout."]
[2021-03-26T09:36:09.563Z] [2021/03/26 17:31:33.740 +08:00] [INFO] [unified_sorter.go:119] ["Unified Sorter: starting cleaning up files"]
[2021-03-26T09:36:09.563Z] [2021/03/26 17:31:33.784 +08:00] [INFO] [backend_pool.go:85] ["Unified Sorter backEnd is being cancelled"]
[2021-03-26T09:36:09.563Z] 
[2021-03-26T09:36:09.563Z] ----------------------------------------------------------------------
[2021-03-26T09:36:09.563Z] FAIL: sorter_test.go:82: sorterSuite.TestSorterCancel
[2021-03-26T09:36:09.563Z] 
[2021-03-26T09:36:09.563Z] OOPS: 10 passed, 1 FAILED
[2021-03-26T09:36:09.563Z] --- FAIL: Test (111.36s)
[2021-03-26T09:36:09.563Z] FAIL
[2021-03-26T09:36:09.563Z] coverage: 77.9% of statements
[2021-03-26T09:36:09.563Z] FAIL	github.com/pingcap/ticdc/cdc/puller/sorter	111.732s

@amyangfei
Copy link
Contributor Author

amyangfei commented Mar 27, 2021 •
edited
Loading

Fixed

integration test

failure: https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/cdc_ghpr_integration_test/detail/cdc_ghpr_integration_test/4907/pipeline

[2021-03-27T04:08:38.626Z] [Sat Mar 27 12:08:38 CST 2021] <<<<<< START cdc server in processor_stop_delay_2 case >>>>>>
[2021-03-27T04:08:38.882Z] check_position_removed http://127.0.0.1:2379
[2021-03-27T04:08:39.138Z] Key: /tidb/cdc/capture/1d9cd715-0122-4524-a46a-55371f35d78f, Value: {"id":"1d9cd715-0122-4524-a46a-55371f35d78f","address":"127.0.0.1:8300","version":"v5.0.0-nightly-15-g9b09f2e"}
[2021-03-27T04:08:39.138Z] Key: /tidb/cdc/changefeed/info/0ccdbf91-a673-4740-9576-3864f222d175, Value: {"sink-uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","opts":{"_changefeed_id":"cli-verify"},"create-time":"2021-03-27T12:08:38.28057801+08:00","start-ts":423839168785809412,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"case-sensitive":true,"enable-old-value":true,"force-replicate":false,"check-gc-safe-point":true,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"ddl-allow-list":null},"mounter":{"worker-num":16},"sink":{"dispatchers":null,"protocol":"default"},"cyclic-replication":{"enable":false,"replica-id":0,"filter-replica-ids":null,"id-buckets":0,"sync-ddl":false},"scheduler":{"type":"table-number","polling-time":-1}},"state":"normal","history":null,"error":null,"sync-point-enabled":false,"sync-point-interval":600000000000,"creator-version":"v5.0.0-nightly-15-g9b09f2e"}
[2021-03-27T04:08:39.138Z] Key: /tidb/cdc/owner/22317871de7a5e1d, Value: 1d9cd715-0122-4524-a46a-55371f35d78f
[2021-03-27T04:08:39.138Z] Show 3 KVs
[2021-03-27T04:08:39.138Z] AAposition: 
[2021-03-27T04:08:39.138Z] AAposition: 0
[2021-03-27T04:08:39.138Z] run task successfully
[2021-03-27T04:08:39.138Z] Replicate lag (449116h8m38.932s) is larger than 1 days, large data may cause OOM, confirm to continue at your own risk [Y/N]
[2021-03-27T04:08:39.138Z] Error: abort changefeed create or resume

Test case problem, fixed in pingcap/ticdc@4147fa5 (5.0) and pingcap/ticdc@3126bf0 (4.0).
master does not exist

Root cause is in test case, if we resume changefeed before the changefeed has been stopped, the checkpoint querying from cdc server may be zero

@amyangfei
Copy link
Contributor Author

leak test

failure: https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/cdc_ghpr_leak_test/detail/cdc_ghpr_leak_test/2710/pipeline

[2021-03-27T05:42:36.423Z] ----------------------------------------------------------------------
[2021-03-27T05:42:36.423Z] FAIL: server_test.go:107: serverSuite.TestEtcdHealthChecker
[2021-03-27T05:42:36.423Z] 
[2021-03-27T05:42:36.423Z] server_test.go:131:
[2021-03-27T05:42:36.423Z]     ...
[2021-03-27T05:42:36.423Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/util/testleak/leaktest.go:159:
[2021-03-27T05:42:36.423Z]     c.Errorf("Test %s check-count %d appears to have leaked: %v", c.TestName(), cnt, g)
[2021-03-27T05:42:36.423Z] ... Error: Test serverSuite.TestEtcdHealthChecker check-count 50 appears to have leaked: internal/poll.runtime_pollWait(0x7f2fcc36dd68, 0x72, 0x5d1d340)
[2021-03-27T05:42:36.423Z] 	/usr/local/go/src/runtime/netpoll.go:184 +0x55
[2021-03-27T05:42:36.423Z] internal/poll.(*pollDesc).wait(0xc000df6318, 0x72, 0x200, 0x200, 0xffffffffffffffff)
[2021-03-27T05:42:36.423Z] 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0xe4
[2021-03-27T05:42:36.423Z] internal/poll.(*pollDesc).waitRead(...)
[2021-03-27T05:42:36.423Z] 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
[2021-03-27T05:42:36.423Z] internal/poll.(*FD).Read(0xc000df6300, 0xc00084e000, 0x200, 0x200, 0x0, 0x0, 0x0)
[2021-03-27T05:42:36.423Z] 	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x253
[2021-03-27T05:42:36.423Z] net.(*netFD).Read(0xc000df6300, 0xc00084e000, 0x200, 0x200, 0x0, 0xc00084e000, 0x200)
[2021-03-27T05:42:36.423Z] 	/usr/local/go/src/net/fd_unix.go:202 +0x66
[2021-03-27T05:42:36.423Z] net.(*conn).Read(0xc000ada078, 0xc00084e000, 0x200, 0x200, 0xc0008a40c0, 0x1491bec, 0x200)
[2021-03-27T05:42:36.423Z] 	/usr/local/go/src/net/net.go:184 +0xa1
[2021-03-27T05:42:36.423Z] net.dnsPacketRoundTrip(0x5dc0a00, 0xc000ada078, 0x696c61766e692220, 0x736f682d64702d64, 0x6e696b6e656a2e74, 0x6376732e69632d73, 0x72657473756c632e, 0x2e6c61636f6c2e, 0x0, 0x0, ...)
[2021-03-27T05:42:36.423Z] 	/usr/local/go/src/net/dnsclient_unix.go:87 +0x143
[2021-03-27T05:42:36.423Z] net.(*Resolver).exchange(0x80a6280, 0x5d89c80, 0xc0003bab40, 0xc000e4a190, 0xe, 0x2d64696c61766e69, 0x2e74736f682d6470, 0x2d736e696b6e656a, 0x632e6376732e6963, 0x6c2e72657473756c, ...)
[2021-03-27T05:42:36.423Z] 	/usr/local/go/src/net/dnsclient_unix.go:166 +0x518
[2021-03-27T05:42:36.423Z] net.(*Resolver).tryOneName(0x80a6280, 0x5d89c80, 0xc0003bab40, 0xc0003bc960, 0xc0001fe480, 0x2d, 0x80a0001, 0x0, 0x0, 0x0, ...)
[2021-03-27T05:42:36.423Z] 	/usr/local/go/src/net/dnsclient_unix.go:256 +0x3ed
[2021-03-27T05:42:36.423Z] net.(*Resolver).goLookupIPCNAMEOrder.func3.1(0x80a6280, 0x5d89c80, 0xc0003bab40, 0xc0003bc960, 0xc0001fe480, 0x2d, 0xc00195f320, 0x1)
[2021-03-27T05:42:36.423Z] 	/usr/local/go/src/net/dnsclient_unix.go:602 +0xa2
[2021-03-27T05:42:36.423Z] created by net.(*Resolver).goLookupIPCNAMEOrder.func3
[2021-03-27T05:42:36.423Z] 	/usr/local/go/src/net/dnsclient_unix.go:601 +0x101
[2021-03-27T05:42:36.423Z] 
[2021-03-27T05:42:36.423Z] server_test.go:131:
[2021-03-27T05:42:36.423Z]     ...
[2021-03-27T05:42:36.423Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/util/testleak/leaktest.go:159:
[2021-03-27T05:42:36.423Z]     c.Errorf("Test %s check-count %d appears to have leaked: %v", c.TestName(), cnt, g)
[2021-03-27T05:42:36.423Z] ... Error: Test serverSuite.TestEtcdHealthChecker check-count 50 appears to have leaked: net.(*Resolver).goLookupIPCNAMEOrder.func4(0xc0001fe480, 0x2d, 0xc00081001c, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
[2021-03-27T05:42:36.423Z] 	/usr/local/go/src/net/dnsclient_unix.go:608 +0xb7
[2021-03-27T05:42:36.423Z] net.(*Resolver).goLookupIPCNAMEOrder(0x80a6280, 0x5d89c80, 0xc0003bab40, 0xc000cbc040, 0xf, 0x1, 0x0, 0x14ad78c, 0x14b38f7, 0x0, ...)
[2021-03-27T05:42:36.423Z] 	/usr/local/go/src/net/dnsclient_unix.go:618 +0x8a0
[2021-03-27T05:42:36.423Z] net.(*Resolver).lookupIP(0x80a6280, 0x5d89c80, 0xc0003bab40, 0x5267146, 0x3, 0xc000cbc040, 0xf, 0xc00061a660, 0x148d312, 0xc00061a750, ...)
[2021-03-27T05:42:36.423Z] 	/usr/local/go/src/net/lookup_unix.go:102 +0x119
[2021-03-27T05:42:36.423Z] net.glob..func1(0x5d89c80, 0xc0003bab40, 0xc000e52060, 0x5267146, 0x3, 0xc000cbc040, 0xf, 0x14c0c6b, 0xc00061a7c8, 0x3650fa6, ...)
[2021-03-27T05:42:36.423Z] 	/usr/local/go/src/net/hook.go:23 +0x9a
[2021-03-27T05:42:36.424Z] net.(*Resolver).lookupIPAddr.func1(0x0, 0x0, 0x0, 0x0)
[2021-03-27T05:42:36.424Z] 	/usr/local/go/src/net/lookup.go:269 +0x15d
[2021-03-27T05:42:36.424Z] internal/singleflight.(*Group).doCall(0x80a6290, 0xc0010c6730, 0xc000cbc060, 0x13, 0xc0003bab80)
[2021-03-27T05:42:36.424Z] 	/usr/local/go/src/internal/singleflight/singleflight.go:95 +0x4d
[2021-03-27T05:42:36.424Z] created by internal/singleflight.(*Group).DoChan
[2021-03-27T05:42:36.424Z] 	/usr/local/go/src/internal/singleflight/singleflight.go:88 +0x4b5
[2021-03-27T05:42:36.424Z] 
[2021-03-27T05:42:36.424Z] {"level":"warn","ts":"2021-03-27T13:42:27.826+0800","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-255c6d9b-59b6-4bd3-b58d-d1bafb64197d/127.0.0.1:17056","attempt":0,"error":"rpc error: code = Canceled desc = context canceled"}
[2021-03-27T05:42:36.424Z] OOPS: 18 passed, 1 FAILED
[2021-03-27T05:42:36.424Z] --- FAIL: TestSuite (50.39s)
[2021-03-27T05:42:36.424Z] FAIL
[2021-03-27T05:42:36.424Z] FAIL	github.com/pingcap/ticdc/cdc	50.540s

@amyangfei
Copy link
Contributor Author

integration test

failure: https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/cdc_ghpr_leak_test/detail/cdc_ghpr_leak_test/2740/pipeline

[2021-03-29T12:43:43.307Z] 
[2021-03-29T12:43:43.307Z] ----------------------------------------------------------------------
[2021-03-29T12:43:43.307Z] FAIL: schema_storage_test.go:445: getUniqueKeysSuite.TestPKShouldBeInTheFirstPlaceWhenPKIsHandle
[2021-03-29T12:43:43.307Z] 
[2021-03-29T12:43:43.307Z] schema_storage_test.go:484:
[2021-03-29T12:43:43.307Z]     ...
[2021-03-29T12:43:43.307Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/util/testleak/leaktest.go:159:
[2021-03-29T12:43:43.307Z]     c.Errorf("Test %s check-count %d appears to have leaked: %v", c.TestName(), cnt, g)
[2021-03-29T12:43:43.307Z] ... Error: Test getUniqueKeysSuite.TestPKShouldBeInTheFirstPlaceWhenPKIsHandle check-count 50 appears to have leaked: go.opencensus.io/stats/view.(*worker).start(0xc000608e10)
[2021-03-29T12:43:43.307Z] 	/go/pkg/mod/go.opencensus.io@v0.22.3/stats/view/worker.go:154 +0x1d6
[2021-03-29T12:43:43.307Z] created by go.opencensus.io/stats/view.init.0
[2021-03-29T12:43:43.307Z] 	/go/pkg/mod/go.opencensus.io@v0.22.3/stats/view/worker.go:32 +0x9a
[2021-03-29T12:43:43.307Z] 
[2021-03-29T12:43:43.307Z] OOPS: 12 passed, 1 FAILED
[2021-03-29T12:43:43.307Z] --- FAIL: Test (32.52s)
[2021-03-29T12:43:43.307Z] FAIL
[2021-03-29T12:43:43.307Z] FAIL	github.com/pingcap/ticdc/cdc/entry	32.895s

@amyangfei
Copy link
Contributor Author

amyangfei commented Apr 3, 2021 •
edited
Loading

Fixed

unit test

failure: https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/cdc_ghpr_test/detail/cdc_ghpr_test/7031/pipeline


[2021-04-02T16:20:07.344Z] [2021/04/03 00:17:07.593 +08:00] [ERROR] [pipeline.go:96] ["found error when running the node"] [name="error node"] [error="error node throw an error, index: 3"] [errorVerbose="error node throw an error, index: 3\ngithub.com/pingcap/ticdc/pkg/pipeline.(*errorNode).Receive\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/pkg/pipeline/pipeline_test.go:197\ngithub.com/pingcap/ticdc/pkg/pipeline.(*nodeRunner).run\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/pkg/pipeline/runner.go:60\ngithub.com/pingcap/ticdc/pkg/pipeline.(*Pipeline).driveRunner\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/pkg/pipeline/pipeline.go:93\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/ticdc/pkg/pipeline.(*Pipeline).driveRunner\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/pkg/pipeline/pipeline.go:96"]
[2021-04-02T16:20:07.344Z] 
[2021-04-02T16:20:07.344Z] ----------------------------------------------------------------------
[2021-04-02T16:20:07.344Z] FAIL: pipeline_test.go:288: pipelineSuite.TestPipelineThrow
[2021-04-02T16:20:07.344Z] 
[2021-04-02T16:20:07.344Z] pipeline_test.go:284:
[2021-04-02T16:20:07.344Z]     n.c.Assert(n.index, check.Equals, 6)
[2021-04-02T16:20:07.344Z] ... obtained int = 4
[2021-04-02T16:20:07.344Z] ... expected int = 6
[2021-04-02T16:20:07.344Z] 
[2021-04-02T16:20:07.344Z] pipeline_test.go:316:
[2021-04-02T16:20:07.344Z]     c.Assert(len(errs), check.Equals, 4)
[2021-04-02T16:20:07.344Z] ... obtained int = 2
[2021-04-02T16:20:07.344Z] ... expected int = 4
[2021-04-02T16:20:07.344Z] 
[2021-04-02T16:20:07.344Z] OOPS: 3 passed, 1 skipped, 1 FAILED
[2021-04-02T16:20:07.344Z] --- FAIL: TestSuite (1.12s)
[2021-04-02T16:20:07.344Z] FAIL
[2021-04-02T16:20:07.344Z] coverage: 73.3% of statements
[2021-04-02T16:20:07.344Z] FAIL	github.com/pingcap/ticdc/pkg/pipeline	1.182s

@amyangfei amyangfei mentioned this issue Apr 3, 2021
Merged
This was referenced Apr 13, 2021
Merged
Merged
@amyangfei
Copy link
Contributor Author

leak test

failure: https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/cdc_ghpr_leak_test/detail/cdc_ghpr_leak_test/3144/pipeline

[2021-04-14T08:14:45.998Z] 
[2021-04-14T08:14:45.998Z] ----------------------------------------------------------------------
[2021-04-14T08:14:45.998Z] FAIL: etcd_test.go:53: etcdSuite.TestEmbedEtcd
[2021-04-14T08:14:45.999Z] 
[2021-04-14T08:14:45.999Z] etcd_test.go:74:
[2021-04-14T08:14:45.999Z]     ...
[2021-04-14T08:14:45.999Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/util/testleak/leaktest.go:159:
[2021-04-14T08:14:45.999Z]     c.Errorf("Test %s check-count %d appears to have leaked: %v", c.TestName(), cnt, g)
[2021-04-14T08:14:45.999Z] ... Error: Test etcdSuite.TestEmbedEtcd check-count 50 appears to have leaked: github.com/pingcap/ticdc/pkg/etcd.(*etcdSuite).SetUpTest.func1(0xc0001f2a50, 0xc00015e580)
[2021-04-14T08:14:45.999Z] 	/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/etcd/etcd_test.go:45 +0x6d
[2021-04-14T08:14:45.999Z] created by github.com/pingcap/ticdc/pkg/etcd.(*etcdSuite).SetUpTest
[2021-04-14T08:14:45.999Z] 	/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/etcd/etcd_test.go:44 +0x16d
[2021-04-14T08:14:45.999Z] 
[2021-04-14T08:14:45.999Z] OOPS: 4 passed, 1 FAILED
[2021-04-14T08:14:45.999Z] --- FAIL: Test (25.57s)
[2021-04-14T08:14:45.999Z] FAIL
[2021-04-14T08:14:45.999Z] FAIL	github.com/pingcap/ticdc/pkg/etcd	25.668s

@amyangfei
Copy link
Contributor Author

amyangfei commented Apr 15, 2021 •
edited
Loading

Fixed

unit test

failure: https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/cdc_ghpr_integration_test/detail/cdc_ghpr_integration_test/5284/pipeline

[2021-04-15T02:30:54.769Z] [2021/04/15 10:21:20.814 +08:00] [ERROR] [client_test.go:2703] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).testKVClientForceReconnect.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:2703\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:222"]
[2021-04-15T02:30:54.769Z] [2021/04/15 10:21:29.039 +08:00] [ERROR] [client_test.go:2703] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).testKVClientForceReconnect.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:2703\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:222"]
[2021-04-15T02:30:54.769Z] [2021/04/15 10:21:29.252 +08:00] [ERROR] [client_v2.go:81] ["received an event but neither pending region nor running region was found"] [regionID=3] [requestID=49] [addr=127.0.0.1:13704] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).sendRegionChangeEventV2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_v2.go:81\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStreamV2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_v2.go:274\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:815\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]

[2021-04-15T02:30:54.770Z] goroutine 198 [select, 9 minutes]:
[2021-04-15T02:30:54.770Z] github.com/pingcap/check.(*suiteRunner).doRun(0xc000899050, 0x0)
[2021-04-15T02:30:54.770Z] 	/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:667 +0x228
[2021-04-15T02:30:54.770Z] github.com/pingcap/check.(*suiteRunner).run(0xc000899050, 0xc000445b40)
[2021-04-15T02:30:54.770Z] 	/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:696 +0x140
[2021-04-15T02:30:54.770Z] github.com/pingcap/check.Run(0x4fc2500, 0xc000445b40, 0xc000647eb8, 0xc000898e10)
[2021-04-15T02:30:54.770Z] 	/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/run.go:150 +0x5b
[2021-04-15T02:30:54.770Z] github.com/pingcap/check.RunAll(0xc000647eb8, 0x0)
[2021-04-15T02:30:54.770Z] 	/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/run.go:113 +0xf3a
[2021-04-15T02:30:54.770Z] github.com/pingcap/check.TestingT(0xc000416600)
[2021-04-15T02:30:54.770Z] 	/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/run.go:99 +0x766
[2021-04-15T02:30:54.770Z] github.com/pingcap/ticdc/cdc/kv.Test(0xc000416600)
[2021-04-15T02:30:54.770Z] 	/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:48 +0x39
[2021-04-15T02:30:54.770Z] testing.tRunner(0xc000416600, 0x50ecc98)
[2021-04-15T02:30:54.770Z] 	/usr/local/go/src/testing/testing.go:909 +0x19a
[2021-04-15T02:30:54.770Z] created by testing.(*T).Run
[2021-04-15T02:30:54.770Z] 	/usr/local/go/src/testing/testing.go:960 +0x652

@amyangfei amyangfei mentioned this issue Apr 15, 2021
Merged
@ti-srebot ti-srebot mentioned this issue Apr 16, 2021
Closed
@amyangfei amyangfei added the component/test Unit tests and integration tests component. label Apr 19, 2021
@amyangfei amyangfei changed the title unstable integration tests some unstable integration/unit tests Apr 19, 2021
@amyangfei amyangfei added the priority/P1 The issue has P1 priority. label Apr 19, 2021
@amyangfei amyangfei mentioned this issue Apr 19, 2021
Closed
17 tasks
@amyangfei
Copy link
Contributor Author 

[2021-05-08T06:59:07.499Z] + GO111MODULE=off
[2021-05-08T06:59:07.499Z] + GOPATH=/go:/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go
[2021-05-08T06:59:07.499Z] + PATH=/go:/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/bin:/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[2021-05-08T06:59:07.499Z] + make leak_test
[2021-05-08T06:59:07.499Z] which bin/failpoint-ctl >/dev/null 2>&1 || CGO_ENABLED=0 GO111MODULE=on go build  -trimpath -o bin/failpoint-ctl github.com/pingcap/failpoint/failpoint-ctl
[2021-05-08T06:59:08.446Z] ./scripts/fix_lib_zstd.sh
[2021-05-08T06:59:08.446Z] rm: cannot remove 'zstd_cgo.go': No such file or directory
[2021-05-08T06:59:08.446Z] $(echo $(for p in $(go list ./...| grep -vE 'vendor|proto|ticdc\/tests|integration|testing_utils'); do echo ${p#"github.com/pingcap/ticdc/"}|grep -v "github.com/pingcap/ticdc"; done) | xargs bin/failpoint-ctl enable >/dev/null)
[2021-05-08T06:59:26.651Z] ok  	github.com/pingcap/ticdc	1.153s
[2021-05-08T07:00:34.578Z] ok  	github.com/pingcap/ticdc/cdc	57.553s
[2021-05-08T07:00:34.578Z] ok  	github.com/pingcap/ticdc/cdc/entry	26.751s
[2021-05-08T07:00:43.037Z] [2021/05/08 14:59:39.967 +08:00] [ERROR] [client.go:1181] ["failed to receive from stream"] [addr=127.0.0.1:18853] [storeID=1] [error="injected error"] [errorVerbose="injected error\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1170\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:833\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1181\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:833\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.037Z] [2021/05/08 14:59:44.571 +08:00] [ERROR] [client_v2.go:237] ["failed to receive from stream"] [addr=127.0.0.1:18108] [storeID=1] [error="injected error"] [errorVerbose="injected error\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStreamV2\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_v2.go:226\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:835\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStreamV2\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_v2.go:237\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:835\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.037Z] [2021/05/08 14:59:50.819 +08:00] [ERROR] [client.go:1181] ["failed to receive from stream"] [addr=127.0.0.1:18635] [storeID=1] [error="injected stream recv error"] [errorVerbose="injected stream recv error\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1170\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:833\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1181\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:833\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.037Z] [2021/05/08 14:59:52.878 +08:00] [ERROR] [client.go:453] ["check tikv version failed"] [error="[CDC:ErrGetAllStoresFailed]context canceled"] [errorVerbose="[CDC:ErrGetAllStoresFailed]context canceled\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByCause\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:279\ngithub.com/pingcap/ticdc/pkg/errors.WrapError\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/errors/helper.go:28\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:128\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:447\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:453\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.037Z] [2021/05/08 14:59:54.565 +08:00] [ERROR] [region_worker.go:512] ["The CommitTs must be greater than the resolvedTs"] ["Event Type"=COMMIT] [CommitTs=98] [resolvedTs=100] [regionID=3] [stack="github.com/pingcap/ticdc/cdc/kv.(*regionWorker).handleEventEntry\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/region_worker.go:512\ngithub.com/pingcap/ticdc/cdc/kv.(*regionWorker).eventHandler\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/region_worker.go:356\ngithub.com/pingcap/ticdc/cdc/kv.(*regionWorker).run.func3\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/region_worker.go:424\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.038Z] [2021/05/08 14:59:56.356 +08:00] [ERROR] [region_worker.go:493] ["The CommitTs must be greater than the resolvedTs"] ["Event Type"=COMMITTED] [CommitTs=98] [resolvedTs=100] [regionID=3] [stack="github.com/pingcap/ticdc/cdc/kv.(*regionWorker).handleEventEntry\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/region_worker.go:493\ngithub.com/pingcap/ticdc/cdc/kv.(*regionWorker).eventHandler\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/region_worker.go:356\ngithub.com/pingcap/ticdc/cdc/kv.(*regionWorker).run.func3\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/region_worker.go:424\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.038Z] [2021/05/08 14:59:58.373 +08:00] [ERROR] [client.go:1092] ["tikv reported compatibility error, which is not expected"] [rpcCtx="region ID: 3, meta: id:3 peers:<id:4 store_id:1 > , peer: id:4 store_id:1 , addr: 127.0.0.1:13863, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv"] [error="required_version:\"v4.0.7\" "] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1092\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func3\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:620\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.038Z] [2021/05/08 15:00:09.000 +08:00] [ERROR] [client.go:1088] ["tikv reported duplicated request to the same region, which is not expected"] [regionID=3] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1088\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func3\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:620\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.038Z] [2021/05/08 15:00:25.039 +08:00] [ERROR] [client.go:453] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:447\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:453\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.038Z] [2021/05/08 15:00:25.090 +08:00] [ERROR] [client.go:453] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:447\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:453\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.039Z] [2021/05/08 15:00:25.166 +08:00] [ERROR] [client.go:453] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 4.0.0-rc is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 4.0.0-rc is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:447\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:453\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.039Z] [2021/05/08 15:00:25.335 +08:00] [ERROR] [client.go:453] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:447\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:453\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.040Z] [2021/05/08 15:00:25.438 +08:00] [ERROR] [client.go:453] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:447\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:453\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.040Z] [2021/05/08 15:00:25.482 +08:00] [ERROR] [client.go:453] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 4.0.0-rc is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 4.0.0-rc is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:447\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:453\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.040Z] [2021/05/08 15:00:25.530 +08:00] [ERROR] [client.go:453] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:447\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:453\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.040Z] [2021/05/08 15:00:25.636 +08:00] [ERROR] [client.go:453] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:447\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:453\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.041Z] [2021/05/08 15:00:25.739 +08:00] [ERROR] [client.go:453] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 4.0.0-rc is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 4.0.0-rc is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:447\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:453\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.041Z] [2021/05/08 15:00:25.767 +08:00] [ERROR] [client.go:453] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:447\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:453\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.041Z] [2021/05/08 15:00:25.809 +08:00] [ERROR] [client.go:453] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:447\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:453\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.041Z] [2021/05/08 15:00:25.882 +08:00] [ERROR] [client.go:453] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:447\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:453\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.042Z] [2021/05/08 15:00:25.985 +08:00] [ERROR] [client.go:453] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:447\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:453\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.042Z] [2021/05/08 15:00:26.037 +08:00] [ERROR] [client.go:453] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:447\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:453\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.042Z] [2021/05/08 15:00:26.130 +08:00] [ERROR] [client.go:453] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:447\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:453\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:441\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:808\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:595\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]
[2021-05-08T07:00:43.042Z] [2021/05/08 15:00:26.900 +08:00] [ERROR] [client_test.go:1541] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).TestIncompatibleTiKV.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:1541\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:222"]
[2021-05-08T07:00:43.042Z] [2021/05/08 15:00:33.989 +08:00] [ERROR] [client_test.go:2730] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).testKVClientForceReconnect.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:2730\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:222"]
[2021-05-08T07:00:43.042Z] [2021/05/08 15:00:35.075 +08:00] [ERROR] [client_test.go:2783] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).testKVClientForceReconnect.func4\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:2783\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:222"]
[2021-05-08T07:00:43.043Z] [2021/05/08 15:00:42.106 +08:00] [ERROR] [client_test.go:2730] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).testKVClientForceReconnect.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:2730\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:222"]
[2021-05-08T07:00:43.043Z] [2021/05/08 15:00:42.109 +08:00] [ERROR] [client_test.go:2730] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).testKVClientForceReconnect.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:2730\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:222"]
[2021-05-08T07:00:43.043Z] panic: close of closed channel
[2021-05-08T07:00:43.043Z] 
[2021-05-08T07:00:43.043Z] goroutine 4727 [running]:
[2021-05-08T07:00:43.043Z] github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).testKVClientForceReconnect.func1.1(0xc000b8d200, 0xc0012bb380, 0xc0004c7800)
[2021-05-08T07:00:43.043Z] 	/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:2723 +0x39
[2021-05-08T07:00:43.043Z] github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).testKVClientForceReconnect.func1(0x5a92400, 0xc000e621f0)
[2021-05-08T07:00:43.043Z] 	/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:2734 +0x281
[2021-05-08T07:00:43.043Z] github.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1(0xc000ffd0e0, 0x5a92400, 0xc000e621f0)
[2021-05-08T07:00:43.043Z] 	/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:222 +0x60
[2021-05-08T07:00:43.043Z] created by github.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed
[2021-05-08T07:00:43.043Z] 	/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:221 +0x463
[2021-05-08T07:00:43.043Z] FAIL	github.com/pingcap/ticdc/cdc/kv	64.535s

@amyangfei
Copy link
Contributor Author

unit test data race

[2021-05-08T10:38:57.782Z] [2021/05/08 18:35:21.692 +08:00] [INFO] [unified_sorter.go:121] ["Unified Sorter: starting cleaning up files"]
[2021-05-08T10:38:57.782Z] [2021/05/08 18:35:21.710 +08:00] [INFO] [backend_pool.go:132] ["Temporary file removed"] [file=/tmp/sorter/sort-284724-33.tmp]
[2021-05-08T10:38:57.782Z] [2021/05/08 18:35:21.712 +08:00] [INFO] [backend_pool.go:85] ["Unified Sorter backEnd is being cancelled"]
[2021-05-08T10:38:57.782Z] ==================
[2021-05-08T10:38:57.782Z] WARNING: DATA RACE
[2021-05-08T10:38:57.782Z] Write at 0x000006b3e2a0 by goroutine 153:
[2021-05-08T10:38:57.782Z]   github.com/pingcap/ticdc/cdc/puller/sorter.UnifiedSorterCleanUp()
[2021-05-08T10:38:57.782Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/unified_sorter.go:123 +0x12d
[2021-05-08T10:38:57.782Z]   runtime.call32()
[2021-05-08T10:38:57.782Z]       /usr/local/go/src/runtime/asm_amd64.s:539 +0x3a
[2021-05-08T10:38:57.782Z]   github.com/pingcap/check.(*C).Fatal()
[2021-05-08T10:38:57.782Z]       /go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/helpers.go:141 +0x1ba
[2021-05-08T10:38:57.782Z]   github.com/pingcap/ticdc/cdc/puller/sorter.(*sorterSuite).TestSorterCancel()
[2021-05-08T10:38:57.782Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/sorter_test.go:115 +0x5ba
[2021-05-08T10:38:57.782Z]   runtime.call32()
[2021-05-08T10:38:57.782Z]       /usr/local/go/src/runtime/asm_amd64.s:539 +0x3a
[2021-05-08T10:38:57.782Z]   reflect.Value.Call()
[2021-05-08T10:38:57.782Z]       /usr/local/go/src/reflect/value.go:321 +0xd3
[2021-05-08T10:38:57.782Z]   github.com/pingcap/check.(*suiteRunner).forkTest.func1()
[2021-05-08T10:38:57.782Z]       /go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:850 +0x9aa
[2021-05-08T10:38:57.782Z]   github.com/pingcap/check.(*suiteRunner).forkCall.func1()
[2021-05-08T10:38:57.782Z]       /go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:739 +0x113
[2021-05-08T10:38:57.782Z] 
[2021-05-08T10:38:57.782Z] Previous read at 0x000006b3e2a0 by goroutine 143:
[2021-05-08T10:38:57.782Z]   github.com/pingcap/ticdc/cdc/puller/sorter.(*fileBackEnd).cleanStats()
[2021-05-08T10:38:57.782Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/file_backend.go:140 +0xd0
[2021-05-08T10:38:57.782Z]   github.com/pingcap/ticdc/cdc/puller/sorter.(*backEndPool).dealloc()
[2021-05-08T10:38:57.782Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/backend_pool.go:202 +0x447
[2021-05-08T10:38:57.782Z]   github.com/pingcap/ticdc/cdc/puller/sorter.(*heapSorter).flush.func1()
[2021-05-08T10:38:57.782Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/heap_sorter.go:157 +0x11c
[2021-05-08T10:38:57.782Z]   github.com/pingcap/ticdc/cdc/puller/sorter.runMerger.func1.1()
[2021-05-08T10:38:57.782Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/merger.go:88 +0x140
[2021-05-08T10:38:57.782Z]   github.com/pingcap/ticdc/cdc/puller/sorter.runMerger.func1()
[2021-05-08T10:38:57.782Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/merger.go:103 +0x221
[2021-05-08T10:38:57.782Z]   github.com/pingcap/ticdc/cdc/puller/sorter.runMerger()
[2021-05-08T10:38:57.782Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/merger.go:480 +0x12f8
[2021-05-08T10:38:57.782Z]   github.com/pingcap/ticdc/cdc/puller/sorter.(*UnifiedSorter).Run.func4()
[2021-05-08T10:38:57.782Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/unified_sorter.go:190 +0x10e
[2021-05-08T10:38:57.782Z]   golang.org/x/sync/errgroup.(*Group).Go.func1()
[2021-05-08T10:38:57.782Z]       /go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57 +0x71
[2021-05-08T10:38:57.782Z] 
[2021-05-08T10:38:57.782Z] Goroutine 153 (running) created at:
[2021-05-08T10:38:57.782Z]   github.com/pingcap/check.(*suiteRunner).forkCall()
[2021-05-08T10:38:57.782Z]       /go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:734 +0x4a3
[2021-05-08T10:38:57.782Z]   github.com/pingcap/check.(*suiteRunner).forkTest()
[2021-05-08T10:38:57.782Z]       /go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:832 +0x1b9
[2021-05-08T10:38:57.782Z]   github.com/pingcap/check.(*suiteRunner).doRun()
[2021-05-08T10:38:57.782Z]       /go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:666 +0x13a
[2021-05-08T10:38:57.782Z]   github.com/pingcap/check.(*suiteRunner).run()
[2021-05-08T10:38:57.782Z]       /go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:696 +0x13f
[2021-05-08T10:38:57.782Z]   github.com/pingcap/check.Run()
[2021-05-08T10:38:57.782Z]       /go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/run.go:150 +0x5a
[2021-05-08T10:38:57.782Z]   github.com/pingcap/check.RunAll()
[2021-05-08T10:38:57.782Z]       /go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/run.go:117 +0x13db
[2021-05-08T10:38:57.782Z]   github.com/pingcap/check.TestingT()
[2021-05-08T10:38:57.782Z]       /go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/run.go:99 +0x765
[2021-05-08T10:38:57.782Z]   github.com/pingcap/ticdc/cdc/puller/sorter.Test()
[2021-05-08T10:38:57.782Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/sorter_test.go:46 +0x38
[2021-05-08T10:38:57.782Z]   testing.tRunner()
[2021-05-08T10:38:57.782Z]       /usr/local/go/src/testing/testing.go:909 +0x199
[2021-05-08T10:38:57.782Z] 
[2021-05-08T10:38:57.782Z] Goroutine 143 (running) created at:
[2021-05-08T10:38:57.782Z]   golang.org/x/sync/errgroup.(*Group).Go()
[2021-05-08T10:38:57.782Z]       /go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:54 +0x73
[2021-05-08T10:38:57.782Z]   github.com/pingcap/ticdc/cdc/puller/sorter.(*UnifiedSorter).Run()
[2021-05-08T10:38:57.782Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/unified_sorter.go:189 +0xd28
[2021-05-08T10:38:57.782Z]   github.com/pingcap/ticdc/cdc/puller/sorter.testSorter.func1()
[2021-05-08T10:38:57.782Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/sorter_test.go:131 +0x5c
[2021-05-08T10:38:57.782Z]   golang.org/x/sync/errgroup.(*Group).Go.func1()
[2021-05-08T10:38:57.782Z]       /go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57 +0x71
[2021-05-08T10:38:57.782Z] ==================
[2021-05-08T10:38:57.782Z] 
[2021-05-08T10:38:57.782Z] ----------------------------------------------------------------------
[2021-05-08T10:38:57.782Z] FAIL: sorter_test.go:84: sorterSuite.TestSorterCancel
[2021-05-08T10:38:57.782Z] 
[2021-05-08T10:38:57.782Z] sorter_test.go:115:
[2021-05-08T10:38:57.782Z]     c.Fatal("TestSorterCancel timed out")
[2021-05-08T10:38:57.782Z] ... Error: TestSorterCancel timed out
[2021-05-08T10:38:57.782Z] 
[2021-05-08T10:38:57.782Z] [2021/05/08 18:35:21.724 +08:00] [INFO] [unified_sorter.go:130] ["sorterDebug: Running Unified Sorter in debug mode"]
[2021-05-08T10:38:57.782Z] [2021/05/08 18:35:21.717 +08:00] [WARN] [merger.go:507] ["Unified Sorter: Error detected"] [error="[CDC:ErrUnifiedSorterBackendTerminating]unified sorter backend is terminating"] [errorVerbose="[CDC:ErrUnifiedSorterBackendTerminating]unified sorter backend is terminating\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ngithub.com/pingcap/ticdc/cdc/puller/sorter.(*backEndPool).dealloc\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/backend_pool.go:208\ngithub.com/pingcap/ticdc/cdc/puller/sorter.(*heapSorter).flush.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/heap_sorter.go:157\ngithub.com/pingcap/ticdc/cdc/puller/sorter.runMerger.func1.1\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/merger.go:88\ngithub.com/pingcap/ticdc/cdc/puller/sorter.runMerger.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/merger.go:103\ngithub.com/pingcap/ticdc/cdc/puller/sorter.runMerger\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/merger.go:480\ngithub.com/pingcap/ticdc/cdc/puller/sorter.(*UnifiedSorter).Run.func4\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/unified_sorter.go:190\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"]
[2021-05-08T10:38:57.783Z] ==================
[2021-05-08T10:38:57.783Z] WARNING: DATA RACE
[2021-05-08T10:38:57.783Z] Write at 0x00c000bbc200 by goroutine 199:
[2021-05-08T10:38:57.783Z]   github.com/pingcap/ticdc/pkg/workerpool.(*defaultAsyncPoolImpl).prepare()
[2021-05-08T10:38:57.783Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/pkg/workerpool/async_pool_impl.go:142 +0x141
[2021-05-08T10:38:57.783Z]   github.com/pingcap/ticdc/pkg/workerpool.(*defaultAsyncPoolImpl).Run()
[2021-05-08T10:38:57.783Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/pkg/workerpool/async_pool_impl.go:95 +0x5b
[2021-05-08T10:38:57.783Z]   github.com/pingcap/ticdc/cdc/puller/sorter.RunWorkerPool.func2()
[2021-05-08T10:38:57.783Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/unified_sorter.go:290 +0x9e
[2021-05-08T10:38:57.783Z]   golang.org/x/sync/errgroup.(*Group).Go.func1()
[2021-05-08T10:38:57.783Z]       /go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57 +0x71
[2021-05-08T10:38:57.783Z] 
[2021-05-08T10:38:57.783Z] Previous read at 0x00c000bbc200 by goroutine 226:
[2021-05-08T10:38:57.783Z]   github.com/pingcap/ticdc/pkg/workerpool.(*defaultAsyncPoolImpl).Run.func3()
[2021-05-08T10:38:57.783Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/pkg/workerpool/async_pool_impl.go:130 +0x19c
[2021-05-08T10:38:57.783Z]   golang.org/x/sync/errgroup.(*Group).Go.func1()
[2021-05-08T10:38:57.783Z]       /go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57 +0x71
[2021-05-08T10:38:57.783Z] 
[2021-05-08T10:38:57.783Z] Goroutine 199 (running) created at:
[2021-05-08T10:38:57.783Z]   golang.org/x/sync/errgroup.(*Group).Go()
[2021-05-08T10:38:57.783Z]       /go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:54 +0x73
[2021-05-08T10:38:57.783Z]   github.com/pingcap/ticdc/cdc/puller/sorter.RunWorkerPool()
[2021-05-08T10:38:57.783Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/unified_sorter.go:289 +0x1f8
[2021-05-08T10:38:57.783Z]   github.com/pingcap/ticdc/cdc/puller/sorter.testSorter.func2()
[2021-05-08T10:38:57.783Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/sorter_test.go:136 +0x4a
[2021-05-08T10:38:57.783Z]   golang.org/x/sync/errgroup.(*Group).Go.func1()
[2021-05-08T10:38:57.783Z]       /go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57 +0x71
[2021-05-08T10:38:57.783Z] 
[2021-05-08T10:38:57.783Z] Goroutine 226 (finished) created at:
[2021-05-08T10:38:57.783Z]   golang.org/x/sync/errgroup.(*Group).Go()
[2021-05-08T10:38:57.783Z]       /go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:54 +0x73
[2021-05-08T10:38:57.783Z]   github.com/pingcap/ticdc/pkg/workerpool.(*defaultAsyncPoolImpl).Run()
[2021-05-08T10:38:57.783Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/pkg/workerpool/async_pool_impl.go:122 +0x426
[2021-05-08T10:38:57.783Z]   github.com/pingcap/ticdc/cdc/puller/sorter.RunWorkerPool.func2()
[2021-05-08T10:38:57.783Z]       /home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/puller/sorter/unified_sorter.go:290 +0x9e
[2021-05-08T10:38:57.783Z]   golang.org/x/sync/errgroup.(*Group).Go.func1()
[2021-05-08T10:38:57.783Z]       /go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57 +0x71
[2021-05-08T10:38:57.783Z] ==================
[2021-05-08T10:38:57.783Z] [2021/05/08 18:35:21.781 +08:00] [INFO] [merger.go:114] ["Merger has exited"]
[2021-05-08T10:38:57.783Z] [2021/05/08 18:35:31.725 +08:00] [INFO] [merger.go:66] ["Unified Sorter: merger exiting, cleaning up resources"] [pending-set-size=48]

internal.pingcap.net.txt

@amyangfei
Copy link
Contributor Author 

[2021-05-13T06:40:47.277Z] + set +x
[2021-05-13T06:40:47.277Z] [Thu May 13 14:40:46 CST 2021] <<<<<< START cdc server in unified_sorter_sort_dir_conflict case >>>>>>
[2021-05-13T06:41:13.786Z] [Thu May 13 14:41:10 CST 2021] <<<<<< START cdc server in unified_sorter_sort_dir_conflict case >>>>>>
[2021-05-13T06:41:13.786Z] check_changefeed_mark_stopped_regex http://127.0.0.1:2379 b5eddeb9-2cd6-4a43-a597-214f4d5a90a2 .*ErrConflictingFileLocks.*
[2021-05-13T06:41:13.786Z] {
[2021-05-13T06:41:13.786Z]  "state": "normal",
[2021-05-13T06:41:13.786Z]  "tso": 424906082122989572,
[2021-05-13T06:41:13.786Z]  "checkpoint": "2021-05-13 14:41:09.622",
[2021-05-13T06:41:13.786Z]  "error": null
[2021-05-13T06:41:13.786Z] }
[2021-05-13T06:41:13.786Z] changefeed state normal does not equal to stopped
[2021-05-13T06:41:13.786Z] run task failed 1-th time, retry later
[2021-05-13T06:41:13.786Z] check_changefeed_mark_stopped_regex http://127.0.0.1:2379 b5eddeb9-2cd6-4a43-a597-214f4d5a90a2 .*ErrConflictingFileLocks.*
[2021-05-13T06:41:13.786Z] {
[2021-05-13T06:41:13.786Z]  "state": "normal",
[2021-05-13T06:41:13.786Z]  "tso": 424906082122989572,
[2021-05-13T06:41:13.786Z]  "checkpoint": "2021-05-13 14:41:09.622",
[2021-05-13T06:41:13.786Z]  "error": {
[2021-05-13T06:41:13.786Z]   "addr": "127.0.0.1:8301",
[2021-05-13T06:41:13.786Z]   "code": "CDC:ErrProcessorUnknown",
[2021-05-13T06:41:13.786Z]   "message": "[ErrSortDirLockError][ErrConflictingFileLocks]file lock conflict: /tmp/cdc_sort_1/ticdc_lock"
[2021-05-13T06:41:13.786Z]  }
[2021-05-13T06:41:13.786Z] }
[2021-05-13T06:41:13.786Z] changefeed state normal does not equal to stopped
[2021-05-13T06:41:13.786Z] run task failed 2-th time, retry later
[2021-05-13T06:41:15.680Z] check_changefeed_mark_stopped_regex http://127.0.0.1:2379 b5eddeb9-2cd6-4a43-a597-214f4d5a90a2 .*ErrConflictingFileLocks.*
[2021-05-13T06:41:15.680Z] {
[2021-05-13T06:41:15.680Z]  "state": "stopped",
[2021-05-13T06:41:15.680Z]  "tso": 424906082122989572,
[2021-05-13T06:41:15.680Z]  "checkpoint": "2021-05-13 14:41:09.622",
[2021-05-13T06:41:15.680Z]  "error": {
[2021-05-13T06:41:15.680Z]   "addr": "127.0.0.1:8301",
[2021-05-13T06:41:15.680Z]   "code": "CDC:ErrProcessorUnknown",
[2021-05-13T06:41:15.680Z]   "message": "[ErrSortDirLockError][ErrConflictingFileLocks]file lock conflict: /tmp/cdc_sort_1/ticdc_lock"
[2021-05-13T06:41:15.680Z]  }
[2021-05-13T06:41:15.680Z] }
[2021-05-13T06:41:15.680Z] run task successfully
[2021-05-13T06:41:15.680Z] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
[2021-05-13T06:41:22.227Z] cdc: no process found
[2021-05-13T06:41:23.597Z] tiflash: no process found
[2021-05-13T06:41:23.597Z] flash_cluster_manager: no process found
[2021-05-13T06:41:35.775Z] make: *** [integration_test_mysql] Error 1

ref: https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/cdc_ghpr_integration_test/detail/cdc_ghpr_integration_test/5870/pipeline

@Rustin170506
Copy link
Member

Fixed

unit test

failure: https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/cdc_ghpr_integration_test/detail/cdc_ghpr_integration_test/5284/pipeline

[2021-04-15T02:30:54.769Z] [2021/04/15 10:21:20.814 +08:00] [ERROR] [client_test.go:2703] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).testKVClientForceReconnect.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:2703\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:222"]
[2021-04-15T02:30:54.769Z] [2021/04/15 10:21:29.039 +08:00] [ERROR] [client_test.go:2703] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).testKVClientForceReconnect.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:2703\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:222"]
[2021-04-15T02:30:54.769Z] [2021/04/15 10:21:29.252 +08:00] [ERROR] [client_v2.go:81] ["received an event but neither pending region nor running region was found"] [regionID=3] [requestID=49] [addr=127.0.0.1:13704] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).sendRegionChangeEventV2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_v2.go:81\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStreamV2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_v2.go:274\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).dispatchRequest.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:815\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]

[2021-04-15T02:30:54.770Z] goroutine 198 [select, 9 minutes]:
[2021-04-15T02:30:54.770Z] github.com/pingcap/check.(*suiteRunner).doRun(0xc000899050, 0x0)
[2021-04-15T02:30:54.770Z] 	/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:667 +0x228
[2021-04-15T02:30:54.770Z] github.com/pingcap/check.(*suiteRunner).run(0xc000899050, 0xc000445b40)
[2021-04-15T02:30:54.770Z] 	/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:696 +0x140
[2021-04-15T02:30:54.770Z] github.com/pingcap/check.Run(0x4fc2500, 0xc000445b40, 0xc000647eb8, 0xc000898e10)
[2021-04-15T02:30:54.770Z] 	/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/run.go:150 +0x5b
[2021-04-15T02:30:54.770Z] github.com/pingcap/check.RunAll(0xc000647eb8, 0x0)
[2021-04-15T02:30:54.770Z] 	/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/run.go:113 +0xf3a
[2021-04-15T02:30:54.770Z] github.com/pingcap/check.TestingT(0xc000416600)
[2021-04-15T02:30:54.770Z] 	/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/run.go:99 +0x766
[2021-04-15T02:30:54.770Z] github.com/pingcap/ticdc/cdc/kv.Test(0xc000416600)
[2021-04-15T02:30:54.770Z] 	/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:48 +0x39
[2021-04-15T02:30:54.770Z] testing.tRunner(0xc000416600, 0x50ecc98)
[2021-04-15T02:30:54.770Z] 	/usr/local/go/src/testing/testing.go:909 +0x19a
[2021-04-15T02:30:54.770Z] created by testing.(*T).Run
[2021-04-15T02:30:54.770Z] 	/usr/local/go/src/testing/testing.go:960 +0x652

Another failure: https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_integration_test/detail/cdc_ghpr_integration_test/1223/pipeline

[2021-06-24T09:10:43.640Z] 

[2021-06-24T09:10:43.640Z] ----------------------------------------------------------------------

[2021-06-24T09:10:43.640Z] PANIC: client_test.go:2907: etcdSuite.TestKVClientForceReconnect

[2021-06-24T09:10:43.640Z] 

[2021-06-24T09:10:43.640Z] ... Panic: send on closed channel (PC=0x1135258)

[2021-06-24T09:10:43.640Z] 

[2021-06-24T09:10:43.640Z] /usr/local/go/src/runtime/panic.go:971

[2021-06-24T09:10:43.640Z]   in gopanic

[2021-06-24T09:10:43.640Z] /usr/local/go/src/runtime/chan.go:204

[2021-06-24T09:10:43.640Z]   in chansend

[2021-06-24T09:10:43.640Z] /usr/local/go/src/runtime/chan.go:143

[2021-06-24T09:10:43.640Z]   in chansend1

[2021-06-24T09:10:43.640Z] client_test.go:2831

[2021-06-24T09:10:43.640Z]   in etcdSuite.testKVClientForceReconnect

[2021-06-24T09:10:43.640Z] client_test.go:2921

[2021-06-24T09:10:43.640Z]   in etcdSuite.TestKVClientForceReconnect

[2021-06-24T09:10:43.640Z] /usr/local/go/src/reflect/value.go:337

[2021-06-24T09:10:43.640Z]   in Value.Call

[2021-06-24T09:10:43.640Z] /nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:850

[2021-06-24T09:10:43.640Z]   in suiteRunner.forkTest.func1

[2021-06-24T09:10:43.640Z] /nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:739

[2021-06-24T09:10:43.640Z]   in suiteRunner.forkCall.func1

[2021-06-24T09:10:43.640Z] /usr/local/go/src/runtime/asm_amd64.s:1371

[2021-06-24T09:10:43.640Z]   in goexit

[2021-06-24T09:10:43.640Z] [2021/06/24 17:09:50.443 +08:00] [ERROR] [client_v2.go:87] ["received an event but neither pending region nor running region was found"] [regionID=3] [requestID=80] [addr=127.0.0.1:40940] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).sendRegionChangeEventV2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_v2.go:87\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStreamV2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_v2.go:262\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:857\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/nfs/cache/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]

[2021-06-24T09:10:43.640Z] [2021/06/24 17:10:19.227 +08:00] [ERROR] [client_test.go:1381] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:1381\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:235"]

[2021-06-24T09:10:43.641Z] [2021/06/24 17:10:19.228 +08:00] [ERROR] [client_v2.go:219] ["failed to receive from stream"] [addr=127.0.0.1:46623] [storeID=1] [error=] [errorVerbose="\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStreamV2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_v2.go:208\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:857\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/nfs/cache/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStreamV2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_v2.go:219\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:857\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/nfs/cache/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]

[2021-06-24T09:10:43.641Z] [2021/06/24 17:10:19.272 +08:00] [ERROR] [client_test.go:1381] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:1381\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:235"]

[2021-06-24T09:10:43.641Z] [2021/06/24 17:10:19.823 +08:00] [ERROR] [client_v2.go:219] ["failed to receive from stream"] [addr=127.0.0.1:41463] [storeID=1] [error=EOF] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStreamV2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_v2.go:219\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:857\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/nfs/cache/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]

[2021-06-24T09:10:43.641Z] [2021/06/24 17:10:20.938 +08:00] [ERROR] [client.go:1279] ["failed to receive from stream"] [addr=127.0.0.1:46385] [storeID=1] [error=EOF] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1279\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:855\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/nfs/cache/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]

[2021-06-24T09:10:43.641Z] [2021/06/24 17:10:24.466 +08:00] [ERROR] [client_v2.go:219] ["failed to receive from stream"] [addr=127.0.0.1:38466] [storeID=1] [error="injected stream recv error"] [errorVerbose="injected stream recv error\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStreamV2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_v2.go:208\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:857\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/nfs/cache/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStreamV2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_v2.go:219\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:857\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/nfs/cache/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]

[2021-06-24T09:10:43.641Z] [2021/06/24 17:10:25.621 +08:00] [ERROR] [client.go:1279] ["failed to receive from stream"] [addr=127.0.0.1:46798] [storeID=1] [error="injected stream recv error"] [errorVerbose="injected stream recv error\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1268\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:855\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/nfs/cache/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1279\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:855\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/nfs/cache/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]

[2021-06-24T09:10:43.641Z] [2021/06/24 17:10:29.042 +08:00] [ERROR] [client_v2.go:219] ["failed to receive from stream"] [addr=127.0.0.1:45152] [storeID=1] [error=EOF] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStreamV2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_v2.go:219\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:857\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/nfs/cache/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57"]

[2021-06-24T09:10:43.642Z] [2021/06/24 17:10:29.304 +08:00] [ERROR] [client_test.go:1193] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).TestStreamSendWithError.func3\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:1193\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:235"]

[2021-06-24T09:10:43.642Z] OOPS: 60 passed, 1 PANICKED

@Rustin170506
Copy link
Member

Use https://github.com/pingcap/ticdc/issues/2246 to track unstable tests. The current issue has too much information and logs.

@AkiraXie AkiraXie added the area/ticdc Issues or PRs related to TiCDC. label Mar 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/ticdc Issues or PRs related to TiCDC. component/test Unit tests and integration tests component. priority/P1 The issue has P1 priority. severity/minor type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@amyangfei @ChenPeng2013 @Rustin170506 @AkiraXie