Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cdc: panic after disable syncpoint through cli update #4934

Closed
ben1009 opened this issue Mar 17, 2022 · 3 comments · Fixed by #6390
Closed

cdc: panic after disable syncpoint through cli update #4934

ben1009 opened this issue Mar 17, 2022 · 3 comments · Fixed by #6390

Comments

@ben1009
Copy link
Contributor

ben1009 commented Mar 17, 2022

What did you do?

create changefeed with syncpoint enable
pause
update with syncpoint disable
resume

What did you expect to see?

not panic

What did you see instead?

cdc panic instead

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x2eac771]

goroutine 543 [running]:
github.com/pingcap/tiflow/cdc/owner.(*ddlSinkImpl).emitSyncPoint(0xc0009ae690, 0x41031c8, 0xc000b75650, 0x5fe5c9046dc0000, 0x3093e9c0, 0xed9c4f221)
	github.com/pingcap/tiflow/cdc/owner/ddl_sink.go:205 +0x71
github.com/pingcap/tiflow/cdc/owner.(*changefeed).handleBarrier(0xc002448a20, 0x41031c8, 0xc000b75650, 0x5fe5c9046dc0000, 0x0, 0x0)
	github.com/pingcap/tiflow/cdc/owner/changefeed.go:464 +0x1bc
github.com/pingcap/tiflow/cdc/owner.(*changefeed).tick(0xc002448a20, 0x41031c8, 0xc000b75650, 0xc00249daa0, 0xc001b22b70, 0x2eb20ba, 0xc00335cf30)
	github.com/pingcap/tiflow/cdc/owner/changefeed.go:178 +0x24d
github.com/pingcap/tiflow/cdc/owner.(*changefeed).Tick(0xc002448a20, 0x41030d8, 0xc000b75638, 0xc00249daa0, 0xc001b22b70)
	github.com/pingcap/tiflow/cdc/owner/changefeed.go:117 +0x149
github.com/pingcap/tiflow/cdc/owner.(*Owner).Tick(0xc000613030, 0x7f19e2a75cf0, 0xc000ba08a0, 0x40af638, 0xc000916440, 0x1, 0x0, 0x1, 0x0)
	github.com/pingcap/tiflow/cdc/owner/owner.go:177 +0x36d
github.com/pingcap/tiflow/pkg/orchestrator.(*EtcdWorker).Run(0xc0004b3780, 0x7f19e2a75cf0, 0xc000ba08a0, 0xc001b22900, 0xbebc200, 0x7fff617b0e66, 0x12, 0x0, 0x0)
	github.com/pingcap/tiflow/pkg/orchestrator/etcd_worker.go:239 +0x591
github.com/pingcap/tiflow/cdc/capture.(*Capture).runEtcdWorker(0xc001e20000, 0x4103178, 0xc000ba08a0, 0x40728a0, 0xc000613030, 0x40af638, 0xc000916440, 0xbebc200, 0x0, 0x0)
	github.com/pingcap/tiflow/cdc/capture/capture.go:450 +0x167
github.com/pingcap/tiflow/cdc/capture.(*Capture).campaignOwner(0xc001e20000, 0x4103178, 0xc00063ce40, 0x0, 0x0)
	github.com/pingcap/tiflow/cdc/capture/capture.go:428 +0xa25
github.com/pingcap/tiflow/cdc/capture.(*Capture).run.func2(0xc000f92100, 0xc001e20000, 0x4103178, 0xc00063ce40, 0xc001b800e0)
	github.com/pingcap/tiflow/cdc/capture/capture.go:292 +0xb5
created by github.com/pingcap/tiflow/cdc/capture.(*Capture).run
	github.com/pingcap/tiflow/cdc/capture/capture.go:286 +0x37b

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

5.4

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

5.4
@asddongmen
Copy link
Contributor

This error never appeared again in v5.4.0 and master branch.

@asddongmen
Copy link
Contributor

This problem is cause by we assignment and use ddlSinkImpl.syncPointStore in different goroutine.

Reproducing steps:

@MimeLyc
Copy link

MimeLyc commented Jul 25, 2022

/remove-label affects-6.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment