cdc server panic since pebble event sorter iter read nil value

[3AceShowHand]

What did you do?

run normal integration test

What did you expect to see?

run normally

What did you see instead?

unexpected fault address 0x7f308dc5318b
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7f308dc5318b pc=0x57defa5]

goroutine 1833 [running]:
runtime.throw({0xaaff5b7?, 0x0?})
	/usr/local/go/src/runtime/panic.go:1077 +0x5c fp=0xc0032953a0 sp=0xc003295370 pc=0x2428e7c
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:875 +0x285 fp=0xc003295400 sp=0xc0032953a0 pc=0x2442725
github.com/tinylib/msgp/msgp.ReadMapHeaderBytes({0x7f308dc5318b, 0x100073, 0x100073})
	/go/pkg/mod/github.com/tinylib/msgp@v1.1.6/msgp/read_bytes.go:166 +0x65 fp=0xc0032954a8 sp=0xc003295400 pc=0x57defa5
github.com/pingcap/tiflow/cdc/model.(*RawKVEntry).UnmarshalMsg(0xc0081c5a40, {0x7f308dc5318b, 0x100073, 0x100073})
	/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/cdc/model/kv_gen.go:245 +0xac fp=0xc003295678 sp=0xc0032954a8 pc=0x60c68cc
github.com/pingcap/tiflow/cdc/processor/sourcemanager/sorter/pebble/encoding.(*MsgPackGenSerde).Unmarshal(0xc002731900?, 0xc001720390, {0x7f308dc5318b, 0x100073, 0x100073})
	/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/cdc/processor/sourcemanager/sorter/pebble/encoding/value.go:43 +0x165 fp=0xc003295700 sp=0xc003295678 pc=0x884d9a5
github.com/pingcap/tiflow/cdc/processor/sourcemanager/sorter/pebble.(*EventIter).Next(0xc001720360)
	/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/cdc/processor/sourcemanager/sorter/pebble/event_sorter.go:326 +0x2dd fp=0xc0032957d0 sp=0xc003295700 pc=0x88548dd
github.com/pingcap/tiflow/cdc/processor/sourcemanager/sorter.(*MountedEventIter).readBatch(0xc00138f200, {0xccd4e30, 0xc00429c0f0})
	/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/cdc/processor/sourcemanager/sorter/mounted_iter.go:106 +0x5db fp=0xc003295900 sp=0xc0032957d0 pc=0x884867b
github.com/pingcap/tiflow/cdc/processor/sourcemanager/sorter.(*MountedEventIter).Next(0xc00138f200, {0xccd4e30, 0xc00429c0f0})
	/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/cdc/processor/sourcemanager/sorter/mounted_iter.go:63 +0x11a fp=0xc0032959b8 sp=0xc003295900 pc=0x8847bda
github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTask(0xc00429c320, {0xccd4e30, 0xc00429c0f0}, 0xc0016d48c0)
	/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:185 +0xa58 fp=0xc003295db8 sp=0xc0032959b8 pc=0x92a9d98
github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTasks(0xc00429c320?, {0xccd4e30, 0xc00429c0f0}, 0xc0046b64e0)
	/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:97 +0x26f fp=0xc003295ec8 sp=0xc003295db8 pc=0x92a90cf
github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*SinkManager).startSinkWorkers.func1()
	/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/cdc/processor/sinkmanager/manager.go:407 +0x8c fp=0xc003295f28 sp=0xc003295ec8 pc=0x928d46c
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:78 +0x98 fp=0xc003295fe0 sp=0xc003295f28 pc=0x4c828f8
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc003295fe8 sp=0xc003295fe0 pc=0x2463661
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1160
	/go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:75 +0x125

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

master

[fubinzh]

/severity major

[3AceShowHand]

0x7f308dc5318b looks like a memory address cannot be visited, the error may comes from the operating system, golang runtime, or the pebble.

[3AceShowHand]

This looks the same as #6367, the error may comes from the pebble.

[asddongmen]

Happens again at this PR:
CI Link: https://do.pingcap.net/jenkins/blue/organizations/jenkins/pingcap%2Ftiflow%2Frelease-7.5%2Fpull_cdc_integration_kafka_test/detail/pull_cdc_integration_kafka_test/444/pipeline/461/

Logs below is found:

unexpected fault address 0x7fe5b06f8193
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7fe5b06f8193 pc=0x52b8605]

goroutine 783 [running]:
runtime.throw({0x9abdacc?, 0xc004864281?})
	/usr/local/go/src/runtime/panic.go:1077 +0x5c fp=0xc005c21310 sp=0xc005c212e0 pc=0x21d147c
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:875 +0x285 fp=0xc005c21370 sp=0xc005c21310 pc=0x21eacc5
github.com/tinylib/msgp/msgp.ReadMapHeaderBytes({0x7fe5b06f8193, 0x10007c, 0x10007c})
	/go/pkg/mod/github.com/tinylib/msgp@v1.1.6/msgp/read_bytes.go:166 +0x65 fp=0xc005c21418 sp=0xc005c21370 pc=0x52b8605
github.com/pingcap/tiflow/cdc/model.(*RawKVEntry).UnmarshalMsg(0xc000ffdab0, {0x7fe5b06f8193, 0x10007c, 0x10007c})
	/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_kafka_test/tiflow/cdc/model/kv_gen.go:245 +0xac fp=0xc005c215f0 sp=0xc005c21418 pc=0x5be53ac
github.com/pingcap/tiflow/cdc/processor/sourcemanager/engine/pebble/encoding.(*MsgPackGenSerde).Unmarshal(0xc004864000?, 0xc005f484e0, {0x7fe5b06f8193, 0x10007c, 0x10007c})
	/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_kafka_test/tiflow/cdc/processor/sourcemanager/engine/pebble/encoding/value.go:43 +0x165 fp=0xc005c21678 sp=0xc005c215f0 pc=0x7fdbac5
github.com/pingcap/tiflow/cdc/processor/sourcemanager/engine/pebble.(*EventIter).Next(0xc005f48450)
	/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_kafka_test/tiflow/cdc/processor/sourcemanager/engine/pebble/event_sorter.go:329 +0x2fe fp=0xc005c21748 sp=0xc005c21678 pc=0x7fe313e
github.com/pingcap/tiflow/cdc/processor/sourcemanager/engine.(*MountedEventIter).readBatch(0xc004521290, {0xb9c2fd8, 0xc003b9c780})
	/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_kafka_test/tiflow/cdc/processor/sourcemanager/engine/mounted_iter.go:106 +0x5e7 fp=0xc005c21898 sp=0xc005c21748 pc=0x7fd6687
github.com/pingcap/tiflow/cdc/processor/sourcemanager/engine.(*MountedEventIter).Next(0xc004521290, {0xb9c2fd8, 0xc003b9c780})
	/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_kafka_test/tiflow/cdc/processor/sourcemanager/engine/mounted_iter.go:63 +0x125 fp=0xc005c21950 sp=0xc005c21898 pc=0x7fd5be5
github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTask(0xc003118600, {0xb9c2fd8, 0xc003b9c780}, 0xc000d26f50)
	/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_kafka_test/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:218 +0xf2c fp=0xc005c21dc8 sp=0xc005c21950 pc=0x8a5e92c
github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTasks(0x40012a0?, {0xb9c2fd8, 0xc003b9c780}, 0xc0014efec0)
	/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_kafka_test/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:107 +0x26f fp=0xc005c21ed8 sp=0xc005c21dc8 pc=0x8a5d78f
github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*SinkManager).startSinkWorkers.func1()
	/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_kafka_test/tiflow/cdc/processor/sinkmanager/manager.go:415 +0x8c fp=0xc005c21f38 sp=0xc005c21ed8 pc=0x8a3cb0c
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x77 fp=0xc005c21fe0 sp=0xc005c21f38 pc=0x4e5cbf7
runtime.goexit()

[seiya-annie]

/report customer