Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

data consistency when running mysql sink changefeed with redo log and injecting S3 network partition #11744

Closed
fubinzh opened this issue Nov 14, 2024 · 2 comments · Fixed by #11747
Closed

data consistency when running mysql sink changefeed with redo log and injecting S3 network partition #11744

fubinzh opened this issue Nov 14, 2024 · 2 comments · Fixed by #11747
Assignees
@CharlesCheung96
Labels
affects-6.5 affects-7.1 affects-7.5 affects-8.1 affects-8.5 area/ticdc Issues or PRs related to TiCDC. severity/critical type/bug The issue is confirmed as a bug.

Comments

@fubinzh
Copy link

fubinzh commented Nov 14, 2024

What did you do?

  1. create tidb sink changefeed with redo and syncpoint on
  2. run workload
  3. inject cdc and redo log storage server network partition for 5m
  4. sleep 2m
  5. pause changefeed
  6. run redo apply
  7. run redo meta command, get resolved ts of changefeed (TS1)
  8. compare data consistency between upstream snapshot TS1 and downstream

What did you expect to see?

Downstream and upstream (snapshot TS1) should be eventually consistent

What did you see instead?

There is a low possibility (4 repreduces out of 50 executions ) that data will be inconsistent.

Versions of the cluster

CDC version:
release-version: v8.5.0-alpha-7-g64decf622
git-hash: 64decf6
@fubinzh fubinzh added area/ticdc Issues or PRs related to TiCDC. type/bug The issue is confirmed as a bug. labels Nov 14, 2024
@fubinzh
Copy link
Author

fubinzh commented Nov 14, 2024

/assign @CharlesCheung96

@fubinzh
Copy link
Author

fubinzh commented Nov 14, 2024

/severity critical

@CharlesCheung96 CharlesCheung96 mentioned this issue Nov 14, 2024
Merged
@ti-chi-bot ti-chi-bot bot closed this as completed in ea35677 Nov 16, 2024
This was referenced Nov 16, 2024
Merged
Merged
This was referenced Nov 16, 2024
Merged
Merged
Merged
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue Nov 16, 2024
@CharlesCheung96
ticdc(redo, sink): return correct error in redo writer & fix default …
0684be7 
…retryer (pingcap#11747)

close pingcap#11744
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue Nov 16, 2024
@CharlesCheung96
ticdc(redo, sink): return correct error in redo writer & fix default …
4ad9544 
…retryer (pingcap#11747)

close pingcap#11744
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue Nov 16, 2024
@CharlesCheung96
ticdc(redo, sink): return correct error in redo writer & fix default …
7bc1754 
…retryer (pingcap#11747)

close pingcap#11744
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue Nov 16, 2024
@CharlesCheung96
ticdc(redo, sink): return correct error in redo writer & fix default …
7f64b80 
…retryer (pingcap#11747)

close pingcap#11744
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue Nov 16, 2024
@CharlesCheung96
ticdc(redo, sink): return correct error in redo writer & fix default …
059f8e1 
…retryer (pingcap#11747)

close pingcap#11744
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue Nov 16, 2024
@CharlesCheung96
ticdc(redo, sink): return correct error in redo writer & fix default …
5ba4480 
…retryer (pingcap#11747)

close pingcap#11744
ti-chi-bot bot pushed a commit that referenced this issue Nov 16, 2024
@ti-chi-bot
ticdc(redo, sink): return correct error in redo writer & fix default …
3f2d1ac 
…retryer (#11747) (#11760)

close #11744
ti-chi-bot bot pushed a commit that referenced this issue Nov 16, 2024
@ti-chi-bot
ticdc(redo, sink): return correct error in redo writer & fix default …
502a4c8 
…retryer (#11747) (#11762)

close #11744
ti-chi-bot bot pushed a commit that referenced this issue Nov 17, 2024
@ti-chi-bot
ticdc(redo, sink): return correct error in redo writer & fix default …
efe4d97 
…retryer (#11747) (#11761)

close #11744
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue Nov 17, 2024
@CharlesCheung96
ticdc(redo, sink): return correct error in redo writer & fix default …
5aac693 
…retryer (pingcap#11747)

close pingcap#11744
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this issue Nov 17, 2024
@CharlesCheung96
ticdc(redo, sink): return correct error in redo writer & fix default …
acd6495 
…retryer (pingcap#11747)

close pingcap#11744
ti-chi-bot bot pushed a commit that referenced this issue Nov 18, 2024
@ti-chi-bot
ticdc(redo, sink): return correct error in redo writer & fix default …
a5a6bb5 
…retryer (#11747) (#11763)

close #11744
ti-chi-bot bot pushed a commit that referenced this issue Nov 18, 2024
@ti-chi-bot
ticdc(redo, sink): return correct error in redo writer & fix default …
cd90b6b 
…retryer (#11747) (#11764)

close #11744
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@CharlesCheung96 CharlesCheung96

Labels
affects-6.5 affects-7.1 affects-7.5 affects-8.1 affects-8.5 area/ticdc Issues or PRs related to TiCDC. severity/critical type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ticdc(redo, sink): return correct error in redo writer & fix default retryer CharlesCheung96/tiflow
2 participants
@fubinzh @CharlesCheung96