Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

owner, redo(ticdc): decouple globalResolvedTs and globalBarrierTs #8964

Merged
merged 5 commits into from
May 18, 2023
Merged

owner, redo(ticdc): decouple globalResolvedTs and globalBarrierTs #8964

merged 5 commits into from
May 18, 2023

Conversation

CharlesCheung96
Copy link
Contributor

@CharlesCheung96 CharlesCheung96 commented May 16, 2023 •
edited
Loading

What problem does this PR solve?

Issue Number: close #8963

What is changed and how it works?

  1. Decouple globalResolvedTs and globalBarrierTs.
  2. Add a redo ddl barrier to prevent resolvedTs from being advanced too early.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note 

`Fix the isuue that resolvedTs may be advanced too early when redo enabled`.

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 16, 2023 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • asddongmen
  • nongfushanquan

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 16, 2023

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ti-chi-bot ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/needs-triage-completed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels May 16, 2023
@CharlesCheung96 CharlesCheung96 marked this pull request as ready for review May 16, 2023 10:20
@ti-chi-bot ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 16, 2023
@CharlesCheung96
Copy link
Contributor Author

/test all

@CharlesCheung96
Copy link
Contributor Author

/test cdc-integration-mysql-test

@ti-chi-bot ti-chi-bot bot added needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed do-not-merge/needs-triage-completed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 16, 2023
@CharlesCheung96
Copy link
Contributor Author

/test all

1 similar comment
@CharlesCheung96
Copy link
Contributor Author

/test all

@ti-chi-bot ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 17, 2023
@CharlesCheung96 CharlesCheung96 changed the title redo(ticdc): support global ddl in redo writer owner, redo(ticdc): decouple globalResolvedTs and globalBarrierTs May 17, 2023
@ti-chi-bot ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 17, 2023
@ti-chi-bot ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 18, 2023
nongfushanquan
nongfushanquan reviewed May 18, 2023
View reviewed changes
Copy link
Contributor

@nongfushanquan nongfushanquan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you add some comments in detail about the issue

nongfushanquan
nongfushanquan reviewed May 18, 2023
View reviewed changes
cdc/owner/changefeed.go
@@ -443,8 +443,8 @@ func (c *changefeed) tick(ctx cdcContext.Context, captures map[model.CaptureID]*
}
})

c.updateStatus(newCheckpointTs, newResolvedTs, minTableBarrierTs)
c.updateMetrics(currentTs, newCheckpointTs, metricsResolvedTs)
c.updateStatus(newCheckpointTs, newResolvedTs, barrier.minDDLBarrierTs)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which one is the globalBarrierTs now?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updateStatus only update status in etcd, however globalBarrierTs is updated by p2p, ref: https://github.com/pingcap/tiflow/pull/8964/files#diff-953746562776cf8aa41cd4a971df8c546649d2721ae787607aed2069f3b44eb0R353-R354

@CharlesCheung96
Copy link
Contributor Author

could you add some comments in detail about the issue

updated, PTAL: #8963

@ti-chi-bot ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 18, 2023
@CharlesCheung96
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 18, 2023

This pull request has been accepted and is ready to merge.

Commit hash: 0f9f299

@ti-chi-bot ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label May 18, 2023
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 18, 2023

@CharlesCheung96: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

trigger some heavy tests which will not run always when PR updated.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@CharlesCheung96
Copy link
Contributor Author

/test all

@ti-chi-bot ti-chi-bot bot merged commit 11202f2 into pingcap:master May 18, 2023
@ti-chi-bot ti-chi-bot mentioned this pull request May 18, 2023
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.5: #8995.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request May 18, 2023
@CharlesCheung96 @ti-chi-bot
This is an automated cherry-pick of pingcap#8964
640f36e 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-7.1: #8996.

@ti-chi-bot ti-chi-bot mentioned this pull request May 18, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this pull request May 19, 2023
@ti-chi-bot
owner, redo(ticdc): decouple globalResolvedTs and globalBarrierTs (#8964
d877c76 
) (#8996)

close #8963
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this pull request May 22, 2023
@CharlesCheung96
owner, redo(ticdc): decouple globalResolvedTs and globalBarrierTs (pi…
104ec92 
…ngcap#8964)

close pingcap#8963
@CharlesCheung96 CharlesCheung96 mentioned this pull request May 24, 2023
Merged
This was referenced May 24, 2023
Closed
Merged
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this pull request May 25, 2023
@CharlesCheung96
owner, redo(ticdc): decouple globalResolvedTs and globalBarrierTs (pi…
d892bcb 
…ngcap#8964)

close pingcap#8963
ti-chi-bot bot pushed a commit that referenced this pull request May 25, 2023
@ti-chi-bot
owner, redo(ticdc): decouple globalResolvedTs and globalBarrierTs (#8964
a2aaa4c 
) (#8995)

close #8963
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nongfushanquan nongfushanquan nongfushanquan approved these changes

@hicqu hicqu Awaiting requested review from hicqu

@asddongmen asddongmen Awaiting requested review from asddongmen

Assignees
No one assigned
Labels
needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

decouple globalResolvedTs and globalBarrierTs
4 participants
@CharlesCheung96 @ti-chi-bot @asddongmen @nongfushanquan