Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sinkv2(ticdc): directly advance checkpoint ts if only resolved ts #7876

Merged
merged 12 commits into from
Dec 12, 2022
Merged

sinkv2(ticdc): directly advance checkpoint ts if only resolved ts #7876

merged 12 commits into from
Dec 12, 2022

Conversation

Rustin170506
Copy link
Member

@Rustin170506 Rustin170506 commented Dec 10, 2022 •
edited
Loading

What problem does this PR solve?

Issue Number: close #7887 ref #7872

What is changed and how it works?

  • Directly advance checkpoint ts if only resolved ts. If there are no events, we can just advance checkpoint ts.
  • Reduce the buffer size from 1024 * 1024 to 4096.

Check List

Tests

  • Unit test
  • Manual test (add detailed scripts or steps below)

Questions

Will it cause performance regression or break compatibility?

No

Do you need to update user documentation, design documentation or monitoring documentation?

No

Release note 

Directly advance checkpoint TS to reduce OOM risk if there is no row change events

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Dec 10, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • amyangfei
  • sdojjy

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-linked-issue release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Dec 10, 2022
@Rustin170506 Rustin170506 marked this pull request as draft December 10, 2022 14:47
@ti-chi-bot ti-chi-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. release-note-none Denotes a PR that doesn't merit a release note. release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note Denotes a PR that will be considered when it comes time to generate release notes. release-note-none Denotes a PR that doesn't merit a release note. labels Dec 10, 2022
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 10, 2022
@Rustin170506
Copy link
Member Author

/run-all-tests

@Rustin170506
Copy link
Member Author

/run-all-tests

@Rustin170506
Copy link
Member Author

/run-all-tests

@Rustin170506
Copy link
Member Author

/run-kafka-integration-test

@Rustin170506
Copy link
Member Author

Rustin170506 commented Dec 12, 2022 •
edited
Loading

before:
image

after:
image

image

@Rustin170506
Copy link
Member Author

/run-all-tests

1 similar comment
@Rustin170506
Copy link
Member Author

/run-all-tests

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 12, 2022
@Rustin170506
Copy link
Member Author

/run-all-tests

@Rustin170506
Copy link
Member Author

/run-all-tests

@Rustin170506
Copy link
Member Author

/run-all-tests

@Rustin170506
Copy link
Member Author

/run-all-tests

@Rustin170506 Rustin170506 marked this pull request as ready for review December 12, 2022 11:28
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 12, 2022
@hicqu
Copy link
Contributor

hicqu commented Dec 12, 2022

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: d48bec0

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 12, 2022
@Rustin170506
Copy link
Member Author

Rustin170506 commented Dec 12, 2022 •
edited
Loading

The lastest test result:
image

@Rustin170506
Copy link
Member Author

/hold

@ti-chi-bot ti-chi-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 12, 2022
@Rustin170506
Copy link
Member Author

/run-kafka-integration-test
/run-integration-test

@Rustin170506
Copy link
Member Author

/run-kafka-integration-test
/run-integration-test

@Rustin170506
Copy link
Member Author

/unhold

@ti-chi-bot ti-chi-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 12, 2022
@ti-chi-bot ti-chi-bot merged commit fb94047 into pingcap:master Dec 12, 2022
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.5: #7888.

@ti-chi-bot ti-chi-bot mentioned this pull request Dec 12, 2022
Merged
ti-chi-bot added a commit that referenced this pull request Dec 12, 2022
@ti-chi-bot
sinkv2(ticdc): directly advance checkpoint ts if only resolved ts (#7876
d520bc9 
) (#7888)

close #7887
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hicqu hicqu hicqu approved these changes

@sdojjy sdojjy sdojjy approved these changes

@amyangfei amyangfei amyangfei approved these changes

Assignees
No one assigned
Labels
needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

v6.5.0 Allocating too much memory makes it easy to OOM when there are many tables
5 participants
@Rustin170506 @ti-chi-bot @hicqu @amyangfei @sdojjy