Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sink, sorter: do not hog memory #2606

Merged
merged 5 commits into from
Aug 26, 2021
Merged

sink, sorter: do not hog memory #2606

merged 5 commits into from
Aug 26, 2021

Conversation

overvenus
Copy link
Member

@overvenus overvenus commented Aug 23, 2021 •
edited
Loading

What problem does this PR solve?

Reduce memory usage for sink node and file backend.

Cc https://github.com/pingcap/ticdc/issues/2553

Check List

Tests

  • Manual test (add detailed scripts or steps below)

Related changes

  • Need to cherry-pick to the release branch

Release note 

None

@overvenus
sink, sorter: do not hog memory in slice
7ab7a31 
Signed-off-by: Neil Shen <overvenus@gmail.com>
@overvenus overvenus added component/sink Sink component. needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. component/sorter Sorter component. needs-cherry-pick-release-5.2 Should cherry pick this PR to release-5.2 branch. labels Aug 23, 2021
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Aug 23, 2021 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • amyangfei
  • asddongmen

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added the release-note-none Denotes a PR that doesn't merit a release note. label Aug 23, 2021
@ti-chi-bot ti-chi-bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Aug 23, 2021
@asddongmen
Copy link
Contributor

/run-integration-tests
/run-kafka-tests

@codecov-commenter
Copy link

Codecov Report

Merging #2606 (7ab7a31) into master (f3fdd65) will increase coverage by 5.0405%.
The diff coverage is 67.5146%.

@@               Coverage Diff                @@
##             master      #2606        +/-   ##
================================================
+ Coverage   56.0786%   61.1191%   +5.0405%     
================================================
  Files           169        161         -8     
  Lines         20605      17924      -2681     
================================================
- Hits          11555      10955       -600     
+ Misses         7919       5970      -1949     
+ Partials       1131        999       -132

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Aug 23, 2021
@asddongmen asddongmen added the status/ptal Could you please take a look? label Aug 23, 2021
@overvenus
sorter/file_backend: do not hold bytes buffer in writer
f3c987f 
Signed-off-by: Neil Shen <overvenus@gmail.com>
@3AceShowHand
Copy link
Contributor

/run-all-tests

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Aug 25, 2021
@overvenus
Copy link
Member Author

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 9433803

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 25, 2021
@overvenus
Copy link
Member Author

/run-kafka-tests
/run-integration-tests

@amyangfei
Copy link
Contributor

/run-integration-tests
/run-kafka-tests

@amyangfei
Copy link
Contributor

/run-kafka-tests

@ti-chi-bot ti-chi-bot merged commit 142ae26 into pingcap:master Aug 26, 2021
@ti-chi-bot ti-chi-bot mentioned this pull request Aug 26, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2641.

@ti-chi-bot ti-chi-bot mentioned this pull request Aug 26, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2642.

@ti-chi-bot ti-chi-bot mentioned this pull request Aug 26, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2643.

@ti-chi-bot ti-chi-bot mentioned this pull request Aug 26, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2644.

ti-chi-bot added a commit that referenced this pull request Aug 29, 2021
@ti-chi-bot
sink, sorter: do not hog memory (#2606) (#2643)
50a7d05
ti-chi-bot added a commit that referenced this pull request Aug 29, 2021
@ti-chi-bot
sink, sorter: do not hog memory (#2606) (#2644)
b6f0c66
ti-chi-bot added a commit that referenced this pull request Aug 30, 2021
@ti-chi-bot
sink, sorter: do not hog memory (#2606) (#2641)
4f8d40d
ti-chi-bot added a commit that referenced this pull request Aug 31, 2021
@ti-chi-bot
sink, sorter: do not hog memory (#2606) (#2642)
e6d6055
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amyangfei amyangfei amyangfei approved these changes

@asddongmen asddongmen asddongmen approved these changes

@lonng lonng Awaiting requested review from lonng

Assignees
No one assigned
Labels
component/sink Sink component. component/sorter Sorter component. needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. needs-cherry-pick-release-5.2 Should cherry pick this PR to release-5.2 branch. release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. status/ptal Could you please take a look?
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@overvenus @ti-chi-bot @asddongmen @codecov-commenter @3AceShowHand @amyangfei