Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

*: reduce memory consumption by reducing channel sizes #1999

Merged
merged 2 commits into from
Jun 10, 2021
Merged

*: reduce memory consumption by reducing channel sizes #1999

merged 2 commits into from
Jun 10, 2021

Conversation

liuzix
Copy link
Contributor

@liuzix liuzix commented Jun 9, 2021

What problem does this PR solve?

  • TiCDC can OOM frequently on machines with memory less than 16GB.

What is changed and how it works?

  • Decreased the channel sizes in two places.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test

Side effects

  • Possible performance regression

Related changes

  • Need to cherry-pick to the release branch

Release note

  • Reduce unnecessary memory consumption

@ti-chi-bot ti-chi-bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Jun 9, 2021
@liuzix
Copy link
Contributor Author

liuzix commented Jun 9, 2021

/run-all-tests

1 similar comment
@liuzix
Copy link
Contributor Author

liuzix commented Jun 9, 2021

/run-all-tests

@liuzix liuzix added this to the v5.1.0 milestone Jun 9, 2021
@liuzix liuzix added component/kv-client TiKV kv log client component. status/ptal Could you please take a look? needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. labels Jun 9, 2021
@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • amyangfei

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 9, 2021
amyangfei
amyangfei reviewed Jun 9, 2021
View reviewed changes
cdc/kv/region_worker.go
@@ -45,7 +45,7 @@ var (
// other components in TiCDC, including worker pool task chan size, mounter
// chan size etc.
// TODO: unified channel buffer mechanism
regionWorkerInputChanSize = 12800
regionWorkerInputChanSize = 128
Copy link
Contributor

@amyangfei amyangfei Jun 9, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note the smaller regionWorkerInputChanSize will tend to use region worker pool. No need to address.

@ti-chi-bot
Copy link
Member

@ben1009: Thanks for your review. The bot only counts approvals from reviewers and higher roles in list, but you're still welcome to leave your comments.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@amyangfei amyangfei added release-blocker This issue blocks a release. Please solve it ASAP. require-LGT1 Indicates that the PR requires an LGTM. labels Jun 10, 2021
@amyangfei
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 802b052

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 10, 2021
@codecov-commenter
Copy link

Codecov Report

Merging #1999 (1ec812a) into master (b6d58ef) will decrease coverage by 0.1536%.
The diff coverage is 64.9243%.

@@               Coverage Diff                @@
##             master      #1999        +/-   ##
================================================
- Coverage   54.0780%   53.9243%   -0.1537%     
================================================
  Files           156        164         +8     
  Lines         16589      17404       +815     
================================================
+ Hits           8971       9385       +414     
- Misses         6667       7016       +349     
- Partials        951       1003        +52

@ti-chi-bot ti-chi-bot merged commit ef9f56e into pingcap:master Jun 10, 2021
ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Jun 10, 2021
@liuzix @ti-chi-bot
This is an automated cherry-pick of pingcap#1999
5f49c63 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Jun 10, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2011.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Jun 10, 2021
@liuzix @ti-chi-bot
This is an automated cherry-pick of pingcap#1999
bfe8d08 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Jun 10, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2012.

@ti-chi-bot ti-chi-bot mentioned this pull request Jun 10, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2013.

ti-chi-bot added a commit that referenced this pull request Jun 10, 2021
@ti-chi-bot
*: reduce memory consumption by reducing channel sizes (#1999) (#2013)
61ff6f3
ti-chi-bot added a commit that referenced this pull request Jun 22, 2021
@ti-chi-bot
*: reduce memory consumption by reducing channel sizes (#1999) (#2012)
9e0d54d
@amyangfei amyangfei mentioned this pull request Jun 25, 2021
Closed
amyangfei pushed a commit to ti-chi-bot/tiflow that referenced this pull request Jul 8, 2021
@liuzix @amyangfei
*: reduce memory consumption by reducing channel sizes (pingcap#1999)
16b8968
ti-chi-bot added a commit that referenced this pull request Jul 8, 2021
@ti-chi-bot
*: reduce memory consumption by reducing channel sizes (#1999) (#2011)
d7802e0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amyangfei amyangfei amyangfei approved these changes

@ben1009 ben1009 ben1009 approved these changes

@lonng lonng Awaiting requested review from lonng

Assignees
No one assigned
Labels
component/kv-client TiKV kv log client component. needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. release-blocker This issue blocks a release. Please solve it ASAP. require-LGT1 Indicates that the PR requires an LGTM. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT1 Indicates that a PR has LGTM 1. status/ptal Could you please take a look?
Projects
None yet
Milestone
v5.1.0
Development

Successfully merging this pull request may close these issues.
5 participants
@liuzix @ti-chi-bot @amyangfei @codecov-commenter @ben1009