Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ticdc: Fix CPU surge problem in sorter #10739

Merged
merged 7 commits into from
Mar 11, 2024

Conversation

hongyunyan
Copy link
Collaborator

@hongyunyan hongyunyan commented Mar 7, 2024

What problem does this PR solve?

Issue Number: close #10738

What is changed and how it works?

The problematic code snippet here:

for {
for len(batch.Repr()) < batchCommitSize && time.Since(startToCollectBatch) < batchCommitInterval {
select {
case item := <-inputCh:
handleItem(item)
case <-s.closed:
return
default:
}
}
batchCh <- &DBBatchEvent{batch, newResolved}
batch = db.NewBatch()
newResolved = spanz.NewHashMap[model.Ts]()
startToCollectBatch = time.Now()
}
}

is a busy loop that continuously polls an input channel and does some work when an item is available.
The issue here is with the 'default' clause in the select statement. If there's no item available in the 'inputCh' channel, it will immediately continue to the next iteration instead of blocking until an item is available or the 's.closed' channel is closed. This could lead to high CPU usage as the loop is continuously running.

This PR replace the 'default' clause with a ticker. This will cause the goroutine to block when there's no work to do, reducing CPU usage.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note

`None`.

@ti-chi-bot ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Mar 7, 2024
@CharlesCheung96
Copy link
Contributor

CharlesCheung96 commented Mar 7, 2024

Please explain the backgorung in pr body

@asddongmen asddongmen self-requested a review March 7, 2024 08:48
@ti-chi-bot ti-chi-bot bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/needs-triage-completed and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Mar 7, 2024
Copy link

codecov bot commented Mar 7, 2024

Codecov Report

Merging #10739 (ad6363f) into master (e5edb19) will increase coverage by 0.1057%.
Report is 3 commits behind head on master.
The diff coverage is 75.0000%.

Additional details and impacted files
Components Coverage Δ
cdc 62.1357% <75.0000%> (+0.2112%) ⬆️
dm 51.2334% <ø> (-0.0081%) ⬇️
engine 63.3724% <ø> (-0.0212%) ⬇️
Flag Coverage Δ
unit 57.6751% <75.0000%> (+0.1057%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master     #10739        +/-   ##
================================================
+ Coverage   57.5694%   57.6751%   +0.1057%     
================================================
  Files           851        852         +1     
  Lines        126793     127027       +234     
================================================
+ Hits          72994      73263       +269     
+ Misses        48361      48324        -37     
- Partials       5438       5440         +2     

@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Mar 8, 2024
if !ok {
return
}
batchCh <- batchEvent
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should an empty batchEvent be ignored?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I check whether batchEvent is empty in L395, because whether the batchEvent is empty, we have to update resolved ts.

@ti-chi-bot ti-chi-bot bot added the lgtm label Mar 11, 2024
Copy link
Contributor

ti-chi-bot bot commented Mar 11, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: asddongmen, CharlesCheung96

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [CharlesCheung96,asddongmen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Mar 11, 2024
Copy link
Contributor

ti-chi-bot bot commented Mar 11, 2024

[LGTM Timeline notifier]

Timeline:

  • 2024-03-08 06:06:49.540524682 +0000 UTC m=+406436.562771072: ☑️ agreed by asddongmen.
  • 2024-03-11 02:05:03.048339963 +0000 UTC m=+651130.070586351: ☑️ agreed by CharlesCheung96.

@ti-chi-bot ti-chi-bot bot merged commit d19833e into pingcap:master Mar 11, 2024
40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved lgtm release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Resolved ts stucks after upgrading to latest CDC build
3 participants