schedule: support patrol region concurrency #8094

lhy1024 · 2024-04-18T08:29:21Z

What problem does this PR solve?

Issue Number: Close #7963 #7706

What is changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

Release note

None.

ti-chi-bot · 2024-04-18T08:29:23Z

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

ti-chi-bot · 2024-04-18T08:29:23Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Signed-off-by: lhy1024 <admin@liudos.us>

nolouch

overall lgtm, is this PR ready?

lhy1024 · 2024-05-14T03:32:03Z

overall lgtm, is this PR ready?

I am preparing some tests for different scenarios

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 · 2024-05-24T02:24:19Z

pkg/schedule/config/config.go

@@ -63,6 +63,8 @@ const (
 	defaultRegionScoreFormulaVersion = "v2"
 	defaultLeaderSchedulePolicy      = "count"
 	defaultStoreLimitVersion         = "v1"
+	defaultPatrolRegionConcurrency   = 1
+	defaultPatrolRegionBatchLimit    = 128


Maybe we can use max(128,region_count/1024)

lhy1024 · 2024-05-24T02:24:38Z

pkg/schedule/operator/operator_controller.go

@@ -461,7 +461,7 @@ func (oc *Controller) checkAddOperator(isPromoting bool, ops ...*Operator) (bool
 			return false, NotInCreateStatus
 		}
 		if !isPromoting && oc.wopStatus.getCount(op.Desc()) >= oc.config.GetSchedulerMaxWaitingOperator() {
-			log.Debug("exceed max return false", zap.Uint64("waiting", oc.wopStatus.ops[op.Desc()]), zap.String("desc", op.Desc()), zap.Uint64("max", oc.config.GetSchedulerMaxWaitingOperator()))
+			log.Debug("exceed max return false", zap.Uint64("waiting", oc.wopStatus.getCount(op.Desc())), zap.String("desc", op.Desc()), zap.Uint64("max", oc.config.GetSchedulerMaxWaitingOperator()))


avoid data race

Signed-off-by: lhy1024 <admin@liudos.us>

codecov · 2024-05-24T04:04:30Z

Codecov Report

Attention: Patch coverage is 85.26316% with 14 lines in your changes are missing coverage. Please review.

Project coverage is 77.36%. Comparing base (4cd42b3) to head (ab9ef1e).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8094      +/-   ##
==========================================
+ Coverage   77.29%   77.36%   +0.07%     
==========================================
  Files         471      471              
  Lines       61445    61515      +70     
==========================================
+ Hits        47491    47590      +99     
+ Misses      10395    10362      -33     
- Partials     3559     3563       +4

Flag	Coverage Δ
unittests	`77.36% <85.26%> (+0.07%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 · 2024-07-04T06:34:48Z

/test pull-integration-realcluster-test

rleungx · 2024-07-04T07:02:15Z

pkg/schedule/coordinator.go

+	hbStreams           *hbstream.HeartbeatStreams
+	pluginInterface     *PluginInterface
+	diagnosticManager   *diagnostic.Manager
+	patrolRegionContext *PatrolRegionContext


Shall we let the checker controller manage the patrol logic?

Move PatrolRegion and patrolRegionContext to checkers
?

lhy1024 · 2024-07-04T09:43:03Z

/test pull-integration-realcluster-test

rleungx · 2024-07-08T07:57:07Z

pkg/schedule/checker/checker_controller.go

+}
+
+func calculateScanLimit(cluster sche.CheckerCluster) int {
+	scanlimit := max(patrolScanRegionMinLimit, cluster.GetTotalRegionCount()/patrolRegionPartition)


Considering that we have 10 million regions, will it be too large?

we test 10 million regions with simulator and it's ok

How about the metrics for CPU and goroutine counts in this case? 🤔

This branch with 1000w regions

Before 20:45 the limit was 128, after 20:45 the limit was increased.

The initialization batch limit was 128. After the first round, getTotalRegionCount got the region of the whole cluster, which is equivalent to adjusting the limit, so the speed was up

CPU has increased roughly 70%(from 160% to 230%).

goroutine seems to have changed very little

memory has increased 7G

operator check

Signed-off-by: lhy1024 <admin@liudos.us>

okJiang

rest lgtm

okJiang · 2024-08-29T03:37:03Z

pkg/schedule/checker/checker_controller.go

+			// wait for the regionChan to be drained
+			if len(c.patrolRegionContext.regionChan) > 0 {
+				continue
+			}


Need we wait for? If the regionChan is full, it will be blocked.

okJiang · 2024-08-29T04:14:43Z

pkg/schedule/checker/checker_controller.go

+}
+
+func (p *PatrolRegionContext) stop() {
+	close(p.regionChan)


It is better to add a failpoint to wait for all rest region consumed in regionChan before close regionChan, rather than time.Sleep(100 * time.Millisecond) in L171. This is more stable.

If possible, we can always wait it even if it is not in testing.

time.Sleep(100 * time.Millisecond) in L171 is used to wait that regions to be consumed.

co.PatrolRegions() re.Empty(oc.GetOperators())

For example, if we enable this failpoint, it will wait 100 ms for goroutine consuming regions. And it will check re.Empty(oc.GetOperators()) immediately after failpoint.

I understand the role of L171. But I think sleep is a destabilizing factor. And it's fine to wait for it to finish consuming, or actively consume all regions before exiting and then exit.

tools/pd-ctl/tests/config/config_test.go

Signed-off-by: lhy1024 <admin@liudos.us>

ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. labels Apr 18, 2024

ti-chi-bot bot requested review from JmPotato and rleungx April 18, 2024 08:29

ti-chi-bot bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Apr 18, 2024

checker: add patrol region concurrency

d1f4b8a

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 force-pushed the patrol-concurrency branch from 4c59016 to d1f4b8a Compare April 22, 2024 09:52

lhy1024 added 3 commits April 22, 2024 18:21

speedup drain

6d0fbd4

Signed-off-by: lhy1024 <admin@liudos.us>

fix config

9b43d61

Signed-off-by: lhy1024 <admin@liudos.us>

make config

b2b4f39

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 force-pushed the patrol-concurrency branch from 2e61d30 to 948dc77 Compare May 9, 2024 15:33

nolouch reviewed May 14, 2024

View reviewed changes

lhy1024 added 4 commits May 16, 2024 19:54

update

cb285f6

Signed-off-by: lhy1024 <admin@liudos.us>

Merge branch 'master' of github.com:tikv/pd into patrol-concurrency-2

2e14405

Signed-off-by: lhy1024 <admin@liudos.us>

fix race

351ef5c

Signed-off-by: lhy1024 <admin@liudos.us>

Merge branch 'master' of github.com:tikv/pd into patrol-concurrency-2

c198b08

lhy1024 force-pushed the patrol-concurrency branch from 948dc77 to c198b08 Compare May 23, 2024 17:56

lhy1024 changed the title ~~Patrol concurrency~~ schedule: support patrol region concurrency May 23, 2024

ti-chi-bot bot removed the do-not-merge/needs-linked-issue label May 23, 2024

lhy1024 marked this pull request as ready for review May 23, 2024 17:58

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 23, 2024

lhy1024 commented May 24, 2024

View reviewed changes

fix test

a0ec33d

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 force-pushed the patrol-concurrency branch from e71f635 to a0ec33d Compare May 24, 2024 03:56

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 4, 2024

address comments

5668d98

Signed-off-by: lhy1024 <admin@liudos.us>

Merge branch 'master' into patrol-concurrency

78e3ba5

rleungx reviewed Jul 4, 2024

View reviewed changes

rleungx reviewed Jul 8, 2024

View reviewed changes

ti-chi-bot bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jul 12, 2024

lhy1024 requested review from JmPotato and rleungx July 15, 2024 08:31

lhy1024 added 5 commits July 16, 2024 15:38

refactor

cf01076

Signed-off-by: lhy1024 <admin@liudos.us>

fix test and add metrics

cc51a2e

Signed-off-by: lhy1024 <admin@liudos.us>

fix failpoint

9f7406a

Signed-off-by: lhy1024 <admin@liudos.us>

Merge branch 'master' of github.com:tikv/pd into patrol-concurrency-4

bd4ca79

Signed-off-by: lhy1024 <admin@liudos.us>

fix conflict

ae0778f

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 force-pushed the patrol-concurrency branch from ff02183 to ae0778f Compare July 16, 2024 09:08

lhy1024 added 2 commits July 22, 2024 23:41

Merge branch 'master' of github.com:tikv/pd into patrol-concurrency

ecb8d8b

Merge branch 'master' of github.com:tikv/pd into patrol-concurrency

8259ad0

Signed-off-by: lhy1024 <admin@liudos.us>

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 1, 2024

lhy1024 added 2 commits August 1, 2024 19:01

fix

18db300

Signed-off-by: lhy1024 <admin@liudos.us>

Merge branch 'master' of github.com:tikv/pd into patrol-concurrency

5ead31e

Signed-off-by: lhy1024 <admin@liudos.us>

ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 8, 2024

lhy1024 and others added 2 commits August 8, 2024 19:18

fix lint

6bdb436

Signed-off-by: lhy1024 <admin@liudos.us>

Merge branch 'master' into patrol-concurrency

40a2e02

okJiang reviewed Aug 29, 2024

View reviewed changes

address comments

8cd8825

Signed-off-by: lhy1024 <admin@liudos.us>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

schedule: support patrol region concurrency #8094

schedule: support patrol region concurrency #8094

lhy1024 commented Apr 18, 2024 •

edited

Loading

ti-chi-bot bot commented Apr 18, 2024

ti-chi-bot bot commented Apr 18, 2024

nolouch left a comment

lhy1024 commented May 14, 2024

lhy1024 May 24, 2024

lhy1024 May 24, 2024

codecov bot commented May 24, 2024 •

edited

Loading

lhy1024 commented Jul 4, 2024

rleungx Jul 4, 2024

lhy1024 Jul 4, 2024

rleungx Jul 4, 2024

lhy1024 commented Jul 4, 2024

rleungx Jul 8, 2024

lhy1024 Jul 9, 2024

JmPotato Jul 9, 2024 •

edited

Loading

HuSharp Jul 9, 2024 •

edited

Loading

okJiang left a comment

okJiang Aug 29, 2024

okJiang Aug 29, 2024

lhy1024 Aug 29, 2024

okJiang Aug 30, 2024

schedule: support patrol region concurrency #8094

Are you sure you want to change the base?

schedule: support patrol region concurrency #8094

Conversation

lhy1024 commented Apr 18, 2024 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Apr 18, 2024

ti-chi-bot bot commented Apr 18, 2024

nolouch left a comment

Choose a reason for hiding this comment

lhy1024 commented May 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented May 24, 2024 • edited Loading

Codecov Report

lhy1024 commented Jul 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lhy1024 commented Jul 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JmPotato Jul 9, 2024 • edited Loading

Choose a reason for hiding this comment

HuSharp Jul 9, 2024 • edited Loading

Choose a reason for hiding this comment

This branch with 1000w regions

okJiang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lhy1024 commented Apr 18, 2024 •

edited

Loading

codecov bot commented May 24, 2024 •

edited

Loading

JmPotato Jul 9, 2024 •

edited

Loading

HuSharp Jul 9, 2024 •

edited

Loading