region_syncer: reduce saveKV of client #3808

HunDunDM · 2021-06-22T04:39:37Z

What problem does this PR solve?

What is changed and how it works?

By introducing part of the heartbeat processing logic in the client of region_syncer, unnecessary saveKV operations are reduced.

Check List

Tests

Unit test

Code changes

Side effects

Related changes

Release note

Improved the performance of synchronizing Region information between PDs.

Signed-off-by: HunDunDM <hundundm@gmail.com>

codecov · 2021-06-22T05:08:16Z

Codecov Report

Merging #3808 (d8b6815) into master (127df63) will decrease coverage by 0.07%.
The diff coverage is 92.50%.

@@            Coverage Diff             @@
##           master    #3808      +/-   ##
==========================================
- Coverage   74.59%   74.51%   -0.08%     
==========================================
  Files         246      246              
  Lines       24885    24905      +20     
==========================================
- Hits        18562    18558       -4     
- Misses       4669     4688      +19     
- Partials     1654     1659       +5

Flag	Coverage Δ
unittests	`74.51% <92.50%> (-0.08%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/region_syncer/client.go	`77.46% <78.57%> (-5.25%)`	⬇️
server/region_syncer/server.go	`83.08% <83.33%> (-6.31%)`	⬇️
server/core/region.go	`91.44% <96.61%> (+0.72%)`	⬆️
server/cluster/cluster.go	`82.19% <100.00%> (-1.07%)`	⬇️
server/kv/etcd_kv.go	`69.56% <0.00%> (-10.15%)`	⬇️
server/tso/local_allocator.go	`71.62% <0.00%> (-6.76%)`	⬇️
pkg/etcdutil/etcdutil.go	`84.70% <0.00%> (-3.53%)`	⬇️
server/schedule/placement/rule_manager.go	`83.13% <0.00%> (-1.21%)`	⬇️
... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 127df63...d8b6815. Read the comment docs.

Signed-off-by: HunDunDM <hundundm@gmail.com>

HunDunDM · 2021-06-22T08:18:45Z

server/region_syncer/server.go

 				}
 				metas = metas[:0]
 				stats = stats[:0]
+				leaders = leaders[:0]


Will this cause the new PD instance to have incorrect leader information? @rleungx

server/core/region.go

Signed-off-by: HunDunDM <hundundm@gmail.com>

ti-chi-bot · 2021-07-14T10:19:05Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

disksing
nolouch

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

nolouch

lgtm

nolouch · 2021-07-28T09:57:31Z

/merge

ti-chi-bot · 2021-07-28T09:57:32Z

@nolouch: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-07-28T09:57:33Z

This pull request has been accepted and is ready to merge.

Commit hash: 9d14833

ti-chi-bot · 2021-08-02T05:24:06Z

In response to a cherrypick label: new pull request created: #3932.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2021-08-02T05:24:07Z

In response to a cherrypick label: new pull request created: #3933.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2021-08-02T05:24:08Z

In response to a cherrypick label: new pull request created: #3934.

HunDunDM · 2021-08-16T08:51:04Z

/cherry-pick release-5.0

ti-chi-bot · 2021-08-16T08:51:38Z

@HunDunDM: new pull request created: #3993.

In response to this:

/cherry-pick release-5.0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

Signed-off-by: HunDunDM <hundundm@gmail.com>

* region_syncer: reduce saveKV of client (#3808) Signed-off-by: HunDunDM <hundundm@gmail.com> * tiny fix Signed-off-by: HunDunDM <hundundm@gmail.com> Co-authored-by: 混沌DM <hundundm@gmail.com>

Signed-off-by: HunDunDM <hundundm@gmail.com>

Signed-off-by: HunDunDM <hundundm@gmail.com> Co-authored-by: HunDunDM <hundundm@gmail.com>

* region_syncer: reduce saveKV of client Signed-off-by: HunDunDM <hundundm@gmail.com> * add comment Signed-off-by: HunDunDM <hundundm@gmail.com> * fix typo Signed-off-by: HunDunDM <hundundm@gmail.com> * fix leader miss bug Signed-off-by: HunDunDM <hundundm@gmail.com> * address comment Signed-off-by: HunDunDM <hundundm@gmail.com> Co-authored-by: HunDunDM <hundundm@gmail.com>

HunDunDM added 2 commits June 22, 2021 12:26

region_syncer: reduce saveKV of client

12e02a3

Signed-off-by: HunDunDM <hundundm@gmail.com>

add comment

03293c0

Signed-off-by: HunDunDM <hundundm@gmail.com>

ti-chi-bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Jun 22, 2021

ti-chi-bot requested review from disksing and rleungx June 22, 2021 04:39

HunDunDM added the component/cluster Cluster logic. label Jun 22, 2021

HunDunDM mentioned this pull request Jun 22, 2021

region_syncer: fix blocking bugs through ctx #3766

Closed

fix typo

217a460

Signed-off-by: HunDunDM <hundundm@gmail.com>

fix leader miss bug

f89cee6

Signed-off-by: HunDunDM <hundundm@gmail.com>

HunDunDM commented Jun 22, 2021

View reviewed changes

disksing reviewed Jun 28, 2021

View reviewed changes

server/core/region.go Outdated Show resolved Hide resolved

HunDunDM added 2 commits July 5, 2021 12:06

address comment

07f370c

Signed-off-by: HunDunDM <hundundm@gmail.com>

Merge branch 'master' into reduce-syncer-storage

9d14833

disksing approved these changes Jul 14, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 14, 2021

nolouch approved these changes Jul 28, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jul 28, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jul 28, 2021

Merge branch 'master' into reduce-syncer-storage

d8b6815

ti-chi-bot merged commit 0affd19 into tikv:master Jul 28, 2021

HunDunDM deleted the reduce-syncer-storage branch July 28, 2021 11:45

disksing mentioned this pull request Jul 30, 2021

LoadRegion from persistent storage should remove overlap region after compare version #2011

Closed

nolouch added needs-cherry-pick-release-4.0 The PR needs to cherry pick to release-4.0 branch. needs-cherry-pick-release-5.0 The PR needs to cherry pick to release-5.0 branch. labels Aug 2, 2021

nolouch added the needs-cherry-pick-release-5.1 Type: Need cherry pick to release-5.1 label Aug 2, 2021

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Aug 2, 2021

This is an automated cherry-pick of tikv#3808

4cfa480

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

This was referenced Aug 2, 2021

region_syncer: reduce saveKV of client (#3808) #3932

Merged

region_syncer: reduce saveKV of client (#3808) #3933

Merged

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Aug 2, 2021

This is an automated cherry-pick of tikv#3808

3872e16

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Aug 2, 2021

region_syncer: reduce saveKV of client (#3808) #3934

Closed

tikv deleted a comment from nolouch Aug 11, 2021

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Aug 16, 2021

This is an automated cherry-pick of tikv#3808

6e4a041

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Aug 16, 2021

region_syncer: reduce saveKV of client (#3808) #3993

Merged

HunDunDM added a commit to ti-chi-bot/pd that referenced this pull request Aug 16, 2021

region_syncer: reduce saveKV of client (tikv#3808)

d125637

Signed-off-by: HunDunDM <hundundm@gmail.com>

HunDunDM added a commit to ti-chi-bot/pd that referenced this pull request Sep 8, 2021

region_syncer: reduce saveKV of client (tikv#3808)

025141e

Signed-off-by: HunDunDM <hundundm@gmail.com>

ti-chi-bot added a commit that referenced this pull request Sep 8, 2021

region_syncer: reduce saveKV of client (#3808) (#3932)

99e700e

Signed-off-by: HunDunDM <hundundm@gmail.com> Co-authored-by: HunDunDM <hundundm@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

region_syncer: reduce saveKV of client #3808

region_syncer: reduce saveKV of client #3808

HunDunDM commented Jun 22, 2021

codecov bot commented Jun 22, 2021 •

edited

Loading

HunDunDM Jun 22, 2021

ti-chi-bot commented Jul 14, 2021 •

edited

Loading

nolouch left a comment

nolouch commented Jul 28, 2021

ti-chi-bot commented Jul 28, 2021

ti-chi-bot commented Jul 28, 2021

ti-chi-bot commented Aug 2, 2021

ti-chi-bot commented Aug 2, 2021

ti-chi-bot commented Aug 2, 2021

HunDunDM commented Aug 16, 2021

ti-chi-bot commented Aug 16, 2021

region_syncer: reduce saveKV of client #3808

region_syncer: reduce saveKV of client #3808

Conversation

HunDunDM commented Jun 22, 2021

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

codecov bot commented Jun 22, 2021 • edited Loading

Codecov Report

HunDunDM Jun 22, 2021

Choose a reason for hiding this comment

ti-chi-bot commented Jul 14, 2021 • edited Loading

nolouch left a comment

Choose a reason for hiding this comment

nolouch commented Jul 28, 2021

ti-chi-bot commented Jul 28, 2021

ti-chi-bot commented Jul 28, 2021

ti-chi-bot commented Aug 2, 2021

ti-chi-bot commented Aug 2, 2021

ti-chi-bot commented Aug 2, 2021

HunDunDM commented Aug 16, 2021

ti-chi-bot commented Aug 16, 2021

codecov bot commented Jun 22, 2021 •

edited

Loading

ti-chi-bot commented Jul 14, 2021 •

edited

Loading