statistics: support hotPeerCache maintained in Peer level #3627

Yisaer · 2021-04-22T11:41:11Z

Signed-off-by: yisaer disxiaofei@163.com

What problem does this PR solve?

What is changed and how it works?

make hotPeerCache maintained by peer level

Release note

No release note

Signed-off-by: yisaer <disxiaofei@163.com>

codecov · 2021-04-22T13:08:05Z

Codecov Report

Merging #3627 (c4d33ca) into master (78d2dc4) will increase coverage by 0.01%.
The diff coverage is 89.19%.

@@            Coverage Diff             @@
##           master    #3627      +/-   ##
==========================================
+ Coverage   74.62%   74.63%   +0.01%     
==========================================
  Files         245      245              
  Lines       24144    24240      +96     
==========================================
+ Hits        18017    18092      +75     
- Misses       4512     4528      +16     
- Partials     1615     1620       +5

Flag	Coverage Δ
unittests	`74.63% <89.19%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/core/kind.go	`61.90% <ø> (ø)`
server/statistics/kind.go	`0.00% <0.00%> (ø)`
server/cluster/coordinator.go	`72.31% <46.66%> (-1.98%)`	⬇️
server/api/trend.go	`66.19% <50.00%> (ø)`
server/tso/tso.go	`70.11% <80.00%> (+0.17%)`	⬆️
server/statistics/flowkind.go	`83.33% <83.33%> (ø)`
server/schedulers/hot_region.go	`80.98% <86.86%> (+<0.01%)`	⬆️
server/core/peer.go	`83.33% <91.66%> (+4.16%)`	⬆️
server/statistics/hot_peer_cache.go	`94.46% <99.11%> (+1.28%)`	⬆️
pkg/mock/mockcluster/mockcluster.go	`94.41% <100.00%> (ø)`
... and 24 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78d2dc4...c4d33ca. Read the comment docs.

server/statistics/hot_peer_cache.go

lhy1024 · 2021-04-23T09:39:24Z

server/statistics/hot_peer_cache.go

@@ -450,3 +437,70 @@ func (f *hotPeerCache) updateHotPeerStat(newItem, oldItem *HotPeerStat, bytes, k
 	}
 	return newItem
 }
+
+// TODO: remove it in future
+func (f *hotPeerCache) getRegionBytes(region *core.RegionInfo) uint64 {


It seems to be related with #3591

Signed-off-by: yisaer <disxiaofei@163.com>

Yisaer · 2021-04-27T13:44:33Z

/run-tests

Yisaer · 2021-04-27T13:44:46Z

/run-test

Signed-off-by: yisaer <disxiaofei@163.com>

…o make_schedule_peer

Signed-off-by: yisaer <disxiaofei@163.com>

server/statistics/hot_peer_cache.go

server/core/peer.go

Signed-off-by: yisaer <disxiaofei@163.com>

server/core/peer.go

Signed-off-by: yisaer <disxiaofei@163.com>

rleungx · 2021-05-08T03:20:19Z

server/statistics/hot_peer_cache.go

+	isExpired := f.isPeerExpired(peer, region)
+	oldItem := f.getOldHotPeerStat(region.GetID(), storeID)
+	if isExpired && oldItem != nil && !f.isInheritExist(region.GetID()) {
+		f.putInheritItem(oldItem)


Will it cause race?

lhy1024 · 2021-05-08T03:56:41Z

server/statistics/hot_peer_cache.go

+	// transfer read leader or remove write peer
+	isExpired := f.isPeerExpired(peer, region)
+	oldItem := f.getOldHotPeerStat(region.GetID(), storeID)
+	if isExpired && oldItem != nil && !f.isInheritExist(region.GetID()) {


why need to judge whether exist in inherit items?

lhy1024 · 2021-05-08T03:57:27Z

server/statistics/hot_peer_cache.go

+
+func (f *hotPeerCache) isInheritExist(regionID uint64) bool {
+	item, ok := f.inheritItem[regionID]
+	return ok && item != nil


when is it nil?

when there is no inherit item.

Signed-off-by: yisaer <disxiaofei@163.com>

rleungx · 2021-05-08T06:02:22Z

server/statistics/hot_cache.go

+	for {
+		select {
+		case <-ctx.Done():
+			close(w.readFlowQueue)


It's better to close it on send side.

Signed-off-by: yisaer <disxiaofei@163.com>

server/cluster/cluster.go

server/statistics/hot_cache.go

server/statistics/hot_peer_cache.go

Signed-off-by: yisaer <disxiaofei@163.com>

sonarcloud · 2021-05-08T09:55:17Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
8 Code Smells

No Coverage information
1.7% Duplication

ti-chi-bot · 2021-05-08T10:05:31Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

lhy1024
rleungx

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

Yisaer · 2021-05-08T10:06:19Z

/merge

ti-chi-bot · 2021-05-08T10:06:20Z

@Yisaer: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-05-08T10:06:21Z

This pull request has been accepted and is ready to merge.

Commit hash: c4d33ca

Yisaer · 2021-05-08T10:16:00Z

/run-integration-ddl-test

support peer

0f9a43b

Signed-off-by: yisaer <disxiaofei@163.com>

ti-chi-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 22, 2021

ti-chi-bot requested review from lhy1024 and rleungx April 22, 2021 11:41

Yisaer marked this pull request as ready for review April 23, 2021 09:27

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 23, 2021

lhy1024 reviewed Apr 23, 2021

View reviewed changes

fix lint

dbd617f

Signed-off-by: yisaer <disxiaofei@163.com>

Yisaer force-pushed the make_schedule_peer branch 2 times, most recently from 701d043 to 6e91990 Compare April 27, 2021 10:52

revise

950e6ea

Signed-off-by: yisaer <disxiaofei@163.com>

Yisaer force-pushed the make_schedule_peer branch from 6e91990 to 950e6ea Compare April 27, 2021 11:18

Merge branch 'master' into make_schedule_peer

3d81c76

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 29, 2021

Yisaer added 4 commits April 30, 2021 13:36

Merge remote-tracking branch 'upstream/master' into make_schedule_peer

4a5e029

revise

af4089e

Signed-off-by: yisaer <disxiaofei@163.com>

revise

5e25d31

Signed-off-by: yisaer <disxiaofei@163.com>

Merge branch 'make_schedule_peer' of https://github.com/Yisaer/pd int…

7c90744

…o make_schedule_peer

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 30, 2021

revise

6a9f7e5

Signed-off-by: yisaer <disxiaofei@163.com>

Yisaer requested a review from lhy1024 April 30, 2021 07:25

lhy1024 reviewed May 6, 2021

View reviewed changes

server/statistics/hot_peer_cache.go Outdated Show resolved Hide resolved

server/statistics/hot_peer_cache.go Outdated Show resolved Hide resolved

rleungx reviewed May 7, 2021

View reviewed changes

server/core/peer.go Outdated Show resolved Hide resolved

address the comment

c83fbc5

Signed-off-by: yisaer <disxiaofei@163.com>

Yisaer force-pushed the make_schedule_peer branch from 590a7fa to c83fbc5 Compare May 7, 2021 04:13

rleungx reviewed May 7, 2021

View reviewed changes

server/core/peer.go Outdated Show resolved Hide resolved

address the comment

27ecc5d

Signed-off-by: yisaer <disxiaofei@163.com>

Yisaer added 4 commits May 7, 2021 17:19

address the comment

df243cf

Signed-off-by: yisaer <disxiaofei@163.com>

address the comment

05376ea

Signed-off-by: yisaer <disxiaofei@163.com>

address the comment

9eb8f46

Signed-off-by: yisaer <disxiaofei@163.com>

address the comment

a002e18

Signed-off-by: yisaer <disxiaofei@163.com>

rleungx reviewed May 8, 2021

View reviewed changes

lhy1024 reviewed May 8, 2021

View reviewed changes

Yisaer added 2 commits May 8, 2021 12:09

asynchronous check

04de5a9

Signed-off-by: yisaer <disxiaofei@163.com>

fix ctx

3f94c65

Signed-off-by: yisaer <disxiaofei@163.com>

Yisaer force-pushed the make_schedule_peer branch from db9a81f to 3f94c65 Compare May 8, 2021 05:28

address the comment

8a30ae6

Signed-off-by: yisaer <disxiaofei@163.com>

rleungx reviewed May 8, 2021

View reviewed changes

address the comment

3cc505e

Signed-off-by: yisaer <disxiaofei@163.com>

lhy1024 approved these changes May 8, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 8, 2021

rleungx reviewed May 8, 2021

View reviewed changes

server/cluster/cluster.go Outdated Show resolved Hide resolved

rleungx reviewed May 8, 2021

View reviewed changes

server/statistics/hot_cache.go Outdated Show resolved Hide resolved

rleungx reviewed May 8, 2021

View reviewed changes

server/statistics/hot_peer_cache.go Outdated Show resolved Hide resolved

address the comment

c4d33ca

Signed-off-by: yisaer <disxiaofei@163.com>

rleungx approved these changes May 8, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 8, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 8, 2021

ti-chi-bot merged commit df1614b into tikv:master May 8, 2021

Yisaer deleted the make_schedule_peer branch May 8, 2021 10:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

statistics: support hotPeerCache maintained in Peer level #3627

statistics: support hotPeerCache maintained in Peer level #3627

Yisaer commented Apr 22, 2021 •

edited

Loading

codecov bot commented Apr 22, 2021 •

edited

Loading

lhy1024 Apr 23, 2021

Yisaer commented Apr 27, 2021

Yisaer commented Apr 27, 2021

rleungx May 8, 2021

lhy1024 May 8, 2021

Yisaer May 8, 2021

lhy1024 May 8, 2021

Yisaer May 8, 2021

rleungx May 8, 2021

Yisaer May 8, 2021

sonarcloud bot commented May 8, 2021

ti-chi-bot commented May 8, 2021

Yisaer commented May 8, 2021

ti-chi-bot commented May 8, 2021

ti-chi-bot commented May 8, 2021

Yisaer commented May 8, 2021

statistics: support hotPeerCache maintained in Peer level #3627

statistics: support hotPeerCache maintained in Peer level #3627

Conversation

Yisaer commented Apr 22, 2021 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Release note

codecov bot commented Apr 22, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Yisaer commented Apr 27, 2021

Yisaer commented Apr 27, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented May 8, 2021

ti-chi-bot commented May 8, 2021

Yisaer commented May 8, 2021

ti-chi-bot commented May 8, 2021

ti-chi-bot commented May 8, 2021

Yisaer commented May 8, 2021

Yisaer commented Apr 22, 2021 •

edited

Loading

codecov bot commented Apr 22, 2021 •

edited

Loading