[new-hotspot-scheduler] add operator influence and more metrics #1713

Luffbee · 2019-08-28T09:43:32Z

changes:

disabled balance hot write by leader.
add operator influence:
- pending influence before operator finish
- zombie influence after operator finish
more metrics to track hotspot operators

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Signed-off-by: disksing <i@disksing.com>

…into new-hotspot-scheduler

* test: fix region syncer test Signed-off-by: disksing <i@disksing.com>

Signed-off-by: cwen0 <cwenyin0@gmail.com>

This reverts commit e74c7a9.

Signed-off-by: Ryan Leung <rleungx@gmail.com>

…#1659) * schedule: Do not send an operator of a region wth a stale epoch Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * schedule: check the version changed by the operator self Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * schedule: fix unit test Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * schedule: fix to avoid dispatching a stale opstep Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * dispatch: refactor "ConsumeConfVer() int" to "ExpectConfVerChange() bool" Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * dispatch: fix typo in comment Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * fix typo Co-Authored-By: Ryan Leung <rleungx@gmail.com> * dispatch: fix unittest Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * dispatch: refine format Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

Signed-off-by: Ryan Leung <rleungx@gmail.com>

…into new-hotspot-scheduler

codecov-io · 2019-08-28T09:52:30Z

Codecov Report

Merging #1713 into new-hotspot-scheduler will decrease coverage by 0.03%.
The diff coverage is 61.26%.

@@                    Coverage Diff                    @@
##           new-hotspot-scheduler    #1713      +/-   ##
=========================================================
- Coverage                  75.71%   75.67%   -0.04%     
=========================================================
  Files                        158      161       +3     
  Lines                      15941    16262     +321     
=========================================================
+ Hits                       12070    12307     +237     
- Misses                      2837     2923      +86     
+ Partials                    1034     1032       -2

Impacted Files	Coverage Δ
server/statistics/region.go	`96% <ø> (ø)`	⬆️
pkg/cache/cache.go	`0% <ø> (ø)`	⬆️
server/statistics/hot_cache.go	`61.3% <ø> (ø)`	⬆️
...ools/pd-simulator/simulator/simutil/case_config.go	`0% <0%> (ø)`
server/schedulers/metrics.go	`100% <100%> (ø)`	⬆️
tools/pd-ctl/pdctl/command/config_command.go	`73.54% <100%> (+2.24%)`	⬆️
server/cluster.go	`84.16% <100%> (+1.31%)`	⬆️
tools/pd-ctl/pdctl/ctl.go	`94.87% <100%> (+10.66%)`	⬆️
server/region_syncer/server.go	`82.05% <100%> (+0.15%)`	⬆️
client/client.go	`69.76% <100%> (+3.98%)`	⬆️
... and 32 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3c4fd92...add662c. Read the comment docs.

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Signed-off-by: nolouch <nolouch@gmail.com>

…th smaller size (tikv#1723) Signed-off-by: disksing <i@disksing.com>

…ikv#1715) Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

Signed-off-by: nolouch <nolouch@gmail.com>

Signed-off-by: Ryan Leung <rleungx@gmail.com>

rleungx and others added 30 commits August 19, 2019 15:41

*: unify get store function everywhere (tikv#1671)

561d004

Signed-off-by: Ryan Leung <rleungx@gmail.com>

remove unnecessary parentheses

7c300ed

server: use leader lease to determine tso service validity (tikv#1676)

16d7464

Signed-off-by: disksing <i@disksing.com>

Merge branch 'master' into new-hotspot-scheduler

83aaff9

Merge branch 'new-hotspot-scheduler' of https://github.com/pingcap/pd …

dd1fa5a

…into new-hotspot-scheduler

Merge branch 'new-hotspot-scheduler' of https://github.com/pingcap/pd …

ffba7e8

…into new-hotspot-scheduler

change internal stat values to float64

b0ffb77

add pending operator influence

dbc36ed

add metrics of pending influence

f4456f7

fix metrics

695d66d

fix panic

7b955b3

adjust pending influence of balanceHotWrite

640448f

change weight of pending influence

7401480

test: fix tests (tikv#1696)

de7e9de

* test: fix region syncer test Signed-off-by: disksing <i@disksing.com>

decrease region rolling window; store pending influence in scheduler

af47f38

add config-check flag for pd-server (tikv#1695)

2e401b6

Signed-off-by: cwen0 <cwenyin0@gmail.com>

decrease possiblility transfer hot write leader

da1fd48

change pending influence weight

ba3d8ca

add unstarted op metrics

63a91f9

add logs for debug

e74c7a9

add log for debug

2876fa0

add logs for debug

c9ef717

add logs for debug

2109c74

add logs for debug

5264b33

add logs for debug

804c4f2

add logs for debug

2b5219b

add logs for debug

7dc19ed

Revert "add logs for debug"

a283d66

This reverts commit e74c7a9.

add metrics for hotspot operators

63e4f0d

operator: rewrite move region related functions (tikv#1667)

84f6a82

Luffbee and others added 12 commits August 22, 2019 21:39

fix metrics thread-safe bug

9ee64d3

fix logic bug

98db272

*: reduce some unnecessary parameters (tikv#1698)

54975cd

Signed-off-by: Ryan Leung <rleungx@gmail.com>

server: fix the dead lock in scatter region (tikv#1706)

14b91e8

Signed-off-by: Ryan Leung <rleungx@gmail.com>

add drop time for operator

10c5d09

use IsDropped to recognize canceled ops

6996440

try to fix trans leader burst

c836b9c

try to fix trans leader burst

63bd217

add zombie influence

b43b783

change select src dst strategy; improve op_controller

ff59fdf

Merge branch 'new-hotspot-scheduler' of https://github.com/pingcap/pd …

b398dc5

…into new-hotspot-scheduler

Luffbee and others added 13 commits August 28, 2019 20:42

change select src strategy

82cd608

fix bug

03ce8cb

tools: fix set namespace in pd-ctl (tikv#1701)

15e2674

Signed-off-by: Ryan Leung <rleungx@gmail.com>

tools: fix parse url without http prefix (tikv#1703)

b1eadf8

Signed-off-by: Ryan Leung <rleungx@gmail.com>

tests: support deadlock detection in make test (tikv#1704)

a1f8d9d

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Makefile: fix failpoint enable (tikv#1722)

a67b5db

Signed-off-by: nolouch <nolouch@gmail.com>

checker: fix the issue that a region does not merge to the sibling wi…

7baffcb

…th smaller size (tikv#1723) Signed-off-by: disksing <i@disksing.com>

tools: balance region simulator (tikv#1708)

9af07b0

scheduler: do not remove the operator when the step does not finish (t…

144031c

…ikv#1715) Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

operator: fix the AddLearner config version judgment (tikv#1732)

38ff102

Signed-off-by: nolouch <nolouch@gmail.com>

tools: fix TLS in pd control (tikv#1729)

ca5cae6

Signed-off-by: Ryan Leung <rleungx@gmail.com>

syncer: support TLS for region syncer (tikv#1728)

f2487bf

Signed-off-by: Ryan Leung <rleungx@gmail.com>

schedule: fix a thread-safe bug and improve code (tikv#1719)

b66ba44

nolouch added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 9, 2019

Merge branch 'master' into new-hotspot-scheduler

add662c

Luffbee merged commit 83ab4c2 into tikv:new-hotspot-scheduler Sep 9, 2019

Luffbee deleted the new-hotspot-scheduler branch September 9, 2019 07:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[new-hotspot-scheduler] add operator influence and more metrics #1713

[new-hotspot-scheduler] add operator influence and more metrics #1713

Luffbee commented Aug 28, 2019

codecov-io commented Aug 28, 2019 •

edited

Loading

[new-hotspot-scheduler] add operator influence and more metrics #1713

[new-hotspot-scheduler] add operator influence and more metrics #1713

Conversation

Luffbee commented Aug 28, 2019

codecov-io commented Aug 28, 2019 • edited Loading

Codecov Report

codecov-io commented Aug 28, 2019 •

edited

Loading