refactor(collector): sort out the structure of partition hotspot detection #597

Smityz · 2020-09-07T09:47:52Z

What problem does this PR solve?

This submission is mainly to merge the previous code, as part of the overall refactoring.

What is changed and how it works?

In the previous design, partition hotspot detection is designed as a 'strategic model'.
in this PR, I will deprecate the formal design, and combine hotspot_policy hotspot_algo_qps_variance andhotspot_calculator into hotspot_partition_calculator, and change some variable names by the way.
The change of data structure and function will be refactored in #592

Rename file:
src/server/table_hotspot_policy.h -> src/server/hotspot_partition_calculator.h
src/server/table_hotspot_policy.cpp -> src/server/hotspot_partition_calculator.cpp
src/server/test/hotspot_partition_test.cpp -> src/server/test/pegasus_tablehotspot_test.cpp

Config changes

[pegasus.collector]
- hotspot_detect_algorithm

…ction

src/server/info_collector.cpp

src/server/hotspot_partition_policy.h

src/server/hotspot_partition_policy.cpp

src/server/hotspot_partition_calculator.cpp

src/server/hotspot_partition_calculator.h

src/server/info_collector.cpp

src/server/info_collector.h

src/server/info_collector.cpp

src/server/hotspot_partition_calculator.h

src/server/info_collector.h

src/server/info_collector.cpp

Co-authored-by: Yingchun Lai <405403881@qq.com>

src/server/hotspot_partition_calculator.h

src/server/test/hotspot_partition_test.cpp

src/server/hotspot_partition_calculator.cpp

src/server/hotspot_partition_calculator.h

src/server/hotspot_partition_calculator.cpp

Co-authored-by: Wu Tao <wutao1@xiaomi.com>

src/server/hotspot_partition_calculator.cpp

src/server/hotspot_partition_calculator.h

src/server/hotspot_partition_calculator.cpp

neverchanje · 2020-09-09T11:45:51Z

src/server/hotspot_partition_calculator.cpp

+    int sample_count = 0;
+    while (!temp_data.empty()) {
+        for (const auto &partition_data : temp_data.front()) {
+            if (partition_data.total_qps - 1.00 > 0) {


Why need such prerequiste? What if a partition qps is 0?

In order to prevent too many 0 elements in the queue from interfering with the accuracy of the historical data, we choose to ignore some small values

Co-authored-by: Wu Tao <wutao1@xiaomi.com>

foreverneverer · 2020-09-10T03:35:25Z

src/server/hotspot_partition_data.h

+enum partition_qps_type
+{
+    READ_HOTSPOT_DATA = 0,
+    WRITE_HOTSPOT_DATA


why READ_HOTSPOT_DATA init explictly, but WRITE_HOTSPOT_DATA no?

in cpp, the enum is increased by one orderly. so we only need to init the first one

The first default 0, so also can not be inited?

neverchanje · 2020-09-10T03:58:37Z

src/server/hotspot_partition_calculator.h

+typedef std::vector<std::vector<std::unique_ptr<dsn::perf_counter_wrapper>>> hot_partition_counters;
+
+// hotspot_partition_calculator is used to find the hot partition in a table.
+class hotspot_partition_calculator


Suggested change

class hotspot_partition_calculator

class hotspot_partition_calculator : public replica_base

replica_base has a app_name() method.

hotspot_partition_calculator stores the statistical data of the correspond table, _app_name is for that table, not itself.

Smityz added 3 commits September 7, 2020 17:40

refactor(collector): sort out the structure of partition hotspot dete…

14f5f52

…ction

add cpp

f65aa72

license

785be3f

Smityz mentioned this pull request Sep 7, 2020

refactor: split read/write hotspot of hotspot_partition_calculator #592

Merged

acelyc111 reviewed Sep 7, 2020

View reviewed changes

src/server/info_collector.cpp Outdated Show resolved Hide resolved

src/server/hotspot_partition_policy.h Outdated Show resolved Hide resolved

src/server/hotspot_partition_policy.cpp Outdated Show resolved Hide resolved

Smityz and others added 3 commits September 8, 2020 10:52

del policy

98bd7f3

header

54135fb

Merge branch 'master' into del_old_alg

76ff93b

levy5307 reviewed Sep 8, 2020

View reviewed changes

src/server/hotspot_partition_calculator.cpp Show resolved Hide resolved

levy5307 reviewed Sep 8, 2020

View reviewed changes

src/server/hotspot_partition_calculator.cpp Outdated Show resolved Hide resolved

levy5307 reviewed Sep 8, 2020

View reviewed changes

src/server/hotspot_partition_calculator.cpp Outdated Show resolved Hide resolved

levy5307 reviewed Sep 8, 2020

View reviewed changes

src/server/hotspot_partition_calculator.cpp Outdated Show resolved Hide resolved

levy5307 reviewed Sep 8, 2020

View reviewed changes

src/server/hotspot_partition_calculator.h Outdated Show resolved Hide resolved

levy5307 reviewed Sep 8, 2020

View reviewed changes

src/server/hotspot_partition_calculator.h Outdated Show resolved Hide resolved

levy5307 reviewed Sep 8, 2020

View reviewed changes

src/server/info_collector.cpp Outdated Show resolved Hide resolved

hycdong reviewed Sep 8, 2020

View reviewed changes

src/server/info_collector.h Outdated Show resolved Hide resolved

src/server/info_collector.cpp Outdated Show resolved Hide resolved

src/server/info_collector.cpp Outdated Show resolved Hide resolved

src/server/hotspot_partition_calculator.h Outdated Show resolved Hide resolved

Smityz added 2 commits September 8, 2020 17:44

update

3c8a516

fix

dfa3091

levy5307 reviewed Sep 8, 2020

View reviewed changes

src/server/info_collector.h Show resolved Hide resolved

acelyc111 reviewed Sep 8, 2020

View reviewed changes

src/server/info_collector.cpp Outdated Show resolved Hide resolved

Smityz and others added 2 commits September 8, 2020 18:15

Update src/server/info_collector.cpp

f31953a

Co-authored-by: Yingchun Lai <405403881@qq.com>

Merge branch 'master' into del_old_alg

6575e40

hycdong reviewed Sep 8, 2020

View reviewed changes

src/server/hotspot_partition_calculator.h Outdated Show resolved Hide resolved

Smityz added 6 commits September 8, 2020 22:57

rename

6d62ad2

Merge branch 'del_old_alg' of github.com:Smityz/pegasus into del_old_alg

8a421bf

merge

e74ee75

rename

ce085fc

rename

6a63987

format

1143c74

levy5307 reviewed Sep 9, 2020

View reviewed changes

src/server/test/hotspot_partition_test.cpp Show resolved Hide resolved

foreverneverer reviewed Sep 9, 2020

View reviewed changes

src/server/hotspot_partition_calculator.cpp Outdated Show resolved Hide resolved

neverchanje reviewed Sep 9, 2020

View reviewed changes

foreverneverer reviewed Sep 9, 2020

View reviewed changes

src/server/hotspot_partition_calculator.cpp Outdated Show resolved Hide resolved

Smityz and others added 9 commits September 9, 2020 18:01

format

99954d7

Update src/server/hotspot_partition_calculator.h

e44e08c

Co-authored-by: Wu Tao <wutao1@xiaomi.com>

Update src/server/hotspot_partition_calculator.h

b2b97e8

Co-authored-by: Wu Tao <wutao1@xiaomi.com>

Update src/server/hotspot_partition_calculator.cpp

7e9a76b

Co-authored-by: Wu Tao <wutao1@xiaomi.com>

Update src/server/hotspot_partition_calculator.cpp

0134f50

Co-authored-by: Wu Tao <wutao1@xiaomi.com>

update

8ac6320

Merge branch 'del_old_alg' of github.com:Smityz/pegasus into del_old_alg

5b1a36a

format

120dab2

format

cec568f

foreverneverer reviewed Sep 9, 2020

View reviewed changes

src/server/hotspot_partition_calculator.cpp Outdated Show resolved Hide resolved

Update hotspot_partition_calculator.cpp

012ce98

neverchanje reviewed Sep 9, 2020

View reviewed changes

src/server/hotspot_partition_calculator.cpp Outdated Show resolved Hide resolved

src/server/hotspot_partition_calculator.h Outdated Show resolved Hide resolved

src/server/hotspot_partition_calculator.cpp Outdated Show resolved Hide resolved

neverchanje reviewed Sep 9, 2020

View reviewed changes

Smityz and others added 2 commits September 9, 2020 20:45

Update src/server/hotspot_partition_calculator.cpp

d27d7a2

Co-authored-by: Wu Tao <wutao1@xiaomi.com>

format

734f290

Smityz force-pushed the del_old_alg branch from 734f290 to 5e201f6 Compare September 10, 2020 02:57

foreverneverer reviewed Sep 10, 2020

View reviewed changes

Smityz force-pushed the del_old_alg branch from 5e201f6 to 734f290 Compare September 10, 2020 04:10

Smityz added 2 commits September 10, 2020 14:39

update

88de731

todo

05dc899

acelyc111 approved these changes Sep 10, 2020

View reviewed changes

Merge branch 'master' into del_old_alg

6ff0822

neverchanje approved these changes Sep 10, 2020

View reviewed changes

neverchanje merged commit b057848 into apache:master Sep 10, 2020

Smityz mentioned this pull request Sep 17, 2020

The way to split PR 502 #588

Closed

Smityz deleted the del_old_alg branch October 20, 2020 09:24

neverchanje mentioned this pull request Mar 1, 2021

Release 2.2.0 #696

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(collector): sort out the structure of partition hotspot detection #597

refactor(collector): sort out the structure of partition hotspot detection #597

Smityz commented Sep 7, 2020 •

edited

Loading

neverchanje Sep 9, 2020

Smityz Sep 9, 2020

foreverneverer Sep 10, 2020

levy5307 Sep 10, 2020 •

edited

Loading

foreverneverer Sep 10, 2020

neverchanje Sep 10, 2020

Smityz Sep 10, 2020

	class hotspot_partition_calculator
	class hotspot_partition_calculator : public replica_base

refactor(collector): sort out the structure of partition hotspot detection #597

refactor(collector): sort out the structure of partition hotspot detection #597

Conversation

Smityz commented Sep 7, 2020 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Config changes

neverchanje Sep 9, 2020

Choose a reason for hiding this comment

Smityz Sep 9, 2020

Choose a reason for hiding this comment

foreverneverer Sep 10, 2020

Choose a reason for hiding this comment

levy5307 Sep 10, 2020 • edited Loading

Choose a reason for hiding this comment

foreverneverer Sep 10, 2020

Choose a reason for hiding this comment

neverchanje Sep 10, 2020

Choose a reason for hiding this comment

Smityz Sep 10, 2020

Choose a reason for hiding this comment

Smityz commented Sep 7, 2020 •

edited

Loading

levy5307 Sep 10, 2020 •

edited

Loading