New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: improve performance of count_data #1091

Merged

empiredan merged 13 commits into apache:master from GehaFearless:lgh_count_data

Aug 30, 2022

Contributor

GehaFearless commented Aug 1, 2022 •

edited

Loading

What problem does this PR solve?

issue: #1090
same job as before: #728

When we precisely count data for a large table, it will cost minutes or hours.
However, it's unnecessarily return key-values from server to client.

What is changed and how does it work?

Actually, we just need the count of data. So we just need transfer the count of data from server to client, but not the detailed data.

In my test, it will 2x on onebox faster than before.
for my bench on onebox：
1> before

2> this commit

Tests

Unit test
Manual test (add detailed scripts or steps below)

Related changes

Need to update the documentation
Need to be included in the release note

github-actions bot added cpp thrift labels

GehaFearless force-pushed the lgh_count_data branch from 42ad016 to 788d2ef Compare

August 1, 2022 08:29


          feat: improve performance of count_data

12798d8

GehaFearless force-pushed the lgh_count_data branch from 788d2ef to 12798d8 Compare

August 1, 2022 11:12

empiredan changed the title ~~improve performance of count_data~~ feat: improve performance of count_data

empiredan reviewed

View reviewed changes

src/shell/commands/data_operations.cpp Outdated Show resolved Hide resolved

src/shell/commands/data_operations.cpp Outdated Show resolved Hide resolved

src/shell/commands/data_operations.cpp Show resolved Hide resolved

src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved

src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved

src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved

src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved

src/shell/command_helper.h Outdated Show resolved Hide resolved

src/shell/command_helper.h Outdated Show resolved Hide resolved


          cr 1

71da038

GehaFearless force-pushed the lgh_count_data branch 2 times, most recently from f4c5af1 to 1be846c Compare

August 3, 2022 11:44

acelyc111 reviewed

View reviewed changes

idl/rrdb.thrift Outdated Show resolved Hide resolved

src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved

src/client_lib/pegasus_scanner_impl.cpp Show resolved Hide resolved

src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved

src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved

idl/rrdb.thrift Outdated Show resolved Hide resolved

src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved


          cr 2

d188ff6

GehaFearless force-pushed the lgh_count_data branch from 1be846c to d188ff6 Compare

August 5, 2022 08:45

WHBANG reviewed

View reviewed changes

src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved


          fix judgment

21d1b66

acelyc111 reviewed

View reviewed changes

src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved

src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved

src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved

src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved

src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved

src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved

src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved

src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved

GehaFearless force-pushed the lgh_count_data branch 3 times, most recently from 8d2fe21 to bb901e0 Compare

August 11, 2022 03:40

cr3

501ce57

GehaFearless force-pushed the lgh_count_data branch from bb901e0 to 501ce57 Compare

August 11, 2022 06:35

Member

acelyc111 commented Aug 12, 2022

Could you give some performance comparation with the old version ?

acelyc111 reviewed

View reviewed changes

src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved

src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved

src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved

src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved

GehaFearless force-pushed the lgh_count_data branch 3 times, most recently from 2e0e6fd to dad6442 Compare

August 15, 2022 14:06

cr4

12d1ec6

GehaFearless force-pushed the lgh_count_data branch from dad6442 to 12d1ec6 Compare

August 16, 2022 04:09

acelyc111 reviewed

View reviewed changes

src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved

src/test/function_test/test_scan.cpp Outdated Show resolved Hide resolved

src/shell/commands/data_operations.cpp Show resolved Hide resolved

idl/rrdb.thrift Outdated Show resolved Hide resolved

src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved

cr5

f45b0d7

acelyc111 reviewed

View reviewed changes

src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved

src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved

src/shell/commands/data_operations.cpp Outdated Show resolved Hide resolved

GehaFearless force-pushed the lgh_count_data branch from 456264a to b0585c3 Compare

August 18, 2022 07:58

cr6

865586c

GehaFearless force-pushed the lgh_count_data branch from b0585c3 to 865586c Compare

August 18, 2022 09:40

acelyc111 reviewed

View reviewed changes

src/client_lib/pegasus_client_impl.h Outdated Show resolved Hide resolved

src/test/function_test/test_scan.cpp Outdated Show resolved Hide resolved

src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved

cr7

0d0c64b

acelyc111 reviewed

View reviewed changes

src/client_lib/pegasus_client_impl.h Outdated Show resolved Hide resolved

src/client_lib/pegasus_client_impl.h Outdated Show resolved Hide resolved

GehaFearless force-pushed the lgh_count_data branch from 3dbc957 to f7de899 Compare

August 23, 2022 07:15

cr8

8de3a44

GehaFearless force-pushed the lgh_count_data branch from f7de899 to 8de3a44 Compare

August 23, 2022 08:05

acelyc111 previously approved these changes

View reviewed changes

empiredan reviewed

View reviewed changes

src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved

src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved

src/shell/commands/data_operations.cpp Outdated Show resolved Hide resolved

acelyc111 dismissed their stale review via

August 29, 2022 03:52

acelyc111 and others added 3 commits

August 29, 2022 11:52


          Update src/client_lib/pegasus_scanner_impl.cpp

Co-authored-by: Dan Wang <wangdan@apache.org>


          Update src/client_lib/pegasus_scanner_impl.cpp

c877cc2

Co-authored-by: Dan Wang <wangdan@apache.org>


          Update src/shell/commands/data_operations.cpp

a298ccc

Co-authored-by: Dan Wang <wangdan@apache.org>

acelyc111 approved these changes

View reviewed changes

empiredan approved these changes

View reviewed changes

empiredan merged commit 7e26c9f into apache:master

acelyc111 mentioned this pull request

improve performance of count_data #728

Closed

empiredan mentioned this pull request

Release 2.5.0 #1584

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels