Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

executor: Improve the performance of WindowExec by using sliding window #14294

Merged
merged 31 commits into from
Feb 7, 2020

Conversation

mmyj
Copy link
Member

@mmyj mmyj commented Dec 31, 2019

What problem does this PR solve?

Improve the performance of WindowExec by using sliding window or segment tree #12967

What is changed and how it works?

1 add a interface named SlidingWindowAggFunc for evaluating the window aggregate functions using sliding window
2 modify the rowFrameWindowProcessor.appendResult2Chunk to use sliding window

Check List

Tests

  • No code

Code changes

  • Has exported function/method change
  • Has interface methods change

Benchmark

windowFuncs := []string{
	ast.AggFuncCount,
}
rows := []int{1000, 100000}
ndvs := []int{10, 1000}
frames := []*core.WindowFrame{
	{Type: ast.Rows, Start: &core.FrameBound{Type: ast.Preceding, Num: 10}, End: &core.FrameBound{Type: ast.Following, Num: 10}},
	{Type: ast.Rows, Start: &core.FrameBound{Type: ast.Preceding, Num: 100}, End: &core.FrameBound{Type: ast.Following, Num: 100}},
}

before

BenchmarkWindowFunctionsWithSlidingWindow/(func:count,_ndv:10,_rows:1000)-8         	    4304	    249372 ns/op
BenchmarkWindowFunctionsWithSlidingWindow/(func:count,_ndv:10,_rows:1000)#01-8      	    1572	    760644 ns/op
BenchmarkWindowFunctionsWithSlidingWindow/(func:count,_ndv:1000,_rows:1000)-8       	    4851	    230780 ns/op
BenchmarkWindowFunctionsWithSlidingWindow/(func:count,_ndv:1000,_rows:1000)#01-8    	    5337	    232417 ns/op
BenchmarkWindowFunctionsWithSlidingWindow/(func:count,_ndv:10,_rows:100000)-8       	      51	  22412645 ns/op
BenchmarkWindowFunctionsWithSlidingWindow/(func:count,_ndv:10,_rows:100000)#01-8    	       8	 139406682 ns/op
BenchmarkWindowFunctionsWithSlidingWindow/(func:count,_ndv:1000,_rows:100000)-8     	      52	  22229838 ns/op
BenchmarkWindowFunctionsWithSlidingWindow/(func:count,_ndv:1000,_rows:100000)#01-8  	      15	  74082167 ns/op

now

BenchmarkWindowFunctionsWithSlidingWindow/(func:count,_ndv:10,_rows:1000)-8         	   13956	     85242 ns/op
BenchmarkWindowFunctionsWithSlidingWindow/(func:count,_ndv:10,_rows:1000)#01-8      	   16231	     76437 ns/op
BenchmarkWindowFunctionsWithSlidingWindow/(func:count,_ndv:1000,_rows:1000)-8       	    7408	    165960 ns/op
BenchmarkWindowFunctionsWithSlidingWindow/(func:count,_ndv:1000,_rows:1000)#01-8    	    7059	    163368 ns/op
BenchmarkWindowFunctionsWithSlidingWindow/(func:count,_ndv:10,_rows:100000)-8       	     181	   6624173 ns/op
BenchmarkWindowFunctionsWithSlidingWindow/(func:count,_ndv:10,_rows:100000)#01-8    	     177	   6605687 ns/op
BenchmarkWindowFunctionsWithSlidingWindow/(func:count,_ndv:1000,_rows:100000)-8     	     183	   6427620 ns/op
BenchmarkWindowFunctionsWithSlidingWindow/(func:count,_ndv:1000,_rows:100000)#01-8  	     206	   5707492 ns/op

benchstat

name                                                                       old time/op  new time/op  delta
WindowFunctionsWithSlidingWindow/(func:count,_ndv:10,_rows:1000)-8          447µs ± 1%   127µs ± 3%  -71.68%  (p=0.000 n=10+10)
WindowFunctionsWithSlidingWindow/(func:count,_ndv:10,_rows:1000)#01-8      1.82ms ± 1%  0.11ms ± 3%  -93.99%  (p=0.000 n=10+10)
WindowFunctionsWithSlidingWindow/(func:count,_ndv:1000,_rows:1000)-8        265µs ± 1%   237µs ± 1%  -10.58%  (p=0.000 n=10+9)
WindowFunctionsWithSlidingWindow/(func:count,_ndv:1000,_rows:1000)#01-8     266µs ± 1%   238µs ± 1%  -10.48%  (p=0.000 n=10+10)
WindowFunctionsWithSlidingWindow/(func:count,_ndv:10,_rows:100000)-8       44.8ms ± 1%  10.6ms ± 2%  -76.24%  (p=0.000 n=10+10)
WindowFunctionsWithSlidingWindow/(func:count,_ndv:10,_rows:100000)#01-8     350ms ± 3%    11ms ± 1%  -96.88%  (p=0.000 n=9+10)
WindowFunctionsWithSlidingWindow/(func:count,_ndv:1000,_rows:100000)-8     43.0ms ± 1%  10.6ms ± 1%  -75.35%  (p=0.000 n=9+10)
WindowFunctionsWithSlidingWindow/(func:count,_ndv:1000,_rows:100000)#01-8   181ms ± 1%     9ms ± 1%  -94.81%  (p=0.000 n=10+9)

mmyj and others added 2 commits November 14, 2019 22:37
…w or segment tree pingcap#12967

just improve slide window of countOriginal4Int
@sre-bot sre-bot added the contribution This PR is from a community contributor. label Dec 31, 2019
Copy link
Contributor

@SunRunAway SunRunAway left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It has some benchmarks for window functions in executor/benchmark_test.go, could you modify it and provide a benchmark result to show how performance we gain.

executor/aggfuncs/aggfuncs.go Show resolved Hide resolved
@SunRunAway
Copy link
Contributor

/run-all-tests

Copy link
Contributor

@SunRunAway SunRunAway left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rest LGTM.

@mmyj

This comment has been minimized.

executor/aggfuncs/func_count.go Outdated Show resolved Hide resolved
executor/aggfuncs/func_count.go Outdated Show resolved Hide resolved
@mmyj

This comment has been minimized.

@mmyj
Copy link
Member Author

mmyj commented Jan 1, 2020

benchstat result

name                                                                       old time/op  new time/op  delta
WindowFunctionsWithSlidingWindow/(func:count,_ndv:10,_rows:1000)-8          447µs ± 1%   127µs ± 3%  -71.68%  (p=0.000 n=10+10)
WindowFunctionsWithSlidingWindow/(func:count,_ndv:10,_rows:1000)#01-8      1.82ms ± 1%  0.11ms ± 3%  -93.99%  (p=0.000 n=10+10)
WindowFunctionsWithSlidingWindow/(func:count,_ndv:1000,_rows:1000)-8        265µs ± 1%   237µs ± 1%  -10.58%  (p=0.000 n=10+9)
WindowFunctionsWithSlidingWindow/(func:count,_ndv:1000,_rows:1000)#01-8     266µs ± 1%   238µs ± 1%  -10.48%  (p=0.000 n=10+10)
WindowFunctionsWithSlidingWindow/(func:count,_ndv:10,_rows:100000)-8       44.8ms ± 1%  10.6ms ± 2%  -76.24%  (p=0.000 n=10+10)
WindowFunctionsWithSlidingWindow/(func:count,_ndv:10,_rows:100000)#01-8     350ms ± 3%    11ms ± 1%  -96.88%  (p=0.000 n=9+10)
WindowFunctionsWithSlidingWindow/(func:count,_ndv:1000,_rows:100000)-8     43.0ms ± 1%  10.6ms ± 1%  -75.35%  (p=0.000 n=9+10)
WindowFunctionsWithSlidingWindow/(func:count,_ndv:1000,_rows:100000)#01-8   181ms ± 1%     9ms ± 1%  -94.81%  (p=0.000 n=10+9)

executor/window.go Outdated Show resolved Hide resolved
@SunRunAway SunRunAway changed the title Feat sliding window executor: Improve the performance of WindowExec by using sliding window Jan 1, 2020
Copy link
Contributor

@SunRunAway SunRunAway left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@SunRunAway
Copy link
Contributor

@qw4990 @XuHuaiyu PTAL

@SunRunAway SunRunAway added the status/LGT1 Indicates that a PR has LGTM 1. label Jan 3, 2020
mmyj added a commit to mmyj/tidb that referenced this pull request Feb 10, 2020
…ndow pingcap#14294

implement range clause of SlidingWindowExec
fix window_test_case
mmyj added a commit to mmyj/tidb that referenced this pull request Mar 1, 2020
mmyj added a commit to mmyj/tidb that referenced this pull request Mar 1, 2020
mmyj added a commit to mmyj/tidb that referenced this pull request Mar 1, 2020
mmyj added a commit to mmyj/tidb that referenced this pull request Mar 1, 2020
mmyj added a commit to mmyj/tidb that referenced this pull request Mar 1, 2020
mmyj added a commit to mmyj/tidb that referenced this pull request Mar 1, 2020
mmyj added a commit to mmyj/tidb that referenced this pull request Mar 4, 2020
mmyj added a commit to mmyj/tidb that referenced this pull request Mar 4, 2020
mmyj added a commit to mmyj/tidb that referenced this pull request Mar 5, 2020
mmyj added a commit to mmyj/tidb that referenced this pull request Mar 5, 2020
mmyj added a commit to mmyj/tidb that referenced this pull request Mar 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
contribution This PR is from a community contributor. sig/execution SIG execution status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants