[feature](agg-func) linear_histogram #39546

joker-star-l · 2024-08-19T03:26:38Z

Proposed changes

add aggregate function: linear_histogram

LINEAR_HISTOGRAM

Description

linear_histogram(expr, DOUBLE interval[, DOUBLE offset)

The linear_histogram function is used to describe the distribution of the data, It uses an "equal width" bucking strategy, and divides the data into buckets according to the value of the data.

Parameter description:

interval: Required. The bucket width.
offset: Optional. Default is 0. It should be in [0, interval).

Example

mysql> select linear_histogram(a, 2) from histogram_test;
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| linear_histogram(a, cast(2 as DOUBLE))                                                                                                                                                                                                                                                                                                           |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| {"num_buckets":6,"buckets":[{"lower":0.0,"upper":2.0,"count":2,"acc_count":2},{"lower":2.0,"upper":4.0,"count":4,"acc_count":6},{"lower":4.0,"upper":6.0,"count":4,"acc_count":10},{"lower":6.0,"upper":8.0,"count":4,"acc_count":14},{"lower":8.0,"upper":10.0,"count":4,"acc_count":18},{"lower":10.0,"upper":12.0,"count":2,"acc_count":20}]} |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.07 sec)

mysql> select linear_histogram(a, 2, 1) from histogram_test;
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| linear_histogram(a, cast(2 as DOUBLE), cast(1 as DOUBLE))                                                                                                                                                                                                                                   |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| {"num_buckets":5,"buckets":[{"lower":1.0,"upper":3.0,"count":4,"acc_count":4},{"lower":3.0,"upper":5.0,"count":4,"acc_count":8},{"lower":5.0,"upper":7.0,"count":4,"acc_count":12},{"lower":7.0,"upper":9.0,"count":4,"acc_count":16},{"lower":9.0,"upper":11.0,"count":4,"acc_count":20}]} |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.04 sec)

Field description:

num_buckets: The number of buckets.
buckets: All buckets.
- lower: Lower bound of the bucket. (included)
- upper: Upper bound of the bucket. (not included)
- count: The number of elements contained in the bucket.
- acc_count: Accumulated count.

Doc PR

linear_histogram in doris-website

doris-robot · 2024-08-19T03:26:42Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

joker-star-l · 2024-08-19T03:27:42Z

@xiaokang

xiaokang · 2024-08-19T03:41:08Z

run buildall

doris-robot · 2024-08-19T04:08:00Z

TPC-H: Total hot run time: 38361 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 721ba15860fea6304ab52a0d162fb9b1a95f6ab6, data reload: false

------ Round 1 ----------------------------------
q1	18134	4448	4390	4390
q2	2295	208	210	208
q3	10425	1137	1103	1103
q4	10186	748	854	748
q5	7760	2842	2768	2768
q6	281	160	161	160
q7	1020	654	646	646
q8	9389	2107	2077	2077
q9	7300	6560	6492	6492
q10	7033	2199	2290	2199
q11	522	279	280	279
q12	431	268	267	267
q13	17971	3000	3009	3000
q14	300	258	260	258
q15	576	522	523	522
q16	512	416	405	405
q17	982	731	727	727
q18	7436	6967	6675	6675
q19	7254	1098	1041	1041
q20	741	356	349	349
q21	3923	2994	3064	2994
q22	1153	1053	1053	1053
Total cold run time: 115624 ms
Total hot run time: 38361 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4472	4309	4291	4291
q2	391	314	313	313
q3	2850	2664	2590	2590
q4	1901	1629	1645	1629
q5	5380	5405	5363	5363
q6	237	145	145	145
q7	2070	1709	1771	1709
q8	3198	3353	3295	3295
q9	8486	8373	8394	8373
q10	3405	3165	3154	3154
q11	611	522	524	522
q12	826	635	652	635
q13	17517	3029	3015	3015
q14	345	310	321	310
q15	553	529	527	527
q16	510	453	442	442
q17	1789	1543	1500	1500
q18	7661	7587	7447	7447
q19	1744	1682	1644	1644
q20	2021	1822	1843	1822
q21	5323	5226	5096	5096
q22	1172	1062	1033	1033
Total cold run time: 72462 ms
Total hot run time: 54855 ms

xiaokang

add function in fe and add .groovy sql testcase

be/src/vec/aggregate_functions/aggregate_function_linear_histogram.h

doris-robot · 2024-08-19T04:20:21Z

TPC-DS: Total hot run time: 190978 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 721ba15860fea6304ab52a0d162fb9b1a95f6ab6, data reload: false

query1	991	394	398	394
query2	6751	1977	1880	1880
query3	6666	228	240	228
query4	29616	23308	23167	23167
query5	4417	722	699	699
query6	314	213	208	208
query7	4621	338	322	322
query8	480	441	430	430
query9	8611	2613	2577	2577
query10	510	361	346	346
query11	17879	15012	15299	15012
query12	183	142	139	139
query13	1716	451	439	439
query14	10039	6840	6745	6745
query15	289	204	192	192
query16	8016	514	497	497
query17	1616	592	578	578
query18	2149	348	357	348
query19	253	171	168	168
query20	142	140	141	140
query21	244	145	142	142
query22	4312	4031	4179	4031
query23	34037	33457	33380	33380
query24	11275	2891	2900	2891
query25	664	417	435	417
query26	1166	179	183	179
query27	2765	303	307	303
query28	7512	2165	2126	2126
query29	843	459	441	441
query30	334	190	183	183
query31	1050	853	805	805
query32	122	83	78	78
query33	808	345	343	343
query34	911	513	504	504
query35	880	752	755	752
query36	1115	958	916	916
query37	178	105	106	105
query38	4083	3907	3844	3844
query39	1548	1544	1468	1468
query40	243	159	155	155
query41	145	142	142	142
query42	140	118	123	118
query43	541	511	503	503
query44	1280	787	797	787
query45	227	199	229	199
query46	1127	785	767	767
query47	1933	1877	1875	1875
query48	412	342	344	342
query49	1263	585	580	580
query50	868	476	463	463
query51	6878	6765	6830	6765
query52	120	110	111	110
query53	299	232	226	226
query54	925	495	495	495
query55	92	90	89	89
query56	326	310	311	310
query57	1236	1131	1118	1118
query58	312	301	316	301
query59	2942	2710	2754	2710
query60	351	329	332	329
query61	151	147	147	147
query62	895	692	688	688
query63	257	229	227	227
query64	5341	2397	1894	1894
query65	3254	3158	3173	3158
query66	1156	691	728	691
query67	15351	14808	15069	14808
query68	6434	580	582	580
query69	726	413	334	334
query70	1239	1183	1151	1151
query71	577	315	324	315
query72	7677	2326	2073	2073
query73	846	353	359	353
query74	9231	8830	8880	8830
query75	4985	2731	2739	2731
query76	4725	1027	1023	1023
query77	896	442	499	442
query78	11269	9693	9876	9693
query79	13109	568	558	558
query80	1028	605	617	605
query81	610	258	252	252
query82	598	159	167	159
query83	416	214	213	213
query84	288	97	96	96
query85	828	358	348	348
query86	461	336	302	302
query87	4475	4440	4315	4315
query88	4998	2456	2646	2456
query89	511	332	330	330
query90	2109	229	231	229
query91	156	131	128	128
query92	89	112	82	82
query93	5833	568	548	548
query94	1040	313	323	313
query95	388	289	294	289
query96	636	290	280	280
query97	3198	3046	3017	3017
query98	250	226	240	226
query99	1517	1334	1293	1293
Total cold run time: 318314 ms
Total hot run time: 190978 ms

doris-robot · 2024-08-19T04:25:55Z

ClickBench: Total hot run time: 30.97 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 721ba15860fea6304ab52a0d162fb9b1a95f6ab6, data reload: false

query1	0.04	0.05	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.08
query5	0.50	0.49	0.49
query6	1.12	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.56	0.50	0.48
query10	0.55	0.55	0.53
query11	0.15	0.12	0.12
query12	0.16	0.12	0.13
query13	0.63	0.62	0.60
query14	0.77	0.78	0.79
query15	0.85	0.82	0.84
query16	0.38	0.36	0.36
query17	1.02	1.04	0.97
query18	0.22	0.23	0.22
query19	1.79	1.67	1.69
query20	0.01	0.01	0.01
query21	15.41	0.69	0.67
query22	4.09	7.93	1.62
query23	18.30	1.38	1.31
query24	2.04	0.25	0.23
query25	0.16	0.09	0.08
query26	0.32	0.24	0.24
query27	0.45	0.23	0.23
query28	13.26	1.03	1.00
query29	12.62	3.35	3.32
query30	0.43	0.26	0.24
query31	2.82	0.41	0.40
query32	3.23	0.49	0.50
query33	2.94	2.96	2.96
query34	17.10	4.39	4.35
query35	4.48	4.46	4.47
query36	0.68	0.49	0.50
query37	0.22	0.18	0.17
query38	0.18	0.17	0.17
query39	0.07	0.06	0.06
query40	0.18	0.14	0.15
query41	0.12	0.07	0.07
query42	0.08	0.08	0.06
query43	0.07	0.05	0.06
Total cold run time: 110.05 s
Total hot run time: 30.97 s

be/src/vec/aggregate_functions/aggregate_function_linear_histogram.cpp

superdiaodiao

Code format is needed, see details in the log:
https://github.com/apache/doris/actions/runs/10446613781/job/28924273137?pr=39546

Tip: you can use the IDE(vscode, clion...) to format the codes automatically.

joker-star-l · 2024-08-21T13:45:55Z

@superdiaodiao @yiguolei Hello, is this class GenerateFunction used to generate new functions? When I run the test function generate(), I got some errors:

java.lang.IllegalStateException: Unsupported generate code by data type: decimalv3(18,0)

	at org.apache.doris.nereids.rules.analysis.GenerateFunction.getDataTypeAndInstance(GenerateFunction.java:1056)
	at org.apache.doris.nereids.rules.analysis.GenerateFunction.generateSignatures(GenerateFunction.java:683)
	at org.apache.doris.nereids.rules.analysis.GenerateFunction.generateFunctions(GenerateFunction.java:596)
	at org.apache.doris.nereids.rules.analysis.GenerateFunction.lambda$collectFunctionCodes$9(GenerateFunction.java:549)
	at org.apache.doris.nereids.rules.analysis.GenerateFunction.collectFunctionCodes(GenerateFunction.java:556)
	at org.apache.doris.nereids.rules.analysis.GenerateFunction.generate(GenerateFunction.java:276)

Am I using it incorrectly?

superdiaodiao · 2024-08-21T15:58:50Z

@superdiaodiao @yiguolei Hello, is this ......
Am I using it incorrectly?

Actually, we have to add FE codes manually rather than generating it, since generating code was out of date and served as a temporary method before.

github-actions

clang-tidy made some suggestions

be/src/vec/aggregate_functions/aggregate_function_linear_histogram.h

be/test/vec/aggregate_functions/agg_linear_histogram_test.cpp

joker-star-l · 2024-08-22T06:50:34Z

@superdiaodiao @yiguolei Hello, is this ......
Am I using it incorrectly?

Actually, we have to add FE codes manually rather than generating it, since generating code was out of date and served as a temporary method before.

Thank you. I add it manually. But when I did the e2e test, I got a strange error. (This error did not occur when I did unit test in the be)
The agg function linear_histogram returns a json str.
I got different double values in json with same query and data.
The test case:

CREATE TABLE IF NOT EXISTS histogram_test
(
    a int
)
ENGINE = OLAP
DUPLICATE KEY(a)
DISTRIBUTED BY HASH(a) BUCKETS AUTO
PROPERTIES (
"replication_num" = "1"
)
;

insert into histogram_test values (1);
insert into histogram_test values (2);
insert into histogram_test values (3);
insert into histogram_test values (4);
insert into histogram_test values (5);
insert into histogram_test values (6);
insert into histogram_test values (7);
insert into histogram_test values (8);

select linear_histogram(a, 2) from histogram_test;

The result of the first execution:

{"num_buckets":5,"buckets":[{"lower":4.6737435351163e-310,"upper":9.34748707015187e-310,"count":1,"acc_count":1},{"lower":9.34748707015187e-310,"upper":1.402123060518747e-309,"count":2,"acc_count":3},{"lower":1.402123060518747e-309,"upper":1.869497414022306e-309,"count":2,"acc_count":5},{"lower":1.869497414022306e-309,"upper":2.336871767525866e-309,"count":2,"acc_count":7},{"lower":2.336871767525866e-309,"upper":2.804246121029425e-309,"count":1,"acc_count":8}]}

The result of the second execution:

{"num_buckets":5,"buckets":[{"lower":9.399408234378048e218,"upper":3.8141098555780214e228,"count":1,"acc_count":1},{"lower":3.8141098555780214e228,"upper":7.628219710216102e228,"count":2,"acc_count":3},{"lower":7.628219710216102e228,"upper":1.1442329564854183e229,"count":2,"acc_count":5},{"lower":1.1442329564854183e229,"upper":1.5256439419492263e229,"count":2,"acc_count":7},{"lower":1.5256439419492263e229,"upper":1.9070549274130343e229,"count":1,"acc_count":8}]}

The result of the third execution:

{"num_buckets":5,"buckets":[{"lower":0.0,"upper":2.0,"count":1,"acc_count":1},{"lower":2.0,"upper":4.0,"count":2,"acc_count":3},{"lower":4.0,"upper":6.0,"count":2,"acc_count":5},{"lower":6.0,"upper":8.0,"count":2,"acc_count":7},{"lower":8.0,"upper":10.0,"count":1,"acc_count":8}]}

...

Only the result of the third execution is correct. I am very confused about this. Why this happens?

joker-star-l · 2024-08-22T11:08:07Z

@superdiaodiao @yiguolei Hello, is this ......
Am I using it incorrectly?

Actually, we have to add FE codes manually rather than generating it, since generating code was out of date and served as a temporary method before.

Thank you. I add it manually. But when I did the e2e test, I got a strange error. (This error did not occur when I did unit test in the be) The agg function linear_histogram returns a json str. I got different double values in json with same query and data. The test case:

CREATE TABLE IF NOT EXISTS histogram_test
(
    a int
)
ENGINE = OLAP
DUPLICATE KEY(a)
DISTRIBUTED BY HASH(a) BUCKETS AUTO
PROPERTIES (
"replication_num" = "1"
)
;

insert into histogram_test values (1);
insert into histogram_test values (2);
insert into histogram_test values (3);
insert into histogram_test values (4);
insert into histogram_test values (5);
insert into histogram_test values (6);
insert into histogram_test values (7);
insert into histogram_test values (8);

select linear_histogram(a, 2) from histogram_test;

The result of the first execution:

{"num_buckets":5,"buckets":[{"lower":4.6737435351163e-310,"upper":9.34748707015187e-310,"count":1,"acc_count":1},{"lower":9.34748707015187e-310,"upper":1.402123060518747e-309,"count":2,"acc_count":3},{"lower":1.402123060518747e-309,"upper":1.869497414022306e-309,"count":2,"acc_count":5},{"lower":1.869497414022306e-309,"upper":2.336871767525866e-309,"count":2,"acc_count":7},{"lower":2.336871767525866e-309,"upper":2.804246121029425e-309,"count":1,"acc_count":8}]}

The result of the second execution:

{"num_buckets":5,"buckets":[{"lower":9.399408234378048e218,"upper":3.8141098555780214e228,"count":1,"acc_count":1},{"lower":3.8141098555780214e228,"upper":7.628219710216102e228,"count":2,"acc_count":3},{"lower":7.628219710216102e228,"upper":1.1442329564854183e229,"count":2,"acc_count":5},{"lower":1.1442329564854183e229,"upper":1.5256439419492263e229,"count":2,"acc_count":7},{"lower":1.5256439419492263e229,"upper":1.9070549274130343e229,"count":1,"acc_count":8}]}

The result of the third execution:

{"num_buckets":5,"buckets":[{"lower":0.0,"upper":2.0,"count":1,"acc_count":1},{"lower":2.0,"upper":4.0,"count":2,"acc_count":3},{"lower":4.0,"upper":6.0,"count":2,"acc_count":5},{"lower":6.0,"upper":8.0,"count":2,"acc_count":7},{"lower":8.0,"upper":10.0,"count":1,"acc_count":8}]}

...

Only the result of the third execution is correct. I am very confused about this. Why this happens?

Sorry, I have fixed it. I just forgot to set parameters in merge function~~

github-actions

clang-tidy made some suggestions

be/test/vec/aggregate_functions/agg_linear_histogram_test.cpp

github-actions

clang-tidy made some suggestions

be/test/vec/aggregate_functions/agg_linear_histogram_test.cpp

joker-star-l · 2024-08-26T09:33:04Z

run buildall

regression-test/data/nereids_function_p0/agg_function/agg.out

doris-robot · 2024-08-26T09:51:40Z

TPC-H: Total hot run time: 37898 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3b2f87cd3b48345dacc417d984aaa716aff3cf40, data reload: false

------ Round 1 ----------------------------------
q1	18151	4461	4383	4383
q2	3112	181	187	181
q3	11535	1142	1164	1142
q4	10350	687	755	687
q5	7792	2898	2884	2884
q6	229	136	138	136
q7	967	614	602	602
q8	9326	2041	2035	2035
q9	7298	6550	6559	6550
q10	6992	2214	2219	2214
q11	466	237	240	237
q12	394	220	216	216
q13	17797	3058	3001	3001
q14	287	233	240	233
q15	527	485	488	485
q16	479	389	386	386
q17	981	682	737	682
q18	7326	6832	6804	6804
q19	1396	1000	1075	1000
q20	669	339	344	339
q21	4010	3100	2691	2691
q22	1134	1036	1010	1010
Total cold run time: 111218 ms
Total hot run time: 37898 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4320	4268	4269	4268
q2	378	270	272	270
q3	2901	2649	2649	2649
q4	1961	1647	1644	1644
q5	5389	5404	5397	5397
q6	218	129	127	127
q7	2123	1685	1690	1685
q8	3206	3341	3315	3315
q9	8382	8398	8385	8385
q10	3440	3210	3167	3167
q11	593	505	493	493
q12	784	578	616	578
q13	11980	3039	2999	2999
q14	312	278	280	278
q15	523	472	482	472
q16	480	430	417	417
q17	1774	1487	1540	1487
q18	7616	7299	7622	7299
q19	1663	1557	1506	1506
q20	2066	1829	1794	1794
q21	5482	5195	5187	5187
q22	1145	1108	1048	1048
Total cold run time: 66736 ms
Total hot run time: 54465 ms

doris-robot · 2024-08-26T10:03:16Z

TPC-DS: Total hot run time: 186595 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3b2f87cd3b48345dacc417d984aaa716aff3cf40, data reload: false

query1	902	376	364	364
query2	6449	2029	1826	1826
query3	6645	204	218	204
query4	26604	23212	23102	23102
query5	4148	497	482	482
query6	257	161	159	159
query7	4588	300	288	288
query8	265	211	199	199
query9	8566	2452	2459	2452
query10	436	285	258	258
query11	16761	14892	14860	14860
query12	151	101	99	99
query13	1619	393	371	371
query14	9795	7068	6921	6921
query15	248	171	174	171
query16	7944	459	482	459
query17	1579	580	591	580
query18	2032	301	311	301
query19	190	145	146	145
query20	117	112	110	110
query21	210	104	102	102
query22	4502	4296	4125	4125
query23	34071	33424	33485	33424
query24	11078	2825	2913	2825
query25	614	379	391	379
query26	1162	159	161	159
query27	2858	283	283	283
query28	7616	2023	2023	2023
query29	816	422	413	413
query30	305	157	151	151
query31	967	768	741	741
query32	98	59	60	59
query33	757	290	291	290
query34	975	491	503	491
query35	866	723	716	716
query36	1081	941	939	939
query37	159	85	80	80
query38	4032	3899	3845	3845
query39	1431	1385	1389	1385
query40	205	118	118	118
query41	47	48	45	45
query42	114	102	96	96
query43	506	472	457	457
query44	1212	742	754	742
query45	199	167	171	167
query46	1106	742	752	742
query47	1879	1777	1805	1777
query48	362	311	298	298
query49	1088	425	437	425
query50	819	415	430	415
query51	7311	7097	7155	7097
query52	99	88	89	88
query53	253	180	181	180
query54	937	457	452	452
query55	78	81	76	76
query56	281	266	260	260
query57	1173	1093	1081	1081
query58	237	234	249	234
query59	2922	2895	2766	2766
query60	296	266	275	266
query61	116	99	97	97
query62	823	664	666	664
query63	219	181	183	181
query64	6437	2330	1768	1768
query65	3202	3112	3148	3112
query66	1365	344	349	344
query67	15593	15230	15278	15230
query68	4044	568	562	562
query69	409	275	277	275
query70	1108	1080	1114	1080
query71	337	287	278	278
query72	6418	2331	2068	2068
query73	749	321	334	321
query74	9022	8867	8741	8741
query75	3377	2666	2673	2666
query76	2276	1036	930	930
query77	483	315	329	315
query78	11047	9451	9080	9080
query79	1023	538	523	523
query80	739	514	519	514
query81	561	228	229	228
query82	239	146	147	146
query83	223	151	157	151
query84	229	76	79	76
query85	733	294	342	294
query86	303	286	297	286
query87	4436	4171	4215	4171
query88	3034	2356	2362	2356
query89	380	294	282	282
query90	1985	201	208	201
query91	126	102	100	100
query92	63	53	53	53
query93	1049	549	538	538
query94	884	304	315	304
query95	358	268	269	268
query96	600	273	273	273
query97	3211	3077	3129	3077
query98	210	207	206	206
query99	1575	1273	1280	1273
Total cold run time: 283004 ms
Total hot run time: 186595 ms

doris-robot · 2024-08-26T10:08:44Z

ClickBench: Total hot run time: 31.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3b2f87cd3b48345dacc417d984aaa716aff3cf40, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.06	0.05
query4	1.67	0.07	0.08
query5	0.51	0.50	0.50
query6	1.12	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.54	0.54	0.55
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.61	0.58	0.58
query14	0.76	0.80	0.77
query15	0.83	0.82	0.81
query16	0.38	0.38	0.39
query17	1.07	1.06	1.00
query18	0.21	0.20	0.21
query19	1.86	1.84	1.76
query20	0.01	0.00	0.01
query21	15.43	0.65	0.65
query22	4.45	6.16	2.33
query23	18.27	1.33	1.22
query24	2.06	0.23	0.22
query25	0.15	0.08	0.08
query26	0.26	0.18	0.17
query27	0.08	0.08	0.07
query28	13.32	1.03	1.00
query29	12.60	3.33	3.31
query30	0.24	0.06	0.06
query31	2.87	0.40	0.39
query32	3.25	0.47	0.47
query33	2.97	3.00	3.00
query34	16.87	4.41	4.50
query35	4.47	4.51	4.42
query36	0.66	0.46	0.50
query37	0.19	0.17	0.16
query38	0.17	0.15	0.16
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.09	0.04	0.04
query42	0.06	0.05	0.05
query43	0.04	0.05	0.04
Total cold run time: 109.56 s
Total hot run time: 31.11 s

joker-star-l · 2024-08-27T05:22:08Z

@superdiaodiao Hello, what 'P0 Regression (Doris Regression) — TeamCity build failed' means? I can not see the details.

superdiaodiao · 2024-08-27T06:27:56Z

@superdiaodiao Hello, what 'P0 Regression (Doris Regression) — TeamCity build failed' means? I can not see the details.

You can click the Details:

then click Login as guest, so that you can see the log.

regression-test/suites/nereids_function_p0/agg_function/agg.groovy

joker-star-l · 2024-08-27T08:20:35Z

run buildall

doris-robot · 2024-08-27T08:56:47Z

TPC-H: Total hot run time: 37709 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 598bfa16d37f64101e2b6a098307112f00859234, data reload: false

------ Round 1 ----------------------------------
q1	17626	4419	4290	4290
q2	2021	183	178	178
q3	11927	960	1132	960
q4	10517	675	722	675
q5	7756	2850	2769	2769
q6	225	133	133	133
q7	970	613	602	602
q8	9326	2045	2072	2045
q9	6932	6522	6549	6522
q10	6991	2208	2223	2208
q11	442	238	242	238
q12	388	216	220	216
q13	18741	3057	3027	3027
q14	286	243	239	239
q15	521	478	492	478
q16	507	398	384	384
q17	970	712	622	622
q18	7421	6884	6861	6861
q19	1388	956	960	956
q20	704	346	337	337
q21	4004	3063	2944	2944
q22	1140	1025	1059	1025
Total cold run time: 110803 ms
Total hot run time: 37709 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4337	4288	4381	4288
q2	378	278	280	278
q3	2851	2668	2653	2653
q4	1962	1668	1646	1646
q5	5585	5644	5674	5644
q6	252	139	130	130
q7	2222	1790	1841	1790
q8	3297	3386	3410	3386
q9	8808	8814	8867	8814
q10	3594	3387	3392	3387
q11	583	509	512	509
q12	832	637	657	637
q13	11701	3230	3214	3214
q14	326	294	303	294
q15	530	496	501	496
q16	507	453	432	432
q17	1843	1526	1524	1524
q18	8128	7873	7998	7873
q19	1733	1681	1595	1595
q20	2146	1886	1900	1886
q21	5834	5462	5486	5462
q22	1201	1056	1045	1045
Total cold run time: 68650 ms
Total hot run time: 56983 ms

doris-robot · 2024-08-27T09:08:51Z

TPC-DS: Total hot run time: 191420 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 598bfa16d37f64101e2b6a098307112f00859234, data reload: false

query1	1275	876	871	871
query2	6167	1923	1920	1920
query3	10716	4146	4073	4073
query4	59781	23907	23251	23251
query5	5403	511	504	504
query6	404	163	176	163
query7	5755	293	295	293
query8	283	210	206	206
query9	9262	2459	2406	2406
query10	488	284	256	256
query11	17929	15110	15219	15110
query12	160	110	104	104
query13	1571	407	387	387
query14	10531	7134	6957	6957
query15	227	174	170	170
query16	7396	537	496	496
query17	1098	557	553	553
query18	2029	306	287	287
query19	296	150	163	150
query20	123	112	109	109
query21	209	110	101	101
query22	4615	4624	4531	4531
query23	34323	33565	33327	33327
query24	5945	2848	2835	2835
query25	517	375	379	375
query26	679	158	156	156
query27	1785	280	277	277
query28	3742	2036	2037	2036
query29	667	399	400	399
query30	228	145	147	145
query31	953	780	763	763
query32	74	52	60	52
query33	448	281	280	280
query34	845	469	474	469
query35	812	708	718	708
query36	1045	932	944	932
query37	136	83	77	77
query38	3955	3908	3827	3827
query39	1447	1391	1378	1378
query40	190	117	113	113
query41	46	43	43	43
query42	109	97	96	96
query43	508	461	468	461
query44	1074	737	735	735
query45	198	170	168	168
query46	1061	731	739	731
query47	1906	1809	1831	1809
query48	371	298	290	290
query49	779	437	461	437
query50	798	411	418	411
query51	7177	7157	7071	7071
query52	100	89	86	86
query53	251	180	183	180
query54	564	465	457	457
query55	78	78	75	75
query56	282	274	260	260
query57	1184	1101	1107	1101
query58	252	230	258	230
query59	3000	2798	2599	2599
query60	300	289	279	279
query61	149	125	120	120
query62	760	655	659	655
query63	210	188	184	184
query64	3388	1757	1742	1742
query65	3160	3142	3133	3133
query66	670	330	336	330
query67	15440	15158	15256	15158
query68	2981	581	578	578
query69	407	315	292	292
query70	1109	1112	1040	1040
query71	343	271	286	271
query72	2571	2094	2085	2085
query73	700	325	324	324
query74	9130	8945	8809	8809
query75	3355	2693	2710	2693
query76	1343	1011	998	998
query77	526	319	314	314
query78	9587	9444	9031	9031
query79	1029	539	527	527
query80	689	503	493	493
query81	529	231	226	226
query82	286	140	135	135
query83	168	146	147	146
query84	267	79	73	73
query85	663	346	286	286
query86	308	279	299	279
query87	4388	4362	4144	4144
query88	3284	2356	2339	2339
query89	370	282	282	282
query90	1807	190	188	188
query91	122	95	98	95
query92	57	51	52	51
query93	1045	522	526	522
query94	604	297	302	297
query95	347	255	264	255
query96	582	269	280	269
query97	3206	3040	3059	3040
query98	219	211	195	195
query99	1719	1255	1279	1255
Total cold run time: 301065 ms
Total hot run time: 191420 ms

doris-robot · 2024-09-19T02:47:41Z

TeamCity be ut coverage result:
Function Coverage: 37.35% (9598/25696)
Line Coverage: 28.75% (79394/276132)
Region Coverage: 28.24% (41135/145665)
Branch Coverage: 24.85% (20953/84318)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ebe864db71c9348be6c10904512bb42e3b9d2e27_ebe864db71c9348be6c10904512bb42e3b9d2e27/report/index.html

doris-robot · 2024-09-19T02:52:10Z

TPC-H: Total hot run time: 41144 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ebe864db71c9348be6c10904512bb42e3b9d2e27, data reload: false

------ Round 1 ----------------------------------
q1	17599	7307	7257	7257
q2	2030	160	158	158
q3	10627	1061	1224	1061
q4	10563	746	723	723
q5	7754	3048	3064	3048
q6	233	151	152	151
q7	1015	636	610	610
q8	9417	2038	2007	2007
q9	6772	6361	6405	6361
q10	7048	2298	2294	2294
q11	449	246	253	246
q12	405	218	221	218
q13	17796	2967	2989	2967
q14	240	204	219	204
q15	587	542	519	519
q16	704	621	616	616
q17	975	832	816	816
q18	7201	6569	6744	6569
q19	1393	933	1034	933
q20	584	295	271	271
q21	3993	3145	3181	3145
q22	1112	996	970	970
Total cold run time: 108497 ms
Total hot run time: 41144 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7238	7235	7265	7235
q2	326	227	227	227
q3	2941	2928	3047	2928
q4	2023	1808	1819	1808
q5	5601	5597	5523	5523
q6	229	147	151	147
q7	2244	1825	1809	1809
q8	3296	3435	3425	3425
q9	8791	8825	8761	8761
q10	3529	3435	3471	3435
q11	602	485	475	475
q12	834	629	630	629
q13	10201	3219	3132	3132
q14	297	284	280	280
q15	576	533	519	519
q16	713	668	689	668
q17	1789	1586	1560	1560
q18	8146	7705	7673	7673
q19	1761	1495	1609	1495
q20	2111	1881	1904	1881
q21	5610	5263	5495	5263
q22	1165	1041	1018	1018
Total cold run time: 70023 ms
Total hot run time: 59891 ms

morrySnow · 2024-09-19T07:56:03Z

run cloud_p0

morrySnow · 2024-09-19T07:56:12Z

run performance

doris-robot · 2024-09-19T09:02:59Z

TPC-H: Total hot run time: 41731 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ebe864db71c9348be6c10904512bb42e3b9d2e27, data reload: false

------ Round 1 ----------------------------------
q1	17720	7410	7304	7304
q2	2046	168	155	155
q3	10698	1065	1327	1065
q4	10406	800	713	713
q5	7766	3125	3137	3125
q6	238	149	145	145
q7	1030	627	618	618
q8	9450	2083	2102	2083
q9	6810	6457	6421	6421
q10	7021	2282	2363	2282
q11	424	243	245	243
q12	412	222	217	217
q13	17817	2993	2973	2973
q14	246	206	215	206
q15	559	530	512	512
q16	654	643	609	609
q17	994	843	835	835
q18	7442	6806	6774	6774
q19	1413	997	995	995
q20	586	283	272	272
q21	4123	3215	3236	3215
q22	1085	1029	969	969
Total cold run time: 108940 ms
Total hot run time: 41731 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7266	7240	7234	7234
q2	320	228	226	226
q3	3041	2992	2979	2979
q4	2138	1918	1755	1755
q5	5622	5650	5674	5650
q6	241	146	142	142
q7	2221	1739	1774	1739
q8	3318	3465	3449	3449
q9	8748	8953	8725	8725
q10	3469	3461	3556	3461
q11	570	496	487	487
q12	848	600	612	600
q13	9287	3179	3170	3170
q14	296	297	273	273
q15	569	538	532	532
q16	721	679	657	657
q17	1838	1620	1625	1620
q18	8454	8070	7878	7878
q19	1752	1601	1536	1536
q20	2149	1894	1883	1883
q21	5525	5461	5659	5461
q22	1152	1048	1046	1046
Total cold run time: 69545 ms
Total hot run time: 60503 ms

doris-robot · 2024-09-19T09:15:28Z

TPC-DS: Total hot run time: 195343 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ebe864db71c9348be6c10904512bb42e3b9d2e27, data reload: false

query1	1257	900	853	853
query2	6355	2046	2044	2044
query3	10789	3864	3837	3837
query4	64370	27154	23799	23799
query5	5336	467	478	467
query6	417	168	166	166
query7	5758	295	304	295
query8	326	235	224	224
query9	9711	2652	2647	2647
query10	510	292	276	276
query11	18480	15135	15990	15135
query12	156	95	106	95
query13	1613	419	413	413
query14	10828	7094	7550	7094
query15	209	179	179	179
query16	7164	480	527	480
query17	1200	623	606	606
query18	1397	323	319	319
query19	294	153	159	153
query20	133	111	113	111
query21	213	105	107	105
query22	4604	4427	4717	4427
query23	34753	33657	33610	33610
query24	6048	2886	2888	2886
query25	503	394	408	394
query26	646	159	157	157
query27	1676	289	281	281
query28	4209	2461	2436	2436
query29	663	437	425	425
query30	230	154	157	154
query31	974	809	821	809
query32	74	53	62	53
query33	402	294	292	292
query34	909	480	495	480
query35	825	735	741	735
query36	1039	943	944	943
query37	146	87	89	87
query38	4011	3934	3957	3934
query39	1450	1442	1417	1417
query40	206	100	95	95
query41	51	48	50	48
query42	115	98	98	98
query43	518	483	498	483
query44	1157	820	799	799
query45	196	165	164	164
query46	1131	735	766	735
query47	1887	1791	1856	1791
query48	483	365	376	365
query49	731	414	414	414
query50	832	413	412	412
query51	7024	6980	6945	6945
query52	98	88	85	85
query53	248	183	181	181
query54	571	468	479	468
query55	74	79	77	77
query56	281	274	271	271
query57	1210	1084	1059	1059
query58	223	232	267	232
query59	3182	3079	3013	3013
query60	291	273	287	273
query61	147	105	101	101
query62	754	655	667	655
query63	213	183	178	178
query64	1605	644	623	623
query65	3263	3160	3181	3160
query66	675	300	298	298
query67	15740	15551	15616	15551
query68	4356	584	583	583
query69	453	301	305	301
query70	1144	1178	1155	1155
query71	337	263	271	263
query72	6457	4072	4058	4058
query73	759	323	326	323
query74	9631	8955	8999	8955
query75	3338	2689	2665	2665
query76	1814	887	906	887
query77	469	300	289	289
query78	10003	9257	9132	9132
query79	1150	546	535	535
query80	847	428	426	426
query81	518	241	240	240
query82	1337	141	138	138
query83	242	133	140	133
query84	281	81	81	81
query85	894	302	283	283
query86	333	301	298	298
query87	4481	4298	4392	4298
query88	2981	2353	2345	2345
query89	396	281	283	281
query90	1912	184	187	184
query91	181	142	137	137
query92	63	46	48	46
query93	1621	533	534	533
query94	748	291	296	291
query95	341	247	249	247
query96	624	275	275	275
query97	3272	3126	3100	3100
query98	216	206	190	190
query99	1773	1312	1279	1279
Total cold run time: 315556 ms
Total hot run time: 195343 ms

doris-robot · 2024-09-19T09:20:58Z

ClickBench: Total hot run time: 32.56 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ebe864db71c9348be6c10904512bb42e3b9d2e27, data reload: false

query1	0.05	0.04	0.04
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.66	0.10	0.10
query5	0.53	0.51	0.51
query6	1.14	0.73	0.72
query7	0.01	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.51	0.50
query10	0.56	0.58	0.55
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.60	0.58	0.58
query14	3.12	2.96	2.96
query15	0.90	0.82	0.82
query16	0.38	0.38	0.38
query17	1.00	1.06	1.03
query18	0.19	0.19	0.19
query19	1.95	1.89	1.97
query20	0.01	0.02	0.01
query21	15.35	0.59	0.58
query22	2.53	1.68	1.83
query23	17.31	0.95	0.92
query24	2.60	1.88	0.75
query25	0.21	0.14	0.04
query26	0.53	0.13	0.13
query27	0.05	0.04	0.04
query28	10.62	1.09	1.06
query29	12.54	3.25	3.28
query30	0.25	0.06	0.06
query31	2.87	0.40	0.38
query32	3.26	0.46	0.46
query33	2.99	3.06	2.99
query34	16.78	4.39	4.38
query35	4.44	4.47	4.46
query36	0.68	0.47	0.51
query37	0.08	0.06	0.05
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.15	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 106.76 s
Total hot run time: 32.56 s

# Versions - [x] dev - [x] 3.0 - [ ] 2.1 - [ ] 2.0 # Languages - [x] Chinese - [x] English # Function PR [linear_histogram](apache/doris#39546) in doris

joker-star-l · 2024-09-30T02:37:52Z

run buildall

doris-robot · 2024-09-30T04:29:58Z

TeamCity be ut coverage result:
Function Coverage: 37.35% (9645/25826)
Line Coverage: 28.73% (80002/278455)
Region Coverage: 28.19% (41412/146881)
Branch Coverage: 24.80% (21090/85026)
Coverage Report: http://coverage.selectdb-in.cc/coverage/91ed344e38af30035b9c3aa8885b6a6ec67dfe9c_91ed344e38af30035b9c3aa8885b6a6ec67dfe9c/report/index.html

joker-star-l · 2024-09-30T09:38:50Z

run buildall

doris-robot · 2024-09-30T10:03:11Z

TeamCity be ut coverage result:
Function Coverage: 37.35% (9647/25826)
Line Coverage: 28.75% (80054/278455)
Region Coverage: 28.21% (41433/146883)
Branch Coverage: 24.81% (21099/85030)
Coverage Report: http://coverage.selectdb-in.cc/coverage/30110b521187e0ae67ae565d657894972bc8b31e_30110b521187e0ae67ae565d657894972bc8b31e/report/index.html

joker-star-l · 2024-09-30T14:01:37Z

run cloud_p0

github-actions · 2024-10-10T04:16:59Z

PR approved by at least one committer and no changes requested.

xiaokang

LGTM

add aggregate function: `linear_histogram(expr, DOUBLE interval[, DOUBLE offset)` The linear_histogram function is used to describe the distribution of the data, It uses an "equal width" bucking strategy, and divides the data into buckets according to the value of the data.

xiaokang reviewed Aug 19, 2024

View reviewed changes

yiguolei reviewed Aug 19, 2024

View reviewed changes

be/src/vec/aggregate_functions/aggregate_function_linear_histogram.cpp Show resolved Hide resolved

superdiaodiao reviewed Aug 19, 2024

View reviewed changes

joker-star-l changed the title ~~[feature] linear_histogram~~ [feature](agg func) linear_histogram Aug 22, 2024

github-actions bot reviewed Aug 22, 2024

View reviewed changes

be/src/vec/aggregate_functions/aggregate_function_linear_histogram.h Show resolved Hide resolved

be/test/vec/aggregate_functions/agg_linear_histogram_test.cpp Show resolved Hide resolved

be/test/vec/aggregate_functions/agg_linear_histogram_test.cpp Show resolved Hide resolved

joker-star-l force-pushed the dev branch from c23f9a3 to e14b50b Compare August 25, 2024 18:53

github-actions bot reviewed Aug 25, 2024

View reviewed changes

be/test/vec/aggregate_functions/agg_linear_histogram_test.cpp Show resolved Hide resolved

joker-star-l changed the title ~~[feature](agg func) linear_histogram~~ [feature](agg-func) linear_histogram Aug 25, 2024

github-actions bot reviewed Aug 25, 2024

View reviewed changes

be/test/vec/aggregate_functions/agg_linear_histogram_test.cpp Show resolved Hide resolved

joker-star-l commented Aug 26, 2024

View reviewed changes

regression-test/data/nereids_function_p0/agg_function/agg.out Outdated Show resolved Hide resolved

superdiaodiao reviewed Aug 27, 2024

View reviewed changes

regression-test/suites/nereids_function_p0/agg_function/agg.groovy Outdated Show resolved Hide resolved

Merge branch 'master' into dev

91ed344

joker-star-l dismissed stale reviews from morrySnow, zhangstar333, and xiaokang via 91ed344 September 30, 2024 02:14

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 30, 2024

fix: typo

30110b5

joker-star-l requested review from morrySnow, zhangstar333 and xiaokang October 1, 2024 12:19

morrySnow approved these changes Oct 10, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 10, 2024

xiaokang approved these changes Oct 14, 2024

View reviewed changes

xiaokang merged commit 6d6c441 into apache:master Oct 14, 2024
23 of 28 checks passed

xiaokang added dev/2.1.x dev/3.0.x labels Oct 14, 2024

[feature](agg-func) linear_histogram #39546

[feature](agg-func) linear_histogram #39546

Conversation

joker-star-l commented Aug 19, 2024 • edited Loading

Proposed changes

LINEAR_HISTOGRAM

Description

Example

Doc PR

doris-robot commented Aug 19, 2024

joker-star-l commented Aug 19, 2024

xiaokang commented Aug 19, 2024

doris-robot commented Aug 19, 2024

xiaokang left a comment

Choose a reason for hiding this comment

doris-robot commented Aug 19, 2024

doris-robot commented Aug 19, 2024

superdiaodiao left a comment • edited Loading

Choose a reason for hiding this comment

joker-star-l commented Aug 21, 2024

superdiaodiao commented Aug 21, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

joker-star-l commented Aug 22, 2024

joker-star-l commented Aug 22, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

joker-star-l commented Aug 26, 2024

doris-robot commented Aug 26, 2024

doris-robot commented Aug 26, 2024

doris-robot commented Aug 26, 2024

joker-star-l commented Aug 27, 2024

superdiaodiao commented Aug 27, 2024 • edited Loading

joker-star-l commented Aug 27, 2024

doris-robot commented Aug 27, 2024

doris-robot commented Aug 27, 2024

doris-robot commented Sep 19, 2024

doris-robot commented Sep 19, 2024

morrySnow commented Sep 19, 2024

morrySnow commented Sep 19, 2024

doris-robot commented Sep 19, 2024

doris-robot commented Sep 19, 2024

doris-robot commented Sep 19, 2024

joker-star-l commented Sep 30, 2024

doris-robot commented Sep 30, 2024

joker-star-l commented Sep 30, 2024

doris-robot commented Sep 30, 2024

joker-star-l commented Sep 30, 2024

github-actions bot commented Oct 10, 2024

xiaokang left a comment

Choose a reason for hiding this comment

joker-star-l commented Aug 19, 2024 •

edited

Loading

superdiaodiao left a comment •

edited

Loading

superdiaodiao commented Aug 27, 2024 •

edited

Loading