Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](mow) merge delete bitmap when compaction #41451

Open
wants to merge 4 commits into
base: branch-2.0
Choose a base branch
from
Open

[opt](mow) merge delete bitmap when compaction #41451

wants to merge 4 commits into from
+51 −14

Conversation

liutang123
Copy link
Contributor

Proposed changes

Issue Number: close #xxx
There are some problem found in #41447.
Before find the real reason, merge the delete bitmaps.

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@liutang123 liutang123 force-pushed the merge-delete-bitmap-2.0 branch 2 times, most recently from 279851b to e7c233c Compare September 30, 2024 07:50
github-actions[bot]
github-actions bot reviewed Sep 30, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/task/engine_storage_migration_task.h Show resolved Hide resolved
github-actions[bot]
github-actions bot reviewed Sep 30, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/task/engine_storage_migration_task.h Show resolved Hide resolved
@liutang123
Copy link
Contributor Author

run buildall

@liutang123
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49000 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fab285c2aebca47f3b792faf06d0964c882f7288, data reload: false

------ Round 1 ----------------------------------
q1	18201	4392	4355	4355
q2	2085	156	152	152
q3	10436	1877	1868	1868
q4	10292	1244	1342	1244
q5	8770	3843	3880	3843
q6	223	123	122	122
q7	2034	1649	1576	1576
q8	9280	2718	2711	2711
q9	10313	9871	9846	9846
q10	8595	3562	3525	3525
q11	418	250	251	250
q12	462	296	299	296
q13	18333	3986	4018	3986
q14	364	330	317	317
q15	506	453	463	453
q16	531	472	450	450
q17	1140	960	930	930
q18	7150	6939	6744	6744
q19	1701	1539	1546	1539
q20	505	311	304	304
q21	4368	4135	4106	4106
q22	498	383	398	383
Total cold run time: 116205 ms
Total hot run time: 49000 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4296	4282	4311	4282
q2	319	224	220	220
q3	4195	4114	4127	4114
q4	2752	2744	2736	2736
q5	7165	7025	7155	7025
q6	238	116	118	116
q7	3248	2837	2851	2837
q8	4330	4428	4435	4428
q9	13728	13685	13643	13643
q10	4231	4282	4241	4241
q11	733	667	672	667
q12	1017	865	858	858
q13	6541	3754	3748	3748
q14	455	430	414	414
q15	506	459	447	447
q16	643	570	571	570
q17	3836	3817	3838	3817
q18	8686	8691	8797	8691
q19	1705	1657	1656	1656
q20	2352	2120	2106	2106
q21	8439	8505	8477	8477
q22	1034	919	967	919
Total cold run time: 80449 ms
Total hot run time: 76012 ms

@doris-robot
Copy link

TPC-H: Total hot run time: 49217 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 853922f1b4e3320ba717c3d7403d3b1efc16dee9, data reload: false

------ Round 1 ----------------------------------
q1	18196	4432	4391	4391
q2	2066	150	148	148
q3	10409	1978	1975	1975
q4	10199	1249	1337	1249
q5	8607	3926	3964	3926
q6	234	123	128	123
q7	2042	1614	1600	1600
q8	9325	2755	2724	2724
q9	10261	9974	9746	9746
q10	8630	3533	3550	3533
q11	417	247	248	247
q12	468	295	291	291
q13	18339	4008	4012	4008
q14	359	321	335	321
q15	500	457	456	456
q16	560	460	463	460
q17	1144	973	940	940
q18	7215	6768	6860	6768
q19	1684	1597	1516	1516
q20	542	302	320	302
q21	4421	4210	4107	4107
q22	500	386	394	386
Total cold run time: 116118 ms
Total hot run time: 49217 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4298	4285	4290	4285
q2	324	220	222	220
q3	4196	4142	4118	4118
q4	2781	2744	2745	2744
q5	7167	7130	7094	7094
q6	240	117	117	117
q7	3189	2801	2874	2801
q8	4372	4550	4497	4497
q9	13724	13621	13569	13569
q10	4225	4255	4274	4255
q11	747	701	682	682
q12	1033	864	866	864
q13	6825	3750	3745	3745
q14	456	422	412	412
q15	496	468	453	453
q16	647	603	593	593
q17	3855	3895	3909	3895
q18	8795	8714	8735	8714
q19	1732	1681	1663	1663
q20	2394	2114	2126	2114
q21	8600	8517	8602	8517
q22	1065	940	963	940
Total cold run time: 81161 ms
Total hot run time: 76292 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.81% (8147/21546)
Line Coverage: 29.55% (67071/226982)
Region Coverage: 29.02% (34575/119135)
Branch Coverage: 24.94% (17816/71448)
Coverage Report: http://coverage.selectdb-in.cc/coverage/853922f1b4e3320ba717c3d7403d3b1efc16dee9_853922f1b4e3320ba717c3d7403d3b1efc16dee9/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 213169 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 853922f1b4e3320ba717c3d7403d3b1efc16dee9, data reload: false

query1	954	390	435	390
query2	6566	2304	2262	2262
query3	6921	202	203	202
query4	23095	21963	21919	21919
query5	19723	6551	6509	6509
query6	274	221	247	221
query7	4349	294	303	294
query8	244	260	248	248
query9	3064	2623	2562	2562
query10	459	305	300	300
query11	15856	15154	15053	15053
query12	125	76	76	76
query13	1035	444	440	440
query14	17160	13433	13624	13433
query15	387	212	230	212
query16	6476	279	262	262
query17	1805	906	892	892
query18	883	308	310	308
query19	214	161	164	161
query20	101	100	95	95
query21	182	93	93	93
query22	5242	5122	5017	5017
query23	34527	33616	33413	33413
query24	7900	6310	6262	6262
query25	513	435	430	430
query26	1263	159	167	159
query27	2456	299	288	288
query28	6060	2287	2259	2259
query29	2983	2844	2587	2587
query30	242	168	162	162
query31	955	773	747	747
query32	68	64	60	60
query33	456	262	275	262
query34	863	474	477	474
query35	1129	888	950	888
query36	1480	1161	1162	1161
query37	179	59	62	59
query38	3083	2884	2872	2872
query39	1376	1324	1321	1321
query40	309	94	97	94
query41	39	36	36	36
query42	79	86	93	86
query43	639	608	668	608
query44	1175	723	728	723
query45	247	229	229	229
query46	1237	967	955	955
query47	1851	1661	1808	1661
query48	506	431	407	407
query49	652	368	374	368
query50	847	635	588	588
query51	4794	4675	4682	4675
query52	101	89	87	87
query53	240	190	197	190
query54	2677	2471	2513	2471
query55	104	80	85	80
query56	232	181	207	181
query57	1353	1238	1169	1169
query58	218	207	210	207
query59	3523	3341	3364	3341
query60	228	202	202	202
query61	94	92	92	92
query62	804	483	430	430
query63	203	175	178	175
query64	3483	1591	1492	1492
query65	3650	3569	3554	3554
query66	773	441	425	425
query67	16442	15923	15404	15404
query68	10167	671	643	643
query69	491	271	260	260
query70	1576	1490	1593	1490
query71	409	309	320	309
query72	6786	4891	4776	4776
query73	763	339	326	326
query74	6246	5818	5864	5818
query75	5401	3686	3659	3659
query76	5829	1100	1170	1100
query77	1015	265	251	251
query78	12509	11824	11549	11549
query79	8159	645	658	645
query80	1537	389	382	382
query81	484	240	238	238
query82	1668	102	95	95
query83	171	129	133	129
query84	255	69	72	69
query85	884	319	316	316
query86	351	289	283	283
query87	3226	3012	3042	3012
query88	4653	2289	2289	2289
query89	479	303	320	303
query90	1965	217	218	217
query91	155	124	124	124
query92	58	50	53	50
query93	6916	593	557	557
query94	707	209	210	209
query95	2004	2011	1987	1987
query96	649	331	328	328
query97	6453	6456	6482	6456
query98	221	213	204	204
query99	2952	814	812	812
Total cold run time: 322287 ms
Total hot run time: 213169 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.72 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 853922f1b4e3320ba717c3d7403d3b1efc16dee9, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.03	0.02
query3	0.25	0.05	0.05
query4	1.78	0.06	0.07
query5	0.52	0.52	0.52
query6	1.23	0.62	0.61
query7	0.01	0.01	0.01
query8	0.04	0.03	0.02
query9	0.52	0.51	0.49
query10	0.53	0.56	0.54
query11	0.12	0.09	0.08
query12	0.11	0.09	0.09
query13	0.63	0.60	0.61
query14	0.79	0.79	0.80
query15	0.79	0.76	0.76
query16	0.35	0.36	0.36
query17	1.00	1.00	1.00
query18	0.21	0.26	0.25
query19	1.93	1.85	1.85
query20	0.02	0.01	0.01
query21	15.46	0.55	0.57
query22	2.02	2.37	1.82
query23	17.51	1.01	1.01
query24	5.67	1.62	2.22
query25	0.36	0.11	0.05
query26	0.78	0.15	0.15
query27	0.04	0.04	0.03
query28	5.80	0.74	0.71
query29	12.61	2.35	2.32
query30	0.57	0.52	0.54
query31	2.84	0.39	0.38
query32	3.38	0.50	0.49
query33	3.08	3.09	3.11
query34	15.24	4.76	4.77
query35	4.85	4.85	4.84
query36	1.07	1.00	1.02
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 102.6 s
Total hot run time: 31.72 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 853922f1b4e3320ba717c3d7403d3b1efc16dee9 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions GitHub Actions GitHub Actions left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@liutang123 @doris-robot