Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](mtmv) Fix written by mv successfully but not get mv lock. If use mv, data maybe wrong #40173

Merged
merged 5 commits into from
Oct 11, 2024
Merged

[fix](mtmv) Fix written by mv successfully but not get mv lock. If use mv, data maybe wrong #40173

merged 5 commits into from
Oct 11, 2024

Conversation

seawinde
Copy link
Contributor

Proposed changes

When refresh partition mv, meanwhile, the query that can be written successfully by the same mv, if we run the query, the result maybe wrong.
this pr fix this.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@seawinde
Copy link
Contributor Author

run buildall

924060929
924060929 previously approved these changes Aug 30, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 30, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 39919 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8bf2b6080a2268e2c1b0737a1d3578afe31835d1, data reload: false

------ Round 1 ----------------------------------
q1	17627	4890	4806	4806
q2	2005	183	170	170
q3	11795	963	1183	963
q4	10516	720	781	720
q5	7792	3019	3026	3019
q6	248	137	140	137
q7	1052	605	593	593
q8	9318	2175	2162	2162
q9	7557	6926	6981	6926
q10	7074	2327	2282	2282
q11	443	252	242	242
q12	437	224	225	224
q13	18159	3045	3041	3041
q14	287	230	247	230
q15	521	486	513	486
q16	591	501	519	501
q17	1036	743	758	743
q18	7399	6994	7014	6994
q19	1413	1107	1108	1107
q20	686	333	335	333
q21	4466	3227	3305	3227
q22	1131	1046	1013	1013
Total cold run time: 111553 ms
Total hot run time: 39919 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4841	4787	4754	4754
q2	414	276	262	262
q3	3048	2769	2752	2752
q4	2107	1776	1775	1775
q5	5825	5976	5784	5784
q6	253	138	129	129
q7	2293	1810	1822	1810
q8	3528	3514	3730	3514
q9	8950	8885	9023	8885
q10	3829	3515	3474	3474
q11	630	544	509	509
q12	888	686	622	622
q13	13327	3231	3204	3204
q14	326	290	295	290
q15	547	506	489	489
q16	642	588	588	588
q17	1993	1632	1645	1632
q18	8103	7971	7997	7971
q19	1897	1826	1817	1817
q20	2243	1886	1948	1886
q21	5829	5724	5733	5724
q22	1136	1040	1018	1018
Total cold run time: 72649 ms
Total hot run time: 58889 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194401 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8bf2b6080a2268e2c1b0737a1d3578afe31835d1, data reload: false

query1	1254	895	861	861
query2	6370	1950	1943	1943
query3	10598	4069	3953	3953
query4	59875	26204	23434	23434
query5	5314	522	514	514
query6	401	160	158	158
query7	5767	293	293	293
query8	291	215	223	215
query9	8974	2480	2467	2467
query10	494	277	265	265
query11	16134	15017	15454	15017
query12	167	103	105	103
query13	1549	402	400	400
query14	11213	7033	7201	7033
query15	233	177	185	177
query16	7588	495	464	464
query17	1233	609	574	574
query18	2059	302	292	292
query19	291	142	143	142
query20	125	110	111	110
query21	207	108	105	105
query22	4716	4813	4881	4813
query23	34278	33291	33434	33291
query24	5957	2877	2930	2877
query25	545	408	416	408
query26	692	163	166	163
query27	1770	287	282	282
query28	3690	2100	2077	2077
query29	692	402	407	402
query30	232	154	159	154
query31	949	774	777	774
query32	89	53	61	53
query33	442	309	288	288
query34	870	480	488	480
query35	843	719	716	716
query36	1078	934	907	907
query37	141	88	90	88
query38	4076	3831	3843	3831
query39	1478	1401	1381	1381
query40	201	116	118	116
query41	47	46	46	46
query42	120	102	100	100
query43	511	477	474	474
query44	1073	743	753	743
query45	214	169	168	168
query46	1079	779	747	747
query47	1886	1832	1811	1811
query48	378	289	295	289
query49	755	431	446	431
query50	814	424	464	424
query51	7193	7057	7089	7057
query52	99	86	85	85
query53	251	181	182	181
query54	562	461	447	447
query55	79	85	77	77
query56	275	254	255	254
query57	1182	1054	1083	1054
query58	234	232	232	232
query59	3063	2894	2871	2871
query60	322	275	293	275
query61	105	104	103	103
query62	773	665	646	646
query63	219	194	187	187
query64	2813	738	685	685
query65	3229	3196	3212	3196
query66	668	339	334	334
query67	15493	15414	15423	15414
query68	2964	585	578	578
query69	413	286	290	286
query70	1172	1196	1089	1089
query71	360	288	278	278
query72	6186	4049	4015	4015
query73	748	324	328	324
query74	9075	8961	8812	8812
query75	3389	2696	2705	2696
query76	1419	1019	1044	1019
query77	559	321	326	321
query78	10109	9705	10257	9705
query79	1130	536	533	533
query80	754	524	514	514
query81	540	236	235	235
query82	289	151	143	143
query83	173	155	151	151
query84	258	82	75	75
query85	722	303	290	290
query86	305	300	300	300
query87	4521	4216	4245	4216
query88	3017	2304	2297	2297
query89	384	295	290	290
query90	1696	198	202	198
query91	126	152	104	104
query92	61	53	56	53
query93	1057	526	528	526
query94	545	290	309	290
query95	352	266	266	266
query96	586	264	265	264
query97	3197	3109	3095	3095
query98	226	215	195	195
query99	1612	1306	1291	1291
Total cold run time: 304258 ms
Total hot run time: 194401 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.89 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8bf2b6080a2268e2c1b0737a1d3578afe31835d1, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.05	0.04
query3	0.22	0.05	0.06
query4	1.68	0.09	0.08
query5	0.51	0.50	0.48
query6	1.14	0.74	0.73
query7	0.02	0.01	0.01
query8	0.06	0.04	0.04
query9	0.54	0.48	0.50
query10	0.55	0.53	0.53
query11	0.16	0.12	0.11
query12	0.16	0.12	0.12
query13	0.61	0.59	0.58
query14	2.03	2.08	2.06
query15	0.90	0.82	0.81
query16	0.36	0.37	0.36
query17	0.99	0.98	1.02
query18	0.22	0.21	0.21
query19	1.80	1.78	1.75
query20	0.01	0.01	0.01
query21	15.38	0.65	0.65
query22	4.06	7.30	1.78
query23	18.28	1.30	1.27
query24	2.06	0.27	0.22
query25	0.15	0.08	0.08
query26	0.26	0.18	0.19
query27	0.07	0.07	0.08
query28	13.19	1.02	1.01
query29	12.64	3.34	3.33
query30	0.24	0.06	0.05
query31	2.88	0.39	0.40
query32	3.29	0.47	0.48
query33	2.95	3.03	3.07
query34	17.09	4.40	4.39
query35	4.48	4.41	4.41
query36	0.65	0.48	0.48
query37	0.20	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.03
query40	0.16	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.05	0.05
Total cold run time: 110.52 s
Total hot run time: 31.89 s

@seawinde seawinde force-pushed the add_mv_read_lock_when_mv_rewrite branch from 8bf2b60 to ec8ff95 Compare September 4, 2024 02:05
@seawinde
Copy link
Contributor Author

seawinde commented Sep 4, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38140 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ec8ff95f4bcd3c1e8624ac58cecbf567692f6577, data reload: false

------ Round 1 ----------------------------------
q1	17607	4692	4316	4316
q2	2023	183	175	175
q3	11643	1019	1111	1019
q4	10510	810	732	732
q5	7767	2898	2805	2805
q6	231	141	137	137
q7	970	628	594	594
q8	9324	2112	2126	2112
q9	7235	6541	6591	6541
q10	7008	2185	2263	2185
q11	481	237	238	237
q12	398	225	223	223
q13	17773	3041	3045	3041
q14	294	248	237	237
q15	531	487	487	487
q16	599	523	508	508
q17	1003	702	816	702
q18	7394	6823	6936	6823
q19	1395	954	1069	954
q20	684	341	339	339
q21	3933	3024	2960	2960
q22	1121	1027	1013	1013
Total cold run time: 109924 ms
Total hot run time: 38140 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4439	4352	4333	4333
q2	377	276	278	276
q3	2908	2646	2695	2646
q4	1980	1681	1641	1641
q5	5706	5714	5763	5714
q6	241	142	143	142
q7	2243	1847	1824	1824
q8	3318	3476	3519	3476
q9	8951	8840	8858	8840
q10	3628	3432	3348	3348
q11	611	503	529	503
q12	816	668	635	635
q13	16255	3346	3309	3309
q14	311	302	278	278
q15	565	491	499	491
q16	645	580	592	580
q17	1868	1555	1527	1527
q18	8343	7833	8043	7833
q19	1760	1530	1524	1524
q20	2140	1940	1908	1908
q21	5710	5454	5434	5434
q22	1105	1103	1079	1079
Total cold run time: 73920 ms
Total hot run time: 57341 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192508 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ec8ff95f4bcd3c1e8624ac58cecbf567692f6577, data reload: false

query1	1242	897	861	861
query2	6400	1982	1907	1907
query3	10617	3905	3939	3905
query4	59719	25669	23109	23109
query5	5403	501	482	482
query6	404	160	172	160
query7	5897	290	280	280
query8	293	199	200	199
query9	8948	2461	2452	2452
query10	488	275	264	264
query11	16703	14925	15264	14925
query12	167	110	116	110
query13	1611	390	374	374
query14	11252	7437	7206	7206
query15	269	178	177	177
query16	7578	476	490	476
query17	1175	581	604	581
query18	2061	304	308	304
query19	311	155	169	155
query20	135	111	111	111
query21	212	102	107	102
query22	4480	4733	4813	4733
query23	34107	33572	33575	33572
query24	5903	2851	2873	2851
query25	556	414	396	396
query26	694	160	155	155
query27	1781	279	291	279
query28	3628	2042	2027	2027
query29	729	424	433	424
query30	236	158	160	158
query31	933	778	786	778
query32	85	52	55	52
query33	471	291	291	291
query34	866	484	488	484
query35	851	714	702	702
query36	1047	927	935	927
query37	154	92	95	92
query38	3994	3854	3922	3854
query39	1459	1404	1377	1377
query40	203	117	116	116
query41	48	52	45	45
query42	113	101	98	98
query43	515	464	468	464
query44	1108	736	727	727
query45	206	168	168	168
query46	1091	741	730	730
query47	1894	1785	1807	1785
query48	379	302	307	302
query49	768	450	451	450
query50	814	422	415	415
query51	7008	6904	6703	6703
query52	94	87	85	85
query53	251	181	178	178
query54	564	460	459	459
query55	78	81	78	78
query56	308	267	263	263
query57	1225	1078	1087	1078
query58	233	242	229	229
query59	2914	2856	2701	2701
query60	299	281	282	281
query61	120	119	118	118
query62	766	666	644	644
query63	214	187	182	182
query64	2927	792	666	666
query65	3199	3149	3157	3149
query66	623	334	337	334
query67	15469	15270	15237	15237
query68	4450	539	559	539
query69	419	285	274	274
query70	1138	1141	1101	1101
query71	337	276	277	276
query72	6630	3984	3983	3983
query73	764	331	332	331
query74	9243	9044	8863	8863
query75	3387	2668	2681	2668
query76	1836	1074	971	971
query77	613	313	340	313
query78	9928	9023	9960	9023
query79	1706	529	531	529
query80	944	514	504	504
query81	575	236	232	232
query82	304	144	147	144
query83	214	143	142	142
query84	264	73	76	73
query85	914	280	305	280
query86	424	291	292	291
query87	4440	4177	4272	4177
query88	3314	2340	2318	2318
query89	384	285	285	285
query90	1803	190	187	187
query91	123	100	96	96
query92	65	46	48	46
query93	1947	522	525	522
query94	772	299	284	284
query95	372	255	250	250
query96	605	265	272	265
query97	3223	3102	3075	3075
query98	219	214	209	209
query99	1534	1286	1280	1280
Total cold run time: 309181 ms
Total hot run time: 192508 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.1 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ec8ff95f4bcd3c1e8624ac58cecbf567692f6577, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.04	0.03
query3	0.23	0.06	0.05
query4	1.68	0.09	0.08
query5	0.51	0.50	0.51
query6	1.13	0.73	0.74
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.54	0.49	0.50
query10	0.55	0.55	0.53
query11	0.16	0.11	0.12
query12	0.15	0.12	0.13
query13	0.60	0.59	0.59
query14	2.07	2.04	2.07
query15	0.89	0.83	0.82
query16	0.39	0.37	0.37
query17	1.05	1.02	1.01
query18	0.21	0.21	0.20
query19	1.92	1.87	1.84
query20	0.02	0.01	0.01
query21	15.39	0.67	0.67
query22	4.16	7.41	1.97
query23	18.28	1.47	1.33
query24	2.15	0.21	0.21
query25	0.16	0.08	0.07
query26	0.27	0.19	0.18
query27	0.08	0.07	0.08
query28	13.25	1.01	0.99
query29	12.64	3.28	3.30
query30	0.24	0.06	0.06
query31	2.87	0.39	0.40
query32	3.26	0.48	0.47
query33	2.96	2.97	3.01
query34	17.06	4.38	4.42
query35	4.41	4.45	4.38
query36	0.66	0.48	0.48
query37	0.19	0.15	0.16
query38	0.16	0.14	0.15
query39	0.04	0.04	0.04
query40	0.16	0.14	0.13
query41	0.08	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.86 s
Total hot run time: 32.1 s

morrySnow
morrySnow reviewed Sep 4, 2024
View reviewed changes
...rc/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java Outdated
if (materializationContext instanceof AsyncMaterializationContext && !materializationContext.isSuccess()) {
// Need get mtmv read lock, to avoid the data of plan which use mv is wrong
cascadesContext.getStatementContext()
.addTableReadLock(((AsyncMaterializationContext) materializationContext).getMtmv());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should lock table before rewrite

@seawinde
Copy link
Contributor Author

seawinde commented Sep 4, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 4, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 38686 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fa6df865ec728ac1e5d449b81523abea492cc1bd, data reload: false

------ Round 1 ----------------------------------
q1	18028	5416	4435	4435
q2	2985	181	179	179
q3	11805	1144	1104	1104
q4	10671	760	729	729
q5	7765	2912	2910	2910
q6	235	140	144	140
q7	984	613	592	592
q8	9352	2062	2113	2062
q9	7020	6558	6592	6558
q10	7004	2281	2183	2183
q11	441	232	245	232
q12	411	227	223	223
q13	18946	3083	3140	3083
q14	290	238	251	238
q15	540	483	504	483
q16	580	511	517	511
q17	989	651	682	651
q18	7495	6922	6893	6893
q19	1405	1039	1099	1039
q20	677	331	353	331
q21	4057	3129	3067	3067
q22	1147	1053	1043	1043
Total cold run time: 112827 ms
Total hot run time: 38686 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4395	4400	4315	4315
q2	399	278	278	278
q3	2896	2732	2759	2732
q4	1990	1706	1701	1701
q5	5418	5417	5427	5417
q6	221	131	131	131
q7	2129	1756	1757	1756
q8	3220	3379	3464	3379
q9	8485	8442	8485	8442
q10	3477	3224	3249	3224
q11	597	491	498	491
q12	771	639	615	615
q13	12282	3106	3122	3106
q14	310	294	288	288
q15	531	478	473	473
q16	599	563	547	547
q17	1808	1522	1504	1504
q18	7814	7351	7402	7351
q19	1686	1590	1590	1590
q20	2069	1850	1844	1844
q21	5599	5299	5449	5299
q22	1160	1077	1052	1052
Total cold run time: 67856 ms
Total hot run time: 55535 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187502 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fa6df865ec728ac1e5d449b81523abea492cc1bd, data reload: false

query1	912	373	367	367
query2	6488	1926	1928	1926
query3	6656	210	215	210
query4	26831	23153	23025	23025
query5	4160	504	498	498
query6	254	166	165	165
query7	4591	294	298	294
query8	242	205	201	201
query9	8617	2503	2500	2500
query10	434	264	262	262
query11	17850	14979	15135	14979
query12	144	103	107	103
query13	1628	367	378	367
query14	10026	6435	6877	6435
query15	291	167	176	167
query16	7946	462	445	445
query17	1574	562	576	562
query18	2120	283	282	282
query19	197	137	144	137
query20	116	108	105	105
query21	211	103	103	103
query22	4526	4542	4389	4389
query23	34088	33425	33220	33220
query24	11133	2896	2838	2838
query25	630	378	390	378
query26	1219	150	155	150
query27	2341	281	283	281
query28	7245	2067	2045	2045
query29	823	415	425	415
query30	307	159	154	154
query31	997	781	796	781
query32	101	53	56	53
query33	773	309	272	272
query34	996	469	478	469
query35	882	723	716	716
query36	1076	947	919	919
query37	168	95	84	84
query38	4043	3981	3821	3821
query39	1455	1440	1401	1401
query40	204	117	111	111
query41	46	48	45	45
query42	122	96	100	96
query43	517	473	480	473
query44	1214	750	748	748
query45	197	171	171	171
query46	1094	764	743	743
query47	1877	1780	1819	1780
query48	369	303	301	301
query49	1088	423	424	423
query50	803	420	405	405
query51	6887	6905	6909	6905
query52	99	86	85	85
query53	253	186	184	184
query54	1014	457	466	457
query55	75	74	80	74
query56	286	264	246	246
query57	1213	1054	1075	1054
query58	252	224	222	222
query59	2941	2721	2929	2721
query60	290	264	265	264
query61	101	101	103	101
query62	835	646	674	646
query63	216	178	185	178
query64	4252	710	646	646
query65	3202	3155	3127	3127
query66	1424	336	354	336
query67	15748	15421	15441	15421
query68	3297	582	589	582
query69	402	275	283	275
query70	1167	1056	1083	1056
query71	334	277	260	260
query72	6473	4151	4185	4151
query73	733	324	334	324
query74	9190	8929	8826	8826
query75	3394	2702	2635	2635
query76	1961	995	1068	995
query77	484	321	325	321
query78	9812	9209	9108	9108
query79	1021	540	521	521
query80	733	525	603	525
query81	597	237	232	232
query82	249	140	147	140
query83	219	151	150	150
query84	223	75	78	75
query85	911	286	283	283
query86	323	294	298	294
query87	4527	4361	4325	4325
query88	3173	2378	2422	2378
query89	370	286	287	286
query90	1896	193	183	183
query91	122	102	101	101
query92	63	49	47	47
query93	1015	547	540	540
query94	954	298	286	286
query95	362	252	251	251
query96	582	264	269	264
query97	3260	3102	3101	3101
query98	227	221	208	208
query99	1657	1304	1297	1297
Total cold run time: 280441 ms
Total hot run time: 187502 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.12 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fa6df865ec728ac1e5d449b81523abea492cc1bd, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.09	0.09
query5	0.50	0.49	0.51
query6	1.12	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.55	0.47	0.49
query10	0.54	0.55	0.54
query11	0.15	0.11	0.12
query12	0.14	0.12	0.13
query13	0.60	0.58	0.58
query14	2.10	2.06	2.06
query15	0.91	0.81	0.82
query16	0.38	0.38	0.36
query17	1.06	1.05	0.99
query18	0.23	0.21	0.20
query19	1.85	1.74	1.75
query20	0.01	0.02	0.01
query21	15.40	0.66	0.67
query22	4.37	7.11	2.14
query23	18.28	1.40	1.23
query24	2.07	0.21	0.24
query25	0.15	0.09	0.08
query26	0.29	0.18	0.17
query27	0.08	0.07	0.08
query28	13.28	1.03	1.01
query29	12.59	3.26	3.27
query30	0.25	0.05	0.05
query31	2.87	0.40	0.39
query32	3.26	0.49	0.48
query33	2.94	3.02	2.97
query34	17.03	4.39	4.42
query35	4.49	4.49	4.47
query36	0.65	0.48	0.49
query37	0.18	0.15	0.15
query38	0.15	0.15	0.15
query39	0.05	0.04	0.03
query40	0.16	0.12	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.97 s
Total hot run time: 32.12 s

@seawinde seawinde force-pushed the add_mv_read_lock_when_mv_rewrite branch from fa6df86 to b204a0c Compare September 6, 2024 03:03
@seawinde
Copy link
Contributor Author

seawinde commented Sep 6, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37963 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b204a0c6de61ab098195e89e4c000ab6d96b69f9, data reload: false

------ Round 1 ----------------------------------
q1	18444	4616	4346	4346
q2	2042	194	184	184
q3	11585	966	1206	966
q4	10373	823	737	737
q5	7733	2935	2788	2788
q6	232	141	137	137
q7	960	628	591	591
q8	9307	2108	2099	2099
q9	6975	6609	6616	6609
q10	7021	2184	2208	2184
q11	451	234	242	234
q12	393	221	219	219
q13	17769	3093	3067	3067
q14	294	230	244	230
q15	551	496	476	476
q16	531	439	440	439
q17	993	732	741	732
q18	7538	6827	6779	6779
q19	1510	1107	1074	1074
q20	663	348	338	338
q21	3957	2730	2792	2730
q22	1145	1011	1004	1004
Total cold run time: 110467 ms
Total hot run time: 37963 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4427	4322	4293	4293
q2	386	269	275	269
q3	2912	2647	2678	2647
q4	1923	1623	1631	1623
q5	5637	5736	5793	5736
q6	238	142	135	135
q7	2203	1860	1846	1846
q8	3303	3487	3554	3487
q9	8851	8987	8817	8817
q10	3606	3411	3343	3343
q11	606	518	506	506
q12	849	696	657	657
q13	13829	3163	3317	3163
q14	327	295	298	295
q15	536	469	511	469
q16	529	509	487	487
q17	1866	1572	1535	1535
q18	8248	7784	7943	7784
q19	1769	1659	1548	1548
q20	2179	1896	1927	1896
q21	5732	5450	5326	5326
q22	1123	1055	1036	1036
Total cold run time: 71079 ms
Total hot run time: 56898 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192843 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b204a0c6de61ab098195e89e4c000ab6d96b69f9, data reload: false

query1	1265	895	889	889
query2	6418	1876	1976	1876
query3	10608	4037	4009	4009
query4	59663	26985	23133	23133
query5	5396	496	506	496
query6	407	172	157	157
query7	5800	297	290	290
query8	313	214	215	214
query9	8700	2470	2477	2470
query10	469	263	275	263
query11	17456	15008	15378	15008
query12	160	110	104	104
query13	1544	411	394	394
query14	11073	7560	7613	7560
query15	268	170	201	170
query16	7468	488	483	483
query17	1134	582	568	568
query18	1643	297	298	297
query19	286	150	151	150
query20	125	113	122	113
query21	213	107	102	102
query22	4724	4560	4578	4560
query23	34273	33623	33275	33275
query24	5950	2882	2844	2844
query25	545	409	403	403
query26	692	157	154	154
query27	1798	270	281	270
query28	3855	2045	2032	2032
query29	683	428	425	425
query30	239	153	155	153
query31	935	797	779	779
query32	75	55	59	55
query33	464	298	297	297
query34	888	479	478	478
query35	829	724	713	713
query36	1067	937	937	937
query37	140	90	91	90
query38	4130	3839	3890	3839
query39	1445	1399	1417	1399
query40	198	117	117	117
query41	48	51	45	45
query42	117	96	95	95
query43	504	463	463	463
query44	1092	753	754	753
query45	199	167	170	167
query46	1103	746	736	736
query47	1945	1795	1808	1795
query48	372	286	293	286
query49	770	432	440	432
query50	823	398	407	398
query51	7050	6860	6879	6860
query52	99	86	87	86
query53	249	176	182	176
query54	568	446	437	437
query55	76	72	72	72
query56	268	265	259	259
query57	1235	1090	1100	1090
query58	209	222	230	222
query59	2987	2747	2658	2658
query60	293	281	274	274
query61	105	104	100	100
query62	777	683	642	642
query63	218	187	184	184
query64	2892	674	665	665
query65	3251	3156	3147	3147
query66	687	342	373	342
query67	15449	15300	15190	15190
query68	2977	580	589	580
query69	412	270	284	270
query70	1149	1120	1115	1115
query71	357	275	274	274
query72	6250	4044	4147	4044
query73	758	326	322	322
query74	9054	8772	8876	8772
query75	3367	2669	2706	2669
query76	1559	1062	928	928
query77	565	327	319	319
query78	9582	9141	9054	9054
query79	1033	543	529	529
query80	701	513	500	500
query81	465	237	230	230
query82	238	142	136	136
query83	170	153	149	149
query84	251	130	75	75
query85	686	284	281	281
query86	302	291	303	291
query87	4449	4289	4372	4289
query88	3201	2315	2334	2315
query89	394	290	284	284
query90	2004	190	185	185
query91	124	100	103	100
query92	59	50	52	50
query93	1084	533	532	532
query94	736	264	301	264
query95	346	283	291	283
query96	592	265	266	265
query97	3169	3090	3131	3090
query98	211	208	209	208
query99	1482	1317	1300	1300
Total cold run time: 304452 ms
Total hot run time: 192843 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.11 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b204a0c6de61ab098195e89e4c000ab6d96b69f9, data reload: false

query1	0.05	0.04	0.03
query2	0.08	0.03	0.04
query3	0.23	0.04	0.05
query4	1.68	0.08	0.07
query5	0.51	0.50	0.51
query6	1.12	0.73	0.73
query7	0.03	0.01	0.01
query8	0.05	0.04	0.05
query9	0.54	0.49	0.48
query10	0.55	0.55	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.59
query14	1.41	1.45	1.41
query15	0.84	0.81	0.82
query16	0.36	0.35	0.35
query17	1.05	1.07	1.00
query18	0.21	0.20	0.19
query19	1.83	1.84	1.79
query20	0.01	0.01	0.01
query21	15.41	0.68	0.65
query22	3.84	7.75	1.83
query23	18.28	1.31	1.25
query24	2.05	0.28	0.21
query25	0.15	0.09	0.08
query26	0.27	0.18	0.18
query27	0.08	0.08	0.07
query28	13.21	1.02	1.00
query29	12.64	3.30	3.25
query30	0.24	0.05	0.05
query31	2.87	0.41	0.39
query32	3.28	0.47	0.47
query33	2.97	2.98	3.03
query34	17.00	4.38	4.39
query35	4.49	4.45	4.42
query36	0.66	0.46	0.50
query37	0.18	0.15	0.16
query38	0.16	0.14	0.14
query39	0.05	0.04	0.04
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.64 s
Total hot run time: 31.11 s

@seawinde seawinde force-pushed the add_mv_read_lock_when_mv_rewrite branch 2 times, most recently from 3a8ee8c to 2fb14d9 Compare September 27, 2024 06:45
@seawinde
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@seawinde seawinde force-pushed the add_mv_read_lock_when_mv_rewrite branch from c2ecf20 to 04c25e5 Compare October 11, 2024 09:04
@seawinde
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Oct 11, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 41277 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 04c25e56d7a47eec2a6ffc0be655e926812399cf, data reload: false

------ Round 1 ----------------------------------
q1	17596	7623	7313	7313
q2	2026	298	281	281
q3	11706	1114	1195	1114
q4	10569	757	730	730
q5	7751	2874	2901	2874
q6	241	152	152	152
q7	995	627	630	627
q8	9372	1913	2056	1913
q9	7008	6488	6466	6466
q10	7009	2352	2328	2328
q11	444	257	259	257
q12	416	226	230	226
q13	17775	3040	3042	3040
q14	238	213	226	213
q15	578	530	521	521
q16	662	595	569	569
q17	984	612	526	526
q18	7189	6690	6695	6690
q19	1352	1069	983	983
q20	500	210	199	199
q21	4039	3267	3350	3267
q22	1122	996	988	988
Total cold run time: 109572 ms
Total hot run time: 41277 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7324	7342	7339	7339
q2	327	229	235	229
q3	3109	2929	2957	2929
q4	2074	1821	1788	1788
q5	5766	5769	5739	5739
q6	229	148	151	148
q7	2277	1869	1858	1858
q8	3389	3554	3446	3446
q9	8990	8917	8899	8899
q10	3572	3556	3511	3511
q11	590	505	496	496
q12	867	656	649	649
q13	9273	3228	3242	3228
q14	319	274	273	273
q15	602	531	521	521
q16	681	649	657	649
q17	1832	1650	1704	1650
q18	8247	7653	7506	7506
q19	1726	1489	1513	1489
q20	2127	1872	1909	1872
q21	5555	5357	5518	5357
q22	1137	1042	1064	1042
Total cold run time: 70013 ms
Total hot run time: 60618 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191592 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 04c25e56d7a47eec2a6ffc0be655e926812399cf, data reload: false

query1	991	396	402	396
query2	6287	2130	2016	2016
query3	8692	196	205	196
query4	34055	23616	23480	23480
query5	3468	486	489	486
query6	265	162	166	162
query7	4201	316	319	316
query8	275	217	210	210
query9	9445	2680	2687	2680
query10	471	320	289	289
query11	17894	15212	15352	15212
query12	162	95	99	95
query13	1574	471	428	428
query14	9517	6984	7627	6984
query15	251	175	182	175
query16	8069	454	460	454
query17	1689	629	604	604
query18	2171	328	327	327
query19	380	163	191	163
query20	123	112	111	111
query21	211	106	116	106
query22	4765	4414	4317	4317
query23	35083	35185	33844	33844
query24	11135	2845	2844	2844
query25	538	408	404	404
query26	915	167	167	167
query27	2609	304	299	299
query28	7649	2435	2416	2416
query29	676	435	437	435
query30	273	155	157	155
query31	1021	803	775	775
query32	95	55	56	55
query33	752	310	297	297
query34	898	514	507	507
query35	853	746	723	723
query36	1094	975	962	962
query37	141	90	93	90
query38	4095	3925	3860	3860
query39	1504	1429	1426	1426
query40	253	103	101	101
query41	46	46	44	44
query42	121	98	97	97
query43	546	491	472	472
query44	1239	814	832	814
query45	196	165	172	165
query46	1176	739	733	733
query47	1920	1847	1830	1830
query48	448	351	347	347
query49	992	438	419	419
query50	821	427	420	420
query51	7098	6967	6916	6916
query52	105	89	94	89
query53	265	189	185	185
query54	1153	476	474	474
query55	82	78	79	78
query56	290	284	274	274
query57	1300	1164	1121	1121
query58	223	237	288	237
query59	3249	2962	3070	2962
query60	295	265	269	265
query61	110	101	100	100
query62	885	698	656	656
query63	226	199	189	189
query64	4882	625	605	605
query65	3247	3196	3226	3196
query66	1242	305	308	305
query67	15898	15633	15739	15633
query68	4328	576	582	576
query69	481	301	321	301
query70	1183	1040	1061	1040
query71	364	275	296	275
query72	6893	3914	3926	3914
query73	780	343	350	343
query74	10408	9144	9028	9028
query75	3391	2726	2676	2676
query76	2813	941	895	895
query77	592	299	312	299
query78	10421	9690	9573	9573
query79	3387	634	615	615
query80	2376	445	450	445
query81	598	234	241	234
query82	696	145	143	143
query83	300	135	134	134
query84	275	83	76	76
query85	1535	296	288	288
query86	410	298	291	291
query87	4613	4393	4268	4268
query88	4320	2376	2367	2367
query89	414	297	289	289
query90	1982	188	184	184
query91	138	110	109	109
query92	60	48	50	48
query93	2398	570	556	556
query94	1061	303	287	287
query95	355	251	257	251
query96	632	280	285	280
query97	3291	3162	3126	3126
query98	225	200	193	193
query99	1694	1307	1295	1295
Total cold run time: 305009 ms
Total hot run time: 191592 ms

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 11, 2024
@doris-robot
Copy link

ClickBench: Total hot run time: 32.79 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 04c25e56d7a47eec2a6ffc0be655e926812399cf, data reload: false

query1	0.04	0.04	0.02
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.49	0.50	0.51
query6	1.14	0.72	0.73
query7	0.02	0.02	0.02
query8	0.03	0.03	0.04
query9	0.56	0.50	0.51
query10	0.57	0.56	0.55
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.60	0.59	0.61
query14	2.71	2.74	2.70
query15	0.91	0.83	0.85
query16	0.39	0.39	0.37
query17	1.05	1.02	1.01
query18	0.20	0.20	0.20
query19	1.86	1.81	1.93
query20	0.01	0.01	0.01
query21	15.37	0.56	0.56
query22	2.22	2.19	1.65
query23	16.98	1.06	0.81
query24	2.99	1.27	1.67
query25	0.19	0.17	0.19
query26	0.50	0.13	0.14
query27	0.04	0.04	0.04
query28	9.96	1.10	1.07
query29	12.53	3.20	3.21
query30	0.24	0.06	0.06
query31	2.86	0.39	0.38
query32	3.26	0.46	0.46
query33	3.03	3.01	3.08
query34	16.88	4.46	4.53
query35	4.51	4.51	4.48
query36	0.66	0.47	0.48
query37	0.08	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.15	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.45 s
Total hot run time: 32.79 s

@morrySnow
Copy link
Contributor

run p0

@morrySnow morrySnow merged commit 0d07e3d into apache:master Oct 11, 2024
25 of 26 checks passed
@morrySnow morrySnow added the p0_w label Oct 17, 2024
seawinde added a commit to seawinde/doris that referenced this pull request Oct 21, 2024
@seawinde
[fix](mtmv) Fix written by mv successfully but not get mv lock. If us…
793ed8c 
…e mv, data maybe wrong (apache#40173)

When refresh partition mv, meanwhile, the query that can be written
successfully by the same mv, if we run the query, the result maybe
wrong.
this pr fix this.
@seawinde seawinde mentioned this pull request Oct 21, 2024
Merged
seawinde added a commit to seawinde/doris that referenced this pull request Oct 23, 2024
@seawinde
[fix](mtmv) Fix written by mv successfully but not get mv lock. If us…
c0a6804 
…e mv, data maybe wrong (apache#40173)

When refresh partition mv, meanwhile, the query that can be written
successfully by the same mv, if we run the query, the result maybe
wrong.
this pr fix this.
seawinde added a commit to seawinde/doris that referenced this pull request Oct 23, 2024
@seawinde
[fix](mtmv) Fix written by mv successfully but not get mv lock. If us…
f2bc4a3 
…e mv, data maybe wrong (apache#40173)

When refresh partition mv, meanwhile, the query that can be written
successfully by the same mv, if we run the query, the result maybe
wrong.
this pr fix this.
@seawinde seawinde mentioned this pull request Oct 23, 2024
Merged
yiguolei pushed a commit that referenced this pull request Oct 24, 2024
@seawinde
Pick some fix from master to 21(#41472) (#40106)(#40173) (#42212)
44e9368 
## Proposed changes

pr: #41472
commitId: 2745e04


pr: #40106
commitId: 0fdb1ee


pr: #40173
commitId: 0d07e3d
morrySnow pushed a commit that referenced this pull request Oct 24, 2024
@seawinde
[pick](mtmv) pick 4 PR from master #41472 #40106 #40173 #42206 (#42325)
a1690f0 
pick from master

pr: #41472
commitId: 2745e04

pr: #40106
commitId: 0fdb1ee

pr: #40173
commitId: 0d07e3d

pr: #42206
commitId: 2bcaa5b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morrySnow morrySnow morrySnow approved these changes

@zddr zddr zddr approved these changes

@924060929 924060929 924060929 left review comments

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged p0_w reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@seawinde @doris-robot @morrySnow @924060929 @zddr @yiguolei