Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](Export) fix dead lock of parallel Exporting #43097

Merged
merged 1 commit into from
Nov 2, 2024

Conversation

BePPPower
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #xxx

bp: #43083

Check List (For Committer)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No colde files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.
  • Release note

    None

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@BePPPower
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the area/load Issues or PRs related to all kinds of load label Nov 1, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 49032 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7a04d53352df5ffc455ae9fec0f827ebd500c020, data reload: false

------ Round 1 ----------------------------------
q1	17719	4382	4320	4320
q2	2101	153	151	151
q3	10259	1934	1939	1934
q4	10376	1211	1314	1211
q5	8753	3849	3947	3849
q6	235	122	123	122
q7	2045	1610	1588	1588
q8	9271	2726	2693	2693
q9	10481	9999	9924	9924
q10	8661	3551	3510	3510
q11	417	237	245	237
q12	460	297	300	297
q13	18363	3970	4035	3970
q14	360	316	333	316
q15	503	466	475	466
q16	540	449	462	449
q17	1123	943	927	927
q18	7186	6752	6886	6752
q19	1661	1554	1549	1549
q20	547	312	300	300
q21	4384	4098	4080	4080
q22	489	395	387	387
Total cold run time: 115934 ms
Total hot run time: 49032 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4338	4287	4296	4287
q2	322	227	225	225
q3	4198	4146	4142	4142
q4	2761	2753	2748	2748
q5	7238	7159	7093	7093
q6	235	123	118	118
q7	3206	2871	2859	2859
q8	4364	4449	4461	4449
q9	14176	13853	14009	13853
q10	4213	4226	4255	4226
q11	736	672	709	672
q12	1019	866	860	860
q13	6722	3736	3736	3736
q14	448	429	418	418
q15	486	457	456	456
q16	630	594	592	592
q17	3773	3830	3826	3826
q18	8696	8733	8694	8694
q19	1724	1638	1646	1638
q20	2398	2161	2138	2138
q21	8631	8512	8481	8481
q22	1032	956	925	925
Total cold run time: 81346 ms
Total hot run time: 76436 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 212891 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7a04d53352df5ffc455ae9fec0f827ebd500c020, data reload: false

query1	929	427	382	382
query2	6531	2292	2257	2257
query3	6916	198	200	198
query4	23123	21884	21404	21404
query5	19725	6496	6576	6496
query6	280	224	239	224
query7	4328	297	304	297
query8	241	254	239	239
query9	3079	2652	2592	2592
query10	474	317	301	301
query11	15723	15289	15200	15200
query12	133	75	73	73
query13	1043	435	449	435
query14	17318	13566	13319	13319
query15	395	214	241	214
query16	6491	274	270	270
query17	1924	950	913	913
query18	900	328	321	321
query19	214	154	148	148
query20	99	101	102	101
query21	186	99	95	95
query22	5227	4996	4929	4929
query23	34317	33753	33538	33538
query24	7716	6287	6301	6287
query25	515	449	427	427
query26	1262	167	167	167
query27	2286	300	300	300
query28	6031	2241	2192	2192
query29	2819	2763	2866	2763
query30	244	170	173	170
query31	968	733	740	733
query32	72	64	60	60
query33	453	268	271	268
query34	842	496	500	496
query35	1117	905	927	905
query36	1258	1179	1468	1179
query37	95	60	61	60
query38	3093	2914	2968	2914
query39	1389	1332	1344	1332
query40	315	113	96	96
query41	42	39	39	39
query42	85	85	81	81
query43	615	642	608	608
query44	1157	724	725	724
query45	249	229	227	227
query46	1230	962	959	959
query47	1797	1801	1817	1801
query48	524	413	405	405
query49	649	388	369	369
query50	866	617	599	599
query51	4757	4671	4678	4671
query52	88	87	75	75
query53	233	182	194	182
query54	2650	2457	2510	2457
query55	94	89	90	89
query56	241	224	198	198
query57	1198	1103	1123	1103
query58	220	207	210	207
query59	3633	3288	3270	3270
query60	223	202	205	202
query61	99	94	107	94
query62	889	498	455	455
query63	203	182	187	182
query64	3577	1618	1518	1518
query65	3669	3539	3563	3539
query66	761	418	413	413
query67	17673	17695	15244	15244
query68	9010	645	637	637
query69	499	281	273	273
query70	1622	1623	1453	1453
query71	394	309	304	304
query72	6798	4914	4832	4832
query73	756	323	315	315
query74	6289	5857	5788	5788
query75	4604	3623	3804	3623
query76	4635	1162	1183	1162
query77	587	252	255	252
query78	12901	11622	12102	11622
query79	8552	638	649	638
query80	1475	402	391	391
query81	501	235	241	235
query82	1685	95	98	95
query83	169	138	133	133
query84	255	75	71	71
query85	944	324	324	324
query86	329	308	283	283
query87	3213	3020	2947	2947
query88	4745	2298	2312	2298
query89	393	289	295	289
query90	1899	223	208	208
query91	167	129	132	129
query92	56	53	54	53
query93	6193	585	570	570
query94	775	202	208	202
query95	2056	1941	2069	1941
query96	652	324	332	324
query97	6527	6404	6456	6404
query98	221	209	208	208
query99	2800	881	905	881
Total cold run time: 319195 ms
Total hot run time: 212891 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.88 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7a04d53352df5ffc455ae9fec0f827ebd500c020, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.02
query3	0.25	0.05	0.04
query4	1.82	0.06	0.06
query5	0.54	0.52	0.52
query6	1.26	0.61	0.60
query7	0.01	0.02	0.01
query8	0.03	0.02	0.02
query9	0.52	0.47	0.47
query10	0.52	0.54	0.54
query11	0.12	0.08	0.09
query12	0.11	0.09	0.09
query13	0.62	0.61	0.61
query14	0.78	0.79	0.79
query15	0.79	0.75	0.76
query16	0.37	0.37	0.36
query17	0.95	1.02	1.01
query18	0.24	0.24	0.26
query19	1.92	1.80	1.84
query20	0.01	0.00	0.01
query21	15.46	0.55	0.54
query22	2.02	2.57	1.60
query23	17.13	1.05	0.98
query24	6.83	1.09	1.60
query25	0.35	0.14	0.05
query26	0.81	0.15	0.14
query27	0.04	0.03	0.04
query28	5.92	0.72	0.72
query29	12.67	2.25	2.26
query30	0.58	0.56	0.53
query31	2.81	0.41	0.36
query32	3.38	0.49	0.50
query33	3.07	3.09	3.10
query34	15.64	4.84	4.83
query35	4.86	4.88	4.89
query36	1.06	1.03	1.02
query37	0.06	0.04	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.15
query41	0.07	0.01	0.01
query42	0.02	0.01	0.02
query43	0.03	0.01	0.02
Total cold run time: 103.99 s
Total hot run time: 30.88 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 7a04d53352df5ffc455ae9fec0f827ebd500c020 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@morningman morningman merged commit d78dbf3 into apache:branch-2.0 Nov 2, 2024
22 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/load Issues or PRs related to all kinds of load
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants