Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[PipelineX](improvement) Prepare tasks in parallel #40844

Merged
merged 5 commits into from
Sep 14, 2024
Merged

[PipelineX](improvement) Prepare tasks in parallel #40844

merged 5 commits into from
Sep 14, 2024

Conversation

Gabriel39
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

@Gabriel39
[PipelineX](improvement) Prepare tasks in parallel (apache#40270)
a4b05fc 
Issue Number: close #xxx

<!--Describe your changes.-->
@Gabriel39
[followup](branch-2.1) Fuzzy parallel_prepare_threshold (apache#40793)
e01b604 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Gabriel39
Copy link
Contributor Author

run buildall

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 42005 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 857748931704211b442b3f497f1bf7a79ae4ae9b, data reload: false

------ Round 1 ----------------------------------
q1	17616	7274	7202	7202
q2	2057	159	155	155
q3	10697	1166	1262	1166
q4	10211	757	899	757
q5	7758	3101	3116	3101
q6	232	148	144	144
q7	1019	613	592	592
q8	9413	1978	2043	1978
q9	6817	6417	6368	6368
q10	7070	2323	2316	2316
q11	422	241	243	241
q12	398	222	211	211
q13	17774	3007	3002	3002
q14	250	217	216	216
q15	569	512	520	512
q16	467	429	422	422
q17	977	929	935	929
q18	7440	6852	6939	6852
q19	1387	1239	1200	1200
q20	579	300	281	281
q21	3905	3339	3469	3339
q22	1092	1024	1021	1021
Total cold run time: 108150 ms
Total hot run time: 42005 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7097	7084	7082	7082
q2	319	225	221	221
q3	3033	3051	3020	3020
q4	2000	1964	2016	1964
q5	5587	5589	5594	5589
q6	242	146	141	141
q7	2265	1838	1783	1783
q8	3342	3401	3406	3401
q9	8767	8891	8698	8698
q10	3540	3501	3566	3501
q11	580	466	474	466
q12	824	600	628	600
q13	8900	3185	3131	3131
q14	320	290	282	282
q15	591	525	516	516
q16	537	476	476	476
q17	1762	1747	1736	1736
q18	8405	8188	8197	8188
q19	1749	1732	1725	1725
q20	2118	1863	1872	1863
q21	5536	5269	5377	5269
q22	1117	1037	1047	1037
Total cold run time: 68631 ms
Total hot run time: 60689 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.92% (9474/25663)
Line Coverage: 28.25% (77869/275681)
Region Coverage: 27.67% (40224/145355)
Branch Coverage: 24.27% (20435/84184)
Coverage Report: http://coverage.selectdb-in.cc/coverage/857748931704211b442b3f497f1bf7a79ae4ae9b_857748931704211b442b3f497f1bf7a79ae4ae9b/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 197950 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 857748931704211b442b3f497f1bf7a79ae4ae9b, data reload: false

query1	1287	866	868	866
query2	6388	1835	1840	1835
query3	10769	3916	3901	3901
query4	54819	26260	24006	24006
query5	5146	508	499	499
query6	403	160	167	160
query7	5730	309	303	303
query8	276	218	216	216
query9	6744	2589	2558	2558
query10	457	291	280	280
query11	16631	15501	15662	15501
query12	150	100	99	99
query13	1497	400	405	400
query14	10651	6585	7213	6585
query15	210	180	181	180
query16	6992	490	495	490
query17	1161	603	592	592
query18	1762	320	308	308
query19	199	154	160	154
query20	119	111	109	109
query21	218	106	114	106
query22	4676	4541	4398	4398
query23	35010	33584	33677	33584
query24	6104	3114	3019	3019
query25	510	409	400	400
query26	622	165	158	158
query27	1585	282	292	282
query28	2847	2105	2067	2067
query29	686	422	442	422
query30	230	148	157	148
query31	962	773	789	773
query32	75	56	56	56
query33	473	296	301	296
query34	886	478	470	470
query35	874	724	733	724
query36	1056	904	934	904
query37	135	77	78	77
query38	4050	3901	3865	3865
query39	1445	1428	1419	1419
query40	211	97	96	96
query41	50	48	49	48
query42	118	96	96	96
query43	512	454	448	448
query44	1183	784	762	762
query45	205	167	164	164
query46	1090	817	804	804
query47	1961	1804	1827	1804
query48	453	357	348	348
query49	695	406	424	406
query50	915	433	432	432
query51	7161	7017	6928	6928
query52	99	85	82	82
query53	245	179	180	179
query54	549	447	457	447
query55	83	74	75	74
query56	290	267	270	267
query57	1228	1074	1111	1074
query58	221	252	248	248
query59	2989	2788	2537	2537
query60	296	257	258	257
query61	104	100	115	100
query62	814	672	678	672
query63	215	182	185	182
query64	1436	646	663	646
query65	3228	3144	3184	3144
query66	680	285	289	285
query67	16150	15526	15571	15526
query68	1573	853	835	835
query69	459	346	347	346
query70	1167	1162	1166	1162
query71	374	334	318	318
query72	6146	3302	3310	3302
query73	592	579	579	579
query74	9308	9025	8957	8957
query75	3114	2953	2977	2953
query76	999	832	829	829
query77	525	344	353	344
query78	9485	9204	9438	9204
query79	891	877	839	839
query80	587	557	560	557
query81	446	281	239	239
query82	197	185	186	185
query83	157	156	159	156
query84	269	100	113	100
query85	683	366	363	363
query86	323	322	331	322
query87	4361	4290	4367	4290
query88	4094	4025	4021	4021
query89	360	353	360	353
query90	1640	299	296	296
query91	162	185	163	163
query92	76	76	76	76
query93	1007	990	987	987
query94	600	395	394	394
query95	419	399	411	399
query96	473	466	463	463
query97	3145	3125	3170	3125
query98	226	230	261	230
query99	1633	1304	1297	1297
Total cold run time: 292507 ms
Total hot run time: 197950 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 857748931704211b442b3f497f1bf7a79ae4ae9b, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.03
query3	0.22	0.05	0.05
query4	1.68	0.06	0.07
query5	0.50	0.50	0.51
query6	1.15	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.56	0.52	0.52
query10	0.57	0.59	0.56
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.62	0.61	0.60
query14	1.47	1.47	1.49
query15	0.91	0.88	0.87
query16	0.35	0.37	0.37
query17	1.04	1.02	1.03
query18	0.16	0.16	0.16
query19	1.94	1.82	1.89
query20	0.01	0.02	0.01
query21	15.40	0.67	0.68
query22	4.20	7.22	1.66
query23	18.11	1.25	1.20
query24	2.29	0.22	0.22
query25	0.19	0.08	0.08
query26	0.29	0.18	0.17
query27	0.08	0.09	0.07
query28	13.17	1.12	1.10
query29	12.56	3.35	3.32
query30	0.25	0.05	0.05
query31	2.87	0.42	0.41
query32	3.22	0.49	0.49
query33	3.02	3.04	3.07
query34	15.46	4.31	4.30
query35	4.33	4.32	4.35
query36	0.70	0.50	0.48
query37	0.19	0.17	0.16
query38	0.15	0.15	0.16
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.10	0.05	0.04
query42	0.06	0.05	0.04
query43	0.05	0.05	0.05
Total cold run time: 108.57 s
Total hot run time: 31.27 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 14, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@Gabriel39 Gabriel39 merged commit 0a2fdce into apache:master Sep 14, 2024
23 of 28 checks passed
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Sep 14, 2024
@Gabriel39
[PipelineX](improvement) Prepare tasks in parallel (apache#40844)
2bc876a
@Gabriel39 Gabriel39 mentioned this pull request Sep 14, 2024
Merged
Gabriel39 added a commit that referenced this pull request Sep 18, 2024
@Gabriel39
[PipelineX](improvement) Prepare tasks in parallel (#40844) (#40874)
5137841 
pick #40844
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Gabriel39 @doris-robot @BiteTheDDDDt @yiguolei