Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improment] Log fatal when compaction memtracker not return zero #40667

Merged
merged 1 commit into from
Sep 12, 2024

Conversation

wangbo
Copy link
Contributor

@wangbo wangbo commented Sep 11, 2024

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wangbo wangbo marked this pull request as draft September 11, 2024 07:21
@wangbo
Copy link
Contributor Author

wangbo commented Sep 11, 2024

run buildall

@wangbo
Copy link
Contributor Author

wangbo commented Sep 11, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.88% (9449/25624)
Line Coverage: 28.24% (77696/275167)
Region Coverage: 27.63% (40102/145118)
Branch Coverage: 24.25% (20375/84024)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c9d02c11fae3e44c9e9891462e065b301411232b_c9d02c11fae3e44c9e9891462e065b301411232b/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 38238 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c9d02c11fae3e44c9e9891462e065b301411232b, data reload: false

------ Round 1 ----------------------------------
q1	18076	4647	4411	4411
q2	2035	191	192	191
q3	11507	963	1142	963
q4	10359	722	754	722
q5	7742	2853	2789	2789
q6	229	142	140	140
q7	972	619	600	600
q8	9340	2105	2050	2050
q9	7029	6552	6611	6552
q10	7001	2212	2296	2212
q11	480	240	238	238
q12	393	223	220	220
q13	18842	3066	3117	3066
q14	294	229	243	229
q15	536	483	486	483
q16	551	425	441	425
q17	995	662	708	662
q18	7473	6994	6849	6849
q19	1400	1110	1099	1099
q20	677	330	329	329
q21	3991	2991	3141	2991
q22	1098	1017	1039	1017
Total cold run time: 111020 ms
Total hot run time: 38238 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4387	4317	4275	4275
q2	384	266	268	266
q3	2918	2689	2682	2682
q4	1942	1677	1681	1677
q5	5765	5737	5878	5737
q6	227	134	134	134
q7	2188	1876	1803	1803
q8	3287	3450	3556	3450
q9	8899	8951	8912	8912
q10	3683	3371	3380	3371
q11	602	540	503	503
q12	834	657	644	644
q13	15418	3264	3292	3264
q14	307	297	292	292
q15	534	479	505	479
q16	556	492	499	492
q17	1849	1528	1564	1528
q18	8175	8066	7878	7878
q19	1734	1563	1559	1559
q20	2202	1899	1934	1899
q21	5791	5455	5382	5382
q22	1117	1054	1022	1022
Total cold run time: 72799 ms
Total hot run time: 57249 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197596 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c9d02c11fae3e44c9e9891462e065b301411232b, data reload: false

query1	1295	900	903	900
query2	6327	1919	1883	1883
query3	10619	3930	4001	3930
query4	59581	28897	23503	23503
query5	5041	509	522	509
query6	389	168	163	163
query7	5618	291	293	291
query8	328	232	243	232
query9	7635	2509	2495	2495
query10	426	290	276	276
query11	17337	15001	15516	15001
query12	164	111	115	111
query13	1453	401	408	401
query14	10375	7324	7105	7105
query15	216	191	178	178
query16	6426	499	469	469
query17	1079	577	566	566
query18	1362	316	301	301
query19	197	156	156	156
query20	124	112	113	112
query21	227	108	104	104
query22	4542	4514	4561	4514
query23	34517	33974	33490	33490
query24	5996	2980	2902	2902
query25	485	381	414	381
query26	598	150	148	148
query27	1595	276	277	276
query28	3595	2048	2034	2034
query29	635	417	411	411
query30	236	155	146	146
query31	944	757	778	757
query32	76	53	52	52
query33	415	287	289	287
query34	860	471	464	464
query35	844	712	707	707
query36	1068	937	951	937
query37	139	84	85	84
query38	4000	3900	3920	3900
query39	1448	1402	1395	1395
query40	196	114	115	114
query41	46	50	44	44
query42	117	98	99	98
query43	500	464	472	464
query44	1076	761	741	741
query45	202	169	167	167
query46	1084	757	736	736
query47	1905	1766	1815	1766
query48	377	294	295	294
query49	779	486	473	473
query50	830	408	417	408
query51	7045	6916	6930	6916
query52	100	88	86	86
query53	252	183	179	179
query54	562	446	448	446
query55	75	77	72	72
query56	285	249	291	249
query57	1185	1069	1061	1061
query58	221	239	236	236
query59	2983	2754	2733	2733
query60	295	266	275	266
query61	110	104	104	104
query62	757	651	656	651
query63	219	188	180	180
query64	1380	665	663	663
query65	3245	3162	3209	3162
query66	680	342	335	335
query67	15735	15365	15193	15193
query68	1562	850	839	839
query69	418	325	319	319
query70	1116	1184	1162	1162
query71	344	340	374	340
query72	4529	3514	3866	3514
query73	582	576	583	576
query74	9076	8902	8977	8902
query75	3062	2978	2964	2964
query76	948	868	837	837
query77	520	407	394	394
query78	9930	9218	9229	9218
query79	882	861	867	861
query80	795	793	793	793
query81	452	259	269	259
query82	264	263	268	263
query83	189	194	190	190
query84	199	107	107	107
query85	608	397	421	397
query86	329	278	322	278
query87	4386	4347	4350	4347
query88	4220	4062	4047	4047
query89	378	362	372	362
query90	1102	318	386	318
query91	123	121	123	121
query92	77	75	76	75
query93	900	885	896	885
query94	442	356	347	347
query95	421	416	413	413
query96	467	467	467	467
query97	3090	3111	3151	3111
query98	229	238	225	225
query99	1337	1291	1257	1257
Total cold run time: 293430 ms
Total hot run time: 197596 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.75 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c9d02c11fae3e44c9e9891462e065b301411232b, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.09
query5	0.50	0.50	0.50
query6	1.13	0.73	0.74
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.55	0.50	0.48
query10	0.53	0.57	0.56
query11	0.16	0.12	0.12
query12	0.14	0.12	0.12
query13	0.60	0.60	0.59
query14	1.39	1.44	1.48
query15	0.83	0.83	0.82
query16	0.37	0.38	0.38
query17	1.03	1.06	1.08
query18	0.20	0.20	0.20
query19	1.86	1.86	1.74
query20	0.02	0.01	0.01
query21	15.39	0.69	0.68
query22	4.45	6.87	2.11
query23	18.26	1.40	1.32
query24	2.07	0.22	0.22
query25	0.16	0.09	0.08
query26	0.26	0.19	0.18
query27	0.08	0.08	0.08
query28	13.25	1.00	1.01
query29	12.60	3.27	3.32
query30	0.24	0.05	0.06
query31	2.88	0.40	0.39
query32	3.24	0.48	0.48
query33	3.02	3.00	3.04
query34	16.97	4.41	4.38
query35	4.48	4.51	4.45
query36	0.65	0.50	0.47
query37	0.19	0.16	0.15
query38	0.15	0.15	0.15
query39	0.05	0.04	0.03
query40	0.16	0.12	0.13
query41	0.09	0.05	0.05
query42	0.06	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.16 s
Total hot run time: 31.75 s

@wangbo wangbo force-pushed the 0911_log_mem_traker branch 2 times, most recently from 7b68646 to 210cfa9 Compare September 12, 2024 02:38
@wangbo wangbo marked this pull request as ready for review September 12, 2024 03:29
@wangbo
Copy link
Contributor Author

wangbo commented Sep 12, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39910 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aef1fcdca8251193eee45aa472349f9a6bada2bf, data reload: false

------ Round 1 ----------------------------------
q1	18779	4362	4301	4301
q2	2691	192	201	192
q3	11524	1393	1390	1390
q4	10545	984	952	952
q5	8050	3214	3166	3166
q6	228	137	137	137
q7	1038	632	627	627
q8	9446	1994	2006	1994
q9	6784	6301	6292	6292
q10	6985	2501	2525	2501
q11	431	242	252	242
q12	400	234	225	225
q13	17750	3036	3012	3012
q14	286	251	257	251
q15	537	493	478	478
q16	507	429	428	428
q17	970	937	929	929
q18	7443	6877	6752	6752
q19	1370	1234	1221	1221
q20	606	328	320	320
q21	3902	3498	3494	3494
q22	1099	1006	1030	1006
Total cold run time: 111371 ms
Total hot run time: 39910 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4173	4150	4169	4150
q2	344	233	226	226
q3	2912	2899	2900	2899
q4	1958	1925	1963	1925
q5	5433	5386	5387	5386
q6	218	130	130	130
q7	2062	1663	1678	1663
q8	3226	3297	3314	3297
q9	8471	8414	8438	8414
q10	3398	3423	3436	3423
q11	570	457	463	457
q12	770	572	583	572
q13	4725	3051	3050	3050
q14	293	273	265	265
q15	527	493	488	488
q16	489	464	448	448
q17	1765	1707	1721	1707
q18	8012	7725	7530	7530
q19	1709	1691	1680	1680
q20	2056	1798	1786	1786
q21	5652	5417	5258	5258
q22	1116	1050	990	990
Total cold run time: 59879 ms
Total hot run time: 55744 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.90% (9456/25629)
Line Coverage: 28.25% (77782/275307)
Region Coverage: 27.66% (40167/145199)
Branch Coverage: 24.27% (20405/84088)
Coverage Report: http://coverage.selectdb-in.cc/coverage/aef1fcdca8251193eee45aa472349f9a6bada2bf_aef1fcdca8251193eee45aa472349f9a6bada2bf/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 194932 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aef1fcdca8251193eee45aa472349f9a6bada2bf, data reload: false

query1	908	395	389	389
query2	6479	1718	1726	1718
query3	6659	214	221	214
query4	26939	24049	24137	24049
query5	5750	545	526	526
query6	255	171	161	161
query7	4580	301	289	289
query8	279	227	227	227
query9	8493	2429	2449	2429
query10	502	286	291	286
query11	16276	15477	15485	15477
query12	163	101	103	101
query13	1709	399	387	387
query14	11297	7139	6578	6578
query15	218	178	170	170
query16	7590	473	468	468
query17	1520	601	567	567
query18	1993	296	303	296
query19	202	166	154	154
query20	125	115	117	115
query21	210	107	104	104
query22	4689	4570	4430	4430
query23	34449	33773	33851	33773
query24	9704	3135	3128	3128
query25	691	415	416	415
query26	1401	161	157	157
query27	2907	281	281	281
query28	6953	2026	2015	2015
query29	1002	433	427	427
query30	304	167	155	155
query31	1009	802	794	794
query32	104	63	65	63
query33	742	313	311	311
query34	925	487	484	484
query35	855	761	742	742
query36	1039	900	922	900
query37	167	91	88	88
query38	4005	3941	4011	3941
query39	1460	1431	1427	1427
query40	297	124	120	120
query41	52	51	49	49
query42	131	101	104	101
query43	493	437	438	437
query44	1203	784	759	759
query45	203	171	173	171
query46	1106	801	802	801
query47	1921	1794	1817	1794
query48	370	294	291	291
query49	1111	475	462	462
query50	921	438	453	438
query51	7055	6759	6927	6759
query52	103	90	90	90
query53	263	191	192	191
query54	780	474	473	473
query55	82	78	80	78
query56	297	290	279	279
query57	1229	1125	1064	1064
query58	271	246	272	246
query59	2762	2609	2534	2534
query60	312	297	280	280
query61	125	125	219	125
query62	903	666	685	666
query63	237	186	194	186
query64	5355	691	679	679
query65	3304	3242	3173	3173
query66	1408	355	361	355
query67	16030	15641	15496	15496
query68	3914	860	854	854
query69	463	332	337	332
query70	1175	1151	1146	1146
query71	357	344	351	344
query72	6206	3368	3463	3368
query73	578	584	580	580
query74	9117	9140	9024	9024
query75	3166	3027	2974	2974
query76	2590	845	862	845
query77	437	423	421	421
query78	9391	9388	9268	9268
query79	923	880	900	880
query80	863	841	837	837
query81	452	261	268	261
query82	269	267	262	262
query83	192	195	195	195
query84	243	108	107	107
query85	732	413	395	395
query86	314	317	328	317
query87	4364	4343	4415	4343
query88	4109	4044	4024	4024
query89	389	370	367	367
query90	1250	318	347	318
query91	126	126	126	126
query92	80	75	78	75
query93	1037	1050	1029	1029
query94	600	390	371	371
query95	475	423	424	423
query96	465	467	465	465
query97	3118	3129	3099	3099
query98	230	230	222	222
query99	1588	1369	1327	1327
Total cold run time: 282090 ms
Total hot run time: 194932 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.9 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aef1fcdca8251193eee45aa472349f9a6bada2bf, data reload: false

query1	0.04	0.05	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.49	0.49
query6	1.15	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.56	0.53	0.51
query10	0.57	0.60	0.60
query11	0.16	0.12	0.12
query12	0.16	0.13	0.13
query13	0.62	0.60	0.61
query14	1.46	1.48	1.44
query15	0.90	0.88	0.88
query16	0.36	0.36	0.35
query17	1.00	1.04	1.02
query18	0.21	0.21	0.21
query19	1.89	1.87	1.87
query20	0.01	0.00	0.01
query21	15.41	0.69	0.69
query22	4.23	7.69	1.03
query23	17.98	1.37	1.35
query24	2.20	0.22	0.21
query25	0.18	0.08	0.08
query26	0.29	0.18	0.18
query27	0.07	0.08	0.08
query28	13.16	1.01	0.98
query29	12.52	3.32	3.30
query30	0.24	0.06	0.05
query31	2.89	0.42	0.42
query32	3.22	0.50	0.49
query33	3.03	3.07	3.02
query34	15.46	4.35	4.36
query35	4.39	4.36	4.37
query36	0.68	0.50	0.50
query37	0.19	0.17	0.16
query38	0.16	0.15	0.16
query39	0.05	0.04	0.04
query40	0.17	0.14	0.14
query41	0.10	0.04	0.05
query42	0.06	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 108.38 s
Total hot run time: 30.9 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 12, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@xinyiZzz xinyiZzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wangbo wangbo merged commit 5eb3c60 into apache:master Sep 12, 2024
23 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.x reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants