Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](MoW) fix MoW & segcompaction conflict on cache of temp segment #37760

Merged

Conversation

freemandealer
Copy link
Contributor

MoW will update delete bitmap during load, and the page cache could be modified by segcompaction. Disable page cache touchs when doing segcompaction could solve this problem.

Proposed changes

Issue Number: close #xxx

MoW will update delete bitmap during load, and the page cache could be
modified by segcompaction. Disable page cache touchs when doing
segcompaction could solve this problem.

Signed-off-by: freemandealer <[email protected]>
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@freemandealer
Copy link
Contributor Author

run buildall

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 14, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 39755 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ca413baa7f0001b9200a1753ce07e5f5a7e466e2, data reload: false

------ Round 1 ----------------------------------
q1	17611	4380	4292	4292
q2	2020	194	191	191
q3	10469	1271	1037	1037
q4	10189	739	801	739
q5	7545	2687	2624	2624
q6	224	143	145	143
q7	988	602	595	595
q8	9212	2054	2073	2054
q9	8729	6575	6519	6519
q10	8797	3804	3758	3758
q11	452	248	245	245
q12	396	227	239	227
q13	17774	2999	2997	2997
q14	286	234	231	231
q15	537	480	487	480
q16	487	395	385	385
q17	965	704	685	685
q18	8089	7493	7478	7478
q19	5682	1403	1339	1339
q20	688	318	310	310
q21	5009	3137	3188	3137
q22	361	295	289	289
Total cold run time: 116510 ms
Total hot run time: 39755 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4469	4252	4231	4231
q2	391	266	274	266
q3	3053	2754	2769	2754
q4	1886	1631	1627	1627
q5	5296	5335	5352	5335
q6	221	132	130	130
q7	2145	1752	1784	1752
q8	3208	3379	3275	3275
q9	8421	8442	8387	8387
q10	3903	3692	3702	3692
q11	606	507	494	494
q12	765	615	620	615
q13	16414	2998	2999	2998
q14	301	273	293	273
q15	530	480	484	480
q16	482	406	419	406
q17	1768	1475	1472	1472
q18	7719	7616	7418	7418
q19	1706	1570	1554	1554
q20	1974	1794	1801	1794
q21	4864	4677	4674	4674
q22	572	487	511	487
Total cold run time: 70694 ms
Total hot run time: 54114 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173469 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ca413baa7f0001b9200a1753ce07e5f5a7e466e2, data reload: false

query1	920	365	361	361
query2	6459	1930	1786	1786
query3	6665	207	217	207
query4	27516	17568	17439	17439
query5	4186	484	483	483
query6	267	156	171	156
query7	4588	305	288	288
query8	244	190	197	190
query9	8514	2372	2334	2334
query10	446	293	270	270
query11	11872	10034	10226	10034
query12	138	88	84	84
query13	1653	378	367	367
query14	9474	7286	7691	7286
query15	215	170	167	167
query16	7812	333	320	320
query17	1764	553	543	543
query18	1972	289	283	283
query19	204	150	154	150
query20	92	85	88	85
query21	207	131	125	125
query22	4424	3944	4011	3944
query23	33918	33127	34333	33127
query24	12048	2846	2915	2846
query25	548	389	391	389
query26	1161	151	152	151
query27	2864	271	290	271
query28	7585	1993	1978	1978
query29	884	630	639	630
query30	288	150	149	149
query31	972	753	764	753
query32	97	56	56	56
query33	781	312	303	303
query34	940	495	490	490
query35	684	596	596	596
query36	1093	930	969	930
query37	226	81	80	80
query38	2843	2739	2789	2739
query39	859	804	795	795
query40	294	122	122	122
query41	49	52	46	46
query42	113	98	101	98
query43	514	474	480	474
query44	1170	747	739	739
query45	190	163	166	163
query46	1090	730	731	730
query47	1854	1778	1785	1778
query48	377	296	299	296
query49	1193	444	425	425
query50	794	395	393	393
query51	6908	6770	6805	6770
query52	110	92	100	92
query53	367	294	296	294
query54	901	456	452	452
query55	77	75	77	75
query56	298	316	286	286
query57	1135	1056	1067	1056
query58	371	241	255	241
query59	2789	2662	2569	2569
query60	298	266	268	266
query61	95	96	93	93
query62	814	649	648	648
query63	335	296	291	291
query64	10238	2218	7449	2218
query65	3143	3108	3126	3108
query66	1228	339	338	338
query67	15631	15113	15040	15040
query68	8799	555	541	541
query69	716	485	390	390
query70	1227	1175	1117	1117
query71	564	282	274	274
query72	8039	5851	5847	5847
query73	820	325	326	325
query74	6051	5572	5599	5572
query75	5218	2712	2647	2647
query76	5460	984	944	944
query77	818	298	297	297
query78	9684	9000	8940	8940
query79	8600	512	517	512
query80	2434	473	470	470
query81	600	228	224	224
query82	779	132	139	132
query83	288	165	165	165
query84	268	88	87	87
query85	1439	315	334	315
query86	405	298	318	298
query87	3305	3119	3127	3119
query88	4847	2483	2450	2450
query89	543	388	387	387
query90	2124	204	197	197
query91	134	105	103	103
query92	62	54	50	50
query93	7191	492	493	492
query94	1359	214	213	213
query95	406	325	316	316
query96	615	274	275	274
query97	3158	3045	3074	3045
query98	235	205	203	203
query99	1545	1237	1256	1237
Total cold run time: 307532 ms
Total hot run time: 173469 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.76 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ca413baa7f0001b9200a1753ce07e5f5a7e466e2, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.49	0.49
query6	1.14	0.72	0.72
query7	0.02	0.02	0.01
query8	0.06	0.04	0.04
query9	0.55	0.48	0.50
query10	0.56	0.55	0.55
query11	0.15	0.11	0.12
query12	0.14	0.12	0.12
query13	0.59	0.59	0.58
query14	0.77	0.77	0.78
query15	0.86	0.80	0.81
query16	0.38	0.37	0.38
query17	1.00	0.97	1.02
query18	0.23	0.22	0.22
query19	1.82	1.76	1.74
query20	0.01	0.02	0.01
query21	15.41	0.74	0.66
query22	4.44	7.11	1.91
query23	18.29	1.32	1.34
query24	2.07	0.23	0.21
query25	0.15	0.09	0.08
query26	0.30	0.20	0.22
query27	0.45	0.23	0.23
query28	13.36	0.99	1.01
query29	12.58	3.35	3.36
query30	0.25	0.06	0.05
query31	2.88	0.39	0.39
query32	3.25	0.48	0.47
query33	2.90	2.99	2.89
query34	16.97	4.42	4.37
query35	4.41	4.40	4.41
query36	0.65	0.48	0.47
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.15	0.12	0.13
query41	0.10	0.05	0.05
query42	0.05	0.06	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.89 s
Total hot run time: 30.76 s

@freemandealer
Copy link
Contributor Author

run external

Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit ba2b371 into apache:master Jul 16, 2024
26 of 30 checks passed
freemandealer added a commit to freemandealer/doris that referenced this pull request Jul 16, 2024
…(pick apache#37760)

MoW will update delete bitmap during load, and the page cache could be
modified by segcompaction. Disable page cache touchs when doing
segcompaction could solve this problem.

Signed-off-by: freemandealer <[email protected]>
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
…pache#37760)

MoW will update delete bitmap during load, and the page cache could be
modified by segcompaction. Disable page cache touchs when doing
segcompaction could solve this problem.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Signed-off-by: freemandealer <[email protected]>
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
…37760)

MoW will update delete bitmap during load, and the page cache could be
modified by segcompaction. Disable page cache touchs when doing
segcompaction could solve this problem.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Signed-off-by: freemandealer <[email protected]>
dataroaring pushed a commit that referenced this pull request Jul 18, 2024
…(pick #37760) (#37938)

MoW will update delete bitmap during load, and the page cache could be
modified by segcompaction. Disable page cache touchs when doing
segcompaction could solve this problem.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Signed-off-by: freemandealer <[email protected]>
dataroaring pushed a commit to dataroaring/incubator-doris that referenced this pull request Aug 7, 2024
…pache#37760)

MoW will update delete bitmap during load, and the page cache could be
modified by segcompaction. Disable page cache touchs when doing
segcompaction could solve this problem.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Signed-off-by: freemandealer <[email protected]>
dataroaring added a commit that referenced this pull request Aug 7, 2024
#38992)

…(#37760)

MoW will update delete bitmap during load, and the page cache could be
modified by segcompaction. Disable page cache touchs when doing
segcompaction could solve this problem.

Signed-off-by: freemandealer <[email protected]>
Co-authored-by: zhengyu <[email protected]>
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
…(pick apache#37760) (apache#37938)

MoW will update delete bitmap during load, and the page cache could be
modified by segcompaction. Disable page cache touchs when doing
segcompaction could solve this problem.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Signed-off-by: freemandealer <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants