Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](inverted index) fix wrong opt for count_on_index #41127 #41153

Merged
merged 3 commits into from
Sep 23, 2024
Merged

[Fix](inverted index) fix wrong opt for count_on_index #41127 #41153

merged 3 commits into from
Sep 23, 2024

Conversation

airborne12
Copy link
Member

cherry pick from #41127

@airborne12
[Fix](inverted index) fix wrong opt for count_on_index (apache#41127)
4dca57e 
## Proposed changes

The pushdownCountOnIndex optimization rule in AggregateStrategies was
incorrectly applied to COUNT functions with complex child expressions,
such as COUNT(CASE WHEN ...). This led to incorrect query results.
@airborne12
Copy link
Member Author

run buildall

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@airborne12
Copy link
Member Author

run buildall

@airborne12
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49037 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e1266e12d9c0a73b07518af3046632118fef7ce8, data reload: false

------ Round 1 ----------------------------------
q1	17876	4366	4337	4337
q2	2075	163	147	147
q3	10260	1876	1923	1876
q4	10231	1201	1296	1201
q5	8997	3899	3935	3899
q6	230	122	126	122
q7	1993	1632	1613	1613
q8	9317	2719	2697	2697
q9	10215	9924	9716	9716
q10	8629	3544	3531	3531
q11	431	249	259	249
q12	468	302	297	297
q13	18574	4006	4081	4006
q14	370	324	325	324
q15	511	463	474	463
q16	542	465	457	457
q17	1124	966	938	938
q18	7291	6843	6943	6843
q19	1673	1592	1532	1532
q20	547	323	303	303
q21	4323	4100	4148	4100
q22	528	386	390	386
Total cold run time: 116205 ms
Total hot run time: 49037 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4471	4368	4347	4347
q2	327	239	228	228
q3	4186	4184	4148	4148
q4	2748	2760	2746	2746
q5	7178	7178	7174	7174
q6	243	119	118	118
q7	3281	2864	2830	2830
q8	4397	4539	4639	4539
q9	14035	13928	13816	13816
q10	4246	4333	4314	4314
q11	759	686	688	686
q12	1041	880	880	880
q13	6928	3793	3760	3760
q14	450	446	434	434
q15	490	464	453	453
q16	625	594	605	594
q17	3823	3856	3880	3856
q18	8716	8680	8773	8680
q19	1706	1655	1691	1655
q20	2374	2123	2099	2099
q21	8481	8416	8497	8416
q22	1033	959	943	943
Total cold run time: 81538 ms
Total hot run time: 76716 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 212579 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e1266e12d9c0a73b07518af3046632118fef7ce8, data reload: false

query1	950	413	378	378
query2	6534	2277	2076	2076
query3	6919	206	200	200
query4	23877	21487	21615	21487
query5	19730	6500	6554	6500
query6	292	227	249	227
query7	4160	296	314	296
query8	253	300	245	245
query9	3121	2665	2582	2582
query10	421	319	306	306
query11	15566	15390	14900	14900
query12	125	78	80	78
query13	1024	459	448	448
query14	17904	13474	13560	13474
query15	379	220	231	220
query16	5688	282	265	265
query17	1721	933	919	919
query18	889	318	314	314
query19	185	159	159	159
query20	106	94	101	94
query21	184	105	92	92
query22	5142	5013	4896	4896
query23	34102	33511	33657	33511
query24	6899	6320	6322	6320
query25	549	436	426	426
query26	1126	161	159	159
query27	2446	292	295	292
query28	6170	2279	2253	2253
query29	2875	2813	2701	2701
query30	246	168	169	168
query31	964	723	743	723
query32	72	65	63	63
query33	436	275	268	268
query34	864	487	496	487
query35	1102	947	946	946
query36	1448	1074	1275	1074
query37	94	61	59	59
query38	3067	2943	2967	2943
query39	1374	1335	1347	1335
query40	203	100	98	98
query41	40	38	37	37
query42	90	93	88	88
query43	615	693	698	693
query44	1177	723	721	721
query45	248	238	236	236
query46	1237	969	960	960
query47	1719	1836	1744	1744
query48	527	423	426	423
query49	640	388	393	388
query50	873	651	604	604
query51	4800	4747	4710	4710
query52	97	86	91	86
query53	247	184	192	184
query54	2648	2424	2469	2424
query55	92	85	80	80
query56	233	201	210	201
query57	1347	1209	1140	1140
query58	227	222	213	213
query59	3528	3068	3379	3068
query60	217	203	213	203
query61	97	104	96	96
query62	830	456	442	442
query63	204	175	175	175
query64	4983	1636	1463	1463
query65	3667	3560	3584	3560
query66	670	383	403	383
query67	18139	15199	15328	15199
query68	11385	662	648	648
query69	525	282	275	275
query70	2065	1480	1405	1405
query71	411	311	315	311
query72	6910	4936	4889	4889
query73	776	314	314	314
query74	6231	5827	5806	5806
query75	5447	3718	3698	3698
query76	7035	1154	1203	1154
query77	1202	255	255	255
query78	12599	11717	22826	11717
query79	4575	627	633	627
query80	711	373	389	373
query81	464	232	238	232
query82	306	102	95	95
query83	172	135	138	135
query84	252	72	72	72
query85	886	328	318	318
query86	337	306	298	298
query87	3238	3013	2965	2965
query88	3389	2309	2295	2295
query89	362	283	285	283
query90	1931	219	222	219
query91	171	133	124	124
query92	57	53	52	52
query93	1506	576	558	558
query94	700	212	211	211
query95	2097	2009	2066	2009
query96	640	324	315	315
query97	6489	6381	6333	6333
query98	232	207	199	199
query99	2537	945	891	891
Total cold run time: 314037 ms
Total hot run time: 212579 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.25 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e1266e12d9c0a73b07518af3046632118fef7ce8, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.03	0.02
query3	0.25	0.05	0.04
query4	1.80	0.08	0.06
query5	0.53	0.53	0.52
query6	1.23	0.65	0.62
query7	0.02	0.01	0.01
query8	0.03	0.03	0.02
query9	0.52	0.48	0.50
query10	0.54	0.55	0.52
query11	0.11	0.08	0.08
query12	0.12	0.09	0.10
query13	0.61	0.61	0.61
query14	0.78	0.79	0.80
query15	0.77	0.75	0.76
query16	0.35	0.36	0.36
query17	0.98	1.03	1.02
query18	0.25	0.24	0.25
query19	1.90	1.85	1.87
query20	0.02	0.01	0.01
query21	15.48	0.55	0.55
query22	2.13	2.60	1.28
query23	16.99	1.12	0.88
query24	4.87	0.91	1.33
query25	0.38	0.08	0.06
query26	0.54	0.15	0.15
query27	0.04	0.04	0.04
query28	8.03	0.72	0.74
query29	12.66	2.32	2.21
query30	0.54	0.49	0.52
query31	2.81	0.39	0.38
query32	3.37	0.50	0.50
query33	3.07	3.10	3.08
query34	15.26	4.80	4.81
query35	4.85	4.88	4.84
query36	1.05	1.01	1.02
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.01	0.02
Total cold run time: 103.36 s
Total hot run time: 30.25 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e1266e12d9c0a73b07518af3046632118fef7ce8 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          57 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       22.0 seconds inserted 10000000 Rows, about 454K ops/s

@airborne12 airborne12 merged commit 5418807 into apache:branch-2.0 Sep 23, 2024
21 of 23 checks passed
@airborne12 airborne12 deleted the pick_41127_to_origin_branch-2.0 branch September 23, 2024 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
area/nereids kind/test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@airborne12 @doris-robot