Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[improvement](statistics)Eliminate null values while sample analyzing ndv.#50574

Merged
Jibing-Li merged 1 commit into
apache:masterfrom
Jibing-Li:null
Jun 9, 2025
Merged

[improvement](statistics)Eliminate null values while sample analyzing ndv.#50574
Jibing-Li merged 1 commit into
apache:masterfrom
Jibing-Li:null

Conversation

@Jibing-Li

@Jibing-Li Jibing-Li commented Apr 30, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

  1. Eliminate null values while sample analyzing ndv.
    For example, if the values of col1 in table1 are all NULL, the ndv of col1 should be 0, not 1.

  2. If ndv=0 and min/max not null and null_num == rowCount, we keep this stats info but set to unknown in stats calculator. This is probably because sampled an all-null tablet.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Jibing-Li

Jibing-Li commented Apr 30, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Apr 30, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Jibing-Li Jibing-Li marked this pull request as ready for review April 30, 2025 09:45
@doris-robot

doris-robot commented Apr 30, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33904 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 90d8d6741001ed49353ec37ecdecb32d201d49b6, data reload: false

------ Round 1 ----------------------------------
q1	25799	4964	4963	4963
q2	2059	271	182	182
q3	10414	1227	693	693
q4	10236	980	515	515
q5	7531	2336	2329	2329
q6	176	164	129	129
q7	908	740	595	595
q8	9317	1279	1132	1132
q9	6868	5134	5113	5113
q10	6870	2283	1886	1886
q11	484	285	264	264
q12	344	345	205	205
q13	17804	3673	3126	3126
q14	217	222	210	210
q15	524	498	496	496
q16	412	433	368	368
q17	598	848	375	375
q18	7561	7293	7162	7162
q19	1439	935	548	548
q20	341	333	225	225
q21	3913	2605	2419	2419
q22	1046	969	993	969
Total cold run time: 114861 ms
Total hot run time: 33904 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5069	4992	5010	4992
q2	237	327	238	238
q3	2140	2616	2320	2320
q4	1337	1759	1332	1332
q5	4456	4460	4482	4460
q6	218	172	131	131
q7	2103	1982	1816	1816
q8	2663	2599	2570	2570
q9	7327	7203	6965	6965
q10	3079	3176	2745	2745
q11	600	498	479	479
q12	676	738	586	586
q13	3507	3902	3307	3307
q14	272	294	277	277
q15	525	469	480	469
q16	450	470	423	423
q17	1158	1560	1360	1360
q18	7749	7598	7502	7502
q19	780	804	998	804
q20	1987	1996	1881	1881
q21	4937	4691	4658	4658
q22	1072	1036	1032	1032
Total cold run time: 52342 ms
Total hot run time: 50347 ms

@doris-robot

doris-robot commented Apr 30, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 191818 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 90d8d6741001ed49353ec37ecdecb32d201d49b6, data reload: false

query1	1392	1075	1080	1075
query2	6205	1738	1791	1738
query3	11020	4478	4513	4478
query4	52309	24900	22908	22908
query5	5171	562	445	445
query6	341	205	206	205
query7	4905	512	283	283
query8	319	256	233	233
query9	5707	2603	2554	2554
query10	456	308	248	248
query11	15022	15048	14867	14867
query12	162	107	102	102
query13	1034	489	370	370
query14	10296	6282	6423	6282
query15	219	202	189	189
query16	7147	659	506	506
query17	1085	750	592	592
query18	1534	442	316	316
query19	192	189	165	165
query20	127	123	132	123
query21	208	131	117	117
query22	4351	4400	4319	4319
query23	34062	33487	33810	33487
query24	6334	2529	2405	2405
query25	460	506	400	400
query26	707	269	153	153
query27	2140	504	326	326
query28	3152	2148	2099	2099
query29	586	570	440	440
query30	269	215	184	184
query31	899	871	788	788
query32	76	71	64	64
query33	506	358	308	308
query34	779	872	520	520
query35	813	831	749	749
query36	940	1002	892	892
query37	111	102	81	81
query38	4141	4183	4079	4079
query39	1519	1448	1448	1448
query40	220	114	107	107
query41	54	51	52	51
query42	129	110	113	110
query43	504	503	468	468
query44	1321	818	807	807
query45	175	178	167	167
query46	851	1055	639	639
query47	1894	1856	1794	1794
query48	388	447	313	313
query49	686	521	427	427
query50	661	691	414	414
query51	4259	4214	4130	4130
query52	108	110	95	95
query53	224	249	187	187
query54	580	566	528	528
query55	88	85	81	81
query56	324	314	301	301
query57	1217	1272	1166	1166
query58	259	257	240	240
query59	2683	2793	2696	2696
query60	329	334	309	309
query61	130	122	128	122
query62	745	735	668	668
query63	223	187	186	186
query64	1806	1040	710	710
query65	4473	4356	4368	4356
query66	782	421	314	314
query67	15837	15700	15220	15220
query68	6981	873	514	514
query69	576	296	264	264
query70	1159	1111	1090	1090
query71	505	327	292	292
query72	6031	4812	4940	4812
query73	1462	706	341	341
query74	9000	9110	8941	8941
query75	4127	3205	2661	2661
query76	4253	1195	778	778
query77	745	354	273	273
query78	10039	9958	9262	9262
query79	2699	825	559	559
query80	618	504	434	434
query81	494	254	211	211
query82	448	134	97	97
query83	293	247	222	222
query84	293	99	83	83
query85	760	362	382	362
query86	369	289	271	271
query87	4429	4373	4299	4299
query88	3622	2261	2260	2260
query89	398	301	279	279
query90	1829	203	222	203
query91	141	144	109	109
query92	78	58	58	58
query93	1989	945	573	573
query94	664	380	308	308
query95	377	291	292	291
query96	473	569	277	277
query97	3217	3232	3086	3086
query98	232	207	207	207
query99	1436	1401	1257	1257
Total cold run time: 297383 ms
Total hot run time: 191818 ms

@doris-robot

doris-robot commented Apr 30, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 28.85 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 90d8d6741001ed49353ec37ecdecb32d201d49b6, data reload: false

query1	0.04	0.04	0.04
query2	0.13	0.11	0.11
query3	0.25	0.19	0.19
query4	1.59	0.19	0.20
query5	0.59	0.61	0.59
query6	1.18	0.72	0.72
query7	0.03	0.02	0.02
query8	0.04	0.04	0.03
query9	0.57	0.53	0.51
query10	0.56	0.58	0.56
query11	0.16	0.10	0.11
query12	0.15	0.11	0.11
query13	0.61	0.60	0.59
query14	0.77	0.80	0.80
query15	0.87	0.84	0.86
query16	0.38	0.38	0.41
query17	0.99	0.99	1.03
query18	0.20	0.20	0.20
query19	1.87	1.75	1.79
query20	0.01	0.01	0.01
query21	15.40	0.90	0.55
query22	0.78	1.33	0.86
query23	14.70	1.38	0.61
query24	7.68	0.80	0.53
query25	0.47	0.22	0.05
query26	0.55	0.17	0.13
query27	0.05	0.05	0.05
query28	9.08	0.85	0.42
query29	12.53	3.82	3.24
query30	0.25	0.09	0.06
query31	2.84	0.58	0.37
query32	3.23	0.53	0.49
query33	2.99	3.03	3.00
query34	15.68	5.10	4.46
query35	4.49	4.53	4.53
query36	0.68	0.49	0.47
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.18	0.14	0.13
query41	0.09	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.02	0.03
Total cold run time: 102.9 s
Total hot run time: 28.85 s

morrySnow
morrySnow reviewed May 6, 2025
View reviewed changes
Comment thread fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java Outdated
@morrySnow morrySnow added usercase Important user case type label dev/2.1.x dev/3.0.x labels May 6, 2025
@Jibing-Li Jibing-Li force-pushed the null branch 2 times, most recently from b798eb5 to 32d6168 Compare May 6, 2025 10:06
@Jibing-Li

Jibing-Li commented May 6, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

doris-robot commented May 6, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33796 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 32d61686e8c2348595f51bd9a02c979526d2589f, data reload: false

------ Round 1 ----------------------------------
q1	26293	5063	5018	5018
q2	2083	269	193	193
q3	10487	1246	682	682
q4	10270	997	532	532
q5	8452	2383	2347	2347
q6	189	162	132	132
q7	942	762	615	615
q8	9319	1310	1100	1100
q9	6899	5140	5059	5059
q10	6794	2287	1892	1892
q11	497	271	263	263
q12	354	348	218	218
q13	17760	3682	3077	3077
q14	229	231	205	205
q15	536	478	481	478
q16	412	414	370	370
q17	581	859	357	357
q18	8037	7147	7108	7108
q19	1214	958	555	555
q20	341	327	220	220
q21	3910	2556	2390	2390
q22	1031	1008	985	985
Total cold run time: 116630 ms
Total hot run time: 33796 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5068	5011	5326	5011
q2	245	342	230	230
q3	2112	2656	2279	2279
q4	1367	1767	1333	1333
q5	4566	4450	4395	4395
q6	216	168	125	125
q7	1960	1925	1748	1748
q8	2564	2518	2538	2518
q9	7103	7162	7110	7110
q10	3006	3170	2730	2730
q11	573	503	474	474
q12	647	739	615	615
q13	3451	3935	3288	3288
q14	270	286	267	267
q15	514	473	502	473
q16	442	488	437	437
q17	1127	1559	1339	1339
q18	7800	7415	7422	7415
q19	793	799	845	799
q20	1941	2052	1862	1862
q21	5233	4563	4587	4563
q22	1056	1016	1005	1005
Total cold run time: 52054 ms
Total hot run time: 50016 ms

@doris-robot

doris-robot commented May 6, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 185104 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 32d61686e8c2348595f51bd9a02c979526d2589f, data reload: false

query1	1013	466	494	466
query2	6584	1771	1797	1771
query3	6745	215	231	215
query4	26591	24086	23211	23211
query5	4387	625	479	479
query6	305	194	181	181
query7	4620	470	281	281
query8	298	263	246	246
query9	8634	2551	2559	2551
query10	488	324	258	258
query11	15592	15006	14883	14883
query12	160	120	107	107
query13	1656	523	391	391
query14	9382	6093	6016	6016
query15	202	190	181	181
query16	7211	628	460	460
query17	1204	749	559	559
query18	1968	392	294	294
query19	186	184	155	155
query20	118	114	125	114
query21	214	125	102	102
query22	4023	4309	4003	4003
query23	34109	33027	32893	32893
query24	8479	2365	2355	2355
query25	543	446	391	391
query26	1232	274	171	171
query27	2728	494	327	327
query28	4378	2084	2058	2058
query29	797	533	431	431
query30	279	210	186	186
query31	986	861	755	755
query32	76	68	64	64
query33	571	358	308	308
query34	793	849	495	495
query35	782	815	750	750
query36	974	987	899	899
query37	117	105	81	81
query38	4138	4173	4115	4115
query39	1486	1407	1407	1407
query40	215	116	108	108
query41	58	54	54	54
query42	127	108	110	108
query43	488	489	472	472
query44	1268	807	788	788
query45	182	169	170	169
query46	820	1017	615	615
query47	1772	1782	1697	1697
query48	374	403	292	292
query49	773	511	425	425
query50	651	677	386	386
query51	4137	4215	4111	4111
query52	107	105	100	100
query53	220	244	183	183
query54	572	586	496	496
query55	81	80	79	79
query56	306	295	281	281
query57	1140	1148	1066	1066
query58	262	254	248	248
query59	2574	2634	2557	2557
query60	338	318	302	302
query61	130	128	126	126
query62	812	750	650	650
query63	225	183	181	181
query64	4375	1027	716	716
query65	4377	4242	4212	4212
query66	1141	407	310	310
query67	15559	15347	15153	15153
query68	7752	885	507	507
query69	481	309	263	263
query70	1224	1114	1086	1086
query71	381	311	317	311
query72	5600	4829	4876	4829
query73	689	685	338	338
query74	8850	9192	8747	8747
query75	3204	3173	2699	2699
query76	3182	1181	750	750
query77	498	369	332	332
query78	9927	9988	9323	9323
query79	2321	814	569	569
query80	788	522	439	439
query81	501	256	215	215
query82	393	129	101	101
query83	259	245	232	232
query84	253	107	82	82
query85	748	389	323	323
query86	415	323	294	294
query87	4402	4393	4317	4317
query88	3319	2198	2171	2171
query89	375	311	284	284
query90	1917	209	206	206
query91	134	144	120	120
query92	79	61	59	59
query93	2623	914	564	564
query94	723	414	315	315
query95	369	301	282	282
query96	473	564	271	271
query97	3208	3235	3109	3109
query98	231	205	208	205
query99	1341	1394	1266	1266
Total cold run time: 273793 ms
Total hot run time: 185104 ms

@doris-robot

doris-robot commented May 6, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.08 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 32d61686e8c2348595f51bd9a02c979526d2589f, data reload: false

query1	0.04	0.03	0.03
query2	0.13	0.10	0.12
query3	0.26	0.20	0.19
query4	1.60	0.19	0.11
query5	0.56	0.56	0.55
query6	1.16	0.72	0.72
query7	0.03	0.02	0.02
query8	0.04	0.04	0.03
query9	0.57	0.51	0.50
query10	0.56	0.57	0.57
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.62	0.60	0.60
query14	0.77	0.81	0.80
query15	0.88	0.83	0.86
query16	0.37	0.38	0.38
query17	1.01	0.98	1.02
query18	0.21	0.19	0.19
query19	1.93	1.83	1.84
query20	0.02	0.01	0.01
query21	15.41	0.90	0.57
query22	0.75	1.21	0.72
query23	14.83	1.38	0.58
query24	7.39	1.76	0.92
query25	0.51	0.21	0.05
query26	0.54	0.16	0.14
query27	0.06	0.05	0.04
query28	9.81	0.80	0.43
query29	12.53	3.84	3.21
query30	0.27	0.10	0.06
query31	2.81	0.58	0.38
query32	3.23	0.55	0.49
query33	3.06	3.07	3.06
query34	15.73	5.08	4.49
query35	4.50	4.48	4.53
query36	0.66	0.49	0.49
query37	0.09	0.07	0.07
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.18	0.14	0.13
query41	0.07	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 103.67 s
Total hot run time: 29.08 s

@Jibing-Li

Jibing-Li commented May 6, 2025

Copy link
Copy Markdown
Contributor Author

run feut

1 similar comment
@Jibing-Li

Jibing-Li commented May 6, 2025

Copy link
Copy Markdown
Contributor Author

run feut

@Jibing-Li

Jibing-Li commented May 6, 2025

Copy link
Copy Markdown
Contributor Author

run external

1 similar comment
@Jibing-Li

Jibing-Li commented May 6, 2025

Copy link
Copy Markdown
Contributor Author

run external

@Jibing-Li Jibing-Li force-pushed the null branch 4 times, most recently from 3540022 to 908c41b Compare May 7, 2025 06:59
@Jibing-Li

Jibing-Li commented May 7, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

doris-robot commented May 7, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33986 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 908c41b0c9f6da33c238a770cd7581bbb104183d, data reload: false

------ Round 1 ----------------------------------
q1	26297	5022	4999	4999
q2	2064	282	195	195
q3	10389	1256	682	682
q4	10223	1005	546	546
q5	7558	2315	2386	2315
q6	185	167	131	131
q7	902	772	621	621
q8	9330	1282	1165	1165
q9	6885	5180	5125	5125
q10	6853	2291	1874	1874
q11	505	290	273	273
q12	360	363	211	211
q13	17764	3626	3040	3040
q14	223	237	211	211
q15	533	501	480	480
q16	421	429	372	372
q17	607	849	358	358
q18	7504	7175	7189	7175
q19	2327	988	554	554
q20	326	331	224	224
q21	3855	3486	2460	2460
q22	1064	1035	975	975
Total cold run time: 116175 ms
Total hot run time: 33986 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5251	5097	5070	5070
q2	230	324	230	230
q3	2183	2633	2309	2309
q4	1358	1808	1411	1411
q5	4441	4428	4402	4402
q6	219	172	127	127
q7	2013	1931	1729	1729
q8	2588	2599	2550	2550
q9	7277	7057	7125	7057
q10	2998	3158	2719	2719
q11	586	508	504	504
q12	668	767	606	606
q13	3515	3911	3394	3394
q14	284	293	268	268
q15	540	473	466	466
q16	480	488	445	445
q17	1162	1555	1403	1403
q18	7796	7736	7433	7433
q19	824	870	951	870
q20	1956	1982	1814	1814
q21	5148	4798	4629	4629
q22	1124	1035	1035	1035
Total cold run time: 52641 ms
Total hot run time: 50471 ms

@doris-robot

doris-robot commented May 7, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 191800 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 908c41b0c9f6da33c238a770cd7581bbb104183d, data reload: false

query1	1415	1097	1099	1097
query2	6391	1775	1759	1759
query3	11009	4463	4595	4463
query4	54957	26458	23386	23386
query5	5010	482	483	482
query6	354	203	195	195
query7	4920	517	285	285
query8	318	249	240	240
query9	5733	2572	2584	2572
query10	431	325	256	256
query11	14955	14953	14866	14866
query12	152	108	107	107
query13	1070	504	390	390
query14	10164	6289	6249	6249
query15	211	193	175	175
query16	7108	659	474	474
query17	1078	706	572	572
query18	1646	407	303	303
query19	196	190	161	161
query20	124	130	118	118
query21	205	130	109	109
query22	4495	4595	4494	4494
query23	34188	33257	33173	33173
query24	6543	2388	2404	2388
query25	462	459	401	401
query26	699	277	151	151
query27	2215	509	343	343
query28	3181	2150	2125	2125
query29	580	579	423	423
query30	273	217	187	187
query31	868	881	803	803
query32	70	69	60	60
query33	475	356	305	305
query34	784	873	527	527
query35	810	825	737	737
query36	944	1000	902	902
query37	120	101	82	82
query38	4175	4182	4124	4124
query39	1512	1411	1434	1411
query40	223	124	112	112
query41	63	55	58	55
query42	126	118	116	116
query43	507	499	472	472
query44	1367	825	834	825
query45	179	178	162	162
query46	844	1043	637	637
query47	1845	1906	1861	1861
query48	402	422	301	301
query49	675	491	423	423
query50	673	692	398	398
query51	4284	4287	4191	4191
query52	110	108	100	100
query53	229	261	188	188
query54	579	565	524	524
query55	85	84	81	81
query56	321	309	286	286
query57	1230	1232	1203	1203
query58	268	263	256	256
query59	2739	2714	2689	2689
query60	327	341	307	307
query61	127	128	124	124
query62	726	737	686	686
query63	227	184	186	184
query64	1773	1037	689	689
query65	4401	4253	4226	4226
query66	695	409	303	303
query67	15804	15618	15160	15160
query68	7939	901	523	523
query69	551	311	276	276
query70	1227	1114	1118	1114
query71	506	330	350	330
query72	5716	4585	4618	4585
query73	1242	576	342	342
query74	8930	9007	8776	8776
query75	3721	3202	2694	2694
query76	4082	1194	762	762
query77	625	354	268	268
query78	10043	10039	9269	9269
query79	4828	801	555	555
query80	663	574	451	451
query81	499	244	213	213
query82	561	126	93	93
query83	381	247	230	230
query84	295	102	87	87
query85	810	349	309	309
query86	412	307	292	292
query87	4350	4387	4259	4259
query88	3944	2218	2227	2218
query89	442	307	280	280
query90	1793	212	212	212
query91	138	151	111	111
query92	77	61	56	56
query93	3263	955	582	582
query94	657	421	310	310
query95	359	296	294	294
query96	490	568	269	269
query97	3146	3220	3116	3116
query98	238	207	198	198
query99	1447	1383	1272	1272
Total cold run time: 304045 ms
Total hot run time: 191800 ms

@doris-robot

doris-robot commented May 7, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 28.69 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 908c41b0c9f6da33c238a770cd7581bbb104183d, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.11
query3	0.27	0.19	0.20
query4	1.60	0.21	0.20
query5	0.59	0.58	0.58
query6	1.18	0.71	0.74
query7	0.03	0.01	0.02
query8	0.04	0.03	0.04
query9	0.58	0.51	0.50
query10	0.56	0.56	0.56
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.61	0.59	0.59
query14	0.78	0.79	0.80
query15	0.86	0.86	0.83
query16	0.39	0.37	0.39
query17	1.04	0.99	1.00
query18	0.21	0.20	0.19
query19	1.90	1.81	1.79
query20	0.01	0.01	0.02
query21	15.46	0.91	0.55
query22	0.74	1.06	0.64
query23	15.12	1.37	0.65
query24	7.90	1.22	0.44
query25	0.50	0.08	0.14
query26	0.67	0.16	0.14
query27	0.05	0.04	0.04
query28	8.79	0.82	0.42
query29	12.58	3.95	3.33
query30	0.26	0.09	0.07
query31	2.81	0.60	0.39
query32	3.23	0.54	0.47
query33	3.04	3.06	3.05
query34	15.79	5.08	4.44
query35	4.53	4.49	4.49
query36	0.66	0.49	0.48
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.03
query40	0.16	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.71 s
Total hot run time: 28.69 s

@Jibing-Li

Jibing-Li commented May 7, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions

github-actions Bot commented Jun 6, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jun 6, 2025
@github-actions

github-actions Bot commented Jun 6, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@doris-robot

doris-robot commented Jun 6, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33815 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 42eae8e3c204eb70f1eb7d8dc80e44949f592e1d, data reload: false

------ Round 1 ----------------------------------
q1	26454	4990	4989	4989
q2	2000	285	174	174
q3	10332	1233	678	678
q4	10232	1006	513	513
q5	7563	2345	2340	2340
q6	184	163	132	132
q7	906	734	612	612
q8	9332	1302	1071	1071
q9	6900	5230	5182	5182
q10	6869	2307	1896	1896
q11	484	289	270	270
q12	336	354	212	212
q13	17780	3624	3153	3153
q14	228	230	217	217
q15	572	477	500	477
q16	445	430	374	374
q17	587	853	356	356
q18	7616	7090	7163	7090
q19	2086	1078	550	550
q20	333	331	223	223
q21	3645	3136	2332	2332
q22	1061	1048	974	974
Total cold run time: 115945 ms
Total hot run time: 33815 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5196	5087	5103	5087
q2	241	324	231	231
q3	2096	2629	2309	2309
q4	1363	1790	1369	1369
q5	4478	4342	4423	4342
q6	214	170	124	124
q7	2022	2062	1780	1780
q8	2544	2562	2548	2548
q9	7304	7161	7011	7011
q10	3060	3183	2725	2725
q11	583	511	507	507
q12	655	778	609	609
q13	3516	3926	3297	3297
q14	306	300	282	282
q15	540	500	502	500
q16	456	498	430	430
q17	1130	1504	1384	1384
q18	7864	7690	7389	7389
q19	778	810	871	810
q20	2057	2053	1831	1831
q21	4788	4609	4353	4353
q22	1149	1104	1025	1025
Total cold run time: 52340 ms
Total hot run time: 49943 ms

@doris-robot

doris-robot commented Jun 6, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 192645 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 42eae8e3c204eb70f1eb7d8dc80e44949f592e1d, data reload: false

query1	1414	1070	1060	1060
query2	6364	1780	1809	1780
query3	11138	4548	4619	4548
query4	53902	25625	22902	22902
query5	5292	486	455	455
query6	397	212	202	202
query7	5198	497	287	287
query8	313	237	212	212
query9	6938	2634	2632	2632
query10	413	327	283	283
query11	15286	14961	14816	14816
query12	171	115	122	115
query13	1191	533	444	444
query14	10037	6282	6268	6268
query15	226	208	197	197
query16	7069	673	535	535
query17	1078	779	609	609
query18	1567	408	316	316
query19	216	203	179	179
query20	132	121	118	118
query21	208	127	111	111
query22	4492	4593	4458	4458
query23	34401	33696	33591	33591
query24	6705	2415	2402	2402
query25	465	474	421	421
query26	714	276	150	150
query27	2407	525	351	351
query28	2989	2166	2167	2166
query29	578	560	442	442
query30	283	231	194	194
query31	874	848	782	782
query32	70	62	65	62
query33	465	368	311	311
query34	810	866	526	526
query35	794	844	778	778
query36	959	1020	877	877
query37	106	98	86	86
query38	4215	4166	4197	4166
query39	1556	1475	1434	1434
query40	217	132	110	110
query41	61	61	62	61
query42	120	109	115	109
query43	498	500	467	467
query44	1345	871	850	850
query45	179	177	169	169
query46	850	1038	647	647
query47	1853	1920	1847	1847
query48	401	443	337	337
query49	689	511	392	392
query50	672	720	408	408
query51	4244	4273	4190	4190
query52	109	107	102	102
query53	230	260	191	191
query54	583	570	513	513
query55	88	86	80	80
query56	295	301	280	280
query57	1229	1266	1191	1191
query58	264	271	262	262
query59	2686	2735	2642	2642
query60	332	313	304	304
query61	154	124	164	124
query62	736	790	697	697
query63	223	203	192	192
query64	1488	1016	676	676
query65	4202	4206	4115	4115
query66	731	394	304	304
query67	16135	15622	15750	15622
query68	7134	892	527	527
query69	539	311	279	279
query70	1193	1185	1116	1116
query71	511	317	287	287
query72	5686	4818	4912	4818
query73	1152	695	356	356
query74	8902	8962	9064	8962
query75	3789	3209	2716	2716
query76	4291	1203	751	751
query77	626	367	271	271
query78	9994	10243	9380	9380
query79	2312	836	592	592
query80	631	518	432	432
query81	514	258	217	217
query82	407	129	94	94
query83	360	247	231	231
query84	288	110	87	87
query85	789	357	310	310
query86	365	315	296	296
query87	4479	4397	4215	4215
query88	3039	2311	2262	2262
query89	403	307	280	280
query90	1964	205	211	205
query91	142	141	113	113
query92	79	60	55	55
query93	1173	940	591	591
query94	697	426	267	267
query95	369	297	286	286
query96	487	567	288	288
query97	2759	2740	2697	2697
query98	225	217	208	208
query99	1432	1400	1308	1308
Total cold run time: 298778 ms
Total hot run time: 192645 ms

@doris-robot

doris-robot commented Jun 6, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.01 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 42eae8e3c204eb70f1eb7d8dc80e44949f592e1d, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.10	0.12
query3	0.26	0.19	0.20
query4	1.60	0.20	0.19
query5	0.45	0.43	0.45
query6	1.43	0.67	0.66
query7	0.02	0.02	0.02
query8	0.04	0.05	0.03
query9	0.59	0.53	0.53
query10	0.57	0.59	0.57
query11	0.16	0.11	0.11
query12	0.15	0.12	0.11
query13	0.61	0.60	0.59
query14	0.79	0.81	0.81
query15	0.88	0.86	0.87
query16	0.39	0.38	0.39
query17	1.00	1.05	1.01
query18	0.22	0.21	0.21
query19	2.00	1.80	1.84
query20	0.02	0.01	0.02
query21	15.39	0.86	0.53
query22	0.76	1.14	0.59
query23	15.06	1.36	0.60
query24	7.29	1.00	0.76
query25	0.50	0.10	0.13
query26	0.63	0.17	0.13
query27	0.05	0.05	0.05
query28	9.44	0.90	0.45
query29	12.55	4.03	3.35
query30	0.26	0.10	0.06
query31	2.83	0.60	0.40
query32	3.23	0.55	0.47
query33	3.00	3.08	3.19
query34	15.79	5.11	4.48
query35	4.54	4.52	4.55
query36	0.66	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.03
query40	0.16	0.14	0.12
query41	0.07	0.02	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.81 s
Total hot run time: 29.01 s

@Jibing-Li

Jibing-Li commented Jun 6, 2025

Copy link
Copy Markdown
Contributor Author

run p0

@Jibing-Li Jibing-Li merged commit f5479b5 into apache:master Jun 9, 2025
26 of 27 checks passed
@Jibing-Li Jibing-Li deleted the null branch June 9, 2025 03:12
github-actions Bot pushed a commit that referenced this pull request Jun 9, 2025
@Jibing-Li
[improvement](statistics)Eliminate null values while sample analyzing…
f93b9be 
… ndv. (#50574)

### What problem does this PR solve?

1. Eliminate null values while sample analyzing ndv.
For example, if the values of col1 in table1 are all NULL, the ndv of
col1 should be 0, not 1.

2. If ndv=0 and min/max not null and null_num == rowCount, we keep this
stats info but set to unknown in stats calculator. This is probably
because sampled an all-null tablet.
@github-actions github-actions Bot mentioned this pull request Jun 9, 2025
Merged
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Jun 11, 2025
@Jibing-Li
[improvement](statistics)Eliminate null values while sample analyzing…
07e0458 
… ndv. (apache#50574)

1. Eliminate null values while sample analyzing ndv.
For example, if the values of col1 in table1 are all NULL, the ndv of
col1 should be 0, not 1.

2. If ndv=0 and min/max not null and null_num == rowCount, we keep this
stats info but set to unknown in stats calculator. This is probably
because sampled an all-null tablet.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Jun 11, 2025
@Jibing-Li
[improvement](statistics)Eliminate null values while sample analyzing…
d98a5cf 
… ndv. (apache#50574)

1. Eliminate null values while sample analyzing ndv.
For example, if the values of col1 in table1 are all NULL, the ndv of
col1 should be 0, not 1.

2. If ndv=0 and min/max not null and null_num == rowCount, we keep this
stats info but set to unknown in stats calculator. This is probably
because sampled an all-null tablet.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Jun 11, 2025
@Jibing-Li
[improvement](statistics)Eliminate null values while sample analyzing…
b24751d 
… ndv. (apache#50574)

1. Eliminate null values while sample analyzing ndv.
For example, if the values of col1 in table1 are all NULL, the ndv of
col1 should be 0, not 1.

2. If ndv=0 and min/max not null and null_num == rowCount, we keep this
stats info but set to unknown in stats calculator. This is probably
because sampled an all-null tablet.
@Jibing-Li Jibing-Li mentioned this pull request Jun 11, 2025
Merged
dataroaring pushed a commit that referenced this pull request Jun 11, 2025
@github-actions @Jibing-Li
branch-3.0: [improvement](statistics)Eliminate null values while samp…
0039311 
…le analyzing ndv. #50574 (#51575)

Cherry-picked from #50574

Co-authored-by: James <[email protected]>
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Jun 11, 2025
@Jibing-Li
[improvement](statistics)Eliminate null values while sample analyzing…
41045b9 
… ndv. (apache#50574)

1. Eliminate null values while sample analyzing ndv.
For example, if the values of col1 in table1 are all NULL, the ndv of
col1 should be 0, not 1.

2. If ndv=0 and min/max not null and null_num == rowCount, we keep this
stats info but set to unknown in stats calculator. This is probably
because sampled an all-null tablet.
yiguolei pushed a commit that referenced this pull request Jun 13, 2025
@Jibing-Li
branch-2.1:[improvement](statistics)Eliminate null values while sampl…
1a16493 
…e analyzing ndv. (#50574) (#51648)

backport: #50574
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Closed
MilanTyagi2004 added a commit to MilanTyagi2004/doris that referenced this pull request Jul 7, 2026
@MilanTyagi2004
[fix](fe) Treat suspicious statistics with nullCount == count as UNKNOWN
74dd71d 
### What problem does this PR solve?

Issue Number: close apache#64122, close apache#64206

Problem Summary:
In PR apache#50574, logic was added to StatsCalculator to treat statistics with ndv=0, min/max not null, and nullCount == count as UNKNOWN (which represents sampling an all-null tablet).
However, this check was only implemented inside a single overload of StatsCalculator#getColumnStatistic, which is bypassed by scan optimization nodes (like OlapScan, HiveScan, and partition-stats merge). As a result, suspicious statistics under the nullCount == count case were not treated as UNKNOWN during CBO estimation.
We unify the validation rules into ColumnStatistic.isSuspiciousStats(...) by removing the `nullCount != count` check. This ensures that any suspicious statistics (ndv == 0 but min/max not null) are automatically deserialized as ColumnStatistic.UNKNOWN at the cache level, making it consistent for all optimizer scan paths. Added unit tests to ColStatsDataTest to cover the nullCount == count case.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - Unit Test (ColStatsDataTest.java)
- Behavior changed: No
- Does this need documentation: No
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@englefly englefly englefly approved these changes
@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.11-merged dev/3.0.7-merged dev/3.1.0-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@Jibing-Li @hello-stephen @doris-robot @englefly @morrySnow @yiguolei @dataroaring