Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[fix](iceberg) fix insert iceberg transform partition table problems and add some cases#54557

Merged
morningman merged 5 commits into
apache:masterfrom
suxiaogang223:fix_decimal_bucket_transform
Aug 13, 2025
Merged

[fix](iceberg) fix insert iceberg transform partition table problems and add some cases#54557
morningman merged 5 commits into
apache:masterfrom
suxiaogang223:fix_decimal_bucket_transform

Conversation

@suxiaogang223

@suxiaogang223 suxiaogang223 commented Aug 11, 2025 •
edited by morningman
Loading

Copy link
Copy Markdown
Member

What problem does this PR solve?

Problem Summary:

  1. Fixed the problem with inserting into Iceberg decimal bucket partition table due to memory issues
  2. Fixed the exception caused by division by zero when selecting count(*) from an empty iceberg partition table
  3. Add cases to cover the logic of Iceberg transform partition table https://iceberg.apache.org/spec/#partition-transforms

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

hello-stephen commented Aug 11, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223

suxiaogang223 commented Aug 11, 2025

Copy link
Copy Markdown
Member Author

run buildall

@doris-robot

doris-robot commented Aug 11, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33676 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9fa9ce1a684029540df31a9474e3733a6176d83b, data reload: false

------ Round 1 ----------------------------------
q1	17594	5231	5181	5181
q2	1914	290	180	180
q3	10322	1405	700	700
q4	10241	1025	545	545
q5	7522	2393	2295	2295
q6	176	169	139	139
q7	890	771	617	617
q8	9322	1299	1088	1088
q9	6764	5050	5076	5050
q10	6887	2357	1991	1991
q11	464	280	263	263
q12	349	370	234	234
q13	17776	3456	2981	2981
q14	231	252	246	246
q15	534	463	453	453
q16	422	423	374	374
q17	570	799	369	369
q18	7498	7052	6954	6954
q19	1098	981	538	538
q20	325	309	212	212
q21	3473	2475	2257	2257
q22	1064	1109	1009	1009
Total cold run time: 105436 ms
Total hot run time: 33676 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5320	5292	5302	5292
q2	235	309	212	212
q3	2068	2609	2172	2172
q4	1312	1704	1303	1303
q5	4138	4203	4482	4203
q6	236	179	136	136
q7	2000	1920	1827	1827
q8	2552	2503	2571	2503
q9	7413	7276	7376	7276
q10	3145	3295	2883	2883
q11	546	531	519	519
q12	702	770	658	658
q13	3461	3803	3194	3194
q14	288	312	277	277
q15	498	477	446	446
q16	474	505	503	503
q17	1146	1421	1403	1403
q18	10057	7842	7753	7753
q19	9563	928	826	826
q20	1945	1982	1947	1947
q21	10254	4516	4163	4163
q22	1032	1042	983	983
Total cold run time: 68385 ms
Total hot run time: 50479 ms

@doris-robot

doris-robot commented Aug 11, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 170455 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9fa9ce1a684029540df31a9474e3733a6176d83b, data reload: false

============================================
query1	982	396	407	396
query2	6508	1796	1685	1685
query3	6742	229	222	222
query4	26888	23610	23358	23358
query5	4363	627	507	507
query6	340	252	231	231
query7	4630	511	300	300
query8	289	233	223	223
query9	8616	2935	2902	2902
query10	460	332	290	290
query11	15868	15037	14881	14881
query12	181	135	127	127
query13	1655	536	432	432
query14	8553	5779	5896	5779
query15	223	195	171	171
query16	7825	643	474	474
query17	1653	784	645	645
query18	2135	441	331	331
query19	279	210	182	182
query20	150	142	143	142
query21	220	130	117	117
query22	3868	3941	3792	3792
query23	34318	34190	34361	34190
query24	8133	2394	2397	2394
query25	599	558	454	454
query26	938	291	159	159
query27	2659	494	358	358
query28	4246	2320	2292	2292
query29	702	596	483	483
query30	298	241	191	191
query31	891	831	735	735
query32	89	73	80	73
query33	564	403	427	403
query34	788	847	513	513
query35	842	817	763	763
query36	1037	1082	946	946
query37	133	109	89	89
query38	3943	3931	3959	3931
query39	1438	1371	1364	1364
query40	237	143	130	130
query41	61	57	54	54
query42	140	125	124	124
query43	512	531	491	491
query44	1377	870	864	864
query45	201	191	179	179
query46	944	1054	669	669
query47	1777	1826	1765	1765
query48	407	427	328	328
query49	693	501	414	414
query50	656	689	416	416
query51	4112	4159	4133	4133
query52	120	127	119	119
query53	258	281	212	212
query54	649	642	571	571
query55	88	84	86	84
query56	345	354	349	349
query57	1232	1208	1132	1132
query58	332	330	329	329
query59	2649	2609	2572	2572
query60	394	399	375	375
query61	122	122	126	122
query62	794	751	646	646
query63	253	213	207	207
query64	3583	1140	813	813
query65	4241	4102	4116	4102
query66	980	455	326	326
query67	query68	17616	986	1120	986
query69	1003	274	287	274
query70	1434	1049	1091	1049
query71	676	323	318	318
query72	9205	2344	4762	2344
query73	3835	780	354	354
query74	9014	8841	8893	8841
query75	7292	3096	2667	2667
query76	8523	1210	785	785
query77	1169	419	335	335
query78	9469	10736	9317	9317
query79	15198	606	594	594
query80	1724	557	484	484
query81	573	266	230	230
query82	442	154	121	121
query83	345	332	265	265
query84	307	97	80	80
query85	935	379	339	339
query86	345	326	301	301
query87	4263	4175	4140	4140
query88	5536	2209	2202	2202
query89	472	377	321	321
query90	2633	226	222	222
query91	146	139	110	110
query92	83	71	68	68
query93	6426	951	662	662
query94	1050	397	274	274
query95	425	326	324	324
query96	516	583	277	277
query97	2701	2737	2608	2608
query98	271	224	225	224
query99	1456	1403	1288	1288
Total cold run time: 306286 ms
Total hot run time: 170455 ms

@doris-robot

doris-robot commented Aug 11, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 32.74 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9fa9ce1a684029540df31a9474e3733a6176d83b, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.26	0.07	0.07
query4	1.60	0.10	0.11
query5	0.42	0.41	0.42
query6	1.17	0.67	0.67
query7	0.03	0.02	0.02
query8	0.05	0.04	0.03
query9	0.54	0.50	0.47
query10	0.54	0.54	0.53
query11	0.15	0.11	0.10
query12	0.15	0.11	0.12
query13	0.66	0.65	0.66
query14	0.92	1.14	1.11
query15	0.91	0.88	0.86
query16	0.40	0.39	0.39
query17	1.07	1.02	1.08
query18	0.22	0.20	0.19
query19	1.93	1.91	1.85
query20	0.02	0.01	0.02
query21	15.37	0.83	0.57
query22	0.77	1.16	0.68
query23	15.01	1.31	0.61
query24	7.29	1.45	0.44
query25	0.50	0.12	0.14
query26	0.62	0.16	0.14
query27	0.06	0.06	0.06
query28	9.64	0.87	0.46
query29	12.58	3.83	3.33
query30	2.99	3.05	2.94
query31	2.82	0.57	0.40
query32	3.24	0.56	0.49
query33	2.99	3.20	3.14
query34	16.09	5.33	4.94
query35	4.87	4.99	4.93
query36	0.71	0.53	0.50
query37	0.11	0.08	0.07
query38	0.06	0.06	0.04
query39	0.03	0.03	0.02
query40	0.18	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 107.25 s
Total hot run time: 32.74 s

@doris-robot

doris-robot commented Aug 11, 2025

Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 100.00% (6/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 59.00% (16604/28141)
Line Coverage 47.87% (150488/314349)
Region Coverage 36.68% (112641/307095)
Branch Coverage 39.62% (49998/126196)

@suxiaogang223 suxiaogang223 changed the title [fix](iceberg) fix insert iceberg partition transform problem [fix](iceberg) fix insert iceberg transform partition table problems and add some cases Aug 11, 2025
@suxiaogang223

suxiaogang223 commented Aug 11, 2025

Copy link
Copy Markdown
Member Author

run buildall

@doris-robot

doris-robot commented Aug 11, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 34040 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4d36df5952f714f29bfa264e6bc8507f2e11b82d, data reload: false

------ Round 1 ----------------------------------
q1	17558	5285	5239	5239
q2	1911	289	177	177
q3	10138	1278	728	728
q4	10196	984	523	523
q5	7534	2353	2336	2336
q6	181	163	132	132
q7	909	741	607	607
q8	9300	1281	1008	1008
q9	6825	5116	5066	5066
q10	6900	2366	1966	1966
q11	476	292	274	274
q12	346	348	211	211
q13	17755	3646	3103	3103
q14	225	236	212	212
q15	571	485	478	478
q16	428	419	364	364
q17	610	844	377	377
q18	7960	7142	7163	7142
q19	1077	937	528	528
q20	336	335	226	226
q21	3699	2568	2333	2333
q22	1041	1044	1010	1010
Total cold run time: 105976 ms
Total hot run time: 34040 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5140	5104	5052	5052
q2	240	321	224	224
q3	2191	2753	2373	2373
q4	1322	1802	1364	1364
q5	4209	4369	4524	4369
q6	223	168	146	146
q7	2082	2012	1853	1853
q8	2613	2552	2493	2493
q9	7386	7271	7267	7267
q10	3161	3262	2876	2876
q11	598	506	500	500
q12	720	775	626	626
q13	3520	4106	3248	3248
q14	309	323	285	285
q15	531	470	468	468
q16	468	476	443	443
q17	1158	1575	1411	1411
q18	7756	7681	7779	7681
q19	817	781	792	781
q20	1884	1991	1820	1820
q21	4706	4417	4298	4298
q22	1105	1049	1017	1017
Total cold run time: 52139 ms
Total hot run time: 50595 ms

@doris-robot

doris-robot commented Aug 11, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 185096 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4d36df5952f714f29bfa264e6bc8507f2e11b82d, data reload: false

query1	1010	376	416	376
query2	6519	1698	1743	1698
query3	6722	218	229	218
query4	26053	23363	22889	22889
query5	4344	610	481	481
query6	312	212	201	201
query7	4621	499	282	282
query8	273	234	221	221
query9	8566	2865	2829	2829
query10	474	330	291	291
query11	15852	15154	14885	14885
query12	160	117	119	117
query13	1656	535	421	421
query14	8771	5826	5795	5795
query15	194	189	161	161
query16	7244	612	480	480
query17	1178	698	559	559
query18	1973	409	341	341
query19	179	188	160	160
query20	122	120	125	120
query21	219	120	105	105
query22	4012	4174	4090	4090
query23	34292	33149	33355	33149
query24	8198	2362	2377	2362
query25	542	473	412	412
query26	1232	275	165	165
query27	2748	497	343	343
query28	4257	2232	2209	2209
query29	767	564	453	453
query30	286	222	199	199
query31	881	795	737	737
query32	86	72	78	72
query33	547	369	361	361
query34	792	837	514	514
query35	840	823	731	731
query36	935	1004	885	885
query37	121	108	89	89
query38	3990	4093	3947	3947
query39	1685	1413	1379	1379
query40	211	131	117	117
query41	59	61	56	56
query42	126	111	121	111
query43	497	488	485	485
query44	1336	850	844	844
query45	187	173	170	170
query46	851	995	648	648
query47	1728	1800	1715	1715
query48	372	412	309	309
query49	734	461	398	398
query50	648	682	419	419
query51	4105	4201	4148	4148
query52	115	110	108	108
query53	236	263	199	199
query54	589	578	538	538
query55	88	87	85	85
query56	312	314	318	314
query57	1173	1194	1124	1124
query58	275	277	290	277
query59	2631	2826	2652	2652
query60	335	334	335	334
query61	128	129	124	124
query62	819	736	641	641
query63	227	191	195	191
query64	4286	1007	702	702
query65	4313	4234	4250	4234
query66	1167	411	332	332
query67	15530	15287	15108	15108
query68	6326	908	574	574
query69	482	321	297	297
query70	1159	1170	1147	1147
query71	407	331	312	312
query72	5334	4887	4918	4887
query73	659	595	346	346
query74	8929	9126	8919	8919
query75	3115	3080	2575	2575
query76	3292	1117	738	738
query77	478	412	319	319
query78	9516	9725	8882	8882
query79	3065	782	581	581
query80	1533	547	472	472
query81	545	358	217	217
query82	671	136	105	105
query83	251	254	243	243
query84	250	95	89	89
query85	826	364	327	327
query86	472	301	321	301
query87	4250	4292	4195	4195
query88	3941	2181	2181	2181
query89	396	310	280	280
query90	1887	228	217	217
query91	137	141	112	112
query92	85	72	66	66
query93	2748	960	625	625
query94	737	435	299	299
query95	389	314	309	309
query96	480	560	271	271
query97	2611	2652	2588	2588
query98	244	219	216	216
query99	1331	1392	1296	1296
Total cold run time: 272277 ms
Total hot run time: 185096 ms

@doris-robot

doris-robot commented Aug 11, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 32.09 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4d36df5952f714f29bfa264e6bc8507f2e11b82d, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.04	0.04
query3	0.24	0.07	0.07
query4	1.62	0.11	0.11
query5	0.45	0.40	0.42
query6	1.16	0.63	0.64
query7	0.02	0.01	0.02
query8	0.05	0.03	0.03
query9	0.59	0.52	0.51
query10	0.58	0.56	0.55
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.63	0.60	0.60
query14	0.85	0.80	0.82
query15	0.88	0.86	0.86
query16	0.37	0.39	0.39
query17	1.05	1.07	1.03
query18	0.21	0.19	0.19
query19	1.94	1.87	1.80
query20	0.02	0.01	0.01
query21	15.42	0.89	0.55
query22	0.76	1.04	0.61
query23	15.15	1.37	0.62
query24	7.34	0.82	0.48
query25	0.48	0.23	0.14
query26	0.67	0.17	0.14
query27	0.05	0.05	0.05
query28	9.50	0.93	0.43
query29	12.56	3.86	3.24
query30	3.11	3.00	3.01
query31	2.81	0.58	0.38
query32	3.24	0.54	0.47
query33	3.07	3.09	3.03
query34	16.00	5.46	4.90
query35	4.86	4.92	4.89
query36	0.68	0.52	0.55
query37	0.09	0.07	0.07
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.18	0.15	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.03	0.04	0.03
Total cold run time: 107.29 s
Total hot run time: 32.09 s

@hello-stephen

hello-stephen commented Aug 11, 2025

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/2) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented Aug 11, 2025

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (6/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 59.00% (16604/28141)
Line Coverage 47.87% (150489/314349)
Region Coverage 36.70% (112712/307095)
Branch Coverage 39.63% (50013/126196)

@hello-stephen

hello-stephen commented Aug 11, 2025

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (6/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 81.48% (22501/27616)
Line Coverage 74.15% (233075/314348)
Region Coverage 61.46% (193822/315357)
Branch Coverage 65.47% (83637/127744)

@morningman morningman requested a review from Copilot August 11, 2025 22:30

This comment was marked as resolved.

Sign in to view

@suxiaogang223

suxiaogang223 commented Aug 12, 2025

Copy link
Copy Markdown
Member Author

run p0

@hello-stephen

hello-stephen commented Aug 12, 2025

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (6/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 81.48% (22501/27616)
Line Coverage 74.15% (233079/314348)
Region Coverage 61.45% (193782/315357)
Branch Coverage 65.47% (83638/127744)

morningman
morningman approved these changes Aug 12, 2025
View reviewed changes

@morningman morningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Aug 12, 2025
@github-actions

github-actions Bot commented Aug 12, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actions Bot commented Aug 12, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

kaka11chen
kaka11chen approved these changes Aug 13, 2025
View reviewed changes

@kaka11chen kaka11chen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 9927b8b into apache:master Aug 13, 2025
32 of 36 checks passed
github-actions Bot pushed a commit that referenced this pull request Aug 13, 2025
@suxiaogang223
[fix](iceberg) fix insert iceberg transform partition table problems …
e169721 
…and add some cases (#54557)

### What problem does this PR solve?

Problem Summary:
1. Fixed the problem with inserting into Iceberg decimal bucket
partition table due to memory issues
2. Fixed the exception caused by division by zero when selecting
count(*) from an empty iceberg partition table
3. Add cases to cover the logic of Iceberg transform partition table
https://iceberg.apache.org/spec/#partition-transforms
@github-actions github-actions Bot mentioned this pull request Aug 13, 2025
Closed
github-actions Bot pushed a commit that referenced this pull request Aug 13, 2025
@suxiaogang223
[fix](iceberg) fix insert iceberg transform partition table problems …
562b28a 
…and add some cases (#54557)

### What problem does this PR solve?

Problem Summary:
1. Fixed the problem with inserting into Iceberg decimal bucket
partition table due to memory issues
2. Fixed the exception caused by division by zero when selecting
count(*) from an empty iceberg partition table
3. Add cases to cover the logic of Iceberg transform partition table
https://iceberg.apache.org/spec/#partition-transforms
@github-actions github-actions Bot mentioned this pull request Aug 13, 2025
Merged
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Aug 14, 2025
@suxiaogang223
[fix](iceberg) fix insert iceberg transform partition table problems …
2521bd5 
…and add some cases (apache#54557)

### What problem does this PR solve?

Problem Summary:
1. Fixed the problem with inserting into Iceberg decimal bucket
partition table due to memory issues
2. Fixed the exception caused by division by zero when selecting
count(*) from an empty iceberg partition table
3. Add cases to cover the logic of Iceberg transform partition table
https://iceberg.apache.org/spec/#partition-transforms
@suxiaogang223 suxiaogang223 mentioned this pull request Aug 14, 2025
Merged
morrySnow pushed a commit that referenced this pull request Aug 15, 2025
@github-actions @suxiaogang223
branch-3.1: [fix](iceberg) fix insert iceberg transform partition tab…
b210d1d 
…le problems and add some cases #54557 (#54723)

Cherry-picked from #54557

Co-authored-by: Socrates <[email protected]>
dataroaring pushed a commit that referenced this pull request Aug 16, 2025
@suxiaogang223
branch-3.0: [fix](iceberg) fix insert iceberg transform partition tab…
c865166 
…le problems and add some cases (#54557) (#54796)

bp: #54557
@gavinchou gavinchou mentioned this pull request Sep 1, 2025
Closed
@suxiaogang223 suxiaogang223 deleted the fix_decimal_bucket_transform branch September 23, 2025 03:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments
@kaka11chen kaka11chen kaka11chen approved these changes
@morningman morningman morningman approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.8-merged dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@suxiaogang223 @hello-stephen @doris-robot @kaka11chen @morningman @dataroaring @morrySnow