Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[Feature](datatype) update be ut codes and fix bugs for IPv4/v6#28670

Merged
starocean999 merged 4 commits into
apache:masterfrom
sjyango:core-bug-fix
Dec 20, 2023
Merged

[Feature](datatype) update be ut codes and fix bugs for IPv4/v6#28670
starocean999 merged 4 commits into
apache:masterfrom
sjyango:core-bug-fix

Conversation

@sjyango

@sjyango sjyango commented Dec 19, 2023

Copy link
Copy Markdown
Contributor

Proposed changes

update from_string_test be ut based on issue #26534

  • update be ut codes
  • fix bugs
  • remove unused codes in DataTypeIPv4/v6

output:
image

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@sjyango

sjyango commented Dec 19, 2023

Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions

github-actions Bot commented Dec 19, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented Dec 19, 2023

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 36.49% (8534/23386)
Line Coverage: 28.63% (69374/242315)
Region Coverage: 27.64% (35898/129859)
Branch Coverage: 24.37% (18338/75250)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8449c02afa1cf61c7c747bb3255da07570f09cfa_8449c02afa1cf61c7c747bb3255da07570f09cfa/report/index.html

@doris-robot

doris-robot commented Dec 19, 2023

Copy link
Copy Markdown

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.54 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.1 seconds inserted 10000000 Rows, about 355K ops/s
storage size: 17183831270 Bytes

@sjyango

sjyango commented Dec 19, 2023

Copy link
Copy Markdown
Contributor Author

run p0

1 similar comment
@sjyango

sjyango commented Dec 19, 2023

Copy link
Copy Markdown
Contributor Author

run p0

@doris-robot

doris-robot commented Dec 19, 2023

Copy link
Copy Markdown

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 8449c02afa1cf61c7c747bb3255da07570f09cfa, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4691	4423	4399	4399
q2	368	158	158	158
q3	1459	1249	1192	1192
q4	1112	891	895	891
q5	3141	3147	3109	3109
q6	247	129	126	126
q7	986	496	485	485
q8	2181	2259	2178	2178
q9	6678	6646	6654	6646
q10	3225	3236	3294	3236
q11	309	185	174	174
q12	349	216	211	211
q13	4556	3782	3776	3776
q14	238	210	211	210
q15	562	531	529	529
q16	451	388	386	386
q17	1003	597	568	568
q18	7244	6947	6985	6947
q19	1505	1406	1355	1355
q20	506	299	273	273
q21	3019	2602	2711	2602
q22	347	273	285	273
Total cold run time: 44177 ms
Total hot run time: 39724 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4344	4352	4319	4319
q2	270	167	172	167
q3	3519	3528	3523	3523
q4	2379	2373	2360	2360
q5	5742	5741	5733	5733
q6	241	121	123	121
q7	2371	1865	1881	1865
q8	3537	3518	3517	3517
q9	8989	9053	8985	8985
q10	3921	3997	4021	3997
q11	495	369	362	362
q12	767	601	593	593
q13	4286	3606	3537	3537
q14	293	254	256	254
q15	563	516	518	516
q16	510	464	453	453
q17	1880	1843	1851	1843
q18	8603	8240	8168	8168
q19	1721	1733	1751	1733
q20	2264	1948	1933	1933
q21	6493	6156	6156	6156
q22	502	428	426	426
Total cold run time: 63690 ms
Total hot run time: 60561 ms

amorynan
amorynan reviewed Dec 20, 2023
View reviewed changes
Comment thread be/test/vec/data_types/from_string_test.cpp Outdated
typedef std::pair<FieldType, string> FieldType_RandStr;
std::vector<FieldType_RandStr> ip_scala_field_types = {
FieldType_RandStr(FieldType::OLAP_FIELD_TYPE_IPV4, "127.0.0.1"),
FieldType_RandStr(FieldType::OLAP_FIELD_TYPE_IPV6, "2405:9800:9800:66::2")};

@amorynan amorynan Dec 20, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add max min case here. to make beut run

amorynan
amorynan reviewed Dec 20, 2023
View reviewed changes
Comment thread be/src/vec/data_types/data_type_ipv6.cpp
ColumnPtr ptr = result.first;
row_num = result.second;
IPv6 value = assert_cast<const ColumnIPv6&>(*ptr).get_element(row_num);
return convert_ipv6_to_string(value);

@amorynan amorynan Dec 20, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why write convert before ?

@github-actions

github-actions Bot commented Dec 20, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@sjyango

sjyango commented Dec 20, 2023

Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions

github-actions Bot commented Dec 20, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions

github-actions Bot commented Dec 20, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented Dec 20, 2023

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 36.50% (8535/23381)
Line Coverage: 28.64% (69393/242332)
Region Coverage: 27.66% (35911/129848)
Branch Coverage: 24.39% (18354/75242)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f3628b20bbc9b968a092fb119a7d27e56f35e1ae_f3628b20bbc9b968a092fb119a7d27e56f35e1ae/report/index.html

@doris-robot

doris-robot commented Dec 20, 2023

Copy link
Copy Markdown

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit f3628b20bbc9b968a092fb119a7d27e56f35e1ae, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4742	4443	4413	4413
q2	362	152	198	152
q3	1467	1237	1205	1205
q4	1119	930	897	897
q5	3156	3204	3152	3152
q6	248	130	129	129
q7	977	492	498	492
q8	2207	2213	2191	2191
q9	6709	6678	6678	6678
q10	3230	3302	3310	3302
q11	306	195	190	190
q12	365	217	211	211
q13	4575	3815	3833	3815
q14	248	219	220	219
q15	575	527	521	521
q16	451	395	394	394
q17	1011	616	515	515
q18	7146	6962	6836	6836
q19	1514	1349	1422	1349
q20	513	308	289	289
q21	3072	2633	2663	2633
q22	358	297	304	297
Total cold run time: 44351 ms
Total hot run time: 39880 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4397	4365	4347	4347
q2	281	168	178	168
q3	3546	3538	3542	3538
q4	2388	2375	2371	2371
q5	5743	5746	5762	5746
q6	242	119	121	119
q7	2372	1894	1914	1894
q8	3518	3527	3533	3527
q9	9043	9029	9001	9001
q10	3916	3992	4007	3992
q11	496	382	384	382
q12	789	608	613	608
q13	4296	3590	3604	3590
q14	298	250	266	250
q15	573	521	530	521
q16	517	463	483	463
q17	1875	1869	1880	1869
q18	8717	8123	8205	8123
q19	1734	1752	1775	1752
q20	2260	1984	1937	1937
q21	6573	6224	6210	6210
q22	529	450	440	440
Total cold run time: 64103 ms
Total hot run time: 60848 ms

@doris-robot

doris-robot commented Dec 20, 2023

Copy link
Copy Markdown

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.15 seconds
stream load tsv: 570 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17184315028 Bytes

amorynan
amorynan approved these changes Dec 20, 2023
View reviewed changes

@amorynan amorynan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actions Bot commented Dec 20, 2023

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Dec 20, 2023
@github-actions

github-actions Bot commented Dec 20, 2023

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@starocean999 starocean999 merged commit 0c9c32c into apache:master Dec 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@starocean999 starocean999 starocean999 approved these changes
+1 more reviewer
@amorynan amorynan amorynan approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sjyango @doris-robot @amorynan @starocean999