Implement tests reordering for TestNG #8467

nikita-tkachenko-datadog · 2025-02-27T19:32:54Z

What Does This Do

Implements fail-fast tests ordering for Test NG framework

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: SDTEST-1512

pr-commenter · 2025-02-27T20:15:37Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/testng-tests-reordering
git_commit_date	1740676228	1740684674
git_commit_sha	`ccc22c5`	`49abe57`
release_version	1.47.0-SNAPSHOT~ccc22c5138	1.47.0-SNAPSHOT~49abe577a1

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1740687153	1740687153
ci_job_id	826737247	826737247
ci_pipeline_id	57266209	57266209
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner--ogofyt-project-304-concurrent-0-seb3wl5x 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner--ogofyt-project-304-concurrent-0-seb3wl5x 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 56 metrics, 7 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.47.0-SNAPSHOT~49abe577a1, baseline=1.47.0-SNAPSHOT~ccc22c5138

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.044 s) : 0, 1043923
Total [baseline] (8.691 s) : 0, 8691173
Agent [candidate] (1.038 s) : 0, 1038005
Total [candidate] (8.651 s) : 0, 8651333
section iast
Agent [baseline] (1.172 s) : 0, 1171664
Total [baseline] (9.258 s) : 0, 9257555
Agent [candidate] (1.172 s) : 0, 1172171
Total [candidate] (9.254 s) : 0, 9254281
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.171 s) : 0, 1171227
Total [baseline] (9.168 s) : 0, 9168037
Agent [candidate] (1.169 s) : 0, 1169274
Total [candidate] (9.164 s) : 0, 9164027
section iast_TELEMETRY_OFF
Agent [baseline] (1.181 s) : 0, 1180792
Total [baseline] (9.304 s) : 0, 9303930
Agent [candidate] (1.18 s) : 0, 1179776
Total [candidate] (9.321 s) : 0, 9320867

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.044 s	-
Agent	iast	1.172 s	127.741 ms (12.2%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.171 s	127.304 ms (12.2%)
Agent	iast_TELEMETRY_OFF	1.181 s	136.869 ms (13.1%)
Total	tracing	8.691 s	-
Total	iast	9.258 s	566.382 ms (6.5%)
Total	iast_HARDCODED_SECRET_DISABLED	9.168 s	476.864 ms (5.5%)
Total	iast_TELEMETRY_OFF	9.304 s	612.757 ms (7.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.038 s	-
Agent	iast	1.172 s	134.166 ms (12.9%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.169 s	131.269 ms (12.6%)
Agent	iast_TELEMETRY_OFF	1.18 s	141.771 ms (13.7%)
Total	tracing	8.651 s	-
Total	iast	9.254 s	602.948 ms (7.0%)
Total	iast_HARDCODED_SECRET_DISABLED	9.164 s	512.694 ms (5.9%)
Total	iast_TELEMETRY_OFF	9.321 s	669.534 ms (7.7%)

gantt
    title insecure-bank - break down per module: candidate=1.47.0-SNAPSHOT~49abe577a1, baseline=1.47.0-SNAPSHOT~ccc22c5138

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (721.023 ms) : 0, 721023
BytebuddyAgent [candidate] (715.641 ms) : 0, 715641
GlobalTracer [baseline] (239.888 ms) : 0, 239888
GlobalTracer [candidate] (238.355 ms) : 0, 238355
AppSec [baseline] (55.854 ms) : 0, 55854
AppSec [candidate] (55.513 ms) : 0, 55513
Remote Config [baseline] (698.713 µs) : 0, 699
Remote Config [candidate] (689.947 µs) : 0, 690
Telemetry [baseline] (11.551 ms) : 0, 11551
Telemetry [candidate] (12.891 ms) : 0, 12891
section iast
BytebuddyAgent [baseline] (837.659 ms) : 0, 837659
BytebuddyAgent [candidate] (837.145 ms) : 0, 837145
GlobalTracer [baseline] (229.915 ms) : 0, 229915
GlobalTracer [candidate] (230.319 ms) : 0, 230319
IAST [baseline] (23.589 ms) : 0, 23589
IAST [candidate] (22.999 ms) : 0, 22999
AppSec [baseline] (56.266 ms) : 0, 56266
AppSec [candidate] (57.304 ms) : 0, 57304
Remote Config [baseline] (612.814 µs) : 0, 613
Remote Config [candidate] (611.946 µs) : 0, 612
Telemetry [baseline] (8.635 ms) : 0, 8635
Telemetry [candidate] (8.795 ms) : 0, 8795
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (836.611 ms) : 0, 836611
BytebuddyAgent [candidate] (835.151 ms) : 0, 835151
GlobalTracer [baseline] (230.199 ms) : 0, 230199
GlobalTracer [candidate] (230.071 ms) : 0, 230071
IAST [baseline] (22.931 ms) : 0, 22931
IAST [candidate] (22.789 ms) : 0, 22789
AppSec [baseline] (57.215 ms) : 0, 57215
AppSec [candidate] (57.098 ms) : 0, 57098
Remote Config [baseline] (613.233 µs) : 0, 613
Remote Config [candidate] (601.772 µs) : 0, 602
Telemetry [baseline] (8.747 ms) : 0, 8747
Telemetry [candidate] (8.658 ms) : 0, 8658
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (844.027 ms) : 0, 844027
BytebuddyAgent [candidate] (843.024 ms) : 0, 843024
GlobalTracer [baseline] (231.573 ms) : 0, 231573
GlobalTracer [candidate] (231.999 ms) : 0, 231999
IAST [baseline] (25.905 ms) : 0, 25905
IAST [candidate] (26.528 ms) : 0, 26528
AppSec [baseline] (54.686 ms) : 0, 54686
AppSec [candidate] (53.575 ms) : 0, 53575
Remote Config [baseline] (644.157 µs) : 0, 644
Remote Config [candidate] (621.19 µs) : 0, 621
Telemetry [baseline] (8.833 ms) : 0, 8833
Telemetry [candidate] (8.678 ms) : 0, 8678

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.47.0-SNAPSHOT~49abe577a1, baseline=1.47.0-SNAPSHOT~ccc22c5138

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.038 s) : 0, 1038448
Total [baseline] (10.364 s) : 0, 10364486
Agent [candidate] (1.04 s) : 0, 1039854
Total [candidate] (10.397 s) : 0, 10397225
section appsec
Agent [baseline] (1.183 s) : 0, 1182636
Total [baseline] (10.694 s) : 0, 10693697
Agent [candidate] (1.184 s) : 0, 1184058
Total [candidate] (10.73 s) : 0, 10729521
section iast
Agent [baseline] (1.169 s) : 0, 1168818
Total [baseline] (11.017 s) : 0, 11016869
Agent [candidate] (1.172 s) : 0, 1172445
Total [candidate] (10.907 s) : 0, 10906744
section profiling
Agent [baseline] (1.261 s) : 0, 1260666
Total [baseline] (10.892 s) : 0, 10891788
Agent [candidate] (1.259 s) : 0, 1258761
Total [candidate] (10.862 s) : 0, 10862225

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.038 s	-
Agent	appsec	1.183 s	144.188 ms (13.9%)
Agent	iast	1.169 s	130.37 ms (12.6%)
Agent	profiling	1.261 s	222.218 ms (21.4%)
Total	tracing	10.364 s	-
Total	appsec	10.694 s	329.211 ms (3.2%)
Total	iast	11.017 s	652.383 ms (6.3%)
Total	profiling	10.892 s	527.303 ms (5.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.04 s	-
Agent	appsec	1.184 s	144.205 ms (13.9%)
Agent	iast	1.172 s	132.592 ms (12.8%)
Agent	profiling	1.259 s	218.908 ms (21.1%)
Total	tracing	10.397 s	-
Total	appsec	10.73 s	332.295 ms (3.2%)
Total	iast	10.907 s	509.519 ms (4.9%)
Total	profiling	10.862 s	465.0 ms (4.5%)

gantt
    title petclinic - break down per module: candidate=1.47.0-SNAPSHOT~49abe577a1, baseline=1.47.0-SNAPSHOT~ccc22c5138

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (716.775 ms) : 0, 716775
BytebuddyAgent [candidate] (717.866 ms) : 0, 717866
GlobalTracer [baseline] (238.416 ms) : 0, 238416
GlobalTracer [candidate] (239.272 ms) : 0, 239272
AppSec [baseline] (55.475 ms) : 0, 55475
AppSec [candidate] (55.52 ms) : 0, 55520
Remote Config [baseline] (692.566 µs) : 0, 693
Remote Config [candidate] (695.662 µs) : 0, 696
Telemetry [baseline] (12.276 ms) : 0, 12276
Telemetry [candidate] (11.475 ms) : 0, 11475
section appsec
BytebuddyAgent [baseline] (735.359 ms) : 0, 735359
BytebuddyAgent [candidate] (735.119 ms) : 0, 735119
GlobalTracer [baseline] (235.472 ms) : 0, 235472
GlobalTracer [candidate] (236.417 ms) : 0, 236417
IAST [baseline] (21.496 ms) : 0, 21496
IAST [candidate] (21.443 ms) : 0, 21443
AppSec [baseline] (176.887 ms) : 0, 176887
AppSec [candidate] (177.86 ms) : 0, 177860
Remote Config [baseline] (653.33 µs) : 0, 653
Remote Config [candidate] (670.821 µs) : 0, 671
Telemetry [baseline] (8.606 ms) : 0, 8606
Telemetry [candidate] (8.319 ms) : 0, 8319
section iast
BytebuddyAgent [baseline] (835.653 ms) : 0, 835653
BytebuddyAgent [candidate] (838.95 ms) : 0, 838950
GlobalTracer [baseline] (229.752 ms) : 0, 229752
GlobalTracer [candidate] (229.227 ms) : 0, 229227
IAST [baseline] (22.646 ms) : 0, 22646
IAST [candidate] (22.841 ms) : 0, 22841
AppSec [baseline] (56.583 ms) : 0, 56583
AppSec [candidate] (57.123 ms) : 0, 57123
Remote Config [baseline] (607.467 µs) : 0, 607
Remote Config [candidate] (620.308 µs) : 0, 620
Telemetry [baseline] (8.665 ms) : 0, 8665
Telemetry [candidate] (8.701 ms) : 0, 8701
section profiling
BytebuddyAgent [baseline] (709.258 ms) : 0, 709258
BytebuddyAgent [candidate] (707.908 ms) : 0, 707908
GlobalTracer [baseline] (349.193 ms) : 0, 349193
GlobalTracer [candidate] (350.49 ms) : 0, 350490
AppSec [baseline] (55.571 ms) : 0, 55571
AppSec [candidate] (54.378 ms) : 0, 54378
Remote Config [baseline] (673.901 µs) : 0, 674
Remote Config [candidate] (679.412 µs) : 0, 679
Telemetry [baseline] (8.867 ms) : 0, 8867
Telemetry [candidate] (8.961 ms) : 0, 8961
ProfilingAgent [baseline] (96.746 ms) : 0, 96746
ProfilingAgent [candidate] (96.023 ms) : 0, 96023
Profiling [baseline] (96.77 ms) : 0, 96770
Profiling [candidate] (96.048 ms) : 0, 96048

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-02-27T19:42:53	2025-02-27T19:50:41
git_branch	master	nikita-tkachenko/testng-tests-reordering
git_commit_date	1740676228	1740684674
git_commit_sha	`ccc22c5`	`49abe57`
release_version	1.47.0-SNAPSHOT~ccc22c5138	1.47.0-SNAPSHOT~49abe577a1
start_time	2025-02-27T19:42:39	2025-02-27T19:50:27

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1740686242	1740686242
ci_job_id	826737249	826737249
ci_pipeline_id	57266209	57266209
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-ut3kn7uf-project-304-concurrent-0-yxf6puko 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-ut3kn7uf-project-304-concurrent-0-yxf6puko 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 14 metrics, 16 unstable metrics.

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.47.0-SNAPSHOT~49abe577a1, baseline=1.47.0-SNAPSHOT~ccc22c5138
    dateFormat X
    axisFormat %s
section baseline
no_agent (396.161 µs) : 376, 416
.   : milestone, 396,
iast (519.116 µs) : 497, 541
.   : milestone, 519,
iast_FULL (737.343 µs) : 715, 759
.   : milestone, 737,
iast_GLOBAL (565.876 µs) : 543, 588
.   : milestone, 566,
iast_HARDCODED_SECRET_DISABLED (520.88 µs) : 499, 543
.   : milestone, 521,
iast_INACTIVE (473.029 µs) : 451, 495
.   : milestone, 473,
iast_TELEMETRY_OFF (506.401 µs) : 483, 529
.   : milestone, 506,
tracing (462.866 µs) : 442, 484
.   : milestone, 463,
section candidate
no_agent (390.896 µs) : 371, 411
.   : milestone, 391,
iast (518.605 µs) : 497, 541
.   : milestone, 519,
iast_FULL (736.783 µs) : 715, 759
.   : milestone, 737,
iast_GLOBAL (569.837 µs) : 548, 592
.   : milestone, 570,
iast_HARDCODED_SECRET_DISABLED (515.171 µs) : 492, 538
.   : milestone, 515,
iast_INACTIVE (470.502 µs) : 449, 492
.   : milestone, 471,
iast_TELEMETRY_OFF (513.97 µs) : 491, 536
.   : milestone, 514,
tracing (466.053 µs) : 444, 488
.   : milestone, 466,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	396.161 µs [376.151 µs, 416.17 µs]	-
iast	519.116 µs [497.47 µs, 540.763 µs]	122.956 µs (31.0%)
iast_FULL	737.343 µs [715.261 µs, 759.425 µs]	341.183 µs (86.1%)
iast_GLOBAL	565.876 µs [543.299 µs, 588.453 µs]	169.715 µs (42.8%)
iast_HARDCODED_SECRET_DISABLED	520.88 µs [499.108 µs, 542.653 µs]	124.72 µs (31.5%)
iast_INACTIVE	473.029 µs [450.993 µs, 495.065 µs]	76.868 µs (19.4%)
iast_TELEMETRY_OFF	506.401 µs [483.336 µs, 529.466 µs]	110.24 µs (27.8%)
tracing	462.866 µs [441.926 µs, 483.806 µs]	66.705 µs (16.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	390.896 µs [371.14 µs, 410.652 µs]	-
iast	518.605 µs [496.706 µs, 540.503 µs]	127.709 µs (32.7%)
iast_FULL	736.783 µs [714.818 µs, 758.749 µs]	345.888 µs (88.5%)
iast_GLOBAL	569.837 µs [547.6 µs, 592.073 µs]	178.941 µs (45.8%)
iast_HARDCODED_SECRET_DISABLED	515.171 µs [492.406 µs, 537.937 µs]	124.276 µs (31.8%)
iast_INACTIVE	470.502 µs [448.997 µs, 492.006 µs]	79.606 µs (20.4%)
iast_TELEMETRY_OFF	513.97 µs [491.456 µs, 536.484 µs]	123.074 µs (31.5%)
tracing	466.053 µs [444.496 µs, 487.61 µs]	75.157 µs (19.2%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.47.0-SNAPSHOT~49abe577a1, baseline=1.47.0-SNAPSHOT~ccc22c5138
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.374 ms) : 1354, 1394
.   : milestone, 1374,
appsec (1.769 ms) : 1746, 1792
.   : milestone, 1769,
appsec_no_iast (1.746 ms) : 1721, 1771
.   : milestone, 1746,
code_origins (1.691 ms) : 1658, 1724
.   : milestone, 1691,
iast (1.521 ms) : 1497, 1545
.   : milestone, 1521,
profiling (1.525 ms) : 1502, 1549
.   : milestone, 1525,
tracing (1.508 ms) : 1483, 1532
.   : milestone, 1508,
section candidate
no_agent (1.367 ms) : 1348, 1387
.   : milestone, 1367,
appsec (1.745 ms) : 1722, 1769
.   : milestone, 1745,
appsec_no_iast (1.756 ms) : 1731, 1780
.   : milestone, 1756,
code_origins (1.703 ms) : 1669, 1736
.   : milestone, 1703,
iast (1.518 ms) : 1493, 1542
.   : milestone, 1518,
profiling (1.572 ms) : 1547, 1597
.   : milestone, 1572,
tracing (1.493 ms) : 1468, 1519
.   : milestone, 1493,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.374 ms [1.354 ms, 1.394 ms]	-
appsec	1.769 ms [1.746 ms, 1.792 ms]	395.175 µs (28.8%)
appsec_no_iast	1.746 ms [1.721 ms, 1.771 ms]	372.274 µs (27.1%)
code_origins	1.691 ms [1.658 ms, 1.724 ms]	317.187 µs (23.1%)
iast	1.521 ms [1.497 ms, 1.545 ms]	147.384 µs (10.7%)
profiling	1.525 ms [1.502 ms, 1.549 ms]	151.108 µs (11.0%)
tracing	1.508 ms [1.483 ms, 1.532 ms]	133.625 µs (9.7%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.367 ms [1.348 ms, 1.387 ms]	-
appsec	1.745 ms [1.722 ms, 1.769 ms]	378.011 µs (27.6%)
appsec_no_iast	1.756 ms [1.731 ms, 1.78 ms]	388.359 µs (28.4%)
code_origins	1.703 ms [1.669 ms, 1.736 ms]	335.654 µs (24.6%)
iast	1.518 ms [1.493 ms, 1.542 ms]	150.355 µs (11.0%)
profiling	1.572 ms [1.547 ms, 1.597 ms]	205.149 µs (15.0%)
tracing	1.493 ms [1.468 ms, 1.519 ms]	125.919 µs (9.2%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/testng-tests-reordering
git_commit_date	1740676228	1740684674
git_commit_sha	`ccc22c5`	`49abe57`
release_version	1.47.0-SNAPSHOT~ccc22c5138	1.47.0-SNAPSHOT~49abe577a1

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1740686708	1740686708
ci_job_id	826737251	826737251
ci_pipeline_id	57266209	57266209
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner--ogofyt-project-304-concurrent-1-9yp63aqq 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner--ogofyt-project-304-concurrent-1-9yp63aqq 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.47.0-SNAPSHOT~49abe577a1, baseline=1.47.0-SNAPSHOT~ccc22c5138
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.466 ms) : 1455, 1478
.   : milestone, 1466,
appsec (2.325 ms) : 2282, 2368
.   : milestone, 2325,
iast (2.097 ms) : 2043, 2152
.   : milestone, 2097,
iast_GLOBAL (2.142 ms) : 2088, 2197
.   : milestone, 2142,
profiling (1.961 ms) : 1918, 2005
.   : milestone, 1961,
tracing (1.944 ms) : 1901, 1986
.   : milestone, 1944,
section candidate
no_agent (1.467 ms) : 1455, 1478
.   : milestone, 1467,
appsec (2.329 ms) : 2286, 2373
.   : milestone, 2329,
iast (2.107 ms) : 2052, 2162
.   : milestone, 2107,
iast_GLOBAL (2.143 ms) : 2088, 2198
.   : milestone, 2143,
profiling (1.959 ms) : 1915, 2004
.   : milestone, 1959,
tracing (1.932 ms) : 1890, 1973
.   : milestone, 1932,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.466 ms [1.455 ms, 1.478 ms]	-
appsec	2.325 ms [2.282 ms, 2.368 ms]	858.507 µs (58.5%)
iast	2.097 ms [2.043 ms, 2.152 ms]	630.919 µs (43.0%)
iast_GLOBAL	2.142 ms [2.088 ms, 2.197 ms]	676.104 µs (46.1%)
profiling	1.961 ms [1.918 ms, 2.005 ms]	495.043 µs (33.8%)
tracing	1.944 ms [1.901 ms, 1.986 ms]	477.127 µs (32.5%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.467 ms [1.455 ms, 1.478 ms]	-
appsec	2.329 ms [2.286 ms, 2.373 ms]	862.216 µs (58.8%)
iast	2.107 ms [2.052 ms, 2.162 ms]	640.052 µs (43.6%)
iast_GLOBAL	2.143 ms [2.088 ms, 2.198 ms]	676.406 µs (46.1%)
profiling	1.959 ms [1.915 ms, 2.004 ms]	492.577 µs (33.6%)
tracing	1.932 ms [1.89 ms, 1.973 ms]	464.989 µs (31.7%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.47.0-SNAPSHOT~49abe577a1, baseline=1.47.0-SNAPSHOT~ccc22c5138
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.358 s) : 15358000, 15358000
.   : milestone, 15358000,
appsec (15.018 s) : 15018000, 15018000
.   : milestone, 15018000,
iast (18.278 s) : 18278000, 18278000
.   : milestone, 18278000,
iast_GLOBAL (18.151 s) : 18151000, 18151000
.   : milestone, 18151000,
profiling (15.523 s) : 15523000, 15523000
.   : milestone, 15523000,
tracing (14.927 s) : 14927000, 14927000
.   : milestone, 14927000,
section candidate
no_agent (14.927 s) : 14927000, 14927000
.   : milestone, 14927000,
appsec (14.972 s) : 14972000, 14972000
.   : milestone, 14972000,
iast (18.944 s) : 18944000, 18944000
.   : milestone, 18944000,
iast_GLOBAL (17.75 s) : 17750000, 17750000
.   : milestone, 17750000,
profiling (15.051 s) : 15051000, 15051000
.   : milestone, 15051000,
tracing (15.063 s) : 15063000, 15063000
.   : milestone, 15063000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.358 s [15.358 s, 15.358 s]	-
appsec	15.018 s [15.018 s, 15.018 s]	-340.0 ms (-2.2%)
iast	18.278 s [18.278 s, 18.278 s]	2.92 s (19.0%)
iast_GLOBAL	18.151 s [18.151 s, 18.151 s]	2.793 s (18.2%)
profiling	15.523 s [15.523 s, 15.523 s]	165.0 ms (1.1%)
tracing	14.927 s [14.927 s, 14.927 s]	-431.0 ms (-2.8%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.927 s [14.927 s, 14.927 s]	-
appsec	14.972 s [14.972 s, 14.972 s]	45.0 ms (0.3%)
iast	18.944 s [18.944 s, 18.944 s]	4.017 s (26.9%)
iast_GLOBAL	17.75 s [17.75 s, 17.75 s]	2.823 s (18.9%)
profiling	15.051 s [15.051 s, 15.051 s]	124.0 ms (0.8%)
tracing	15.063 s [15.063 s, 15.063 s]	136.0 ms (0.9%)

daniel-mohedano

LGTM, nice refactoring in CiVisibilityInstrumentationTest btw, it does look much cleaner now

| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | [com.datadoghq:dd-trace-api](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.46.1` -> `1.47.0` | | [com.datadoghq:dd-trace-ot](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.46.1` -> `1.47.0` | | [software.amazon.awssdk:sdk-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:sqs](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:dynamodb-enhanced](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:dynamodb](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:aws-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:bom](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:auth](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | --- ### Release Notes <details> <summary>datadog/dd-trace-java (com.datadoghq:dd-trace-api)</summary> ### [`v1.47.0`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.47.0): 1.47.0 ##### Components ##### Application Security Management (IAST) - 🐛 Exclude com.stripe.net.HttpURLConnectionClient to solve IAST SSRF vulnerability false positives ([#8483](DataDog/dd-trace-java#8483) - [@jandro996](https://github.com/jandro996)) - 🐛 Add exclusion to solve IAST weak randomness vulnerability false positives ([#8462](DataDog/dd-trace-java#8462) - [@jandro996](https://github.com/jandro996)) - ✨ Fix weak randomness false positive in Kafka client ([#8408](DataDog/dd-trace-java#8408) - [@smola](https://github.com/smola)) - ✨ Fix location for SSRF with Kong Unirest ([#8407](DataDog/dd-trace-java#8407) - [@smola](https://github.com/smola)) - ✨ Exclude IBM Instana from IAST ([#8406](DataDog/dd-trace-java#8406) - [@smola](https://github.com/smola)) - 🐛 Fix org.json iast instrumentation test for latest dependency ([#8347](DataDog/dd-trace-java#8347) - [@jandro996](https://github.com/jandro996)) - ✨ Configuration to Disable APM Tracing ([#8219](DataDog/dd-trace-java#8219) - [@jandro996](https://github.com/jandro996)) - ✨ Address cookie vulnerability cardinality issues ([#8210](DataDog/dd-trace-java#8210) - [@jandro996](https://github.com/jandro996)) - ✨ Email HTML Injection detection in IAST ([#8205](DataDog/dd-trace-java#8205) - [@sezen-datadog](https://github.com/sezen-datadog)) ##### Application Security Management (WAF) - 🐛✨ Ensure usr.exists tag is not overridden when UsernameNotFoundException is thrown ([#8376](DataDog/dd-trace-java#8376) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - 🐛✨ Ensure usr.exists tag is not overridden by auto instrumentation ([#8374](DataDog/dd-trace-java#8374) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Update appsec metrics with event_rules_version tag ([#8354](DataDog/dd-trace-java#8354) - [@sezen-datadog](https://github.com/sezen-datadog)) - ✨ Update metrics: appsec.waf.requests ([#8353](DataDog/dd-trace-java#8353) - [@Mariovido](https://github.com/Mariovido)) - ✨ Improve ASM support in vert.x 5.0 ([#8285](DataDog/dd-trace-java#8285) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Update metrics: appsec.waf.updates and appsec.waf.init ([#8280](DataDog/dd-trace-java#8280) - [@Mariovido](https://github.com/Mariovido)) - ✨ Configuration to Disable APM Tracing ([#8219](DataDog/dd-trace-java#8219) - [@jandro996](https://github.com/jandro996)) ##### Build & Tooling - 🐛 Do not generate Muzzle references for primitive arrays in method body ([#8361](DataDog/dd-trace-java#8361) - [@amarziali](https://github.com/amarziali)) - 📖 Improve dev env setup documentation for Windows ([#8180](DataDog/dd-trace-java#8180) - [@lucaspimentel](https://github.com/lucaspimentel)) ##### Continuous Integration Visibility - ✨ Add support for skip-EFD tagging ([#8487](DataDog/dd-trace-java#8487) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Fix an NPE in Gradle Android instrumentation ([#8484](DataDog/dd-trace-java#8484) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Consider modified tests when applying fail-fast tests ordering ([#8474](DataDog/dd-trace-java#8474) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Implement tests reordering for TestNG ([#8467](DataDog/dd-trace-java#8467) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Fix Gradle Launcher instrumentation to not interfere with Gradle Test Kit ([#8465](DataDog/dd-trace-java#8465) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Use separate TestEventHandlers per framework in CI Vis instrumentations ([#8451](DataDog/dd-trace-java#8451) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Remove warning log when JUnit 4 test method cannot be retrieved ([#8445](DataDog/dd-trace-java#8445) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Fix Scalatest tracing for tests that are reported asynchronously ([#8444](DataDog/dd-trace-java#8444) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Implement attempt to fix tests ([#8393](DataDog/dd-trace-java#8393) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Implement test disabling ([#8377](DataDog/dd-trace-java#8377) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Update CODEOWNERS parser to not log errors on comments with leading whitespace ([#8349](DataDog/dd-trace-java#8349) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Request Test Management tests list ([#8345](DataDog/dd-trace-java#8345) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Receive test management settings from CIVis settings request ([#8331](DataDog/dd-trace-java#8331) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Implement quarantined tests tagging ([#8326](DataDog/dd-trace-java#8326) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Implement tests quarantining ([#8320](DataDog/dd-trace-java#8320) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Add tag to specify if the user is setting DD_SERVICE ([#8318](DataDog/dd-trace-java#8318) - [@daniel-mohedano](https://github.com/daniel-mohedano)) ##### Crash tracking - ✨ Only fork jps when required ([#8419](DataDog/dd-trace-java#8419) - [@mcculls](https://github.com/mcculls)) - 🐛 Use Java home of the crashed process to launch crash uploader ([#8348](DataDog/dd-trace-java#8348) - [@jbachorik](https://github.com/jbachorik)) ##### Data Streams Monitoring - 🐛 Fix error happening when sqs message attributes are readonly ([#8473](DataDog/dd-trace-java#8473) - [@vandonr](https://github.com/vandonr)) - 🐛 Fix bug on proto schema extraction ([#8403](DataDog/dd-trace-java#8403) - [@vandonr](https://github.com/vandonr)) - 🐛 Fix service name overrides in consumers ([#8387](DataDog/dd-trace-java#8387) - [@piochelepiotr](https://github.com/piochelepiotr)) ##### Database Monitoring - ✨ Add DBMTracePreparedStatements to tracer configuration log ([#8508](DataDog/dd-trace-java#8508) - [@cecile75](https://github.com/cecile75)) ##### Dynamic Instrumentation - ✨ Look in another location for grpc service methods ([#8468](DataDog/dd-trace-java#8468) - [@evanchooly](https://github.com/evanchooly)) - 🐛 Fix Exception Replay with Lambda proxy classes ([#8452](DataDog/dd-trace-java#8452) - [@jpbempel](https://github.com/jpbempel)) - ✨ Add code origin support for spring-webmvc ([#8416](DataDog/dd-trace-java#8416) - [@evanchooly](https://github.com/evanchooly)) - ✨ Add support for scanning jar from loaded class ([#8370](DataDog/dd-trace-java#8370) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Disable capture of entry values ([#8369](DataDog/dd-trace-java#8369) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Fix CodeOrigin for `@Trace` annotation ([#8344](DataDog/dd-trace-java#8344) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Fix equals/hashCode for CodeOrigin probe ([#8319](DataDog/dd-trace-java#8319) - [@jpbempel](https://github.com/jpbempel)) - ✨ Add code origin support to kafka message listeners ([#8301](DataDog/dd-trace-java#8301) - [@evanchooly](https://github.com/evanchooly)) ##### Metrics - ✨ Create metric: appsec.waf.error ([#8381](DataDog/dd-trace-java#8381) - [@sezen-datadog](https://github.com/sezen-datadog)) - ✨ Create metric: appsec.rasp.error ([#8364](DataDog/dd-trace-java#8364) - [@sezen-datadog](https://github.com/sezen-datadog)) ##### Profiling - ✨ Bump ddprof library to 1.22.0 ([#8463](DataDog/dd-trace-java#8463) - [@jbachorik](https://github.com/jbachorik)) - IBM J9 8u361 corresponds to OpenJDK 8u362 by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#187 - Fix compatibility with musl libc 1.2.4 by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#189 - Modify version extraction by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#179 - Do not write null values to jvminfo event by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#184 - Productize VMStructs-based stack walker by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#177 - A few minor downport issues by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#180 - Enable ASGCT by default on fairly safe J9 JDK versions by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#181 - 🐛 Exclude OrderedThreadPoolExecutor from queue-time measurements ([#8456](DataDog/dd-trace-java#8456) - [@jbachorik](https://github.com/jbachorik)) - ✨ Record JVM info on JVMs without JFR ([#8431](DataDog/dd-trace-java#8431) - [@jbachorik](https://github.com/jbachorik)) - 🐛 Actually use CleanupTask in TempLocationManager ([#8420](DataDog/dd-trace-java#8420) - [@mcculls](https://github.com/mcculls)) - ✨ Only fork jps when required ([#8419](DataDog/dd-trace-java#8419) - [@mcculls](https://github.com/mcculls)) - 🐛 Adjust JFR checks for J9 ([#8405](DataDog/dd-trace-java#8405) - [@jbachorik](https://github.com/jbachorik)) - 🧹 Disable smap RSS parsing by default ([#8342](DataDog/dd-trace-java#8342) - [@MattAlp](https://github.com/MattAlp)) ##### Telemetry - 🐛 Add support for JBoss jar:file format to DependencyResolver ([#8428](DataDog/dd-trace-java#8428) - [@jandro996](https://github.com/jandro996)) - ✨ Update metrics: appsec.waf.requests ([#8353](DataDog/dd-trace-java#8353) - [@Mariovido](https://github.com/Mariovido)) ##### Trace context propagation - ✨ Introduce tracing propagator ([#8313](DataDog/dd-trace-java#8313) - [@PerfectSlayer](https://github.com/PerfectSlayer)) ##### Tracer core - 🐛 Fix Stable Config telemetry source names ([#8460](DataDog/dd-trace-java#8460) - [@BaptisteFoy](https://github.com/BaptisteFoy)) - ✨ Probe trace endpoints with a valid payload of empty arrays ([#8414](DataDog/dd-trace-java#8414) - [@mcculls](https://github.com/mcculls)) - ✨ Add 1 minute fail-safe to JUL/JMX class-loading callback ([#8399](DataDog/dd-trace-java#8399) - [@mcculls](https://github.com/mcculls)) - ✨ Migrate DSM injection calls to context-first APIs ([#8383](DataDog/dd-trace-java#8383) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - 🧹 Move continuation capture methods from scope to tracer ([#8371](DataDog/dd-trace-java#8371) - [@mcculls](https://github.com/mcculls)) - ✨ Migrate context extraction calls to context-first APIs ([#8368](DataDog/dd-trace-java#8368) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - 🧹 Migrate context injection calls to context-first APIs ([#8358](DataDog/dd-trace-java#8358) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - 💡 Support reading configurations from files ([#8338](DataDog/dd-trace-java#8338) - [@mtoffl01](https://github.com/mtoffl01)) - 💡 Implementation of BaggagePropagator and BaggageContext ([#8330](DataDog/dd-trace-java#8330) - [@mhlidd](https://github.com/mhlidd)) - 🧹 Combine continuation implementations into one which supports multiple activations ([#8324](DataDog/dd-trace-java#8324) - [@mcculls](https://github.com/mcculls)) - ✨ Introduce tracing propagator ([#8313](DataDog/dd-trace-java#8313) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ✨ Remove old context propagation API ([#8271](DataDog/dd-trace-java#8271) - [@PerfectSlayer](https://github.com/PerfectSlayer)) ##### Instrumentations ##### AWS Lambda instrumentation - 🐛 Send error message and stack to Lambda extension ([#8417](DataDog/dd-trace-java#8417) - [@nhulston](https://github.com/nhulston)) ##### AWS SDK instrumentation - 🐛 Fix error happening when sqs message attributes are readonly ([#8473](DataDog/dd-trace-java#8473) - [@vandonr](https://github.com/vandonr)) - 💡 Inject trace context into AWS Step Functions input ([#7585](DataDog/dd-trace-java#7585) - [@DylanLovesCoffee](https://github.com/DylanLovesCoffee)) ##### Core Java language instrumentation - ✨ Look in another location for grpc service methods ([#8468](DataDog/dd-trace-java#8468) - [@evanchooly](https://github.com/evanchooly)) - ✨ Add code origin support for spring-webmvc ([#8416](DataDog/dd-trace-java#8416) - [@evanchooly](https://github.com/evanchooly)) - 💡 Implementation of BaggagePropagator and BaggageContext ([#8330](DataDog/dd-trace-java#8330) - [@mhlidd](https://github.com/mhlidd)) - ✨ Add code origin support to kafka message listeners ([#8301](DataDog/dd-trace-java#8301) - [@evanchooly](https://github.com/evanchooly)) ##### gRPC instrumentation - ✨ Look in another location for grpc service methods ([#8468](DataDog/dd-trace-java#8468) - [@evanchooly](https://github.com/evanchooly)) ##### Kafka instrumentation - ✨ Add messaging.destination.name tag to kafka integrations ([#8366](DataDog/dd-trace-java#8366) - [@rarguelloF](https://github.com/rarguelloF)) ##### Protocol Buffer instrumentation - 🐛 Fix bug on proto schema extraction ([#8403](DataDog/dd-trace-java#8403) - [@vandonr](https://github.com/vandonr)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: 108a0f86aa59ab4c938cbac0688dd4c19cb301fa

Implement tests reordering for TestNG

49abe57

nikita-tkachenko-datadog added type: enhancement comp: ci visibility Continuous Integration Visibility labels Feb 27, 2025

nikita-tkachenko-datadog marked this pull request as ready for review February 27, 2025 20:23

nikita-tkachenko-datadog requested a review from a team as a code owner February 27, 2025 20:23

nikita-tkachenko-datadog requested review from juan-fernandez and daniel-mohedano February 27, 2025 20:23

daniel-mohedano approved these changes Feb 28, 2025

View reviewed changes

nikita-tkachenko-datadog merged commit 5328e2b into master Feb 28, 2025
220 of 221 checks passed

nikita-tkachenko-datadog deleted the nikita-tkachenko/testng-tests-reordering branch February 28, 2025 10:44

github-actions bot added this to the 1.47.0 milestone Feb 28, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement tests reordering for TestNG #8467

Implement tests reordering for TestNG #8467

nikita-tkachenko-datadog commented Feb 27, 2025 •

edited by jira bot

Loading

pr-commenter bot commented Feb 27, 2025

daniel-mohedano left a comment

Implement tests reordering for TestNG #8467

Implement tests reordering for TestNG #8467

Conversation

nikita-tkachenko-datadog commented Feb 27, 2025 • edited by jira bot Loading

What Does This Do

Contributor Checklist

pr-commenter bot commented Feb 27, 2025

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

daniel-mohedano left a comment

Choose a reason for hiding this comment

nikita-tkachenko-datadog commented Feb 27, 2025 •

edited by jira bot

Loading