Perf release gate #9068

bric3 · 2025-06-30T16:59:23Z

What Does This Do

Check then notify if the release do not meet expected thresholds.

Motivation

Ensure performance thresholds.

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

pr-commenter · 2025-06-30T17:33:00Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	bdu/r-gate
git_commit_date	1752761628	1752763864
git_commit_sha	`d1794c1`	`df915f0`
release_version	1.52.0-SNAPSHOT~d1794c17af	1.52.0-SNAPSHOT~df915f0a03

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1752765625	1752765625
ci_job_id	1034135474	1034135474
ci_pipeline_id	70870999	70870999
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-j5jmyhp9 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-j5jmyhp9 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 49 metrics, 10 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.52.0-SNAPSHOT~df915f0a03, baseline=1.52.0-SNAPSHOT~d1794c17af

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.043 s) : 0, 1043390
Total [baseline] (10.692 s) : 0, 10692369
Agent [candidate] (1.04 s) : 0, 1039928
Total [candidate] (10.702 s) : 0, 10702188
section appsec
Agent [baseline] (1.214 s) : 0, 1213686
Total [baseline] (10.824 s) : 0, 10823993
Agent [candidate] (1.216 s) : 0, 1216395
Total [candidate] (10.77 s) : 0, 10770023
section iast
Agent [baseline] (1.186 s) : 0, 1185650
Total [baseline] (10.897 s) : 0, 10896987
Agent [candidate] (1.171 s) : 0, 1171480
Total [candidate] (10.982 s) : 0, 10981897
section profiling
Agent [baseline] (1.184 s) : 0, 1183973
Total [baseline] (10.93 s) : 0, 10929792
Agent [candidate] (1.184 s) : 0, 1183632
Total [candidate] (10.937 s) : 0, 10936649

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.043 s	-
Agent	appsec	1.214 s	170.296 ms (16.3%)
Agent	iast	1.186 s	142.26 ms (13.6%)
Agent	profiling	1.184 s	140.582 ms (13.5%)
Total	tracing	10.692 s	-
Total	appsec	10.824 s	131.623 ms (1.2%)
Total	iast	10.897 s	204.618 ms (1.9%)
Total	profiling	10.93 s	237.423 ms (2.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.04 s	-
Agent	appsec	1.216 s	176.467 ms (17.0%)
Agent	iast	1.171 s	131.551 ms (12.7%)
Agent	profiling	1.184 s	143.703 ms (13.8%)
Total	tracing	10.702 s	-
Total	appsec	10.77 s	67.836 ms (0.6%)
Total	iast	10.982 s	279.709 ms (2.6%)
Total	profiling	10.937 s	234.461 ms (2.2%)

gantt
    title petclinic - break down per module: candidate=1.52.0-SNAPSHOT~df915f0a03, baseline=1.52.0-SNAPSHOT~d1794c17af

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.438 ms) : 0, 1438
crashtracking [candidate] (1.428 ms) : 0, 1428
BytebuddyAgent [baseline] (733.131 ms) : 0, 733131
BytebuddyAgent [candidate] (730.932 ms) : 0, 730932
GlobalTracer [baseline] (242.267 ms) : 0, 242267
GlobalTracer [candidate] (241.265 ms) : 0, 241265
AppSec [baseline] (30.577 ms) : 0, 30577
AppSec [candidate] (30.506 ms) : 0, 30506
Debugger [baseline] (6.031 ms) : 0, 6031
Debugger [candidate] (6.016 ms) : 0, 6016
Remote Config [baseline] (662.859 µs) : 0, 663
Remote Config [candidate] (649.928 µs) : 0, 650
Telemetry [baseline] (8.247 ms) : 0, 8247
Telemetry [candidate] (8.192 ms) : 0, 8192
section appsec
crashtracking [baseline] (1.439 ms) : 0, 1439
crashtracking [candidate] (1.434 ms) : 0, 1434
BytebuddyAgent [baseline] (748.779 ms) : 0, 748779
BytebuddyAgent [candidate] (750.853 ms) : 0, 750853
GlobalTracer [baseline] (233.218 ms) : 0, 233218
GlobalTracer [candidate] (234.472 ms) : 0, 234472
AppSec [baseline] (166.829 ms) : 0, 166829
AppSec [candidate] (168.234 ms) : 0, 168234
Debugger [baseline] (9.45 ms) : 0, 9450
Debugger [candidate] (7.164 ms) : 0, 7164
Remote Config [baseline] (612.387 µs) : 0, 612
Remote Config [candidate] (607.646 µs) : 0, 608
Telemetry [baseline] (8.895 ms) : 0, 8895
Telemetry [candidate] (8.794 ms) : 0, 8794
IAST [baseline] (23.3 ms) : 0, 23300
IAST [candidate] (23.7 ms) : 0, 23700
section iast
crashtracking [baseline] (1.452 ms) : 0, 1452
crashtracking [candidate] (1.427 ms) : 0, 1427
BytebuddyAgent [baseline] (856.305 ms) : 0, 856305
BytebuddyAgent [candidate] (845.804 ms) : 0, 845804
GlobalTracer [baseline] (234.801 ms) : 0, 234801
GlobalTracer [candidate] (230.774 ms) : 0, 230774
AppSec [baseline] (28.988 ms) : 0, 28988
AppSec [candidate] (27.562 ms) : 0, 27562
Debugger [baseline] (6.633 ms) : 0, 6633
Debugger [candidate] (6.737 ms) : 0, 6737
Remote Config [baseline] (594.273 µs) : 0, 594
Remote Config [candidate] (597.083 µs) : 0, 597
Telemetry [baseline] (8.025 ms) : 0, 8025
Telemetry [candidate] (7.921 ms) : 0, 7921
IAST [baseline] (27.734 ms) : 0, 27734
IAST [candidate] (29.827 ms) : 0, 29827
section profiling
crashtracking [baseline] (1.428 ms) : 0, 1428
crashtracking [candidate] (1.409 ms) : 0, 1409
BytebuddyAgent [baseline] (759.06 ms) : 0, 759060
BytebuddyAgent [candidate] (759.498 ms) : 0, 759498
GlobalTracer [baseline] (220.833 ms) : 0, 220833
GlobalTracer [candidate] (220.32 ms) : 0, 220320
AppSec [baseline] (30.4 ms) : 0, 30400
AppSec [candidate] (30.339 ms) : 0, 30339
Debugger [baseline] (6.293 ms) : 0, 6293
Debugger [candidate] (6.322 ms) : 0, 6322
Remote Config [baseline] (700.313 µs) : 0, 700
Remote Config [candidate] (675.378 µs) : 0, 675
Telemetry [baseline] (10.974 ms) : 0, 10974
Telemetry [candidate] (12.394 ms) : 0, 12394
ProfilingAgent [baseline] (106.018 ms) : 0, 106018
ProfilingAgent [candidate] (104.396 ms) : 0, 104396
Profiling [baseline] (106.04 ms) : 0, 106040
Profiling [candidate] (104.418 ms) : 0, 104418

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.52.0-SNAPSHOT~df915f0a03, baseline=1.52.0-SNAPSHOT~d1794c17af

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.042 s) : 0, 1041547
Total [baseline] (8.636 s) : 0, 8635628
Agent [candidate] (1.037 s) : 0, 1037324
Total [candidate] (8.602 s) : 0, 8601976
section iast
Agent [baseline] (1.171 s) : 0, 1171232
Total [baseline] (9.282 s) : 0, 9282471
Agent [candidate] (1.19 s) : 0, 1190343
Total [candidate] (9.35 s) : 0, 9349969

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.042 s	-
Agent	iast	1.171 s	129.685 ms (12.5%)
Total	tracing	8.636 s	-
Total	iast	9.282 s	646.843 ms (7.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.037 s	-
Agent	iast	1.19 s	153.02 ms (14.8%)
Total	tracing	8.602 s	-
Total	iast	9.35 s	747.993 ms (8.7%)

gantt
    title insecure-bank - break down per module: candidate=1.52.0-SNAPSHOT~df915f0a03, baseline=1.52.0-SNAPSHOT~d1794c17af

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.43 ms) : 0, 1430
crashtracking [candidate] (1.432 ms) : 0, 1432
BytebuddyAgent [baseline] (731.047 ms) : 0, 731047
BytebuddyAgent [candidate] (728.97 ms) : 0, 728970
GlobalTracer [baseline] (242.523 ms) : 0, 242523
GlobalTracer [candidate] (240.887 ms) : 0, 240887
AppSec [baseline] (30.641 ms) : 0, 30641
AppSec [candidate] (30.305 ms) : 0, 30305
Debugger [baseline] (6.037 ms) : 0, 6037
Debugger [candidate] (5.989 ms) : 0, 5989
Remote Config [baseline] (657.752 µs) : 0, 658
Remote Config [candidate] (654.164 µs) : 0, 654
Telemetry [baseline] (8.3 ms) : 0, 8300
Telemetry [candidate] (8.183 ms) : 0, 8183
section iast
crashtracking [baseline] (1.425 ms) : 0, 1425
crashtracking [candidate] (1.453 ms) : 0, 1453
BytebuddyAgent [baseline] (845.701 ms) : 0, 845701
BytebuddyAgent [candidate] (861.146 ms) : 0, 861146
GlobalTracer [baseline] (230.673 ms) : 0, 230673
GlobalTracer [candidate] (233.319 ms) : 0, 233319
AppSec [baseline] (28.144 ms) : 0, 28144
AppSec [candidate] (27.675 ms) : 0, 27675
Debugger [baseline] (8.328 ms) : 0, 8328
Debugger [candidate] (6.75 ms) : 0, 6750
Remote Config [baseline] (589.455 µs) : 0, 589
Remote Config [candidate] (615.116 µs) : 0, 615
Telemetry [baseline] (7.962 ms) : 0, 7962
Telemetry [candidate] (8.071 ms) : 0, 8071
IAST [baseline] (27.497 ms) : 0, 27497
IAST [candidate] (30.198 ms) : 0, 30198

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	bdu/r-gate
git_commit_date	1752761628	1752763864
git_commit_sha	`d1794c1`	`df915f0`
release_version	1.52.0-SNAPSHOT~d1794c17af	1.52.0-SNAPSHOT~df915f0a03

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1752765370	1752765370
ci_job_id	1034135475	1034135475
ci_pipeline_id	70870999	70870999
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-hrg2yowu 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-hrg2yowu 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 2 performance regressions! Performance is the same for 9 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:tracing:high_load	worse [+334.443µs; +587.842µs] or [+4.381%; +7.700%]	unstable [-109.476op/s; +41.038op/s] or [-18.043%; +6.764%]	8.095ms	572.531op/s	7.634ms	606.750op/s
scenario:load:insecure-bank:iast:high_load	worse [+304.279µs; +637.376µs] or [+3.341%; +6.998%]	unstable [-83.520op/s; +33.833op/s] or [-16.392%; +6.640%]	9.579ms	484.688op/s	9.109ms	509.531op/s
scenario:load:petclinic:appsec:high_load	better [-4.214ms; -3.277ms] or [-8.415%; -6.545%]	unstable [+0.324op/s; +14.776op/s] or [+0.347%; +15.801%]	46.325ms	101.062op/s	50.070ms	93.513op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.52.0-SNAPSHOT~df915f0a03, baseline=1.52.0-SNAPSHOT~d1794c17af
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.452 ms) : 4391, 4512
.   : milestone, 4452,
iast (9.109 ms) : 8960, 9257
.   : milestone, 9109,
iast_FULL (14.038 ms) : 13761, 14315
.   : milestone, 14038,
iast_GLOBAL (10.033 ms) : 9840, 10227
.   : milestone, 10033,
profiling (8.733 ms) : 8583, 8884
.   : milestone, 8733,
tracing (7.634 ms) : 7518, 7750
.   : milestone, 7634,
section candidate
no_agent (4.462 ms) : 4412, 4512
.   : milestone, 4462,
iast (9.579 ms) : 9419, 9740
.   : milestone, 9579,
iast_FULL (13.78 ms) : 13505, 14055
.   : milestone, 13780,
iast_GLOBAL (10.197 ms) : 10015, 10379
.   : milestone, 10197,
profiling (9.002 ms) : 8862, 9143
.   : milestone, 9002,
tracing (8.095 ms) : 7976, 8215
.   : milestone, 8095,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.452 ms [4.391 ms, 4.512 ms]	-
iast	9.109 ms [8.96 ms, 9.257 ms]	4.657 ms (104.6%)
iast_FULL	14.038 ms [13.761 ms, 14.315 ms]	9.587 ms (215.4%)
iast_GLOBAL	10.033 ms [9.84 ms, 10.227 ms]	5.582 ms (125.4%)
profiling	8.733 ms [8.583 ms, 8.884 ms]	4.282 ms (96.2%)
tracing	7.634 ms [7.518 ms, 7.75 ms]	3.183 ms (71.5%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.462 ms [4.412 ms, 4.512 ms]	-
iast	9.579 ms [9.419 ms, 9.74 ms]	5.118 ms (114.7%)
iast_FULL	13.78 ms [13.505 ms, 14.055 ms]	9.318 ms (208.8%)
iast_GLOBAL	10.197 ms [10.015 ms, 10.379 ms]	5.735 ms (128.5%)
profiling	9.002 ms [8.862 ms, 9.143 ms]	4.541 ms (101.8%)
tracing	8.095 ms [7.976 ms, 8.215 ms]	3.633 ms (81.4%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.52.0-SNAPSHOT~df915f0a03, baseline=1.52.0-SNAPSHOT~d1794c17af
    dateFormat X
    axisFormat %s
section baseline
no_agent (37.981 ms) : 37667, 38296
.   : milestone, 37981,
appsec (50.07 ms) : 49607, 50534
.   : milestone, 50070,
code_origins (44.599 ms) : 44209, 44989
.   : milestone, 44599,
iast (44.459 ms) : 44079, 44839
.   : milestone, 44459,
profiling (47.137 ms) : 46689, 47586
.   : milestone, 47137,
tracing (44.719 ms) : 44338, 45100
.   : milestone, 44719,
section candidate
no_agent (37.54 ms) : 37237, 37844
.   : milestone, 37540,
appsec (46.325 ms) : 45920, 46730
.   : milestone, 46325,
code_origins (44.838 ms) : 44464, 45213
.   : milestone, 44838,
iast (44.708 ms) : 44315, 45101
.   : milestone, 44708,
profiling (47.536 ms) : 47083, 47990
.   : milestone, 47536,
tracing (44.137 ms) : 43765, 44508
.   : milestone, 44137,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	37.981 ms [37.667 ms, 38.296 ms]	-
appsec	50.07 ms [49.607 ms, 50.534 ms]	12.089 ms (31.8%)
code_origins	44.599 ms [44.209 ms, 44.989 ms]	6.618 ms (17.4%)
iast	44.459 ms [44.079 ms, 44.839 ms]	6.477 ms (17.1%)
profiling	47.137 ms [46.689 ms, 47.586 ms]	9.156 ms (24.1%)
tracing	44.719 ms [44.338 ms, 45.1 ms]	6.738 ms (17.7%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	37.54 ms [37.237 ms, 37.844 ms]	-
appsec	46.325 ms [45.92 ms, 46.73 ms]	8.785 ms (23.4%)
code_origins	44.838 ms [44.464 ms, 45.213 ms]	7.298 ms (19.4%)
iast	44.708 ms [44.315 ms, 45.101 ms]	7.168 ms (19.1%)
profiling	47.536 ms [47.083 ms, 47.99 ms]	9.996 ms (26.6%)
tracing	44.137 ms [43.765 ms, 44.508 ms]	6.597 ms (17.6%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	bdu/r-gate
git_commit_date	1752761628	1752763864
git_commit_sha	`d1794c1`	`df915f0`
release_version	1.52.0-SNAPSHOT~d1794c17af	1.52.0-SNAPSHOT~df915f0a03

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1752765790	1752765790
ci_job_id	1034135476	1034135476
ci_pipeline_id	70870999	70870999
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-2-c35tsyqg 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-2-c35tsyqg 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.52.0-SNAPSHOT~df915f0a03, baseline=1.52.0-SNAPSHOT~d1794c17af
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.476 ms) : 1464, 1487
.   : milestone, 1476,
appsec (3.652 ms) : 3434, 3871
.   : milestone, 3652,
iast (2.202 ms) : 2139, 2264
.   : milestone, 2202,
iast_GLOBAL (2.236 ms) : 2174, 2299
.   : milestone, 2236,
profiling (2.035 ms) : 1985, 2085
.   : milestone, 2035,
tracing (2.012 ms) : 1963, 2061
.   : milestone, 2012,
section candidate
no_agent (1.476 ms) : 1465, 1488
.   : milestone, 1476,
appsec (3.573 ms) : 3361, 3786
.   : milestone, 3573,
iast (2.209 ms) : 2146, 2272
.   : milestone, 2209,
iast_GLOBAL (2.244 ms) : 2181, 2308
.   : milestone, 2244,
profiling (2.044 ms) : 1994, 2095
.   : milestone, 2044,
tracing (2.02 ms) : 1971, 2068
.   : milestone, 2020,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.476 ms [1.464 ms, 1.487 ms]	-
appsec	3.652 ms [3.434 ms, 3.871 ms]	2.177 ms (147.5%)
iast	2.202 ms [2.139 ms, 2.264 ms]	725.646 µs (49.2%)
iast_GLOBAL	2.236 ms [2.174 ms, 2.299 ms]	760.395 µs (51.5%)
profiling	2.035 ms [1.985 ms, 2.085 ms]	558.867 µs (37.9%)
tracing	2.012 ms [1.963 ms, 2.061 ms]	536.156 µs (36.3%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.476 ms [1.465 ms, 1.488 ms]	-
appsec	3.573 ms [3.361 ms, 3.786 ms]	2.097 ms (142.1%)
iast	2.209 ms [2.146 ms, 2.272 ms]	732.582 µs (49.6%)
iast_GLOBAL	2.244 ms [2.181 ms, 2.308 ms]	768.271 µs (52.0%)
profiling	2.044 ms [1.994 ms, 2.095 ms]	568.107 µs (38.5%)
tracing	2.02 ms [1.971 ms, 2.068 ms]	543.416 µs (36.8%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.52.0-SNAPSHOT~df915f0a03, baseline=1.52.0-SNAPSHOT~d1794c17af
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.758 s) : 14758000, 14758000
.   : milestone, 14758000,
appsec (14.983 s) : 14983000, 14983000
.   : milestone, 14983000,
iast (18.089 s) : 18089000, 18089000
.   : milestone, 18089000,
iast_GLOBAL (18.002 s) : 18002000, 18002000
.   : milestone, 18002000,
profiling (16.055 s) : 16055000, 16055000
.   : milestone, 16055000,
tracing (14.959 s) : 14959000, 14959000
.   : milestone, 14959000,
section candidate
no_agent (15.672 s) : 15672000, 15672000
.   : milestone, 15672000,
appsec (14.891 s) : 14891000, 14891000
.   : milestone, 14891000,
iast (18.632 s) : 18632000, 18632000
.   : milestone, 18632000,
iast_GLOBAL (18.276 s) : 18276000, 18276000
.   : milestone, 18276000,
profiling (15.21 s) : 15210000, 15210000
.   : milestone, 15210000,
tracing (15.015 s) : 15015000, 15015000
.   : milestone, 15015000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.758 s [14.758 s, 14.758 s]	-
appsec	14.983 s [14.983 s, 14.983 s]	225.0 ms (1.5%)
iast	18.089 s [18.089 s, 18.089 s]	3.331 s (22.6%)
iast_GLOBAL	18.002 s [18.002 s, 18.002 s]	3.244 s (22.0%)
profiling	16.055 s [16.055 s, 16.055 s]	1.297 s (8.8%)
tracing	14.959 s [14.959 s, 14.959 s]	201.0 ms (1.4%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.672 s [15.672 s, 15.672 s]	-
appsec	14.891 s [14.891 s, 14.891 s]	-781.0 ms (-5.0%)
iast	18.632 s [18.632 s, 18.632 s]	2.96 s (18.9%)
iast_GLOBAL	18.276 s [18.276 s, 18.276 s]	2.604 s (16.6%)
profiling	15.21 s [15.21 s, 15.21 s]	-462.0 ms (-2.9%)
tracing	15.015 s [15.015 s, 15.015 s]	-657.0 ms (-4.2%)

ddyurchenko

Looks good from my side! 🎉
Need also @igoragoli review for the final approval.

igoragoli

Thanks @bric3! 🙌

There are just some points regarding startup:petclinic.* thresholds that need to be addressed before merging.

.gitlab/benchmarks/bp-runner.fail-on-breach.yml

.gitlab/macrobenchmarks.yml

…can debug

igoragoli

Looks good, thanks Brice!

I think it's a good idea to include the source for the SLOs in the thresholds file, nice.

bric3 · 2025-07-11T14:12:06Z

.gitlab/macrobenchmarks.yml

-      when: always
-    - when: manual
-      allow_failure: true
+    - when: on_success  # TODO: PLEASE revert before merging the PR


todo:

To revert before merging

ddyurchenko · 2025-07-14T15:30:27Z

Still see some yellow in results, so while job won't block releases, it will send warning messages in Java guild channel.
Augusto practically finalized the changes to reporting (so now confidence intervals and thresholds are clearly displayed, incl. warning threshold). I propose to rebuild Java image, so the changes are included, and update the SLOs once more for p50 latency, p99 latency and startup time mean value (execution_time metric), so they are no longer in yellow zone.

SLO breach check  | 
SLO breach check  | #### high_load--only-tracing
SLO breach check  | 
SLO breach check  | - 🟩 `throughput` 1[250](https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-java/-/jobs/1025486085#L250).55 op/s > 1100.00 op/s
SLO breach check  | 
SLO breach check  | #### high_load--otel-latest
SLO breach check  | 
SLO breach check  | - 🟩 `throughput` 1245.11 op/s > 1100.00 op/s
SLO breach check  | 
SLO breach check  | #### normal_operation--only-tracing
SLO breach check  | 
SLO breach check  | - 🟩 `agg_http_req_duration_p50` 2.12 ms < 2.36 ms
SLO breach check  | - 🟨 `agg_http_req_duration_p99` 7.10 ms < 7.89 ms
SLO breach check  | 
SLO breach check  | #### normal_operation--otel-latest
SLO breach check  | 
SLO breach check  | - 🟨 `agg_http_req_duration_p50` 2.12 ms < 2.34 ms
SLO breach check  | - 🟨 `agg_http_req_duration_p99` 8.75 ms < 9.50 ms
SLO breach check  | 
SLO breach check  | #### startup:petclinic:appsec:GlobalTracer
SLO breach check  | 
SLO breach check  | - 🟨 `execution_time` 235.01 ms < [260](https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-java/-/jobs/1025486085#L260).00 ms
SLO breach check  | 
SLO breach check  | #### startup:petclinic:iast:GlobalTracer
SLO breach check  | 
SLO breach check  | - 🟩 `execution_time` 231.70 ms < 260.00 ms
SLO breach check  | 
SLO breach check  | #### startup:petclinic:profiling:GlobalTracer
SLO breach check  | 
SLO breach check  | - 🟨 `execution_time` 361.27 ms < 368.00 ms
SLO breach check  | 
SLO breach check  | #### startup:petclinic:tracing:GlobalTracer
SLO breach check  | 
SLO breach check  | - 🟨 `execution_time` 243.13 ms < 260.00 ms
SLO breach check  | 
SLO breach check  | ---
SLO breach check  | 
SLO breach check  | Legend:
SLO breach check  | - 🟩 pass
SLO breach check  | - 🟥 breach
SLO breach check  | - 🟨 warning
SLO breach check  | - (unstable) unstable

igoragoli · 2025-07-16T12:16:09Z

Augusto practically finalized the changes to reporting

Changes for reporting done! 🙂

I'm updating the registry.ddbuild.io/images/benchmarking-platform-tools-ubuntu:latest image (the one used in the check-slo-breaches job across the board) here: https://gitlab.ddbuild.io/DataDog/benchmarking-platform-tools/-/jobs/1031270236

ddyurchenko · 2025-07-16T13:35:43Z

Thanks @igoragoli ! I reran the job https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-java/-/jobs/1031460223, based on its results, will provide suggestions to @bric3 .

ddyurchenko · 2025-07-16T13:39:14Z

.gitlab/benchmarks/bp-runner.fail-on-breach.yml

+        #   https://datadoghq.atlassian.net/wiki/x/LgI1LgE#Specification
+        # Measurements
+        #   https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario
+        scenarios:


Based on the results of https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-java/-/jobs/1031460223, I suggest to update the SLOs to the following values:

# Standard macrobenchmarks # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=normal_operation%2Fonly-tracing&trendsType=scenario - name: normal_operation/only-tracing thresholds: - agg_http_req_duration_p50 < 2.36 ms - agg_http_req_duration_p99 < 7.89 ms # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=normal_operation%2Fotel-latest&trendsType=scenario - name: normal_operation/otel-latest thresholds: - agg_http_req_duration_p50 < 2.5 ms - agg_http_req_duration_p99 < 10 ms # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=high_load%2Fonly-tracing&trendsType=scenario - name: high_load/only-tracing thresholds: - throughput > 1100.0 op/s # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=high_load%2Fotel-latest&trendsType=scenario - name: high_load/otel-latest thresholds: - throughput > 1100.0 op/s # Startup macrobenchmarks # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Atracing%3AGlobalTracer&trendsType=scenario # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Aappsec%3AGlobalTracer&trendsType=scenario # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Aiast%3AGlobalTracer&trendsType=scenario - name: "startup:petclinic:(tracing|appsec|iast):GlobalTracer" thresholds: - execution_time < 280 ms # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Aprofiling%3AGlobalTracer&trendsType=scenario - name: "startup:petclinic:profiling:GlobalTracer" thresholds: - execution_time < 420 ms

Another tweak

ddyurchenko

Need to update some of SLO thresholds before merge.

See https://github.com/DataDog/dd-trace-java/pull/9068/files#r2210474360

…robenchmarks)

bric3 requested a review from a team as a code owner June 30, 2025 16:59

bric3 requested review from colin-higgins and removed request for a team June 30, 2025 16:59

bric3 added tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling labels Jun 30, 2025

bric3 marked this pull request as draft June 30, 2025 16:59

bric3 changed the title ~~chore(ci): Basic slo breach prototype~~ Perf release gate Jun 30, 2025

ddyurchenko approved these changes Jul 1, 2025

View reviewed changes

ddyurchenko requested a review from igoragoli July 1, 2025 13:43

bric3 requested a review from a team July 1, 2025 14:07

igoragoli requested changes Jul 1, 2025

View reviewed changes

.gitlab/benchmarks/bp-runner.fail-on-breach.yml Outdated Show resolved Hide resolved

.gitlab/benchmarks/bp-runner.fail-on-breach.yml Outdated Show resolved Hide resolved

.gitlab/macrobenchmarks.yml Show resolved Hide resolved

bric3 requested a review from a team July 2, 2025 15:24

bric3 force-pushed the bdu/r-gate branch 7 times, most recently from 7e61d82 to feb60ef Compare July 10, 2025 12:48

bric3 added 3 commits July 11, 2025 11:35

chore(ci): Basic slo breach prototype

fdd30e5

chore(ci): PR review

6407c42

chore: Collect reports from benchmarks

736aefd

bric3 force-pushed the bdu/r-gate branch from feb60ef to 736aefd Compare July 11, 2025 09:35

ddyurchenko added 2 commits July 11, 2025 15:28

tweak: Include benchmarkType in file name

361aaf6

tweak: Store artifacts that were used for checking regression, so we …

92f9783

…can debug

ddyurchenko self-requested a review July 11, 2025 14:39

chore(ci): Tweak thresholds

67435a9

bric3 marked this pull request as ready for review July 11, 2025 15:45

igoragoli approved these changes Jul 11, 2025

View reviewed changes

bric3 commented Jul 11, 2025

View reviewed changes

ddyurchenko reviewed Jul 16, 2025

View reviewed changes

ddyurchenko requested changes Jul 16, 2025

View reviewed changes

bric3 added 3 commits July 17, 2025 11:26

chore(ci): Tweak to recommended thresholds

616ae97

See https://github.com/DataDog/dd-trace-java/pull/9068/files#r2210474360

Merge remote-tracking branch 'origin/master' into bdu/r-gate

0708dfb

chore(ci): Another tweak to recommended thresholds

98f9a31

See https://github.com/DataDog/dd-trace-java/pull/9068/files#r2210474360

ddyurchenko approved these changes Jul 17, 2025

View reviewed changes

bric3 and others added 2 commits July 17, 2025 14:49

chore(ci): Revert hack to run the release gate (it needed the all mac…

098e012

…robenchmarks)

Merge branch 'master' into bdu/r-gate

df915f0

bric3 enabled auto-merge (squash) July 22, 2025 10:08

bric3 merged commit 4e4c286 into master Jul 22, 2025
503 checks passed

bric3 deleted the bdu/r-gate branch July 22, 2025 10:33

github-actions bot added this to the 1.52.0 milestone Jul 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Perf release gate #9068

Perf release gate #9068

Uh oh!

bric3 commented Jun 30, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Jun 30, 2025 •

edited

Loading

Uh oh!

ddyurchenko left a comment

Uh oh!

igoragoli left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

igoragoli left a comment

Uh oh!

bric3 Jul 11, 2025

Uh oh!

ddyurchenko commented Jul 14, 2025

Uh oh!

igoragoli commented Jul 16, 2025 •

edited

Loading

Uh oh!

ddyurchenko commented Jul 16, 2025

Uh oh!

ddyurchenko Jul 16, 2025

Uh oh!

bric3 Jul 17, 2025

Uh oh!

bric3 Jul 17, 2025

Uh oh!

ddyurchenko left a comment

Uh oh!

Uh oh!

Uh oh!

Perf release gate #9068

Perf release gate #9068

Uh oh!

Conversation

bric3 commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

ddyurchenko left a comment

Choose a reason for hiding this comment

Uh oh!

igoragoli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

igoragoli left a comment

Choose a reason for hiding this comment

Uh oh!

bric3 Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

ddyurchenko commented Jul 14, 2025

Uh oh!

igoragoli commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ddyurchenko commented Jul 16, 2025

Uh oh!

ddyurchenko Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

bric3 Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

bric3 Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

ddyurchenko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bric3 commented Jun 30, 2025 •

edited

Loading

pr-commenter bot commented Jun 30, 2025 •

edited

Loading

igoragoli commented Jul 16, 2025 •

edited

Loading