Remove E2E system tests execution from CircleCI #8768

PerfectSlayer · 2025-05-05T14:13:35Z

What Does This Do

Remove the execution of E2E system test from CircleCI.

Motivation

This should matches the GitHub workflow behavior (new system tests CI for now) and unblock the CI following the CircleCI environment variable clean up.

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

amarziali · 2025-05-05T14:26:04Z

.circleci/config.continue.yml.j2

          # Stop the job after 5m to avoid excessive overhead. Will need adjustment as more tests are added.
          no_output_timeout: 5m
          command: |
            cd system-tests
-            DD_SITE=datadoghq.com DD_API_KEY=$SYSTEM_TESTS_E2E_DD_API_KEY DD_APPLICATION_KEY=$SYSTEM_TESTS_E2E_DD_APP_KEY ./run.sh INTEGRATIONS
+            DD_API_KEY=$SYSTEM_TESTS_DD_API_KEY ./run.sh INTEGRATIONS


I think DD_SITE can be left to be sure it's not overridden but it anyway defaults to that

Those keys do not match the datadoghq.com org. It must be removed according to Charles.

pr-commenter · 2025-05-05T15:04:26Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	bbujon/system-tests
git_commit_date	1746263227	1746507169
git_commit_sha	`12da6fe`	`a127379`
release_version	1.49.0-SNAPSHOT~12da6fe499	1.49.0-SNAPSHOT~a12737982e

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1746510190	1746510190
ci_job_id	924873054	924873054
ci_pipeline_id	64200768	64200768
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-z2euzavw-project-304-concurrent-0-kmi0ygs7 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-z2euzavw-project-304-concurrent-0-kmi0ygs7 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 67 metrics, 4 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.49.0-SNAPSHOT~a12737982e, baseline=1.49.0-SNAPSHOT~12da6fe499

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.019 s) : 0, 1019389
Total [baseline] (8.668 s) : 0, 8668042
Agent [candidate] (1.004 s) : 0, 1003770
Total [candidate] (8.638 s) : 0, 8638061
section iast
Agent [baseline] (1.136 s) : 0, 1135617
Total [baseline] (9.191 s) : 0, 9190865
Agent [candidate] (1.156 s) : 0, 1156242
Total [candidate] (9.251 s) : 0, 9250805
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.144 s) : 0, 1143687
Total [baseline] (9.212 s) : 0, 9211645
Agent [candidate] (1.138 s) : 0, 1138161
Total [candidate] (9.164 s) : 0, 9163996
section iast_TELEMETRY_OFF
Agent [baseline] (1.147 s) : 0, 1147014
Total [baseline] (9.2 s) : 0, 9199895
Agent [candidate] (1.141 s) : 0, 1140934
Total [candidate] (9.189 s) : 0, 9189339

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.019 s	-
Agent	iast	1.136 s	116.227 ms (11.4%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.144 s	124.298 ms (12.2%)
Agent	iast_TELEMETRY_OFF	1.147 s	127.625 ms (12.5%)
Total	tracing	8.668 s	-
Total	iast	9.191 s	522.823 ms (6.0%)
Total	iast_HARDCODED_SECRET_DISABLED	9.212 s	543.603 ms (6.3%)
Total	iast_TELEMETRY_OFF	9.2 s	531.853 ms (6.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.004 s	-
Agent	iast	1.156 s	152.472 ms (15.2%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.138 s	134.392 ms (13.4%)
Agent	iast_TELEMETRY_OFF	1.141 s	137.164 ms (13.7%)
Total	tracing	8.638 s	-
Total	iast	9.251 s	612.743 ms (7.1%)
Total	iast_HARDCODED_SECRET_DISABLED	9.164 s	525.935 ms (6.1%)
Total	iast_TELEMETRY_OFF	9.189 s	551.277 ms (6.4%)

gantt
    title insecure-bank - break down per module: candidate=1.49.0-SNAPSHOT~a12737982e, baseline=1.49.0-SNAPSHOT~12da6fe499

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (676.426 ms) : 0, 676426
BytebuddyAgent [candidate] (670.513 ms) : 0, 670513
GlobalTracer [baseline] (243.333 ms) : 0, 243333
GlobalTracer [candidate] (239.387 ms) : 0, 239387
AppSec [baseline] (55.441 ms) : 0, 55441
AppSec [candidate] (54.517 ms) : 0, 54517
Debugger [baseline] (8.406 ms) : 0, 8406
Debugger [candidate] (6.853 ms) : 0, 6853
Remote Config [baseline] (705.415 µs) : 0, 705
Remote Config [candidate] (699.325 µs) : 0, 699
Telemetry [baseline] (11.406 ms) : 0, 11406
Telemetry [candidate] (8.464 ms) : 0, 8464
section iast
BytebuddyAgent [baseline] (788.676 ms) : 0, 788676
BytebuddyAgent [candidate] (803.008 ms) : 0, 803008
GlobalTracer [baseline] (229.945 ms) : 0, 229945
GlobalTracer [candidate] (234.159 ms) : 0, 234159
IAST [baseline] (22.608 ms) : 0, 22608
IAST [candidate] (23.467 ms) : 0, 23467
AppSec [baseline] (56.661 ms) : 0, 56661
AppSec [candidate] (57.23 ms) : 0, 57230
Debugger [baseline] (5.922 ms) : 0, 5922
Debugger [candidate] (6.059 ms) : 0, 6059
Remote Config [baseline] (579.012 µs) : 0, 579
Remote Config [candidate] (596.984 µs) : 0, 597
Telemetry [baseline] (7.862 ms) : 0, 7862
Telemetry [candidate] (8.051 ms) : 0, 8051
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (795.628 ms) : 0, 795628
BytebuddyAgent [candidate] (790.404 ms) : 0, 790404
GlobalTracer [baseline] (230.5 ms) : 0, 230500
GlobalTracer [candidate] (230.529 ms) : 0, 230529
IAST [baseline] (23.012 ms) : 0, 23012
IAST [candidate] (22.884 ms) : 0, 22884
AppSec [baseline] (56.594 ms) : 0, 56594
AppSec [candidate] (56.54 ms) : 0, 56540
Debugger [baseline] (5.929 ms) : 0, 5929
Debugger [candidate] (5.848 ms) : 0, 5848
Remote Config [baseline] (585.366 µs) : 0, 585
Remote Config [candidate] (584.422 µs) : 0, 584
Telemetry [baseline] (7.909 ms) : 0, 7909
Telemetry [candidate] (7.94 ms) : 0, 7940
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (797.098 ms) : 0, 797098
BytebuddyAgent [candidate] (792.635 ms) : 0, 792635
GlobalTracer [baseline] (232.451 ms) : 0, 232451
GlobalTracer [candidate] (231.509 ms) : 0, 231509
IAST [baseline] (22.761 ms) : 0, 22761
IAST [candidate] (22.683 ms) : 0, 22683
AppSec [baseline] (56.622 ms) : 0, 56622
AppSec [candidate] (56.251 ms) : 0, 56251
Debugger [baseline] (6.001 ms) : 0, 6001
Debugger [candidate] (5.998 ms) : 0, 5998
Remote Config [baseline] (610.679 µs) : 0, 611
Remote Config [candidate] (619.058 µs) : 0, 619
Telemetry [baseline] (7.873 ms) : 0, 7873
Telemetry [candidate] (7.791 ms) : 0, 7791

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.49.0-SNAPSHOT~a12737982e, baseline=1.49.0-SNAPSHOT~12da6fe499

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.006 s) : 0, 1006177
Total [baseline] (10.584 s) : 0, 10584447
Agent [candidate] (1.01 s) : 0, 1009773
Total [candidate] (10.497 s) : 0, 10496838
section appsec
Agent [baseline] (1.15 s) : 0, 1149567
Total [baseline] (10.679 s) : 0, 10678764
Agent [candidate] (1.153 s) : 0, 1153246
Total [candidate] (10.684 s) : 0, 10684030
section iast
Agent [baseline] (1.147 s) : 0, 1147423
Total [baseline] (10.956 s) : 0, 10955638
Agent [candidate] (1.14 s) : 0, 1140018
Total [candidate] (10.859 s) : 0, 10859084
section profiling
Agent [baseline] (1.256 s) : 0, 1256489
Total [baseline] (10.726 s) : 0, 10726260
Agent [candidate] (1.259 s) : 0, 1258512
Total [candidate] (10.778 s) : 0, 10777857

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.006 s	-
Agent	appsec	1.15 s	143.39 ms (14.3%)
Agent	iast	1.147 s	141.245 ms (14.0%)
Agent	profiling	1.256 s	250.312 ms (24.9%)
Total	tracing	10.584 s	-
Total	appsec	10.679 s	94.317 ms (0.9%)
Total	iast	10.956 s	371.191 ms (3.5%)
Total	profiling	10.726 s	141.812 ms (1.3%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.01 s	-
Agent	appsec	1.153 s	143.472 ms (14.2%)
Agent	iast	1.14 s	130.244 ms (12.9%)
Agent	profiling	1.259 s	248.738 ms (24.6%)
Total	tracing	10.497 s	-
Total	appsec	10.684 s	187.192 ms (1.8%)
Total	iast	10.859 s	362.246 ms (3.5%)
Total	profiling	10.778 s	281.02 ms (2.7%)

gantt
    title petclinic - break down per module: candidate=1.49.0-SNAPSHOT~a12737982e, baseline=1.49.0-SNAPSHOT~12da6fe499

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (670.619 ms) : 0, 670619
BytebuddyAgent [candidate] (674.698 ms) : 0, 674698
GlobalTracer [baseline] (240.064 ms) : 0, 240064
GlobalTracer [candidate] (239.598 ms) : 0, 239598
AppSec [baseline] (54.657 ms) : 0, 54657
AppSec [candidate] (54.638 ms) : 0, 54638
Debugger [baseline] (6.117 ms) : 0, 6117
Debugger [candidate] (6.144 ms) : 0, 6144
Remote Config [baseline] (709.043 µs) : 0, 709
Remote Config [candidate] (696.379 µs) : 0, 696
Telemetry [baseline] (10.66 ms) : 0, 10660
Telemetry [candidate] (10.583 ms) : 0, 10583
section appsec
BytebuddyAgent [baseline] (688.728 ms) : 0, 688728
BytebuddyAgent [candidate] (691.34 ms) : 0, 691340
GlobalTracer [baseline] (236.484 ms) : 0, 236484
GlobalTracer [candidate] (237.757 ms) : 0, 237757
IAST [baseline] (21.701 ms) : 0, 21701
IAST [candidate] (21.753 ms) : 0, 21753
AppSec [baseline] (174.323 ms) : 0, 174323
AppSec [candidate] (175.403 ms) : 0, 175403
Debugger [baseline] (5.926 ms) : 0, 5926
Debugger [candidate] (5.895 ms) : 0, 5895
Remote Config [baseline] (640.811 µs) : 0, 641
Remote Config [candidate] (632.554 µs) : 0, 633
Telemetry [baseline] (9.274 ms) : 0, 9274
Telemetry [candidate] (7.816 ms) : 0, 7816
section iast
BytebuddyAgent [baseline] (796.535 ms) : 0, 796535
BytebuddyAgent [candidate] (792.27 ms) : 0, 792270
GlobalTracer [baseline] (232.388 ms) : 0, 232388
GlobalTracer [candidate] (230.783 ms) : 0, 230783
IAST [baseline] (23.166 ms) : 0, 23166
IAST [candidate] (22.935 ms) : 0, 22935
AppSec [baseline] (57.167 ms) : 0, 57167
AppSec [candidate] (56.143 ms) : 0, 56143
Debugger [baseline] (6.015 ms) : 0, 6015
Debugger [candidate] (5.92 ms) : 0, 5920
Remote Config [baseline] (603.287 µs) : 0, 603
Remote Config [candidate] (591.828 µs) : 0, 592
Telemetry [baseline] (8.064 ms) : 0, 8064
Telemetry [candidate] (7.887 ms) : 0, 7887
section profiling
BytebuddyAgent [baseline] (661.934 ms) : 0, 661934
BytebuddyAgent [candidate] (662.513 ms) : 0, 662513
GlobalTracer [baseline] (378.53 ms) : 0, 378530
GlobalTracer [candidate] (378.889 ms) : 0, 378889
AppSec [baseline] (54.434 ms) : 0, 54434
AppSec [candidate] (54.059 ms) : 0, 54059
Debugger [baseline] (6.134 ms) : 0, 6134
Debugger [candidate] (6.1 ms) : 0, 6100
Remote Config [baseline] (646.086 µs) : 0, 646
Remote Config [candidate] (646.475 µs) : 0, 646
Telemetry [baseline] (8.12 ms) : 0, 8120
Telemetry [candidate] (8.139 ms) : 0, 8139
ProfilingAgent [baseline] (96.598 ms) : 0, 96598
ProfilingAgent [candidate] (97.222 ms) : 0, 97222
Profiling [baseline] (96.622 ms) : 0, 96622
Profiling [candidate] (97.247 ms) : 0, 97247

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-05-06T05:13:40	2025-05-06T05:21:26
git_branch	master	bbujon/system-tests
git_commit_date	1746263227	1746507169
git_commit_sha	`12da6fe`	`a127379`
release_version	1.49.0-SNAPSHOT~12da6fe499	1.49.0-SNAPSHOT~a12737982e
start_time	2025-05-06T05:13:26	2025-05-06T05:21:11

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1746509287	1746509287
ci_job_id	924873055	924873055
ci_pipeline_id	64200768	64200768
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-a5c54cuo-project-304-concurrent-0-a1a81qcz 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-a5c54cuo-project-304-concurrent-0-a1a81qcz 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 18 unstable metrics.

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.49.0-SNAPSHOT~a12737982e, baseline=1.49.0-SNAPSHOT~12da6fe499
    dateFormat X
    axisFormat %s
section baseline
no_agent (381.842 µs) : 362, 402
.   : milestone, 382,
iast (524.011 µs) : 501, 547
.   : milestone, 524,
iast_FULL (742.362 µs) : 719, 766
.   : milestone, 742,
iast_GLOBAL (579.296 µs) : 556, 603
.   : milestone, 579,
iast_HARDCODED_SECRET_DISABLED (519.021 µs) : 496, 542
.   : milestone, 519,
iast_INACTIVE (467.133 µs) : 446, 489
.   : milestone, 467,
iast_TELEMETRY_OFF (509.184 µs) : 486, 532
.   : milestone, 509,
tracing (471.009 µs) : 449, 494
.   : milestone, 471,
section candidate
no_agent (386.238 µs) : 366, 407
.   : milestone, 386,
iast (531.612 µs) : 508, 555
.   : milestone, 532,
iast_FULL (741.99 µs) : 719, 765
.   : milestone, 742,
iast_GLOBAL (576.809 µs) : 553, 600
.   : milestone, 577,
iast_HARDCODED_SECRET_DISABLED (520.354 µs) : 498, 543
.   : milestone, 520,
iast_INACTIVE (471.292 µs) : 449, 494
.   : milestone, 471,
iast_TELEMETRY_OFF (509.024 µs) : 486, 532
.   : milestone, 509,
tracing (466.179 µs) : 444, 489
.   : milestone, 466,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	381.842 µs [362.073 µs, 401.61 µs]	-
iast	524.011 µs [501.185 µs, 546.837 µs]	142.169 µs (37.2%)
iast_FULL	742.362 µs [718.655 µs, 766.07 µs]	360.52 µs (94.4%)
iast_GLOBAL	579.296 µs [555.688 µs, 602.905 µs]	197.454 µs (51.7%)
iast_HARDCODED_SECRET_DISABLED	519.021 µs [496.03 µs, 542.012 µs]	137.179 µs (35.9%)
iast_INACTIVE	467.133 µs [445.666 µs, 488.6 µs]	85.291 µs (22.3%)
iast_TELEMETRY_OFF	509.184 µs [486.2 µs, 532.167 µs]	127.342 µs (33.3%)
tracing	471.009 µs [448.509 µs, 493.508 µs]	89.167 µs (23.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	386.238 µs [365.885 µs, 406.591 µs]	-
iast	531.612 µs [508.384 µs, 554.84 µs]	145.374 µs (37.6%)
iast_FULL	741.99 µs [718.944 µs, 765.036 µs]	355.752 µs (92.1%)
iast_GLOBAL	576.809 µs [553.202 µs, 600.416 µs]	190.571 µs (49.3%)
iast_HARDCODED_SECRET_DISABLED	520.354 µs [497.584 µs, 543.125 µs]	134.117 µs (34.7%)
iast_INACTIVE	471.292 µs [448.63 µs, 493.954 µs]	85.054 µs (22.0%)
iast_TELEMETRY_OFF	509.024 µs [486.352 µs, 531.695 µs]	122.786 µs (31.8%)
tracing	466.179 µs [443.652 µs, 488.705 µs]	79.941 µs (20.7%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.49.0-SNAPSHOT~a12737982e, baseline=1.49.0-SNAPSHOT~12da6fe499
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.348 ms) : 1329, 1368
.   : milestone, 1348,
appsec (1.733 ms) : 1709, 1757
.   : milestone, 1733,
appsec_no_iast (1.733 ms) : 1709, 1757
.   : milestone, 1733,
code_origins (1.694 ms) : 1667, 1720
.   : milestone, 1694,
iast (1.531 ms) : 1506, 1555
.   : milestone, 1531,
profiling (1.573 ms) : 1548, 1597
.   : milestone, 1573,
tracing (1.488 ms) : 1464, 1513
.   : milestone, 1488,
section candidate
no_agent (1.377 ms) : 1357, 1396
.   : milestone, 1377,
appsec (1.731 ms) : 1708, 1754
.   : milestone, 1731,
appsec_no_iast (1.736 ms) : 1712, 1760
.   : milestone, 1736,
code_origins (1.694 ms) : 1668, 1721
.   : milestone, 1694,
iast (1.526 ms) : 1502, 1550
.   : milestone, 1526,
profiling (1.583 ms) : 1557, 1608
.   : milestone, 1583,
tracing (1.493 ms) : 1468, 1517
.   : milestone, 1493,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.348 ms [1.329 ms, 1.368 ms]	-
appsec	1.733 ms [1.709 ms, 1.757 ms]	384.629 µs (28.5%)
appsec_no_iast	1.733 ms [1.709 ms, 1.757 ms]	384.797 µs (28.5%)
code_origins	1.694 ms [1.667 ms, 1.72 ms]	345.172 µs (25.6%)
iast	1.531 ms [1.506 ms, 1.555 ms]	182.269 µs (13.5%)
profiling	1.573 ms [1.548 ms, 1.597 ms]	224.288 µs (16.6%)
tracing	1.488 ms [1.464 ms, 1.513 ms]	140.034 µs (10.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.377 ms [1.357 ms, 1.396 ms]	-
appsec	1.731 ms [1.708 ms, 1.754 ms]	354.502 µs (25.8%)
appsec_no_iast	1.736 ms [1.712 ms, 1.76 ms]	359.581 µs (26.1%)
code_origins	1.694 ms [1.668 ms, 1.721 ms]	317.777 µs (23.1%)
iast	1.526 ms [1.502 ms, 1.55 ms]	149.249 µs (10.8%)
profiling	1.583 ms [1.557 ms, 1.608 ms]	206.009 µs (15.0%)
tracing	1.493 ms [1.468 ms, 1.517 ms]	116.054 µs (8.4%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	bbujon/system-tests
git_commit_date	1746263227	1746507169
git_commit_sha	`12da6fe`	`a127379`
release_version	1.49.0-SNAPSHOT~12da6fe499	1.49.0-SNAPSHOT~a12737982e

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1746509784	1746509784
ci_job_id	924873056	924873056
ci_pipeline_id	64200768	64200768
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-yl2rg8ey-project-304-concurrent-0-erisslao 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-yl2rg8ey-project-304-concurrent-0-erisslao 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.49.0-SNAPSHOT~a12737982e, baseline=1.49.0-SNAPSHOT~12da6fe499
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.478 ms) : 1466, 1489
.   : milestone, 1478,
appsec (2.393 ms) : 2345, 2440
.   : milestone, 2393,
iast (2.174 ms) : 2113, 2234
.   : milestone, 2174,
iast_GLOBAL (2.212 ms) : 2152, 2273
.   : milestone, 2212,
profiling (2.466 ms) : 2290, 2643
.   : milestone, 2466,
tracing (1.987 ms) : 1941, 2033
.   : milestone, 1987,
section candidate
no_agent (1.48 ms) : 1469, 1492
.   : milestone, 1480,
appsec (2.391 ms) : 2343, 2438
.   : milestone, 2391,
iast (2.162 ms) : 2102, 2222
.   : milestone, 2162,
iast_GLOBAL (2.205 ms) : 2145, 2265
.   : milestone, 2205,
profiling (2.043 ms) : 1994, 2092
.   : milestone, 2043,
tracing (1.992 ms) : 1946, 2038
.   : milestone, 1992,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.478 ms [1.466 ms, 1.489 ms]	-
appsec	2.393 ms [2.345 ms, 2.44 ms]	914.94 µs (61.9%)
iast	2.174 ms [2.113 ms, 2.234 ms]	695.928 µs (47.1%)
iast_GLOBAL	2.212 ms [2.152 ms, 2.273 ms]	734.823 µs (49.7%)
profiling	2.466 ms [2.29 ms, 2.643 ms]	988.865 µs (66.9%)
tracing	1.987 ms [1.941 ms, 2.033 ms]	509.453 µs (34.5%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.48 ms [1.469 ms, 1.492 ms]	-
appsec	2.391 ms [2.343 ms, 2.438 ms]	910.272 µs (61.5%)
iast	2.162 ms [2.102 ms, 2.222 ms]	681.56 µs (46.0%)
iast_GLOBAL	2.205 ms [2.145 ms, 2.265 ms]	724.688 µs (48.9%)
profiling	2.043 ms [1.994 ms, 2.092 ms]	562.241 µs (38.0%)
tracing	1.992 ms [1.946 ms, 2.038 ms]	511.607 µs (34.6%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.49.0-SNAPSHOT~a12737982e, baseline=1.49.0-SNAPSHOT~12da6fe499
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.653 s) : 15653000, 15653000
.   : milestone, 15653000,
appsec (15.014 s) : 15014000, 15014000
.   : milestone, 15014000,
iast (19.158 s) : 19158000, 19158000
.   : milestone, 19158000,
iast_GLOBAL (18.161 s) : 18161000, 18161000
.   : milestone, 18161000,
profiling (15.385 s) : 15385000, 15385000
.   : milestone, 15385000,
tracing (15.026 s) : 15026000, 15026000
.   : milestone, 15026000,
section candidate
no_agent (14.682 s) : 14682000, 14682000
.   : milestone, 14682000,
appsec (14.94 s) : 14940000, 14940000
.   : milestone, 14940000,
iast (18.697 s) : 18697000, 18697000
.   : milestone, 18697000,
iast_GLOBAL (18.165 s) : 18165000, 18165000
.   : milestone, 18165000,
profiling (15.116 s) : 15116000, 15116000
.   : milestone, 15116000,
tracing (15.117 s) : 15117000, 15117000
.   : milestone, 15117000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.653 s [15.653 s, 15.653 s]	-
appsec	15.014 s [15.014 s, 15.014 s]	-639.0 ms (-4.1%)
iast	19.158 s [19.158 s, 19.158 s]	3.505 s (22.4%)
iast_GLOBAL	18.161 s [18.161 s, 18.161 s]	2.508 s (16.0%)
profiling	15.385 s [15.385 s, 15.385 s]	-268.0 ms (-1.7%)
tracing	15.026 s [15.026 s, 15.026 s]	-627.0 ms (-4.0%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.682 s [14.682 s, 14.682 s]	-
appsec	14.94 s [14.94 s, 14.94 s]	258.0 ms (1.8%)
iast	18.697 s [18.697 s, 18.697 s]	4.015 s (27.3%)
iast_GLOBAL	18.165 s [18.165 s, 18.165 s]	3.483 s (23.7%)
profiling	15.116 s [15.116 s, 15.116 s]	434.0 ms (3.0%)
tracing	15.117 s [15.117 s, 15.117 s]	435.0 ms (3.0%)

PerfectSlayer requested a review from a team as a code owner May 5, 2025 14:13

PerfectSlayer requested a review from dougqh May 5, 2025 14:13

PerfectSlayer added type: bug tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling labels May 5, 2025

PerfectSlayer force-pushed the bbujon/system-tests branch from f4f2da4 to 9793532 Compare May 5, 2025 14:17

PerfectSlayer marked this pull request as draft May 5, 2025 14:18

PerfectSlayer force-pushed the bbujon/system-tests branch from 9793532 to e49e409 Compare May 5, 2025 14:22

PerfectSlayer marked this pull request as ready for review May 5, 2025 14:22

amarziali reviewed May 5, 2025

View reviewed changes

amarziali approved these changes May 5, 2025

View reviewed changes

PerfectSlayer force-pushed the bbujon/system-tests branch 2 times, most recently from daa1c2f to 105e00f Compare May 6, 2025 03:19

fix(ci): Remove E2E system tests execution from CircleCI

a127379

PerfectSlayer force-pushed the bbujon/system-tests branch from 105e00f to a127379 Compare May 6, 2025 04:52

PerfectSlayer merged commit 0e18e0e into master May 6, 2025
454 checks passed

PerfectSlayer deleted the bbujon/system-tests branch May 6, 2025 07:25

github-actions bot added this to the 1.49.0 milestone May 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove E2E system tests execution from CircleCI #8768

Remove E2E system tests execution from CircleCI #8768

PerfectSlayer commented May 5, 2025

amarziali May 5, 2025

PerfectSlayer May 5, 2025

pr-commenter bot commented May 5, 2025 •

edited

Loading

Remove E2E system tests execution from CircleCI #8768

Remove E2E system tests execution from CircleCI #8768

Conversation

PerfectSlayer commented May 5, 2025

What Does This Do

Motivation

Additional Notes

Contributor Checklist

amarziali May 5, 2025

Choose a reason for hiding this comment

PerfectSlayer May 5, 2025

Choose a reason for hiding this comment

pr-commenter bot commented May 5, 2025 • edited Loading

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

pr-commenter bot commented May 5, 2025 •

edited

Loading