Configure OpenLineage if present in Spark instrumentation #8541
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hi! π Thanks for your pull request! π To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 65 metrics, 6 unstable metrics. Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.057 s) : 0, 1056860
Total [baseline] (8.756 s) : 0, 8756395
Agent [candidate] (1.052 s) : 0, 1052036
Total [candidate] (8.669 s) : 0, 8668550
section iast
Agent [baseline] (1.179 s) : 0, 1179159
Total [baseline] (9.25 s) : 0, 9249768
Agent [candidate] (1.18 s) : 0, 1180295
Total [candidate] (9.241 s) : 0, 9240989
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.18 s) : 0, 1179974
Total [baseline] (9.193 s) : 0, 9193198
Agent [candidate] (1.189 s) : 0, 1189252
Total [candidate] (9.257 s) : 0, 9257106
section iast_TELEMETRY_OFF
Agent [baseline] (1.175 s) : 0, 1175267
Total [baseline] (9.225 s) : 0, 9224944
Agent [candidate] (1.18 s) : 0, 1180245
Total [candidate] (9.235 s) : 0, 9234735
gantt
title insecure-bank - break down per module: candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (722.977 ms) : 0, 722977
BytebuddyAgent [candidate] (719.423 ms) : 0, 719423
GlobalTracer [baseline] (241.072 ms) : 0, 241072
GlobalTracer [candidate] (239.602 ms) : 0, 239602
AppSec [baseline] (54.787 ms) : 0, 54787
AppSec [candidate] (54.277 ms) : 0, 54277
Debugger [baseline] (4.466 ms) : 0, 4466
Debugger [candidate] (4.411 ms) : 0, 4411
Remote Config [baseline] (695.648 Β΅s) : 0, 696
Remote Config [candidate] (698.178 Β΅s) : 0, 698
Telemetry [baseline] (12.221 ms) : 0, 12221
Telemetry [candidate] (13.046 ms) : 0, 13046
section iast
BytebuddyAgent [baseline] (837.265 ms) : 0, 837265
BytebuddyAgent [candidate] (838.061 ms) : 0, 838061
GlobalTracer [baseline] (229.65 ms) : 0, 229650
GlobalTracer [candidate] (229.073 ms) : 0, 229073
IAST [baseline] (22.626 ms) : 0, 22626
IAST [candidate] (23.535 ms) : 0, 23535
AppSec [baseline] (55.637 ms) : 0, 55637
AppSec [candidate] (55.685 ms) : 0, 55685
Debugger [baseline] (4.134 ms) : 0, 4134
Debugger [candidate] (4.173 ms) : 0, 4173
Remote Config [baseline] (609.876 Β΅s) : 0, 610
Remote Config [candidate] (610.992 Β΅s) : 0, 611
Telemetry [baseline] (8.752 ms) : 0, 8752
Telemetry [candidate] (8.808 ms) : 0, 8808
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (837.669 ms) : 0, 837669
BytebuddyAgent [candidate] (845.656 ms) : 0, 845656
GlobalTracer [baseline] (229.522 ms) : 0, 229522
GlobalTracer [candidate] (230.526 ms) : 0, 230526
IAST [baseline] (22.735 ms) : 0, 22735
IAST [candidate] (22.921 ms) : 0, 22921
AppSec [baseline] (56.071 ms) : 0, 56071
AppSec [candidate] (56.053 ms) : 0, 56053
Debugger [baseline] (4.159 ms) : 0, 4159
Debugger [candidate] (4.134 ms) : 0, 4134
Remote Config [baseline] (619.782 Β΅s) : 0, 620
Remote Config [candidate] (608.445 Β΅s) : 0, 608
Telemetry [baseline] (8.746 ms) : 0, 8746
Telemetry [candidate] (8.735 ms) : 0, 8735
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (833.838 ms) : 0, 833838
BytebuddyAgent [candidate] (839.013 ms) : 0, 839013
GlobalTracer [baseline] (229.238 ms) : 0, 229238
GlobalTracer [candidate] (229.025 ms) : 0, 229025
IAST [baseline] (22.287 ms) : 0, 22287
IAST [candidate] (22.331 ms) : 0, 22331
AppSec [baseline] (56.115 ms) : 0, 56115
AppSec [candidate] (55.92 ms) : 0, 55920
Debugger [baseline] (4.188 ms) : 0, 4188
Debugger [candidate] (4.163 ms) : 0, 4163
Remote Config [baseline] (616.429 Β΅s) : 0, 616
Remote Config [candidate] (617.675 Β΅s) : 0, 618
Telemetry [baseline] (8.65 ms) : 0, 8650
Telemetry [candidate] (8.664 ms) : 0, 8664
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.049 s) : 0, 1049442
Total [baseline] (10.53 s) : 0, 10530093
Agent [candidate] (1.052 s) : 0, 1052209
Total [candidate] (10.461 s) : 0, 10461334
section appsec
Agent [baseline] (1.194 s) : 0, 1194103
Total [baseline] (10.722 s) : 0, 10722065
Agent [candidate] (1.194 s) : 0, 1193507
Total [candidate] (10.727 s) : 0, 10726572
section iast
Agent [baseline] (1.188 s) : 0, 1187776
Total [baseline] (11.07 s) : 0, 11070306
Agent [candidate] (1.18 s) : 0, 1179747
Total [candidate] (11.041 s) : 0, 11040520
section profiling
Agent [baseline] (1.273 s) : 0, 1272804
Total [baseline] (10.863 s) : 0, 10862747
Agent [candidate] (1.283 s) : 0, 1283341
Total [candidate] (10.905 s) : 0, 10905045
gantt
title petclinic - break down per module: candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (717.637 ms) : 0, 717637
BytebuddyAgent [candidate] (718.733 ms) : 0, 718733
GlobalTracer [baseline] (238.937 ms) : 0, 238937
GlobalTracer [candidate] (239.567 ms) : 0, 239567
AppSec [baseline] (54.411 ms) : 0, 54411
AppSec [candidate] (54.482 ms) : 0, 54482
Debugger [baseline] (5.153 ms) : 0, 5153
Debugger [candidate] (4.433 ms) : 0, 4433
Remote Config [baseline] (685.957 Β΅s) : 0, 686
Remote Config [candidate] (701.371 Β΅s) : 0, 701
Telemetry [baseline] (12.132 ms) : 0, 12132
Telemetry [candidate] (13.817 ms) : 0, 13817
section appsec
BytebuddyAgent [baseline] (738.003 ms) : 0, 738003
BytebuddyAgent [candidate] (737.564 ms) : 0, 737564
GlobalTracer [baseline] (235.92 ms) : 0, 235920
GlobalTracer [candidate] (235.483 ms) : 0, 235483
AppSec [baseline] (175.841 ms) : 0, 175841
AppSec [candidate] (175.629 ms) : 0, 175629
Debugger [baseline] (4.279 ms) : 0, 4279
Debugger [candidate] (4.294 ms) : 0, 4294
Remote Config [baseline] (640.17 Β΅s) : 0, 640
Remote Config [candidate] (640.948 Β΅s) : 0, 641
Telemetry [baseline] (8.204 ms) : 0, 8204
Telemetry [candidate] (8.541 ms) : 0, 8541
IAST [baseline] (21.415 ms) : 0, 21415
IAST [candidate] (21.708 ms) : 0, 21708
section iast
BytebuddyAgent [baseline] (844.286 ms) : 0, 844286
BytebuddyAgent [candidate] (837.964 ms) : 0, 837964
GlobalTracer [baseline] (230.656 ms) : 0, 230656
GlobalTracer [candidate] (229.547 ms) : 0, 229547
AppSec [baseline] (55.641 ms) : 0, 55641
AppSec [candidate] (55.889 ms) : 0, 55889
Debugger [baseline] (4.162 ms) : 0, 4162
Debugger [candidate] (4.109 ms) : 0, 4109
Remote Config [baseline] (622.265 Β΅s) : 0, 622
Remote Config [candidate] (615.666 Β΅s) : 0, 616
Telemetry [baseline] (8.795 ms) : 0, 8795
Telemetry [candidate] (8.625 ms) : 0, 8625
IAST [baseline] (22.99 ms) : 0, 22990
IAST [candidate] (22.555 ms) : 0, 22555
section profiling
ProfilingAgent [baseline] (102.23 ms) : 0, 102230
ProfilingAgent [candidate] (103.317 ms) : 0, 103317
BytebuddyAgent [baseline] (709.029 ms) : 0, 709029
BytebuddyAgent [candidate] (715.025 ms) : 0, 715025
GlobalTracer [baseline] (348.306 ms) : 0, 348306
GlobalTracer [candidate] (351.262 ms) : 0, 351262
AppSec [baseline] (53.761 ms) : 0, 53761
AppSec [candidate] (53.734 ms) : 0, 53734
Debugger [baseline] (4.252 ms) : 0, 4252
Debugger [candidate] (4.297 ms) : 0, 4297
Remote Config [baseline] (694.487 Β΅s) : 0, 694
Remote Config [candidate] (707.246 Β΅s) : 0, 707
Telemetry [baseline] (8.93 ms) : 0, 8930
Telemetry [candidate] (9.033 ms) : 0, 9033
Profiling [baseline] (102.256 ms) : 0, 102256
Profiling [candidate] (103.342 ms) : 0, 103342
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 18 unstable metrics. Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed
dateFormat X
axisFormat %s
section baseline
no_agent (386.813 Β΅s) : 367, 406
. : milestone, 387,
iast (516.4 Β΅s) : 495, 538
. : milestone, 516,
iast_FULL (735.829 Β΅s) : 714, 758
. : milestone, 736,
iast_GLOBAL (573.642 Β΅s) : 550, 597
. : milestone, 574,
iast_HARDCODED_SECRET_DISABLED (511.929 Β΅s) : 490, 533
. : milestone, 512,
iast_INACTIVE (467.099 Β΅s) : 445, 489
. : milestone, 467,
iast_TELEMETRY_OFF (508.127 Β΅s) : 486, 530
. : milestone, 508,
tracing (470.344 Β΅s) : 447, 493
. : milestone, 470,
section candidate
no_agent (376.262 Β΅s) : 357, 396
. : milestone, 376,
iast (514.304 Β΅s) : 493, 536
. : milestone, 514,
iast_FULL (737.11 Β΅s) : 715, 759
. : milestone, 737,
iast_GLOBAL (560.891 Β΅s) : 539, 583
. : milestone, 561,
iast_HARDCODED_SECRET_DISABLED (515.283 Β΅s) : 493, 537
. : milestone, 515,
iast_INACTIVE (468.495 Β΅s) : 447, 490
. : milestone, 468,
iast_TELEMETRY_OFF (500.898 Β΅s) : 479, 522
. : milestone, 501,
tracing (462.803 Β΅s) : 442, 484
. : milestone, 463,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed
dateFormat X
axisFormat %s
section baseline
no_agent (1.359 ms) : 1339, 1379
. : milestone, 1359,
appsec (1.743 ms) : 1719, 1767
. : milestone, 1743,
appsec_no_iast (1.748 ms) : 1725, 1772
. : milestone, 1748,
code_origins (1.698 ms) : 1672, 1724
. : milestone, 1698,
iast (1.524 ms) : 1499, 1549
. : milestone, 1524,
profiling (1.548 ms) : 1523, 1573
. : milestone, 1548,
tracing (1.508 ms) : 1483, 1534
. : milestone, 1508,
section candidate
no_agent (1.346 ms) : 1326, 1366
. : milestone, 1346,
appsec (1.736 ms) : 1712, 1760
. : milestone, 1736,
appsec_no_iast (1.757 ms) : 1732, 1782
. : milestone, 1757,
code_origins (1.685 ms) : 1657, 1712
. : milestone, 1685,
iast (1.525 ms) : 1501, 1550
. : milestone, 1525,
profiling (1.565 ms) : 1539, 1590
. : milestone, 1565,
tracing (1.508 ms) : 1483, 1533
. : milestone, 1508,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed
dateFormat X
axisFormat %s
section baseline
no_agent (15.454 s) : 15454000, 15454000
. : milestone, 15454000,
appsec (15.176 s) : 15176000, 15176000
. : milestone, 15176000,
iast (18.992 s) : 18992000, 18992000
. : milestone, 18992000,
iast_GLOBAL (17.605 s) : 17605000, 17605000
. : milestone, 17605000,
profiling (15.549 s) : 15549000, 15549000
. : milestone, 15549000,
tracing (14.945 s) : 14945000, 14945000
. : milestone, 14945000,
section candidate
no_agent (15.421 s) : 15421000, 15421000
. : milestone, 15421000,
appsec (14.938 s) : 14938000, 14938000
. : milestone, 14938000,
iast (18.812 s) : 18812000, 18812000
. : milestone, 18812000,
iast_GLOBAL (18.259 s) : 18259000, 18259000
. : milestone, 18259000,
profiling (15.206 s) : 15206000, 15206000
. : milestone, 15206000,
tracing (14.78 s) : 14780000, 14780000
. : milestone, 14780000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed
dateFormat X
axisFormat %s
section baseline
no_agent (1.472 ms) : 1460, 1483
. : milestone, 1472,
appsec (2.371 ms) : 2324, 2418
. : milestone, 2371,
iast (2.166 ms) : 2106, 2226
. : milestone, 2166,
iast_GLOBAL (2.201 ms) : 2141, 2260
. : milestone, 2201,
profiling (2.046 ms) : 1998, 2095
. : milestone, 2046,
tracing (1.981 ms) : 1935, 2027
. : milestone, 1981,
section candidate
no_agent (1.47 ms) : 1458, 1481
. : milestone, 1470,
appsec (2.373 ms) : 2327, 2420
. : milestone, 2373,
iast (2.153 ms) : 2094, 2213
. : milestone, 2153,
iast_GLOBAL (2.19 ms) : 2131, 2249
. : milestone, 2190,
profiling (2.013 ms) : 1965, 2062
. : milestone, 2013,
tracing (1.985 ms) : 1939, 2030
. : milestone, 1985,
|
harels
reviewed
Mar 13, 2025
...on/spark/src/main/java/datadog/trace/instrumentation/spark/AbstractDatadogSparkListener.java
Outdated
Show resolved
Hide resolved
d36e910 to
d48bb91
Compare
kr-igor
approved these changes
Mar 13, 2025
| @SuppressForbidden // called at most once per spark application | ||
| private static String removeUuidFromEndOfString(String input) { | ||
| return input.replaceAll( | ||
| "_[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$", ""); | ||
| } | ||
|
|
||
| private Optional<Class> loadClass(String className) { |
There was a problem hiding this comment.
Why don't we just instrument OL listener instead? I.e. add listener setup code which sets all required settings if not present?
| @@ -74,6 +77,7 @@ public abstract class AbstractDatadogSparkListener extends SparkListener { | |||
| private final int MAX_COLLECTION_SIZE = 5000; | |||
| private final int MAX_ACCUMULATOR_SIZE = 50000; | |||
| private final String RUNTIME_TAGS_PREFIX = "spark.datadog.tags."; | |||
| private static final String AGENT_OL_ENDPOINT = "openlineage/api/v1/lineage"; | |||
There was a problem hiding this comment.
Not directly related to this PR since it is an agent change, but to keep the same default as openlineage, wondering if it could be easier to keep the endpoint name as /api/v1/lineage in the agent
There was a problem hiding this comment.
My idea was that /api/v1/lineage endpoint was too confusing, and that as we're setting this - not customer - the additional endpoint setting does not matter that much. We can do a change on the client still, it was not yet released.
| @@ -151,8 +156,50 @@ public AbstractDatadogSparkListener(SparkConf sparkConf, String appId, String sp | |||
| finishApplication(System.currentTimeMillis(), null, 0, null); | |||
| } | |||
| })); | |||
| initApplicationSpanIfNotInitialized(); | |||
There was a problem hiding this comment.
I don't think we should create the application span in the constructor/onApplicationStart.
In the case of Databricks or streaming jobs, we are currently not creating a application span:
- in databricks the parent of a spark job is the databricks task that launched it
- in streaming, we don't attach the streaming span to the application span
In those cases, I think we should skip overriding OpenLineage for now
There was a problem hiding this comment.
Removed this here, added check for databricks/streaming
c3e99e1 to
a582374
Compare
55d9b0d to
ccd94f3
Compare
b4637a8 to
1c43c96
Compare
amarziali
reviewed
Mar 27, 2025
| public static void exit(@Advice.This Object self) throws IllegalAccessException { | ||
| LoggerFactory.getLogger(Config.class).debug("Checking for OpenLineageSparkListener"); | ||
| try { | ||
| Field conf = self.getClass().getDeclaredField("conf"); |
There was a problem hiding this comment.
can you use @FieldValue to have the same without using the reflection?
There was a problem hiding this comment.
I would like to - but the instrumented class clones the object so it's not the same one as what's passed to the method.
There was a problem hiding this comment.
I was not proposing to capture the argument of the constructor but to use @FieldValue to directly map the field (https://javadoc.io/doc/net.bytebuddy/byte-buddy/1.10.2/net/bytebuddy/asm/Advice.FieldValue.html) like this:
public static void exit(@Advice.This Object self, @FieldValue("conf") SparkConf conf) {
...
amarziali
reviewed
Mar 27, 2025
| @Override | ||
| public String[] knownMatchingTypes() { | ||
| return new String[] { | ||
| "io.openlineage.spark.agent.OpenLineageSparkListener", "org.apache.spark.util.Utils" |
There was a problem hiding this comment.
why org.apache.spark.util.Utils is also matching the advice activation while the method advice is only filtering by io.openlineage.spark.agent.OpenLineageSparkListener?
amarziali
reviewed
Mar 27, 2025
...on/spark/src/main/java/datadog/trace/instrumentation/spark/AbstractSparkInstrumentation.java
Outdated
Show resolved
Hide resolved
amarziali
reviewed
Mar 27, 2025
...on/spark/src/main/java/datadog/trace/instrumentation/spark/AbstractSparkInstrumentation.java
Outdated
Show resolved
Hide resolved
amarziali
reviewed
Mar 27, 2025
...on/spark/src/main/java/datadog/trace/instrumentation/spark/AbstractDatadogSparkListener.java
Outdated
Show resolved
Hide resolved
amarziali
reviewed
Mar 27, 2025
...rk/spark_2.12/src/main/java/datadog/trace/instrumentation/spark/Spark212Instrumentation.java
Outdated
Show resolved
Hide resolved
amarziali
reviewed
Mar 27, 2025
...rk/spark_2.13/src/main/java/datadog/trace/instrumentation/spark/Spark213Instrumentation.java
Outdated
Show resolved
Hide resolved
amarziali
requested changes
Mar 27, 2025
1c43c96 to
34d6cb4
Compare
|
Thanks @amarziali, applied the changes. Added one more change - using |
34d6cb4 to
5cd3c14
Compare
amarziali
approved these changes
Mar 31, 2025
5cd3c14 to
f596261
Compare
7 tasks
bdb681c to
c34a423
Compare
Signed-off-by: Maciej Obuchowski <[email protected]>
Signed-off-by: Maciej Obuchowski <[email protected]>
c34a423 to
83cf093
Compare
svc-squareup-copybara
pushed a commit
to cashapp/misk
that referenced
this pull request
Apr 11, 2025
| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | org.flywaydb.flyway | plugin | misk/gradle/libs.versions.toml | gradle | minor | `11.6.0` -> `11.7.0` | | [com.squareup.okio:okio-fakefilesystem](https://github.com/square/okio) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `3.10.2` -> `3.11.0` | | [com.squareup.okio:okio](https://github.com/square/okio) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `3.10.2` -> `3.11.0` | | [com.autonomousapps.dependency-analysis](https://github.com/autonomousapps/dependency-analysis-android-gradle-plugin) | plugin | misk/gradle/libs.versions.toml | gradle | minor | `2.15.0` -> `2.16.0` | | [com.datadoghq:dd-trace-api](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.47.3` -> `1.48.1` | | [com.datadoghq:dd-trace-ot](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.47.3` -> `1.48.1` | | [software.amazon.awssdk:sdk-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.18` -> `2.31.20` | | [software.amazon.awssdk:sqs](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.18` -> `2.31.20` | | [software.amazon.awssdk:dynamodb-enhanced](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.18` -> `2.31.20` | | [software.amazon.awssdk:dynamodb](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.18` -> `2.31.20` | | [software.amazon.awssdk:aws-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.18` -> `2.31.20` | | [software.amazon.awssdk:bom](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.18` -> `2.31.20` | | [software.amazon.awssdk:auth](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.18` -> `2.31.20` | --- ### Release Notes <details> <summary>square/okio (com.squareup.okio:okio-fakefilesystem)</summary> ### [`v3.11.0`](https://github.com/square/okio/blob/HEAD/CHANGELOG.md#Version-3110) *2025-04-09* - Fix: Clear the deflater's byte array reference - New: Faster implementation of `String.decodeHex()` on Kotlin/JS. - New: Declare `EXACTLY_ONCE` execution for blocks like `Closeable.use {}` and `FileSystem.read {}`. - Upgrade: \[Kotlin 2.1.20]\[kotlin\_2\_1\_20]. </details> <details> <summary>autonomousapps/dependency-analysis-android-gradle-plugin (com.autonomousapps.dependency-analysis)</summary> ### [`v2.16.0`](https://github.com/autonomousapps/dependency-analysis-android-gradle-plugin/blob/HEAD/CHANGELOG.md#Version-2160) - \[Feat]: support `com.android.test` projects. - \[Feat]: support typesafe project accessors with opt-in. ```kotlin dependencyAnalysis { useTypesafeProjectAccessors(true) // false by default } ``` </details> <details> <summary>datadog/dd-trace-java (com.datadoghq:dd-trace-api)</summary> ### [`v1.48.1`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.48.1): 1.48.1 ### Components #### Tracer internal logging - π Remove print line causing unnecessary logs ([#​8687](DataDog/dd-trace-java#8687) - [@​sarahchen6](https://github.com/sarahchen6)) ### [`v1.48.0`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.48.0): 1.48.0 ### Known Bugs > \[!NOTE] > If you are experiencing issues with spamming timeout logs, please update to the [latest version](https://github.com/DataDog/dd-trace-java/releases/latest) or set [JDK_SOCKET_ENABLED](https://github.com/DataDog/dd-trace-java/blob/33fc3c9a9b7cda3beda88b8b3e5224ae2b10764a/dd-trace-api/src/main/java/datadog/trace/api/config/GeneralConfig.java#L98) to false. ### Components #### Application Security Management (IAST) - β¨ Fix vulnerability location org.jose4j.lang.HashUtil ([#​8610](DataDog/dd-trace-java#8610) - [@​jandro996](https://github.com/jandro996)) - β¨ Fix weak randomness in oracle.ucp.util.OpaqueString ([#​8609](DataDog/dd-trace-java#8609) - [@​jandro996](https://github.com/jandro996)) - β¨ Fix weak hash false positive in oracle.security.o5logon.O5Logon ([#​8608](DataDog/dd-trace-java#8608) - [@​jandro996](https://github.com/jandro996)) - π Prevent before callsites targeting constructors in super calls ([#​8549](DataDog/dd-trace-java#8549) - [@​manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) #### Application Security Management (WAF) - β¨ Update login events public SDK to V2 ([#​8620](DataDog/dd-trace-java#8620) - [@​manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - π Send RASP LFI capability only when AppSec is statically enabled ([#​8573](DataDog/dd-trace-java#8573) - [@​jandro996](https://github.com/jandro996)) - β¨ Improve detection of missing request end events ([#​8510](DataDog/dd-trace-java#8510) - [@​smola](https://github.com/smola)) - π§Ή Remove remote configuration for API Security sampling rate ([#​8486](DataDog/dd-trace-java#8486) - [@​smola](https://github.com/smola)) - β¨ Add setUser to user monitoring SDK ([#​8482](DataDog/dd-trace-java#8482) - [@​manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - β¨ Add missing address for signup event ([#​8469](DataDog/dd-trace-java#8469) - [@​manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - β¨ Allow login events SDK to be used with appsec disabled ([#​8464](DataDog/dd-trace-java#8464) - [@​manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - β¨ Add support for endpoint discovery in spring mvc ([#​8352](DataDog/dd-trace-java#8352) - [@​manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - β¨ New API Security sampling algorithm ([#​8178](DataDog/dd-trace-java#8178) - [@​ValentinZakharov](https://github.com/ValentinZakharov)) #### Build & Tooling - β¨ Add buffer size customizability to JDK UDS support ([#​8629](DataDog/dd-trace-java#8629) - [@​sarahchen6](https://github.com/sarahchen6)) - β¨ Add JDK built-in support for UDS on Java 16+ ([#​8314](DataDog/dd-trace-java#8314) - [@​sarahchen6](https://github.com/sarahchen6)) #### Configuration at Runtime - π Send RASP LFI capability only when AppSec is statically enabled ([#​8573](DataDog/dd-trace-java#8573) - [@​jandro996](https://github.com/jandro996)) #### Continuous Integration Visibility - π Prevent double reporting of Scalatest events when using SBT with test forking ([#​8682](DataDog/dd-trace-java#8682) - [@​nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - π Shutdown CI Visibility test event handlers before tracer ([#​8677](DataDog/dd-trace-java#8677) - [@​nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - π Do not apply JUnit 4 instrumentation to MUnit runners ([#​8675](DataDog/dd-trace-java#8675), [#​8683](DataDog/dd-trace-java#8683) - [@​nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - β¨ Remove error log when source path resolution fails on isModified check ([#​8663](DataDog/dd-trace-java#8663) - [@​daniel-mohedano](https://github.com/daniel-mohedano)) - β¨ Implement tests reordering for JUnit 4 ([#​8650](DataDog/dd-trace-java#8650) - [@​daniel-mohedano](https://github.com/daniel-mohedano)) - π Set default Attempt to Fix retries if none provided from the backend ([#​8615](DataDog/dd-trace-java#8615) - [@​daniel-mohedano](https://github.com/daniel-mohedano)) - β¨ Allow to manually set PR info ([#​8566](DataDog/dd-trace-java#8566) - [@​nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - π Fix Test Optimization init when repo root cannot be determined ([#​8533](DataDog/dd-trace-java#8533) - [@​nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - β¨ Add capabilities tagging ([#​8499](DataDog/dd-trace-java#8499), [#​8540](DataDog/dd-trace-java#8540) - [@​daniel-mohedano](https://github.com/daniel-mohedano)) #### Crash tracking - π Remove dependency on bash from crash/oome uploder scripts ([#​8652](DataDog/dd-trace-java#8652) - [@​jbachorik](https://github.com/jbachorik)) #### Data Streams Monitoring - β¨ e2e pipeline configuration when data jobs is enabled ([#​8553](DataDog/dd-trace-java#8553) - [@​kr-igor](https://github.com/kr-igor)) #### Dynamic Instrumentation - π Fix In-Product when config is empty ([#​8679](DataDog/dd-trace-java#8679) - [@​jpbempel](https://github.com/jpbempel)) - β¨ Add support for filtering shaded third-party libs ([#​8612](DataDog/dd-trace-java#8612) - [@​jpbempel](https://github.com/jpbempel)) - β¨ Add In-Product Enablement ([#​8587](DataDog/dd-trace-java#8587) - [@​jpbempel](https://github.com/jpbempel)) - β¨β‘ Reduce footprint of SourceFile tracking ([#​8524](DataDog/dd-trace-java#8524) - [@​jpbempel](https://github.com/jpbempel)) - β¨β‘ Optimize the SourceFile tracking ([#​8520](DataDog/dd-trace-java#8520) - [@​jpbempel](https://github.com/jpbempel)) #### OpenTracing - π§Ή Remove activeScope() use in OpenTracing shim ([#​8478](DataDog/dd-trace-java#8478) - [@​mcculls](https://github.com/mcculls)) #### Profiling - β¨ Add profiler env check command to AgentCLI ([#​8671](DataDog/dd-trace-java#8671) - [@​jbachorik](https://github.com/jbachorik)) - β¨ Bump ddprof to 1.23.0 ([#​8668](DataDog/dd-trace-java#8668) - [@​jbachorik](https://github.com/jbachorik)) - Fix a crash related to ElfParser::loadSymbolTable ([#​191](DataDog/dd-trace-java#191)) by [@​yanglong1010](https://github.com/yanglong1010) in DataDog/java-profiler#192 - Unwind String.indexOf intrinsic on AArch64 by [@​MattAlp](https://github.com/MattAlp) in DataDog/java-profiler#193 - Fix Java 24 support by [@​jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#194 - A set of fixes related to clang, aarch64 and musl pecularities of vmstructs stack unwinder by [@​jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#199 - π Remove process information from JFR recording ([#​8661](DataDog/dd-trace-java#8661) - [@​r1viollet](https://github.com/r1viollet)) - π Make TempLocationManager USER aware ([#​8605](DataDog/dd-trace-java#8605) - [@​jbachorik](https://github.com/jbachorik)) - β¨ Extract git tags from embedded git.properties and datadog_git.properties ([#​8561](DataDog/dd-trace-java#8561) - [@​wmouchere](https://github.com/wmouchere)) #### Telemetry - π Fix appsec.rasp.error and appsec.waf.error telemetry metrics ([#​8624](DataDog/dd-trace-java#8624) - [@​jandro996](https://github.com/jandro996)) - β¨ Create metric: appsec.rasp.rule.skipped ([#​8618](DataDog/dd-trace-java#8618) - [@​jandro996](https://github.com/jandro996)) - β¨ Extract git tags from embedded git.properties and datadog_git.properties ([#​8561](DataDog/dd-trace-java#8561) - [@​wmouchere](https://github.com/wmouchere)) #### Testing - π§Ή Simplify ssi tests one-pipeline ([#​8558](DataDog/dd-trace-java#8558) - [@​robertomonteromiguel](https://github.com/robertomonteromiguel)) - β¨ Add smoke tests for java's concurrent API ([#​8438](DataDog/dd-trace-java#8438) - [@​sarahchen6](https://github.com/sarahchen6)) #### Trace context propagation - β¨ Adding Support for `TRACE_PROPAGATION_BEHAVIOR_EXTRACT` ([#​8535](DataDog/dd-trace-java#8535) - [@​mhlidd](https://github.com/mhlidd)) #### Tracer core - π Ensure shaded helpers have unique names ([#​8559](DataDog/dd-trace-java#8559) - [@​amarziali](https://github.com/amarziali)) - β¨ Support common config sources for user-provided git info ([#​8547](DataDog/dd-trace-java#8547) - [@​nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - β¨ Make the default config sources more robust when a security manager is installed ([#​8544](DataDog/dd-trace-java#8544) - [@​mcculls](https://github.com/mcculls)) - β¨ Support targeting services with configurations in stable configuration file ([#​8526](DataDog/dd-trace-java#8526) - [@​mtoffl01](https://github.com/mtoffl01)) - β¨ Add new parser for `DD_TAGS` and prioritizing `DD_SERVICE` ([#​8296](DataDog/dd-trace-java#8296) - [@​mhlidd](https://github.com/mhlidd)) #### Tracer internal logging - π Add missing debug log for the cloudPayloadTaggingServices config ([#​8600](DataDog/dd-trace-java#8600) - [@​ygree](https://github.com/ygree)) - β¨ Add the possibility to output the logs of the Java tracer in JSON ([#​8083](DataDog/dd-trace-java#8083) - [@​cecile75](https://github.com/cecile75)) #### Tracer public API - β¨ Introducing `DD_TRACE_EXPERIMENTAL_FEATURES_ENABLED` Config ([#​8536](DataDog/dd-trace-java#8536) - [@​mhlidd](https://github.com/mhlidd)) - β¨ Config Consistency Round 2 ([#​8489](DataDog/dd-trace-java#8489) - [@​mhlidd](https://github.com/mhlidd)) ### Instrumentations #### - π Fix NPE in getMdcCopy of LoggingEventInstrumentation ([#​8599](DataDog/dd-trace-java#8599) - [@​ygree](https://github.com/ygree)) #### Apache Spark instrumentation - β¨ Instrument Runtime.exit() to finish spark application spans ([#​8572](DataDog/dd-trace-java#8572) - [@​paul-laffon-dd](https://github.com/paul-laffon-dd)) - β¨ Configure OpenLineage if present in Spark instrumentation ([#​8541](DataDog/dd-trace-java#8541) - [@​mobuchowski](https://github.com/mobuchowski)) #### Armeria Instrumentation - β¨ Support armeria grpc 1.32.3 ([#​8606](DataDog/dd-trace-java#8606) - [@​github-actions](https://github.com/github-actions)\[bot]) #### AWS DynamoDB Instrumentation - β¨ Create DynamoDB instrumentation + add span pointers for `updateItem` and `deleteItem` ([#​8490](DataDog/dd-trace-java#8490) - [@​nhulston](https://github.com/nhulston)) #### AWS SDK instrumentation - β¨ Add DynamoDB in DEFAULT_TRACE_CLOUD_PAYLOAD_TAGGING_SERVICES ([#​8595](DataDog/dd-trace-java#8595) - [@​joeyzhao2018](https://github.com/joeyzhao2018)) #### Azure Functions instrumentation - β¨ Enable tracer computed trace metrics by default for Azure Functions ([#​8518](DataDog/dd-trace-java#8518) - [@​duncanpharvey](https://github.com/duncanpharvey)) - π‘ Add azure-functions instrumentation ([#​8432](DataDog/dd-trace-java#8432) - [@​duncanpharvey](https://github.com/duncanpharvey)) #### Core Java language instrumentation - π Fix ForkJoinPool.execute() instrumentation on Java 21+ ([#​8560](DataDog/dd-trace-java#8560) - [@​PerfectSlayer](https://github.com/PerfectSlayer)) #### Eclipse Vert.x instrumentation - β¨ Add vertx postgresql client instrumentation ([#​8471](DataDog/dd-trace-java#8471) - [@​vandonr](https://github.com/vandonr) - thanks for the contribution!) #### Kafka instrumentation - β¨ Support and test kafka-clients 4 ([#​8581](DataDog/dd-trace-java#8581) - [@​amarziali](https://github.com/amarziali)) #### Kotlin instrumentation - β¨ Avoid disconnected traces when using Kotlin flowOn ([#​8651](DataDog/dd-trace-java#8651) - [@​mcculls](https://github.com/mcculls)) #### OpenTelemetry instrumentation - π§Ή Migrate OtelContext wrapper to new internal Context API ([#​8645](DataDog/dd-trace-java#8645) - [@​mcculls](https://github.com/mcculls)) #### Spring instrumentation - π Support CompletableFuture on spring webmvc controllers ([#​8659](DataDog/dd-trace-java#8659) - [@​amarziali](https://github.com/amarziali)) - β¨ Add support for endpoint discovery in spring mvc ([#​8352](DataDog/dd-trace-java#8352) - [@​manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) #### WebSocket Instrumentation - β¨ Instrument Jetty websocket pojo ([#​8562](DataDog/dd-trace-java#8562) - [@​amarziali](https://github.com/amarziali)) - π‘ Instrument Java Websocket API (JSR356) ([#​8440](DataDog/dd-trace-java#8440) - [@​amarziali](https://github.com/amarziali)) #### All other instrumentations - β¨ Introduce cache for peer.hostname lookup ([#​8601](DataDog/dd-trace-java#8601) - [@​mcculls](https://github.com/mcculls)) - β¨ Support pekko http 1.1 ([#​8532](DataDog/dd-trace-java#8532) - [@​amarziali](https://github.com/amarziali)) </details> --- ### Configuration π **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). π¦ **Automerge**: Disabled by config. Please merge this manually once you are satisfied. β» **Rebasing**: Never, or you tick the rebase/retry checkbox. π» **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: 331314f71acaced3adc75ea5d7e855c248d593fc
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
If OpenLineage library is present on the classpath, instrument SparkListenerBus to inject the OpenLineageSparkListener with additional tags ourselves.
Motivation
https://docs.google.com/document/d/14nmf3UcqhzoOfooiveoEBUO-HHejKalLR-qAKzbR2PI/edit?tab=t.0#heading=h.dpfdi5vyt85h
https://docs.google.com/document/d/1EwNCsnhnilL6YHYMg4UYwqK_yKgBEUsxys669h8l_-M/edit?tab=t.0#heading=h.o8z5ox632qxx
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]