Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Add GC cause to jvm GC related metric #13750

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Apr 28, 2025
Merged

Add GC cause to jvm GC related metric #13750

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
d787b9b
Add GC cause to jvm GC related metric
steverao Apr 22, 2025
61db9ea
Add the configuration to related README.md
steverao Apr 22, 2025
6be7963
Address review comments
steverao Apr 24, 2025
05128b5
Address review comments
steverao Apr 24, 2025
1812bd2
Address review comments
steverao Apr 24, 2025
a312ef2
Address review comments
steverao Apr 24, 2025
8dc57b9
Parameterized test
steverao Apr 24, 2025
4c6cc09
Reformatting table
steverao Apr 24, 2025
3bbfa4f
Address review comments
steverao Apr 25, 2025
d84e697
Address review comments
steverao Apr 27, 2025
3555002
simplify
laurit Apr 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions instrumentation/runtime-telemetry/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
# Settings for the Runtime Telemetry instrumentation

| System property | Type | Default | Description |
|--------------------------------------------------------------------------|---------|---------|-------------------------------------------------------------------|
| `otel.instrumentation.runtime-telemetry.emit-experimental-telemetry` | Boolean | `false` | Enable the capture of experimental metrics. |
| `otel.instrumentation.runtime-telemetry-java17.enable-all` | Boolean | `false` | Enable the capture of all JFR based metrics. |
| `otel.instrumentation.runtime-telemetry-java17.enabled` | Boolean | `false` | Enable the capture of JFR based metrics. |
| `otel.instrumentation.runtime-telemetry.package-emitter.enabled` | Boolean | `false` | Enable creating events for JAR libraries used by the application. |
| `otel.instrumentation.runtime-telemetry.package-emitter.jars-per-second` | Integer | 10 | The number of JAR files processed per second. |
| System property | Type | Default | Description |
|--------------------------------------------------------------------------|---------|---------|-----------------------------------------------------------------------------------|
| `otel.instrumentation.runtime-telemetry.capture-gc-cause` | Boolean | `false` | Enable the capture of the jvm.gc.cause attribute with the jvm.gc.duration metric. |
| `otel.instrumentation.runtime-telemetry.emit-experimental-telemetry` | Boolean | `false` | Enable the capture of experimental metrics. |
| `otel.instrumentation.runtime-telemetry-java17.enable-all` | Boolean | `false` | Enable the capture of all JFR based metrics. |
| `otel.instrumentation.runtime-telemetry-java17.enabled` | Boolean | `false` | Enable the capture of JFR based metrics. |
| `otel.instrumentation.runtime-telemetry.package-emitter.enabled` | Boolean | `false` | Enable creating events for JAR libraries used by the application. |
| `otel.instrumentation.runtime-telemetry.package-emitter.jars-per-second` | Integer | 10 | The number of JAR files processed per second. |
10 changes: 9 additions & 1 deletion ...in/java/io/opentelemetry/instrumentation/runtimemetrics/java17/RuntimeMetricsBuilder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ public final class RuntimeMetricsBuilder {

private boolean disableJmx = false;
private boolean enableExperimentalJmxTelemetry = false;
private boolean captureGcCause = false;

RuntimeMetricsBuilder(OpenTelemetry openTelemetry) {
this.openTelemetry = openTelemetry;
Expand Down Expand Up @@ -81,13 +82,20 @@ public RuntimeMetricsBuilder enableExperimentalJmxTelemetry() {
return this;
}

/** Enable the capture of the jvm.gc.cause attribute with the jvm.gc.duration metric. */
@CanIgnoreReturnValue
public RuntimeMetricsBuilder captureGcCause() {
captureGcCause = true;
return this;
}

/** Build and start an {@link RuntimeMetrics} with the config from this builder. */
public RuntimeMetrics build() {
List<AutoCloseable> observables =
disableJmx
? List.of()
: JmxRuntimeMetricsFactory.buildObservables(
openTelemetry, enableExperimentalJmxTelemetry);
openTelemetry, enableExperimentalJmxTelemetry, captureGcCause);
RuntimeMetrics.JfrRuntimeMetrics jfrRuntimeMetrics = buildJfrMetrics();
return new RuntimeMetrics(openTelemetry, observables, jfrRuntimeMetrics);
}
Expand Down
4 changes: 4 additions & 0 deletions ...pentelemetry/instrumentation/runtimemetrics/java17/internal/RuntimeMetricsConfigUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@ public static RuntimeMetrics configure(
builder.enableExperimentalJmxTelemetry();
}

if (config.getBoolean("otel.instrumentation.runtime-telemetry.capture-gc-cause", false)) {
builder.captureGcCause();
}

return builder.build();
}
}
32 changes: 23 additions & 9 deletions ...src/main/java/io/opentelemetry/instrumentation/runtimemetrics/java8/GarbageCollector.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,9 @@

import com.sun.management.GarbageCollectionNotificationInfo;
import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.common.AttributeKey;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.common.AttributesBuilder;
import io.opentelemetry.api.metrics.DoubleHistogram;
import io.opentelemetry.api.metrics.Meter;
import io.opentelemetry.instrumentation.runtimemetrics.java8.internal.JmxRuntimeMetricsUtil;
Expand Down Expand Up @@ -55,14 +57,17 @@ public final class GarbageCollector {

static final List<Double> GC_DURATION_BUCKETS = unmodifiableList(asList(0.01, 0.1, 1., 10.));

private static final AttributeKey<String> JVM_GC_CAUSE = AttributeKey.stringKey("jvm.gc.cause");

private static final NotificationFilter GC_FILTER =
notification ->
notification
.getType()
.equals(GarbageCollectionNotificationInfo.GARBAGE_COLLECTION_NOTIFICATION);

/** Register observers for java runtime memory metrics. */
public static List<AutoCloseable> registerObservers(OpenTelemetry openTelemetry) {
public static List<AutoCloseable> registerObservers(
OpenTelemetry openTelemetry, boolean captureGcCause) {
if (!isNotificationClassPresent()) {
logger.fine(
"The com.sun.management.GarbageCollectionNotificationInfo class is not available;"
Expand All @@ -73,14 +78,16 @@ public static List<AutoCloseable> registerObservers(OpenTelemetry openTelemetry)
return registerObservers(
openTelemetry,
ManagementFactory.getGarbageCollectorMXBeans(),
GarbageCollector::extractNotificationInfo);
GarbageCollector::extractNotificationInfo,
captureGcCause);
}

// Visible for testing
static List<AutoCloseable> registerObservers(
OpenTelemetry openTelemetry,
List<GarbageCollectorMXBean> gcBeans,
Function<Notification, GarbageCollectionNotificationInfo> notificationInfoExtractor) {
Function<Notification, GarbageCollectionNotificationInfo> notificationInfoExtractor,
boolean captureGcCause) {
Meter meter = JmxRuntimeMetricsUtil.getMeter(openTelemetry);

DoubleHistogram gcDuration =
Expand All @@ -98,7 +105,7 @@ static List<AutoCloseable> registerObservers(
}
NotificationEmitter notificationEmitter = (NotificationEmitter) gcBean;
GcNotificationListener listener =
new GcNotificationListener(gcDuration, notificationInfoExtractor);
new GcNotificationListener(gcDuration, notificationInfoExtractor, captureGcCause);
notificationEmitter.addNotificationListener(listener, GC_FILTER, null);
result.add(() -> notificationEmitter.removeNotificationListener(listener));
}
Expand All @@ -107,13 +114,16 @@ static List<AutoCloseable> registerObservers(

private static final class GcNotificationListener implements NotificationListener {

private final boolean captureGcCause;
private final DoubleHistogram gcDuration;
private final Function<Notification, GarbageCollectionNotificationInfo>
notificationInfoExtractor;

private GcNotificationListener(
DoubleHistogram gcDuration,
Function<Notification, GarbageCollectionNotificationInfo> notificationInfoExtractor) {
Function<Notification, GarbageCollectionNotificationInfo> notificationInfoExtractor,
boolean captureGcCause) {
this.captureGcCause = captureGcCause;
this.gcDuration = gcDuration;
this.notificationInfoExtractor = notificationInfoExtractor;
}
Expand All @@ -126,10 +136,14 @@ public void handleNotification(Notification notification, Object unused) {
String gcName = notificationInfo.getGcName();
String gcAction = notificationInfo.getGcAction();
double duration = notificationInfo.getGcInfo().getDuration() / MILLIS_PER_S;

gcDuration.record(
duration,
Attributes.of(JvmAttributes.JVM_GC_NAME, gcName, JvmAttributes.JVM_GC_ACTION, gcAction));
AttributesBuilder builder = Attributes.builder();
builder.put(JvmAttributes.JVM_GC_NAME, gcName);
builder.put(JvmAttributes.JVM_GC_ACTION, gcAction);
if (captureGcCause) {
String gcCause = notificationInfo.getGcCause();
builder.put(JVM_GC_CAUSE, gcCause);
}
gcDuration.record(duration, builder.build());
}
}

Expand Down
11 changes: 10 additions & 1 deletion ...ain/java/io/opentelemetry/instrumentation/runtimemetrics/java8/RuntimeMetricsBuilder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ public final class RuntimeMetricsBuilder {
private final OpenTelemetry openTelemetry;

private boolean enableExperimentalJmxTelemetry = false;
private boolean captureGcCause = false;

RuntimeMetricsBuilder(OpenTelemetry openTelemetry) {
this.openTelemetry = openTelemetry;
Expand All @@ -28,10 +29,18 @@ public RuntimeMetricsBuilder enableExperimentalJmxTelemetry() {
return this;
}

/** Enable the capture of the jvm.gc.cause attribute with the jvm.gc.duration metric. */
@CanIgnoreReturnValue
public RuntimeMetricsBuilder captureGcCause() {
captureGcCause = true;
return this;
}

/** Build and start an {@link RuntimeMetrics} with the config from this builder. */
public RuntimeMetrics build() {
List<AutoCloseable> observables =
JmxRuntimeMetricsFactory.buildObservables(openTelemetry, enableExperimentalJmxTelemetry);
JmxRuntimeMetricsFactory.buildObservables(
openTelemetry, enableExperimentalJmxTelemetry, captureGcCause);
return new RuntimeMetrics(observables);
}
}
4 changes: 2 additions & 2 deletions ...opentelemetry/instrumentation/runtimemetrics/java8/internal/JmxRuntimeMetricsFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@
public class JmxRuntimeMetricsFactory {
@SuppressWarnings("CatchingUnchecked")
public static List<AutoCloseable> buildObservables(
OpenTelemetry openTelemetry, boolean enableExperimentalJmxTelemetry) {
OpenTelemetry openTelemetry, boolean enableExperimentalJmxTelemetry, boolean captureGcCause) {
// Set up metrics gathered by JMX
List<AutoCloseable> observables = new ArrayList<>();
observables.addAll(Classes.registerObservers(openTelemetry));
observables.addAll(Cpu.registerObservers(openTelemetry));
observables.addAll(GarbageCollector.registerObservers(openTelemetry));
observables.addAll(GarbageCollector.registerObservers(openTelemetry, captureGcCause));
observables.addAll(MemoryPools.registerObservers(openTelemetry));
observables.addAll(Threads.registerObservers(openTelemetry));
if (enableExperimentalJmxTelemetry) {
Expand Down
4 changes: 4 additions & 0 deletions ...opentelemetry/instrumentation/runtimemetrics/java8/internal/RuntimeMetricsConfigUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,10 @@ public static RuntimeMetrics configure(
builder.enableExperimentalJmxTelemetry();
}

if (config.getBoolean("otel.instrumentation.runtime-telemetry.capture-gc-cause", false)) {
builder.captureGcCause();
}

return builder.build();
}
}
49 changes: 30 additions & 19 deletions ...test/java/io/opentelemetry/instrumentation/runtimemetrics/java8/GarbageCollectorTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,12 @@

package io.opentelemetry.instrumentation.runtimemetrics.java8;

import static io.opentelemetry.api.common.AttributeKey.stringKey;
import static io.opentelemetry.instrumentation.runtimemetrics.java8.ScopeUtil.EXPECTED_SCOPE;
import static io.opentelemetry.sdk.testing.assertj.OpenTelemetryAssertions.assertThat;
import static io.opentelemetry.sdk.testing.assertj.OpenTelemetryAssertions.equalTo;
import static io.opentelemetry.semconv.JvmAttributes.JVM_GC_ACTION;
import static io.opentelemetry.semconv.JvmAttributes.JVM_GC_NAME;
import static java.util.Collections.singletonList;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.mock;
Expand All @@ -15,17 +19,17 @@

import com.sun.management.GarbageCollectionNotificationInfo;
import com.sun.management.GcInfo;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.instrumentation.testing.junit.InstrumentationExtension;
import io.opentelemetry.instrumentation.testing.junit.LibraryInstrumentationExtension;
import java.lang.management.GarbageCollectorMXBean;
import java.util.concurrent.atomic.AtomicLong;
import javax.management.Notification;
import javax.management.NotificationEmitter;
import javax.management.NotificationListener;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.junit.jupiter.api.extension.RegisterExtension;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import org.mockito.ArgumentCaptor;
import org.mockito.Captor;
import org.mockito.Mock;
Expand All @@ -48,23 +52,27 @@ class GarbageCollectorTest {

@Captor private ArgumentCaptor<NotificationListener> listenerCaptor;

@Test
void registerObservers() {
@ParameterizedTest
@ValueSource(booleans = {true, false})
void registerObservers(boolean captureGcCause) {
GarbageCollector.registerObservers(
testing.getOpenTelemetry(),
singletonList(gcBean),
GarbageCollectorTest::getGcNotificationInfo);
GarbageCollectorTest::getGcNotificationInfo,
captureGcCause);

NotificationEmitter notificationEmitter = (NotificationEmitter) gcBean;
verify(notificationEmitter).addNotificationListener(listenerCaptor.capture(), any(), any());
NotificationListener listener = listenerCaptor.getValue();

listener.handleNotification(
createTestNotification("G1 Young Generation", "end of minor GC", 10), null);
createTestNotification("G1 Young Generation", "end of minor GC", "Allocation Failure", 10),
null);
listener.handleNotification(
createTestNotification("G1 Young Generation", "end of minor GC", 12), null);
createTestNotification("G1 Young Generation", "end of minor GC", "Allocation Failure", 12),
null);
listener.handleNotification(
createTestNotification("G1 Old Generation", "end of major GC", 11), null);
createTestNotification("G1 Old Generation", "end of major GC", "System.gc()", 11), null);

testing.waitAndAssertMetrics(
"io.opentelemetry.runtime-telemetry-java8",
Expand All @@ -83,30 +91,33 @@ void registerObservers() {
point
.hasCount(2)
.hasSum(0.022)
.hasAttributes(
Attributes.builder()
.put("jvm.gc.name", "G1 Young Generation")
.put("jvm.gc.action", "end of minor GC")
.build())
.hasAttributesSatisfyingExactly(
equalTo(JVM_GC_NAME, "G1 Young Generation"),
equalTo(JVM_GC_ACTION, "end of minor GC"),
equalTo(
stringKey("jvm.gc.cause"),
captureGcCause ? "Allocation Failure" : null))
.hasBucketBoundaries(GC_DURATION_BUCKETS),
point ->
point
.hasCount(1)
.hasSum(0.011)
.hasAttributes(
Attributes.builder()
.put("jvm.gc.name", "G1 Old Generation")
.put("jvm.gc.action", "end of major GC")
.build())
.hasAttributesSatisfyingExactly(
equalTo(JVM_GC_NAME, "G1 Old Generation"),
equalTo(JVM_GC_ACTION, "end of major GC"),
equalTo(
stringKey("jvm.gc.cause"),
captureGcCause ? "System.gc()" : null))
.hasBucketBoundaries(GC_DURATION_BUCKETS)))));
}

private static Notification createTestNotification(
String gcName, String gcAction, long duration) {
String gcName, String gcAction, String gcCause, long duration) {
GarbageCollectionNotificationInfo gcNotificationInfo =
mock(GarbageCollectionNotificationInfo.class);
when(gcNotificationInfo.getGcName()).thenReturn(gcName);
when(gcNotificationInfo.getGcAction()).thenReturn(gcAction);
when(gcNotificationInfo.getGcCause()).thenReturn(gcCause);
GcInfo gcInfo = mock(GcInfo.class);
when(gcInfo.getDuration()).thenReturn(duration);
when(gcNotificationInfo.getGcInfo()).thenReturn(gcInfo);
Expand Down
6 changes: 6 additions & 0 deletions ...t-autoconfigure/src/main/resources/META-INF/additional-spring-configuration-metadata.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -482,6 +482,12 @@
"description": "Enables the DB statement sanitization.",
"defaultValue": true
},
{
"name": "otel.instrumentation.runtime-telemetry.capture-gc-cause",
"type": "java.lang.Boolean",
"description": "Enable the capture of the jvm.gc.cause attribute with the jvm.gc.duration metric.",
"defaultValue": false
},
{
"name": "otel.instrumentation.runtime-telemetry.enabled",
"type": "java.lang.Boolean",
Expand Down
Loading