Thanks to visit codestin.com
Credit goes to github.com

Skip to content

fix(datastore): reduce flakiness of E2E tracing tests #13645

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lqiu96 merged 4 commits into from
Jul 6, 2026
Merged

fix(datastore): reduce flakiness of E2E tracing tests #13645

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1e5a18c
fix(datastore): reduce flakiness of E2E tracing tests
lqiu96 Jul 2, 2026
9493964
fix(datastore): use openTelemetrySdk.close() in teardown
lqiu96 Jul 6, 2026
91b9e21
fix(datastore): make tracer and SDK fields non-static in E2E tests
lqiu96 Jul 6, 2026
6d01781
fix(datastore): address PR comments on E2E tracing tests
lqiu96 Jul 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 39 additions & 17 deletions .../google-cloud-datastore/src/test/java/com/google/cloud/datastore/it/ITE2ETracingTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,14 +227,12 @@ private boolean dfsContainsCallStack(long spanId, List<String> expectedCallStack

private static final int NUM_SPAN_ID_BYTES = 16;

private static final int GET_TRACE_RETRY_COUNT = 60;
private static final int GET_TRACE_RETRY_COUNT = 120;

@blakeli0 blakeli0 Jul 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that we do Thread.sleep after each unsuccessful check, consider migrate the test to use Awaitility?

@lqiu96 lqiu96 Jul 6, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Raised a PR #13668


private static final int GET_TRACE_RETRY_BACKOFF_MILLIS = 1000;

private static final int TRACE_FORCE_FLUSH_MILLIS = 5000;

private static final int TRACE_PROVIDER_SHUTDOWN_MILLIS = 1000;

private static Key KEY1;

private static Key KEY2;
Expand All @@ -246,22 +244,24 @@ private boolean dfsContainsCallStack(long spanId, List<String> expectedCallStack
// Random int generator for trace ID and span ID
private static Random random;

private static TraceExporter traceExporter;
private static Credentials credentials;

private TraceExporter traceExporter;

// Required for reading back traces from Cloud Trace for validation
private static TraceServiceClient traceClient;

// Custom SpanContext for each test, required for TraceID injection
private static SpanContext customSpanContext;
private SpanContext customSpanContext;

// Trace read back from Cloud Trace using traceClient for verification
private static Trace retrievedTrace;
private Trace retrievedTrace;

private static String rootSpanName;
private static Tracer tracer;
private String rootSpanName;
private Tracer tracer;

// Required to set custom-root span
private static OpenTelemetrySdk openTelemetrySdk;
private OpenTelemetrySdk openTelemetrySdk;

private static String projectId;

Expand All @@ -287,14 +287,7 @@ public static void setup() throws IOException {
// Share the same credentials used by Datastore client with the TraceExporter and
// TraceServiceClient to ensure consistency and avoid auth issues in environments
// where default ADC resolution might fail for the exporter.
Credentials credentials = DatastoreOptions.getDefaultInstance().getCredentials();

TraceConfiguration.Builder traceConfigurationBuilder =
TraceConfiguration.builder().setProjectId(projectId);
if (credentials != null) {
traceConfigurationBuilder.setCredentials(credentials);
}
traceExporter = TraceExporter.createWithConfiguration(traceConfigurationBuilder.build());
credentials = DatastoreOptions.getDefaultInstance().getCredentials();

TraceServiceSettings.Builder clientBuilder = TraceServiceSettings.newBuilder();
if (credentials != null) {
Expand All @@ -310,6 +303,13 @@ public void before() throws Exception {
Resource resource =
Resource.getDefault().merge(Resource.builder().put(SERVICE_NAME, "Sparky").build());

TraceConfiguration.Builder traceConfigurationBuilder =
TraceConfiguration.builder().setProjectId(projectId);
if (credentials != null) {
traceConfigurationBuilder.setCredentials(credentials);
}
traceExporter = TraceExporter.createWithConfiguration(traceConfigurationBuilder.build());

if (isUsingGlobalOpenTelemetrySDK()) {
openTelemetrySdk =
OpenTelemetrySdk.builder()
Expand Down Expand Up @@ -393,7 +393,22 @@ public void after() throws Exception {
tracer = null;
retrievedTrace = null;
customSpanContext = null;
try {
if (openTelemetrySdk != null) {
openTelemetrySdk.close();
}
} finally {
if (traceExporter != null) {
try {
// Attempt to shut down traceExporter.
traceExporter.shutdown();
} catch (Exception e) {
logger.log(Level.WARNING, "Failed to shut down traceExporter", e);
}
}
}
openTelemetrySdk = null;
traceExporter = null;
}

@AfterClass
Expand Down Expand Up @@ -437,10 +452,17 @@ protected Span getNewRootSpanWithContext() {
}

protected void waitForTracesToComplete() throws Exception {
if (openTelemetrySdk == null) {
logger.warning("OpenTelemetrySdk is null, cannot flush traces");
return;
}
logger.info("Flushing traces...");
CompletableResultCode completableResultCode =
openTelemetrySdk.getSdkTracerProvider().forceFlush();
completableResultCode.join(TRACE_FORCE_FLUSH_MILLIS, TimeUnit.MILLISECONDS);
if (!completableResultCode.isSuccess()) {
logger.warning("Force flush did not complete successfully");
}
Comment on lines 460 to +465

@gemini-code-assist gemini-code-assist Bot Jul 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If isUsingGlobalOpenTelemetrySDK() is false, openTelemetrySdk will be null, leading to a NullPointerException when calling openTelemetrySdk.getSdkTracerProvider(). Adding a null check and falling back to flushing traceExporter directly ensures robust behavior.

    if (openTelemetrySdk != null) {
      CompletableResultCode completableResultCode =
          openTelemetrySdk.getSdkTracerProvider().forceFlush();
      completableResultCode.join(TRACE_FORCE_FLUSH_MILLIS, TimeUnit.MILLISECONDS);
      if (!completableResultCode.isSuccess()) {
        logger.warning("Force flush did not complete successfully");
      }
    } else if (traceExporter != null) {
      CompletableResultCode completableResultCode = traceExporter.flush();
      completableResultCode.join(TRACE_FORCE_FLUSH_MILLIS, TimeUnit.MILLISECONDS);
      if (!completableResultCode.isSuccess()) {
        logger.warning("Trace exporter flush did not complete successfully");
      }
    }
References
  1. When using lazily initialized resources (such as ExecutorService), ensure that teardown or close methods perform explicit null checks before invoking methods on them to prevent NullPointerException.

}

// Validates `retrievedTrace`. Cloud Trace indexes traces w/ eventual consistency, even when
Expand Down
Loading