Thanks to visit codestin.com
Credit goes to github.com

Skip to content

fix(datastore): reduce flakiness of E2E tracing tests#13645

Merged
lqiu96 merged 4 commits into
mainfrom
investigate-issue-13644-java
Jul 6, 2026
Merged

fix(datastore): reduce flakiness of E2E tracing tests#13645
lqiu96 merged 4 commits into
mainfrom
investigate-issue-13644-java

Conversation

@lqiu96

@lqiu96 lqiu96 commented Jul 2, 2026

Copy link
Copy Markdown
Member

Fixes #13644

Increase the trace retry count to 120 (from 60) to allow more time for eventual consistency in Cloud Trace. Log a warning if forceFlush() does not complete successfully. Shutdown SdkTracerProvider in after() to ensure clean resource release.

TAG=agy
CONV=c8697e3d-c06b-4d5b-86e9-f8950e4298fd

Fixes #13644

Increase the trace retry count to 120 (from 60) to allow more time for
eventual consistency in Cloud Trace. Log a warning if forceFlush() does
not complete successfully. Shutdown SdkTracerProvider in after() to
ensure clean resource release.

TAG=agy
CONV=c8697e3d-c06b-4d5b-86e9-f8950e4298fd

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request increases the trace retry count, adds warning logging for unsuccessful trace force flushes, and introduces teardown logic for the OpenTelemetry SDK. The reviewer suggested a more robust and idiomatic way to shut down the SDK by calling openTelemetrySdk.close() instead of manually shutting down the tracer provider.

Use openTelemetrySdk.close() instead of manually shutting down the
tracer provider, as suggested by the reviewer. This is more robust and
idiomatic. Also remove the now unused constant
TRACE_PROVIDER_SHUTDOWN_MILLIS.

TAG=agy
CONV=c8697e3d-c06b-4d5b-86e9-f8950e4298fd
@lqiu96 lqiu96 requested review from blakeli0 and jinseopkim0 July 6, 2026 14:58
@lqiu96 lqiu96 marked this pull request as ready for review July 6, 2026 14:58
@lqiu96 lqiu96 requested review from a team as code owners July 6, 2026 14:58
@lqiu96 lqiu96 enabled auto-merge (squash) July 6, 2026 15:02
@lqiu96 lqiu96 added the release-please:force-run To run release-please label Jul 6, 2026
@release-please release-please Bot removed the release-please:force-run To run release-please label Jul 6, 2026
@lqiu96 lqiu96 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 6, 2026
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 6, 2026
@lqiu96 lqiu96 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 6, 2026
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 6, 2026
@lqiu96 lqiu96 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 6, 2026
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 6, 2026
Make openTelemetrySdk, traceExporter, tracer, and other tracing-related
fields non-static to avoid sharing them (and closing them) across tests,
which was causing "Channel shutdown invoked" errors. Store credentials
statically since they can be shared safely. Initialize traceExporter in
before() instead of setup().

TAG=agy
CONV=c8697e3d-c06b-4d5b-86e9-f8950e4298fd
@lqiu96

lqiu96 commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

/gemini review

private static final int NUM_SPAN_ID_BYTES = 16;

private static final int GET_TRACE_RETRY_COUNT = 60;
private static final int GET_TRACE_RETRY_COUNT = 120;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that we do Thread.sleep after each unsuccessful check, consider migrate the test to use Awaitility?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Raised a PR #13668

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors ITE2ETracingTest.java by converting several static fields (including traceExporter and openTelemetrySdk) to instance fields, increasing the trace retry count, and closing openTelemetrySdk during teardown. The review feedback highlights two critical issues: first, traceExporter is not closed in the after() method, which can lead to resource leaks; second, a potential NullPointerException could occur in waitForTracesToComplete() if openTelemetrySdk is null, which requires adding a null check and a fallback flush mechanism.

Comment on lines 396 to 399
if (openTelemetrySdk != null) {
openTelemetrySdk.close();
}
openTelemetrySdk = null;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The traceExporter resource is created in before() but is not closed in after(), which can lead to resource leaks (such as gRPC channels and threads). Additionally, following the LIFO order for closing resources and ensuring exception safety is recommended.

    try {
      if (openTelemetrySdk != null) {
        openTelemetrySdk.close();
      }
    } finally {
      if (traceExporter != null) {
        traceExporter.close();
      }
    }
    openTelemetrySdk = null;
    traceExporter = null;
References
  1. When managing a collection of closeable resources (e.g., scopes), ensure they are closed in the reverse order of their creation (LIFO). The implementation must be exception-safe to prevent resource leaks, meaning all opened resources should be closed even if exceptions occur during their creation or closing.

Comment on lines 444 to +449
CompletableResultCode completableResultCode =
openTelemetrySdk.getSdkTracerProvider().forceFlush();
completableResultCode.join(TRACE_FORCE_FLUSH_MILLIS, TimeUnit.MILLISECONDS);
if (!completableResultCode.isSuccess()) {
logger.warning("Force flush did not complete successfully");
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If isUsingGlobalOpenTelemetrySDK() is false, openTelemetrySdk will be null, leading to a NullPointerException when calling openTelemetrySdk.getSdkTracerProvider(). Adding a null check and falling back to flushing traceExporter directly ensures robust behavior.

    if (openTelemetrySdk != null) {
      CompletableResultCode completableResultCode =
          openTelemetrySdk.getSdkTracerProvider().forceFlush();
      completableResultCode.join(TRACE_FORCE_FLUSH_MILLIS, TimeUnit.MILLISECONDS);
      if (!completableResultCode.isSuccess()) {
        logger.warning("Force flush did not complete successfully");
      }
    } else if (traceExporter != null) {
      CompletableResultCode completableResultCode = traceExporter.flush();
      completableResultCode.join(TRACE_FORCE_FLUSH_MILLIS, TimeUnit.MILLISECONDS);
      if (!completableResultCode.isSuccess()) {
        logger.warning("Trace exporter flush did not complete successfully");
      }
    }
References
  1. When using lazily initialized resources (such as ExecutorService), ensure that teardown or close methods perform explicit null checks before invoking methods on them to prevent NullPointerException.

- Shut down traceExporter in after() to avoid resource leaks.
- Add null check for openTelemetrySdk in waitForTracesToComplete() to avoid NPE.

TAG=agy
CONV=c8697e3d-c06b-4d5b-86e9-f8950e4298fd
@lqiu96 lqiu96 merged commit 8afc0ab into main Jul 6, 2026
208 checks passed
@lqiu96 lqiu96 deleted the investigate-issue-13644-java branch July 6, 2026 19:12
lqiu96 added a commit that referenced this pull request Jul 7, 2026
Also removes Datastore's MultipleAttemptsRule to use the version from
java-core.

From:
#13645 (comment)
lqiu96 pushed a commit that referenced this pull request Jul 7, 2026
🤖 I have created a release *beep* *boop*
---


<details><summary>1.88.0</summary>

##
[1.88.0](v1.87.1...v1.88.0)
(2026-07-06)


### Features

* **auth:** Add support for Regional Access Boundaries
([#13499](#13499))
([b721e43](b721e43))
* automate libraries-bom tag and release creation on release publish
([#13655](#13655))
([7a7e2ff](7a7e2ff))
* **bigquery-jdbc:** add `EnableProjectDiscovery` connection property
for metadata methods
([#13344](#13344))
([ab9669a](ab9669a))
* **bigquery-jdbc:** migrate statement execution thread tracking to
connection-scoped executor
([#13454](#13454))
([341e51a](341e51a))
* **bigquery-jdbc:** respect standard JVM trustStore properties by
default
([#13435](#13435))
([7af3224](7af3224))
* **bigquery-jdbc:** support connection-scoped executor isolation and
dynamic queue warnings
([#13413](#13413))
([79e26b8](79e26b8))
* **bigquery:** add internal listProjects API to core client
([#13429](#13429))
([3580407](3580407))
* **bigtable:** route point read rows to shim
([#13542](#13542))
([2c7e5f5](2c7e5f5))
* **deps:** add jspecify to shared dependencies
([#13412](#13412))
([582053d](582053d))
* enable self-signed JWTs by default in ServiceOptions
([#13338](#13338))
([110d2b7](110d2b7))
* **gapic-generator:** Add NullMarked annotation to generated classes
([#13517](#13517))
([825dadd](825dadd))
* **google/cloud/agentregistry/v1:** onboard a new library
([#13509](#13509))
([6728816](6728816))
* scale up connection worker pool based on latency
([#13384](#13384))
([eb379b3](eb379b3))
* **spanner:** add settings for gRCP keep-alive
([#13643](#13643))
([9d78307](9d78307))
* **spanner:** auth login support for Spanner Omni endpoints
([#13470](#13470))
([55930b4](55930b4))
* **storage:** add checksum validation in the json read channel
([#13270](#13270))
([872d7b7](872d7b7))
* **storage:** add checksum validation on json read paths
([#13269](#13269))
([396b042](396b042))
* **storage:** add full object checksum validation for bidi flow
([#13266](#13266))
([9113d80](9113d80))
* **storage:** add full object checksum validation for grpc flow
([#13265](#13265))
([a5ba606](a5ba606))
* **storage:** Adding CumulativeHasher wrapper class for Full object …
([#13239](#13239))
([bd40324](bd40324))
* **storage:** adding disabling option for bidi reads
([#13506](#13506))
([591cae0](591cae0))
* **storage:** log additional bytes received from GCS in read path
([#13427](#13427))
([9492aa2](9492aa2))
* **storage:** update compose sample to support deleteSourceObjects
option
([#13493](#13493))
([50a8658](50a8658))
* use self-signed JWTs in Spanner MutableCredentials
([#13381](#13381))
([29543a2](29543a2))


### Bug Fixes

* Add logging-logback to gapic-libraries-bom
([#13663](#13663))
([512e370](512e370))
* **auth:** Fix UserCredentials serialization clientSecret leak
([#13465](#13465))
([2d9d01c](2d9d01c))
* **auth:** handle missing APPDATA on Windows gracefully
([#13471](#13471))
([77e84a9](77e84a9)),
closes
[#12565](#12565)
* **bigquery-jdbc:** add proper version to BigQueryDriver
([#13294](#13294))
([9fd84fc](9fd84fc))
* **bigquery-jdbc:** avoid rollback on statement close in manual commit
mode
([#13503](#13503))
([79bbee1](79bbee1))
* **bigquery-jdbc:** correct `FilterTablesOnDefaultDataset` fallback
logic
([#13625](#13625))
([f7eb23d](f7eb23d))
* **bigquery-jdbc:** ensure largeResults are handled in
PreparedStatement
([#13496](#13496))
([d3e7a19](d3e7a19))
* **bigquery-jdbc:** ensure test uses unique dataset & cleans up
([#13587](#13587))
([a6bb362](a6bb362))
* **bigquery-jdbc:** explicitly set location when available for
temporary dataset
([#13508](#13508))
([40cd0a7](40cd0a7))
* **bigquery-jdbc:** propagate connection proxy settings to auth library
([#13539](#13539))
([87dda5d](87dda5d))
* **bigquery-jdbc:** shade org.slf4j
([#13547](#13547))
([7ef1312](7ef1312))
* **bigquery-jdbc:** update format validation tests to create tables for
the test
([#13588](#13588))
([4bb3a61](4bb3a61))
* **bigquery-jdbc:** update Maven Central artifact link in the README
([#13563](#13563))
([8707a65](8707a65))
* **bigquery:** retry cancel job and routine creation on transient HTTP
5xx errors
([#13416](#13416))
([35eb041](35eb041))
* **bigquery:** route JOB_CREATION_REQUIRED through fast query path
([#13437](#13437))
([64cde7f](64cde7f))
* **bigquery:** Update fast query path to allow jobTimeoutMs in the
request
([#13502](#13502))
([1a6f4d5](1a6f4d5))
* **bigtable:** honour session_load override when server-returned
session_load is 0
([#13629](#13629))
([b56f9b8](b56f9b8))
* **bqjdbc:** pass rowsInPage with TableResult
([#13238](#13238))
([203a91e](203a91e))
* **ci:** build dependencies before convergence check
([#13638](#13638))
([9de3209](9de3209))
* **ci:** ignore SNAPSHOT suffix in flatten-plugin check
([#13639](#13639))
([bb65627](bb65627))
* **ci:** run root install in dependency convergence check
([#13666](#13666))
([86d35e4](86d35e4))
* **ci:** skip existing-versions-check for gapic-generator-java-root
([#13590](#13590))
([a1f3b8a](a1f3b8a))
* **datastore:** Configure TraceExporter with Datastore credentials in
E2E tests
([#13627](#13627))
([d281a62](d281a62))
* **datastore:** disable built-in OpenTelemetry SDK when metrics export
is disabled
([#13543](#13543))
([53b7142](53b7142))
* **datastore:** reduce flakiness of E2E tracing tests
([#13645](#13645))
([8afc0ab](8afc0ab))
* **deps:** update dependency
com.google.apis:google-api-services-bigquery to v2-rev20260612-2.0.0
([#13570](#13570))
([ca5ff79](ca5ff79))
* **deps:** update dependency com.google.apis:google-api-services-dns to
v1-rev20260616-2.0.0
([#12796](#12796))
([0dd60ca](0dd60ca))
* **deps:** update dependency com.google.cloud:google-cloud-pubsub-bom
to v1.151.0
([#12097](#12097))
([d169006](d169006))
* **deps:** update dependency com.google.cloud:libraries-bom to v26.84.0
([#12663](#12663))
([95c7774](95c7774))
* **deps:** update first-party storage dependencies to
v1-rev20260524-2.0.0
([#13406](#13406))
([bdeedb8](bdeedb8))
* exclude java-vertexai from existing versions check
([#13632](#13632))
([edaa4e3](edaa4e3))
* fallback on VPC
([#13567](#13567))
([e8c428d](e8c428d))
* **java-cloud-bom:** execute pre-install pass in GHA release notes
generator
([#13608](#13608))
([c365c10](c365c10))
* **release:** check remote tags using git ls-remote to prevent already
exists error
([#13574](#13574))
([6cf0567](6cf0567))
* **spanner:** fail fast when inline-begin DML returns no transaction id
([#13536](#13536))
([94315ce](94315ce))
* **spanner:** pin inline read-write transactions to default endpoint
([#13562](#13562))
([6908dee](6908dee))
* **spanner:** remove explicit tink version to resolve dependency
convergence conflict
([#13602](#13602))
([863c26e](863c26e))
* **spanner:** update dynamic channel pool default configuration
([#13646](#13646))
([37b77c3](37b77c3))


### Dependencies

* Add org.bouncycastle:bcprov-jdk18on to java-shared-dependencies
([#13531](#13531))
([e3d2082](e3d2082))
* **auth:** Update commons-codec to v1.18.0
([#13598](#13598))
([fd89957](fd89957))
* Update gapic-showcase to v0.41.0
([#13582](#13582))
([09faaac](09faaac))
* Upgrade google-http-client to v2.1.1
([#13578](#13578))
([6abef19](6abef19))


### Documentation

* add instructions for manual code generation checks in CI
([#13596](#13596))
([f6162e2](f6162e2))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Datastore Flaky Test: Trace NOT_FOUND

4 participants