Fix concurrent span reusable data marshaler#7041
Merged
jack-berg merged 2 commits intoJan 24, 2025
Merged
Conversation
ArrayDeque specifies that: > Array deques ... are not thread-safe; in the absence of external > synchronization, they do not support concurrent access by multiple > threads. `marshalerPool` is concurrently added to by the OkHttp threadpool without synchronization, along with all threads that end spans (with synchronisation in `SimpleSpanProcessor.exportLock`, which is not used to synchronize with the OkHttp threadpool). Just making the ArrayQueue synchronous internally removes all need to worry about upstream locks. Fixes open-telemetry#7019
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #7041 +/- ##
============================================
- Coverage 89.95% 89.89% -0.06%
- Complexity 6636 6655 +19
============================================
Files 745 748 +3
Lines 20010 20077 +67
Branches 1962 1969 +7
============================================
+ Hits 17999 18048 +49
- Misses 1415 1435 +20
+ Partials 596 594 -2 ☔ View full report in Codecov by Sentry. |
trask
approved these changes
Jan 24, 2025
Houlong66
added a commit
to Houlong66/rocketmq
that referenced
this pull request
Apr 20, 2026
OpenTelemetry Java 1.44.0 ~ 1.46.x ships OtlpGrpcMetricExporter with MemoryMode.REUSABLE_DATA by default. The underlying MetricReusableDataMarshaler.marshalerPool is a non-thread-safe ArrayDeque accessed concurrently by the reader thread (poll) and the OkHttp callback thread (add, via whenComplete). With BatchSplittingMetricExporter issuing N concurrent sub-batch exports per cycle, the pool races and leaks marshalers (~132 KiB each) until OOM. Fixed upstream in 1.47.0 via open-telemetry/opentelemetry-java#7041 (ArrayDeque -> ConcurrentLinkedDeque). - Bump OpenTelemetry to 1.47.0 in pom.xml so the upstream race fix is in effect. - Default OtlpGrpcMetricExporter to MemoryMode.IMMUTABLE_DATA to preserve the pre-1.44 default behavior; exposed via brokerConfig.metricsExportOtelMemoryMode ("IMMUTABLE_DATA" / "REUSABLE_DATA", case-insensitive). Operators may opt in to REUSABLE_DATA when running on OTel >= 1.47. - Cap concurrent in-flight sub-batches in BatchSplittingMetricExporter with a Semaphore controlled by brokerConfig.metricsExportBatchMaxConcurrent (default 4; set to 1 to serialize and match pre-batch behavior; 0 or Integer.MAX_VALUE means unlimited). - Add brokerConfig.metricsExportBatchSplitEnabled (default true) as an escape hatch to bypass BatchSplittingMetricExporter entirely, restoring the raw OtlpGrpcMetricExporter wiring. - Defensively snapshot MetricData points before export to avoid ArrayIndexOutOfBoundsException in NumberDataPointMarshaler when async instrument callbacks mutate point collections during export.
This was referenced Apr 20, 2026
lizhimins
pushed a commit
to apache/rocketmq
that referenced
this pull request
Apr 21, 2026
…orter pool race (#10267) OpenTelemetry Java 1.44.0 ~ 1.46.x ships OtlpGrpcMetricExporter with MemoryMode.REUSABLE_DATA by default. The underlying MetricReusableDataMarshaler.marshalerPool is a non-thread-safe ArrayDeque accessed concurrently by the reader thread (poll) and the OkHttp callback thread (add, via whenComplete). With BatchSplittingMetricExporter issuing N concurrent sub-batch exports per cycle, the pool races and leaks marshalers (~132 KiB each) until OOM. Fixed upstream in 1.47.0 via open-telemetry/opentelemetry-java#7041 (ArrayDeque -> ConcurrentLinkedDeque). - Bump OpenTelemetry to 1.47.0 in pom.xml so the upstream race fix is in effect. - Default OtlpGrpcMetricExporter to MemoryMode.IMMUTABLE_DATA to preserve the pre-1.44 default behavior; exposed via brokerConfig.metricsExportOtelMemoryMode ("IMMUTABLE_DATA" / "REUSABLE_DATA", case-insensitive). Operators may opt in to REUSABLE_DATA when running on OTel >= 1.47. - Cap concurrent in-flight sub-batches in BatchSplittingMetricExporter with a Semaphore controlled by brokerConfig.metricsExportBatchMaxConcurrent (default 4; set to 1 to serialize and match pre-batch behavior; 0 or Integer.MAX_VALUE means unlimited). - Add brokerConfig.metricsExportBatchSplitEnabled (default true) as an escape hatch to bypass BatchSplittingMetricExporter entirely, restoring the raw OtlpGrpcMetricExporter wiring. - Defensively snapshot MetricData points before export to avoid ArrayIndexOutOfBoundsException in NumberDataPointMarshaler when async instrument callbacks mutate point collections during export.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #7027.
Fixes #7019.