[pull] master from DataDog:master#753
Merged
Merged
Conversation
…rt (#8645) ## Summary of changes Adds `http/protobuf` support to the OTLP traces exporter introduced in #8211, which introduced the exporter and `http/json` support. This feature is enabled by setting `OTEL_TRACES_EXPORTER=otlp` and `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf` (or the parent `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf`). ## Reason for change When #8211 landed, `http/json` was the only functional OTLP traces protocol — setting any other protocol would fall back (with a log message) to the Datadog MessagePack encoding. `http/protobuf` is the OTLP default protocol per the OpenTelemetry specification and is what most OTel collectors, Datadog Agent included, expect to receive, so we need to support it to be useful out-of-the-box for users adopting the OTLP traces export. ## Implementation details ### Serializer Adds `Datadog.Trace.OpenTelemetry.Traces.OtlpTracesProtobufSerializer`, a new `ISpanBufferSerializer` that emits the [OTLP `ExportTraceServiceRequest`](https://github.com/open-telemetry/opentelemetry-proto/blob/v1.2.0/opentelemetry/proto/trace/v1/trace.proto) protobuf payload directly into `SpanBuffer`'s output buffer: - The `resource_spans` and `scope_spans` envelopes are opened on the first `SerializeSpans` call, span entries are appended per chunk, and `FinishBody` patches the two length-prefix placeholders using absolute positions into the destination buffer. - The serializer respects the same per-span / per-event / per-link attribute count limits (`128`) as `OtlpTracesJsonSerializer`, and re-uses the `OtlpMapper` logic to map resource attributes and tags onto the serialized spans. - `OtlpTracesProtobufSerializer` is **stateful** (it tracks the two length-placeholder positions), so `AgentWriter` now creates a fresh instance for the front and back buffers rather than sharing one — the previous shared `OtlpTracesJsonSerializer` was stateless and didn't need this. ### Vendored OpenTelemetry SDK Builds on the protobuf primitives vendored from the OpenTelemetry .NET SDK in #7653 (`ProtobufSerializer`, `ProtobufWireType`): - Adds new field-number constant files: `ProtobufOtlpTraceFieldNumberConstants` and `ProtobufOtlpResourceFieldNumberConstants` (split out so the OTLP trace serializer only depends on what it needs). - Removes the `#if NETCOREAPP3_1_OR_GREATER` guard from the vendored `ProtobufSerializer` — now that we vendor a `Span<>` polyfill the serializer compiles and runs on all supported runtimes (.NET Framework 4.6.1+). - Drops the matching `#if NET6_0_OR_GREATER` guard around the `SubmitsOtlpTraces` integration test (`http/protobuf` and `http/json` both run on .NET Framework now too). ### Wiring - `ExporterSettings`: `OtlpProtocol.HttpProtobuf` now resolves to `TracesEncoding.OtlpProtobuf` instead of falling back to `DatadogV0_4`. - `AgentWriter.GetSpanSerializer`: selects `OtlpTracesProtobufSerializer` when `TracesEncoding == OtlpProtobuf`. - `ApiOtlp.SendTracesAsync`: chooses `application/x-protobuf` vs `application/json` for the request `Content-Type` based on `_tracesEncoding`. Adds the `application/x-protobuf` MIME type constant to `Datadog.Trace.Agent.Transports.MimeTypes`. - `supported-configurations.yaml`: documents `http/protobuf` as a valid value for `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` and regenerates `ConfigurationKeys.OpenTelemetry.g.cs` for all TFMs. ## Test coverage - **Unit tests**: New `OtlpTracesProtobufSerializerTests` exercises the serializer end-to-end using `Google.Protobuf` and a little custom parsing. Test cases include: - Empty buffer (`FinishBody` returns 0) - Multiple traces reported in a single-chunk - Serializing span events/links/status - Exceeding buffer `maxSize` honoring behavior - Exceeding span limits - **Integration tests**: `OpenTelemetrySdkTests.SubmitsOtlpTraces` now runs both `http/json` and `http/protobuf` cases for the Datadog SDK on all TFMs. - **Snapshot unification**: To keep one Datadog-SDK snapshot regardless of OTLP protocol, snapshots for `http/protobuf` are normalized to the `http/json` shape via scrubbers that translate the test agent's protobuf rendering (snake_case field names, string-form enum values like `SPAN_KIND_SERVER`) into the JSON serializer's shape (camelCase field names, integer enum values). - **Snapshot rename**: `SubmitsOtlpTraces_DD_http_json.verified.txt` renamed to `SubmitsOtlpTraces_DD.verified.txt` (one snapshot now covers both protocols). ## Other details With this PR, `http/json` and `http/protobuf` are both functional; `grpc` still falls back to Datadog MessagePack encoding with a startup warning. ### Follow-up work - Implement the `grpc` OTLP traces protocol - Add OTLP traces telemetry counts (the `TODO: Telemetry - Record OTLP Traces API submissions` in `ApiOtlp.SendTracesAsync` still applies) - Honor `OTEL_EXPORTER_OTLP_TRACES_TIMEOUT` in `ApiOtlp` (still unused) --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )