feat: support backoff/retry in OTLP #3126

scottgerring · 2025-08-12T07:05:58Z

Fixes #3081, building on the work started by @AaronRM 🤝

Changes

A new retry module added to `opentelemetry-sdk`

Models the sorts of retry an operation may request (retry / can't retry / throttle), and provides a helper retry_with_backoff mechanism that can be used to wrap up a retryable operation and retry it. The helper relies on experimental_async_runtime for its runtime abstraction, to provide the actual pausing. It also takes a lambda to classify the error, so the caller can inform the retry mechanism if a retry is required.

A new retry_classification module added to `opentelemetry-otlp`

This bit takes the actual error responses that we get back over OTLP and maps them back to the retry model. Because this is OTLP-specific stuff it belongs here rather than alongside the retry code.

Retry binding

... happens in each one of the concrete exporters to tie it all together.

Also ...

Extended exporter builders to allow the user to customise default retry policy
Added new feature flags experimental-http-retry and experimental-grpc-retry which pull in the experimental-async-runtime dep and set everything up. This way we can get going with this now without having to stabilise the experimental-async-runtime feature.

Open Questions

Merge requirement checklist

CONTRIBUTING guidelines followed
Unit tests added/updated (if applicable)
Appropriate CHANGELOG.md files updated for non-trivial, user-facing changes
Changes in public API reviewed (if applicable)

codecov · 2025-08-12T07:43:42Z

Codecov Report

❌ Patch coverage is 70.36753% with 774 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.7%. Comparing base (ad88615) to head (27d303a).
⚠️ Report is 205 commits behind head on main.

Files with missing lines	Patch %	Lines
opentelemetry-otlp/src/exporter/http/mod.rs	67.0%	161 Missing ⚠️
opentelemetry-sdk/src/metrics/data/mod.rs	13.4%	154 Missing ⚠️
opentelemetry-proto/src/transform/metrics.rs	11.1%	64 Missing ⚠️
...-sdk/src/metrics/internal/exponential_histogram.rs	65.1%	52 Missing ⚠️
opentelemetry-otlp/src/exporter/tonic/metrics.rs	0.0%	50 Missing ⚠️
opentelemetry-otlp/src/exporter/tonic/trace.rs	0.0%	48 Missing ⚠️
opentelemetry-otlp/src/exporter/tonic/logs.rs	0.0%	46 Missing ⚠️
opentelemetry-otlp/src/exporter/tonic/mod.rs	73.4%	42 Missing ⚠️
opentelemetry-sdk/src/metrics/instrument.rs	88.9%	29 Missing ⚠️
opentelemetry-sdk/src/logs/logger_provider.rs	92.0%	12 Missing ⚠️
... and 29 more

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #3126     +/-   ##
=======================================
+ Coverage   79.6%   80.7%   +1.1%     
=======================================
  Files        124     128      +4     
  Lines      23174   22850    -324     
=======================================
+ Hits       18456   18460      +4     
+ Misses      4718    4390    -328

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

opentelemetry-otlp/src/exporter/tonic/logs.rs

scottgerring · 2025-08-12T11:39:37Z

opentelemetry-otlp/Cargo.toml

@@ -35,6 +35,7 @@ tracing = {workspace = true, optional = true}

 prost = { workspace = true, optional = true }
 tonic = { workspace = true, optional = true }
+tonic-types = { workspace = true, optional = true }


Needed for gRPC error type

opentelemetry-otlp/src/exporter/http/logs.rs

scottgerring · 2025-08-12T11:45:00Z

opentelemetry-otlp/src/exporter/http/logs.rs

+        .map_err(|e| OTelSdkError::InternalFailure(e.message))
+    }
+
+    #[cfg(not(feature = "http-retry"))]


This is a massive duplication of code.

If we decide that http export always has retry behaviour, which means we include the unstable runtime feature, then we can remove all of this.

Alternatively we can provide export once, with all the extra pomp and fanfare to support retry, and then just use the stub "don't actually retry" impl. There would be some slight runtime overhead to this, but the codebase would be much simpler.

Comment above applies to the other HTTP exporters also.

Wouldn't it be possible to use the same code for the logic (broken out into a method) and construct the HttpError instances but just have different variants of retry_with_backoff and the HttpError classification depending on if we have the unstable runtime feature enabled?

@bantonsson something like this commit? --> ba1f9e2

I've factored the retry behaviour out into otlp::exporter::http, and then switch out the implementation based on the retry-retry flag. This removes huge volumes of code duplication and means that we could get this in without stabilising the async runtime feature.

and the HttpError classification depending on if we have the unstable runtime feature enabled?

I'm not sure the classification needs to change independently of the "do we use the retry variant or not" bit - but I might be missing something?

I've also not touched this for the tonic exporters yet, but once we have the HTTP ones in shape I can continue the pattern over there.

I'm not sure the classification needs to change independently of the "do we use the retry variant or not" bit - but I might be missing something?

You're not missing anything. That refactoring is cleaner than what I was thinking about.

Because of the way the tonic exporters are structured its harder to cleanly pull it apart. I've struck what I think is a sensible compromise over there, adding a wrapper for the retry function that subs in retry/no-op based on feature flags. I think this is better than going too wild with traits/generics/lambdas to squeeze the last little bit of code re-use out in terms of clarity.

Copilot

Pull Request Overview

This PR implements retry logic with exponential backoff and jitter for OTLP exporters to handle transient failures gracefully, addressing issue #3081. The implementation supports both HTTP and gRPC protocols with protocol-specific error classification and server-provided throttling hints.

Adds a new retry module to opentelemetry-sdk with configurable retry policies and exponential backoff
Implements protocol-specific error classification in opentelemetry-otlp for HTTP and gRPC responses
Integrates retry functionality into all OTLP exporters (traces, metrics, logs) for both HTTP and gRPC transports

Reviewed Changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
opentelemetry-sdk/src/retry.rs	Core retry module with exponential backoff, jitter, and error classification
opentelemetry-otlp/src/retry_classification.rs	Protocol-specific error classification for HTTP and gRPC responses
opentelemetry-otlp/src/exporter/tonic/*.rs	gRPC exporter integration with retry functionality
opentelemetry-otlp/src/exporter/http/*.rs	HTTP exporter integration with retry functionality
opentelemetry-otlp/Cargo.toml	Feature flags and dependencies for retry support

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

opentelemetry-sdk/src/retry.rs

Copilot · 2025-09-01T18:51:52Z

opentelemetry-otlp/src/retry_classification.rs

+    fn parse_http_date_to_delay(date_str: &str) -> Result<u64, ()> {
+        // For now, return error - would need proper HTTP date parsing
+        // This could be implemented with chrono or similar
+        let _ = date_str;
+        Err(())
+    }


The HTTP date parsing function is a stub that always returns an error. This means Retry-After headers with HTTP date format will be ignored, falling back to retryable behavior instead of respecting server-specified delays.

opentelemetry-otlp/src/exporter/tonic/trace.rs

opentelemetry-otlp/src/exporter/http/trace.rs

opentelemetry-otlp/src/exporter/http/logs.rs

bantonsson

I think the HTTP exporters look good now. Love all those red lines.

scottgerring · 2025-09-05T10:58:42Z

opentelemetry-otlp/src/exporter/tonic/mod.rs

+    feature = "experimental-grpc-retry",
+    any(feature = "trace", feature = "metrics", feature = "logs")
+))]
+async fn tonic_retry_with_backoff<R, F, Fut, T>(


Because of the way the tonic path works, it is much harder to cleanly extract as much from the signal-specific exporters as we managed with HTTP.

I've done this as a kind of middleground - this way the feature flagging we use to sub between retry-available / no-retry is all in one place - as a thin wrapper around the global retry function retry_with_backoff.

I keep trying to slice and dice this in some nice way, but the generated clients do not implement a common interface, so it's complicated to do common code without changing them.

Yeah, I know what you mean. I came up with something that saved a few lines by forcing some intermediate abstractions but it didn't reduce line count and certainly didn't improve clarity.

I feel like the way it is strikes a reasonable balance even if it isn't particularly satisfying.

bantonsson

👍🏼 for the HTTP code. I can't see a clear way to reuse more of the Tonic code.

lalitb · 2025-09-16T03:41:25Z

Sorry for delay. I would like to review during this week - assigning to myself.

… we don't want retry

scottgerring force-pushed the feat/retry-logic branch 4 times, most recently from 3847b26 to fb141db Compare August 12, 2025 10:22

scottgerring commented Aug 12, 2025

View reviewed changes

opentelemetry-otlp/src/exporter/tonic/logs.rs Show resolved Hide resolved

scottgerring force-pushed the feat/retry-logic branch from 603c31e to 75f0d71 Compare August 12, 2025 10:36

scottgerring commented Aug 12, 2025

View reviewed changes

opentelemetry-otlp/src/exporter/http/logs.rs Outdated Show resolved Hide resolved

scottgerring commented Aug 12, 2025

View reviewed changes

scottgerring changed the title ~~[not ready!] feat: support backoff/retry~~ feat: support backoff/retry in OTLP Aug 12, 2025

scottgerring requested review from cijothomas and lalitb August 19, 2025 05:52

scottgerring marked this pull request as ready for review August 19, 2025 14:32

scottgerring requested a review from a team as a code owner August 19, 2025 14:32

lalitb requested a review from Copilot September 1, 2025 18:50

Copilot AI reviewed Sep 1, 2025

View reviewed changes

scottgerring force-pushed the feat/retry-logic branch 2 times, most recently from af933a2 to f1636a0 Compare September 2, 2025 09:11

bantonsson reviewed Sep 2, 2025

View reviewed changes

opentelemetry-otlp/src/exporter/http/logs.rs Outdated Show resolved Hide resolved

scottgerring force-pushed the feat/retry-logic branch from ba1f9e2 to ff33723 Compare September 2, 2025 11:54

bantonsson reviewed Sep 2, 2025

View reviewed changes

scottgerring force-pushed the feat/retry-logic branch from c0dba75 to 059943f Compare September 5, 2025 10:52

scottgerring commented Sep 5, 2025

View reviewed changes

scottgerring force-pushed the feat/retry-logic branch from 059943f to 693765b Compare September 5, 2025 11:02

bantonsson approved these changes Sep 15, 2025

View reviewed changes

lalitb self-assigned this Sep 16, 2025

AaronRM added 2 commits September 16, 2025 17:22

Fix comment about how to only run unit tests

c924eb3

Remove extraneous lines from test_with_gzip_compression unit test

b2b4518

AaronRM and others added 21 commits September 16, 2025 17:22

Add retry_with_exponential_backoff method; Use in tonic logs client

9364bfa

Add tests and comments to retry.rs

f848dd8

Move retry to opentelemetry-sdk (WIP)

63df3f8

Merge branch 'main' into aaronm-retry

f3a319f

Fix build warning

c5fd833

Scope retry to just logs+tonic

9c11248

clean up after rebase

3a15eba

migrate to the async runtime abstraction

d868a0d

add retry to traces

b4e3c2d

add retry to metrics

2e0bf3e

handle failure hints

5e49f2d

simplify retry interface

861343a

Implement retry for http trace

f48d0b5

Implement retry for http metrics and logs

4a701f5

changelog

2877126

chore: wrap headers in arc to save cloning per batch

6f8f023

chore: simplify HttpExportError creation

9bde68f

chore: factor retry logic out

a8e7ad8

chore: configurable retry policy with default

b55b369

chore: modify tonic exporters so we don't _need_ the async runtime if…

c4fb3d9

… we don't want retry

chore: fix missing dep

27d303a

scottgerring force-pushed the feat/retry-logic branch from af8f832 to 27d303a Compare September 16, 2025 15:22

feat: support backoff/retry in OTLP #3126

Are you sure you want to change the base?

feat: support backoff/retry in OTLP #3126

Conversation

scottgerring commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

A new retry module added to opentelemetry-sdk

A new retry_classification module added to opentelemetry-otlp

Retry binding

Open Questions

Merge requirement checklist

Uh oh!

codecov bot commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

scottgerring Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Copilot AI Sep 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bantonsson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bantonsson left a comment

Choose a reason for hiding this comment

Uh oh!

lalitb commented Sep 16, 2025

Uh oh!

Uh oh!

scottgerring commented Aug 12, 2025 •

edited

Loading

A new retry module added to `opentelemetry-sdk`

A new retry_classification module added to `opentelemetry-otlp`

codecov bot commented Aug 12, 2025 •

edited

Loading

scottgerring Sep 2, 2025 •

edited

Loading