Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat(lib-retry, lib-httpx): honor server-supplied Retry-After on retries#72

Closed
pditommaso wants to merge 4 commits into
masterfrom
feat/retry-after-honoring
Closed

feat(lib-retry, lib-httpx): honor server-supplied Retry-After on retries#72
pditommaso wants to merge 4 commits into
masterfrom
feat/retry-after-honoring

Conversation

@pditommaso

Copy link
Copy Markdown
Contributor

Summary

Adds server-driven retry timing to the HTTP retry pipeline so callers automatically honor Retry-After headers on 429/503 responses, instead of running their own (typically much shorter) exponential backoff inside the server rate-limit window.

  • lib-retry: new generic builder hook Retryable.retryDelayHint(Function<R, Duration>). When set, the policy waits min(maxDelay, max(scheduled_backoff, hint)). The hint is a strict lower bound, the configured maxDelay is a strict upper bound, and Failsafe withJitter continues to apply on top. A null hint falls back to the regular exponential backoff — fully backward-compatible for callers that do not opt in.
  • lib-httpx: both sendWithRetry and sendWithRetryAsync now wire HxClient::parseRetryAfter as the hint extractor. Only the RFC 9110 §10.2.3 delta-seconds form is parsed (e.g. Retry-After: 41); HTTP-date form degrades to backoff (not used by Seqera services).

Motivation

Nextflow issue nextflow-io/nextflow#7176: the plugin registry returns 429 + Retry-After: 41 under burst CI load, but the HxClient generic exponential backoff (5 attempts ~5.6s total wait) lands every retry inside the same 60-second rate-limit window and they all fail. Honoring Retry-After at the library layer fixes this for every Seqera HTTP caller — Wave, Tower, registry — not just plugin metadata.

Design notes

  • withDelayFn and withBackoff are mutually exclusive in Failsafe 3.3.2, so when a hint extractor is registered we compute exponential backoff manually inside the lambda. maxDelay is also enforced manually because withDelayFn does not honor the policy maxDelay field.
  • Failsafe rethrows exceptions from delayFn, so the hint extractor invocation is wrapped in try/catch — a misbehaving extractor degrades to backoff rather than breaking the retry.
  • Async parity verified with a wiremock test that uses sendAsync(...).get().

Test plan

  • :lib-retry:test — 3 new Spock tests for retryDelayHint: floor (hint > backoff), fallback (null hint), cap (hint > maxDelay)
  • :lib-httpx:test — 2 new wiremock integration tests (sync and async 429 + Retry-After honored as lower bound), 1 new wiremock test for the maxDelay cap, and 9-row data-driven unit test for parseRetryAfter edge cases (null response, whitespace, zero, negative, non-numeric, fractional, HTTP-date, missing header)
  • No regressions in the existing HxClientRetryIntegrationTest, HxClientTest, and RetryableTest suites
  • The pre-existing should retry on 429 rate limit integration test pins a small maxDelay so it stays fast under the new behavior

🤖 Generated with Claude Code

pditommaso and others added 4 commits May 27, 2026 13:38
Adds a new builder hook on Retryable that extracts a per-attempt delay
lower bound from the just-failed result. When set, the policy waits
min(maxDelay, max(scheduled_backoff, hint)) — the hint floors the wait
but the configured maxDelay still caps it, and Failsafe's withJitter
keeps applying on top. A null hint falls back to regular exponential
backoff, preserving existing behavior for callers that don't opt in.

The motivating use case is honoring HTTP Retry-After headers on 429/503
responses, but the API is generic over the result type R so it can
extract delay hints from any retried operation.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: Paolo Di Tommaso <[email protected]>
Wires the new Retryable.retryDelayHint API into both the sync and async
retry paths of HxClient. A server-supplied Retry-After header is now used
as a lower bound for the next retry delay, capped by the configured
maxDelay. Without this, the existing exponential backoff (delay=500ms,
multiplier=2) routinely sleeps far less than the server requests on 429,
so retries land inside the same rate-limit window and all fail.

Only the RFC 9110 §10.2.3 delta-seconds form is parsed; HTTP-date is not
used by Seqera services and degrades to backoff. The pre-existing
'should retry on 429' integration test pins a small maxDelay so it stays
fast — its intent is verifying the retry mechanic, not Retry-After
honoring (which has dedicated sync and async tests).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: Paolo Di Tommaso <[email protected]>
Failsafe lets withBackoff and withDelayFn coexist on the same policy;
the delayFn takes precedence at execution time when set. Use that to
avoid restructuring the builder chain: the original withBackoff line
stays as-is, and withDelayFn is appended only when a hint extractor
is registered.

Pure refactor, no behavior change. All tests still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: Paolo Di Tommaso <[email protected]>
Pulls the delay-with-hint computation out of the withDelayFn lambda into
a dedicated static method, so the policy-builder chain reads as a single
line. The semantics — min(maxDelay, max(scheduled_backoff, hint)) — now
live in one place with focused javadoc.

Pure refactor, no behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: Paolo Di Tommaso <[email protected]>
pditommaso added a commit that referenced this pull request May 27, 2026
…ies (#72)

* lib-retry: add retryDelayHint to let callers override the backoff delay
  with a server-supplied value (additive — withBackoff still applies).
* lib-retry: extract computeRetryDelay helper for readability.
* lib-httpx: parse Retry-After (delta-seconds) on retried HTTP responses
  and feed it into Retryable.retryDelayHint; debug-log unparseable headers.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@pditommaso

Copy link
Copy Markdown
Contributor Author

Merged manually

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant