feat(lib-retry, lib-httpx): honor server-supplied Retry-After on retries#72
Closed
pditommaso wants to merge 4 commits into
Closed
feat(lib-retry, lib-httpx): honor server-supplied Retry-After on retries#72pditommaso wants to merge 4 commits into
pditommaso wants to merge 4 commits into
Conversation
Adds a new builder hook on Retryable that extracts a per-attempt delay lower bound from the just-failed result. When set, the policy waits min(maxDelay, max(scheduled_backoff, hint)) — the hint floors the wait but the configured maxDelay still caps it, and Failsafe's withJitter keeps applying on top. A null hint falls back to regular exponential backoff, preserving existing behavior for callers that don't opt in. The motivating use case is honoring HTTP Retry-After headers on 429/503 responses, but the API is generic over the result type R so it can extract delay hints from any retried operation. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> Signed-off-by: Paolo Di Tommaso <[email protected]>
Wires the new Retryable.retryDelayHint API into both the sync and async retry paths of HxClient. A server-supplied Retry-After header is now used as a lower bound for the next retry delay, capped by the configured maxDelay. Without this, the existing exponential backoff (delay=500ms, multiplier=2) routinely sleeps far less than the server requests on 429, so retries land inside the same rate-limit window and all fail. Only the RFC 9110 §10.2.3 delta-seconds form is parsed; HTTP-date is not used by Seqera services and degrades to backoff. The pre-existing 'should retry on 429' integration test pins a small maxDelay so it stays fast — its intent is verifying the retry mechanic, not Retry-After honoring (which has dedicated sync and async tests). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> Signed-off-by: Paolo Di Tommaso <[email protected]>
Failsafe lets withBackoff and withDelayFn coexist on the same policy; the delayFn takes precedence at execution time when set. Use that to avoid restructuring the builder chain: the original withBackoff line stays as-is, and withDelayFn is appended only when a hint extractor is registered. Pure refactor, no behavior change. All tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> Signed-off-by: Paolo Di Tommaso <[email protected]>
Pulls the delay-with-hint computation out of the withDelayFn lambda into a dedicated static method, so the policy-builder chain reads as a single line. The semantics — min(maxDelay, max(scheduled_backoff, hint)) — now live in one place with focused javadoc. Pure refactor, no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> Signed-off-by: Paolo Di Tommaso <[email protected]>
pditommaso
added a commit
that referenced
this pull request
May 27, 2026
…ies (#72) * lib-retry: add retryDelayHint to let callers override the backoff delay with a server-supplied value (additive — withBackoff still applies). * lib-retry: extract computeRetryDelay helper for readability. * lib-httpx: parse Retry-After (delta-seconds) on retried HTTP responses and feed it into Retryable.retryDelayHint; debug-log unparseable headers. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Contributor
Author
|
Merged manually |
Merged
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds server-driven retry timing to the HTTP retry pipeline so callers automatically honor
Retry-Afterheaders on 429/503 responses, instead of running their own (typically much shorter) exponential backoff inside the server rate-limit window.lib-retry: new generic builder hookRetryable.retryDelayHint(Function<R, Duration>). When set, the policy waitsmin(maxDelay, max(scheduled_backoff, hint)). The hint is a strict lower bound, the configuredmaxDelayis a strict upper bound, and FailsafewithJittercontinues to apply on top. Anullhint falls back to the regular exponential backoff — fully backward-compatible for callers that do not opt in.lib-httpx: bothsendWithRetryandsendWithRetryAsyncnow wireHxClient::parseRetryAfteras the hint extractor. Only the RFC 9110 §10.2.3 delta-seconds form is parsed (e.g.Retry-After: 41); HTTP-date form degrades to backoff (not used by Seqera services).Motivation
Nextflow issue nextflow-io/nextflow#7176: the plugin registry returns
429 + Retry-After: 41under burst CI load, but the HxClient generic exponential backoff (5 attempts ~5.6s total wait) lands every retry inside the same 60-second rate-limit window and they all fail. HonoringRetry-Afterat the library layer fixes this for every Seqera HTTP caller — Wave, Tower, registry — not just plugin metadata.Design notes
withDelayFnandwithBackoffare mutually exclusive in Failsafe 3.3.2, so when a hint extractor is registered we compute exponential backoff manually inside the lambda.maxDelayis also enforced manually becausewithDelayFndoes not honor the policymaxDelayfield.delayFn, so the hint extractor invocation is wrapped in try/catch — a misbehaving extractor degrades to backoff rather than breaking the retry.sendAsync(...).get().Test plan
:lib-retry:test— 3 new Spock tests forretryDelayHint: floor (hint > backoff), fallback (null hint), cap (hint > maxDelay):lib-httpx:test— 2 new wiremock integration tests (sync and async 429 +Retry-Afterhonored as lower bound), 1 new wiremock test for themaxDelaycap, and 9-row data-driven unit test forparseRetryAfteredge cases (null response, whitespace, zero, negative, non-numeric, fractional, HTTP-date, missing header)HxClientRetryIntegrationTest,HxClientTest, andRetryableTestsuitesshould retry on 429 rate limitintegration test pins a smallmaxDelayso it stays fast under the new behavior🤖 Generated with Claude Code