fix(aibridge/provider): disable keep-alive on the STS assume-role client - #26971
Conversation
583a2b2 to
bd4b826
Compare
|
/coder-agents-review |
|
Chat: Review posted | View chat Review history
deep-review v0.9.0 | Round 1 | Last posted: Round 1, 3 findings (1 P3, 2 Nit), COMMENT. Review Finding inventoryFindings
Round logRound 1Panel. 0 P0-P2, 1 P3, 2 Nit. Reviewed against dee41c3..bd4b826. About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
Well-scoped fix for a non-obvious operational issue. The PR correctly identifies connection reuse as the amplifier that turns AWS's seconds-scale eventual consistency into a minutes-scale outage. The intervention (one transport config knob) is proportional, and the comment honestly separates confirmed observation from unconfirmed inference.
Pariston's differential diagnosis tested four framings (connection pinning, DNS pinning, AWS eventual consistency, idle-connection defaults) and confirmed no simpler alternative exists. In particular, IdleConnTimeout would not help during the failure window when the credential cache triggers rapid back-to-back AssumeRole calls with no idle gap. DisableKeepAlives is the correct knob.
Leorio called the comment at lines 82-93 "exactly what every workaround comment should look like." Knov independently praised it as "good documentation of a non-obvious operational fix." Worth noting because CRF-2 below disagrees on length but not substance.
Severity count: 1 P3, 2 Nit.
🤖 This review was automatically generated with Coder Agents.
| // only; Bedrock requests use a separate client and keep pooling. | ||
| stsClient := sts.NewFromConfig(base, func(o *sts.Options) { | ||
| o.HTTPClient = awshttp.NewBuildableClient().WithTransportOptions(func(t *http.Transport) { | ||
| t.DisableKeepAlives = true |
There was a problem hiding this comment.
P3 [CRF-1] No test asserts that the STS client disables keep-alives, the entire point of this PR.
Delete the
DisableKeepAlivesline and every test still passes green. The fix could regress silently during a future refactor.
The existing TestBuildBedrockCredentialsAssumeRole mock handler already receives the HTTP request. When DisableKeepAlives is true, Go's HTTP client sends a Connection: close header. A single assertion in the mock handler proves the transport is wired correctly:
require.Equal(t, "close", r.Header.Get("Connection"),
"STS client should disable keep-alives so each AssumeRole opens a fresh connection")The thorough comment at lines 82-93 mitigates accidental removal, but comments don't prevent regressions. A regression here means multi-minute production recovery windows after trust-policy changes, a failure mode that only surfaces under specific conditions. (Bisky)
🤖
| // Disable keep-alive on the STS client so each AssumeRole opens a | ||
| // fresh connection. Observed: with keep-alive, AssumeRole calls reuse | ||
| // one connection pinned to a single STS endpoint, and after a | ||
| // trust-policy change that connection kept returning AccessDenied for | ||
| // minutes while a fresh connection (e.g. the AWS CLI) accepted the | ||
| // identical request at once; the gateway recovered only when that | ||
| // connection recycled or the process restarted. The STS-internal reason is | ||
| // unconfirmed (likely per-endpoint propagation of the change); what we | ||
| // verified is that a fresh connection per call recovers in seconds | ||
| // instead of minutes. AssumeRole runs at most once per credential-cache | ||
| // lifetime, so keep-alive saves nothing here. Scoped to the STS client | ||
| // only; Bedrock requests use a separate client and keep pooling. |
There was a problem hiding this comment.
Nit [CRF-2] Comment is 12 lines; 4 carry the why-not-what and trap, the rest is narrative padding.
Gon proposed trimming to:
// Disable keep-alive: with connection reuse, AssumeRole pins to one STS
// endpoint and returns stale AccessDenied for minutes after a trust-policy
// change. A fresh connection recovers in seconds. AssumeRole runs once per
// credential-cache lifetime, so keep-alive saves nothing.This cuts the "Observed:" investigation narrative, the AWS CLI verification detail, the "STS-internal reason is unconfirmed" speculation, and the scope sentence that repeats what the local stsClient variable already shows.
Orchestrator note: downgraded from Gon's P2. Three reviewers (Leorio, Knov, Mafu-san) independently praised this comment's structure, specifically its observation-inference separation and honest uncertainty disclosure. The trim is good advice; the original is not a defect. (Gon P2, downgraded by orchestrator)
🤖
| // verified is that a fresh connection per call recovers in seconds | ||
| // instead of minutes. AssumeRole runs at most once per credential-cache | ||
| // lifetime, so keep-alive saves nothing here. Scoped to the STS client | ||
| // only; Bedrock requests use a separate client and keep pooling. |
There was a problem hiding this comment.
Nit [CRF-3] The commit subject names the mechanism ("disable keep-alive on the STS assume-role client") instead of the condition it fixes. A git blame reader needs the condition, not the knob. Consider: fix(aibridge/provider): prevent stale AccessDenied after STS trust-policy changes. The PR description is exemplary; the subject should carry the same intent. (Leorio)
🤖
dannykopping
left a comment
There was a problem hiding this comment.
LGTM but would be good to validate with a test.
4f1c2dc to
fa282e4
Compare
A Bedrock provider that assumes an IAM role kept failing with
AssumeRoleAccessDeniedfor several minutes after its target role's trust policy was changed, and only recovered on a gateway restart or a long wait. The request itself was correct: the AWS CLI, using the same identity and the sameExternalId/role/region, accepted the identical request immediately against the same endpoint.The difference is the connection. The Go SDK reuses a keep-alive connection for the STS client, so every
AssumeRolerides one connection pinned to a single STS endpoint. After a trust-policy change, that connection kept returningAccessDeniedfor minutes while a fresh connection (the AWS CLI) accepted the identical request at once; it recovered only when the connection recycled or the process restarted. The exact STS-internal reason is unconfirmed (likely per-endpoint propagation of the change) — what is verified is that a fresh connection per call recovers promptly.Disable keep-alive on the STS client so each
AssumeRoleopens a fresh connection and a trust-policy update takes effect quickly.AssumeRoleruns at most once per credential-cache lifetime, so keep-alive bought nothing here. The change is scoped to the STS client only; Bedrock model requests are signed by a separate client and keep their connection pooling.What the data proves
bedrock-base-user-useless/AKIA…44NLsts.us-east-2.amazonaws.comExternalId=QL53…, role, session, 900AssumeRoles, all deniedSame identity, params, and endpoint, concurrent — yet the gateway was denied for ~4.5 minutes while the CLI recovered in 7 seconds, and the gateway made a fresh
AssumeRoleon every request (so it was not caching a failure). The only difference was connection reuse.After disabling keep-alive, the same break/fix experiment brought gateway recovery down from ~4.5 minutes to ~7 seconds, in lockstep with the AWS CLI.