fix(coderd/x/chatd): classify bedrock credential errors as non-retryable - #27913
Conversation
|
/coder-agents-review |
|
Chat: Review posted | View chat Review historydeep-review v0.9.0 | Round 2 | Last posted: Round 2, 6 findings (1 P2, 2 P3, 2 Nit, 1 Note), COMMENT. Review Finding inventoryFinding inventory, PR #27913Findings
Contested and acknowledgedCRF-2 (P3, classify.go:184) - text/plain detail truncated to first line
CRF-1 (P2, classify.go:181) - partial widening, remainder re-raised
Round logRound 1Netero-only first pass (P2 present, panel gated). 1 P2, 2 P3, 3 Nit, 1 Note. Reviewed against f96671d..a8008f0. Round 2Churn guard: PROCEED (6 addressed, 1 deferred with ticket, 0 silent). Branch was rewritten; head b927f1f carries a single commit. Netero-only again: a P2 remains on a pre-panel round, so the panel is still gated. 1 P2 re-raised (CRF-1), 1 new P2 (CRF-9, CI), 1 Nit, 1 Note. Reviewed against f96671d..b927f1f. About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
First-pass review only. These are mechanical findings from the pre-panel reviewer; the full review panel has not reviewed this PR yet and will do so after these are addressed.
The diagnosis is precise and the test proves it: restoring classify.go from the base SHA makes TestClassify_BedrockCredentialResolutionDeadline fail with expected "timeout", actual "generic", so the test is not tautological. Renaming usageLimitText to combinedText de-duplicates rather than adds a copy, and git grep usageLimitText returns nothing, so the rename left nothing dangling. go test ./coderd/x/chatd/chaterror/ -count=1 and go vet pass at head.
Counts: 1 P2, 2 P3, 3 Nit, 1 Note.
The P2 is the shape of the fix, not its correctness. Netero: "The PR establishes the fact that for Bedrock via the fantasy adapter the transport wrapper carries no usable text and the body is the only signal. That fact does not stop at 'context deadline exceeded'." A throwaway in-package test showed four more permanent failures still arriving as retryable generic 500s. Either widen the remaining checks to combinedText or record in the comment why deadline and usage limits are the only two signals allowed to read the body.
Two findings with no diff line to attach to:
Nit [CRF-5] Commit scope fix(chatd) is not a real filesystem path. AGENTS.md requires a scope that is a real path containing every changed file; the changed files live under coderd/x/chatd/chaterror/ and there is no root chatd/. The PR title already uses fix(coderd/x/chatd); make the commit match.
Nit [CRF-6] Em-dash (U+2014) in the PR description, in "ProviderError.ResponseBody — but ProviderError.Error()". AGENTS.md bans em-dash, en-dash, and -- as punctuation; PR prose is held to the same rule. A byte-grep for U+2014 and U+2013 over the full diff is clean, so this is body text only.
Note [CRF-7] Both failing checks are the title job. Reading .github/workflows/contrib.yaml:121-222, the current title fix(coderd/x/chatd) satisfies both the type allowlist and the scope-path rule, so the red runs look stale. This is an inference, not an observation: gh returns HTTP 401 in this environment, so the failing run logs could not be read. Confirm with a re-run.
🤖 This review was automatically generated with Coder Agents.
| (statusCode != 429 && containsAny(usageLimitText, usageLimitPatterns...)) | ||
| usageLimitMatch := containsAny(combinedText, usageLimitAnyStatusPatterns...) || | ||
| (statusCode != 429 && containsAny(combinedText, usageLimitPatterns...)) | ||
| authStrong := statusCode == 401 || containsAny(lower, authStrongPatterns...) |
There was a problem hiding this comment.
P2 [CRF-1] The body-is-the-only-signal fact applies to every signal, but only deadline and usageLimitMatch were widened. Auth, config, rate-limit, timeout, and overloaded text in the same Bedrock plain-text 500 shape still misclassifies. (Netero)
authStrong(:192),configMatch(:193),authWeak(:194),rateLimitMatch(:195),timeoutPatternMatch(:196),overloadedMatch(:186), andproviderDisabledMatch(:181) all still match againstloweralone.
Verified with a throwaway in-package test (added, run, removed):
body="...resolve AWS credentials: ...GetMetadata, request timed out" kind=generic retryable=true
body="create bedrock client: invalid model identifier for this region" kind=generic retryable=true
body="...unauthorized: the security token included in the request is invalid" kind=generic retryable=true
body="upstream provider is overloaded, please retry" kind=generic retryable=true
I confirmed each of those seven expressions reads lower in the current head. All four cases are permanent or better-classified failures arriving as retryable generic 500s, which is exactly the bug this PR fixes for one string. The fix is instance-level where the evidence is class-level. Either feed combinedText to the remaining checks, the ordering rules at :209-218 already handle precedence, or state in the comment why deadline and usage limits are the only two signals allowed to read the body.
🤖
There was a problem hiding this comment.
Scope creep. This is a job for a follow-up.
There was a problem hiding this comment.
Compromise applied: widened overloaded, authStrong, authWeak, config, and timeoutPattern to combinedText with test coverage in TestClassify_BedrockBodyOnlySignals. rateLimitMatch and providerDisabledMatch left on lower for a follow-up.
Filed #27918 for the broader refactor: normalize all signal checks through a single evidence struct so the two-source inconsistency can't reintroduce silently.
There was a problem hiding this comment.
Re-raising the deferred remainder of CRF-1, narrowed to providerDisabledMatch at classify.go:181. The deferral rests on the remainder being a kind-label issue. That is true for rateLimitMatch and false for providerDisabledMatch. (Netero)
Reproduced at head b927f1f with a throwaway in-package test (added, run, removed, git status clean):
input: wrapper `POST "...": 503 Service Unavailable`, statusCode 503,
text/plain body `provider_disabled: AI provider "anthropic" is disabled`
output: kind=timeout retryable=true status=503
message="AWS Bedrock is temporarily unavailable."
aibridge writes the sentinel via
http.Erroras a plain-text 503 body. The anthropic SDK'sError()isstatusInfo + JSON.raw, so a non-JSON body does not appear in the wrapper, the same channel gap this PR fixes for credential errors.providerDisabledMatchstill readsloweronly (classify.go:181), so the sentinel instructured.detailis invisible; statusCode 503 then satisfies the timeout rule, and the code's own ordering comment (classify.go:218-219, "Provider-disabled must precede timeout because disabled providers return 503") documents that this ordering exists precisely to prevent it.
I confirmed the SDK mechanism by reading coder/anthropic-sdk-go internal/apierror/apierror.go:36-45 (Error() returns statusInfo plus r.JSON.raw) and coder/fantasy providers/anthropic/error.go:20-28 (Message: apiErr.Error(), ResponseBody: apiErr.DumpResponse(true)).
Consequence: an operator disables a provider, the worker retries up to chatretry.MaxAttempts against a provider that will never answer, and the user sees a retryable "temporarily unavailable" instead of "The provider has been disabled. Contact your Coder administrator." retryable flips false to true, so this is behavioral, not a label.
Why the existing suite misses it: the ProviderDisabled503* cases (classify_test.go:353-383) build ProviderError with the sentinel mirrored into Message, which is the shape this PR's own premise says does not occur for text/plain aibridge responses. Those tests and this PR disagree about the wrapper contents; the SDK source above settles it.
The fix is lower to combinedText at classify.go:181 plus a body-only test case, in the expression block this PR already edits. rateLimitMatch can stay deferred: body-only rate-limit text under a 500 falls to the generic rule, which is also retryable=true, and RetryAfter comes from response headers regardless, so that half really is a label.
🤖
There was a problem hiding this comment.
Good catch. Widened providerDisabledMatch to combinedText and added TestClassify_ProviderDisabledBodyOnly with the exact 503 + provider_disabled sentinel reproduction. Confirmed it classifies as ProviderDisabled with retryable=false instead of Timeout with retryable=true.
| deadline := errors.Is(err, context.DeadlineExceeded) || strings.Contains(lower, "context deadline exceeded") | ||
| // combinedText merges the transport wrapper text with the structured | ||
| // provider response body so signal patterns in either are detected. | ||
| combinedText := lower + "\n" + strings.ToLower(structured.detail) |
There was a problem hiding this comment.
P3 [CRF-2] structured.detail for a text/plain body is only the body's first line, so the new deadline signal misses any credential error whose deadline text is not on line 1. (Netero)
providerErrorResponseMessagecuts plain-text bodies at the first newline (coderd/x/chatd/chaterror/provider_error.go:75).combinedTexttherefore merges the wrapper with one line of body, not the body.
Verified:
body="line one of the failure\ncreate anthropic client: ...: context deadline exceeded"
-> kind=generic retryable=true detail="line one of the failure"
I read providerErrorResponseMessage and confirmed the strings.Cut(..., "\n") truncation. The observed AWS SDK error is single-line so the shipped test passes, but AWS credential chain errors are not guaranteed single-line. The signal source and the user-facing detail are currently the same string; match on the full body and keep the first line for Detail to remove the coupling.
🤖
There was a problem hiding this comment.
Deferred to #27918. The first-line truncation coupling between signal matching and detail display is a structural issue that the follow-up refactor will address by decoupling the signal text from the display text.
| // combinedText merges the transport wrapper text with the structured | ||
| // provider response body so signal patterns in either are detected. | ||
| combinedText := lower + "\n" + strings.ToLower(structured.detail) | ||
| deadline := errors.Is(err, context.DeadlineExceeded) || strings.Contains(combinedText, "context deadline exceeded") |
There was a problem hiding this comment.
P3 [CRF-3] Any provider 500 whose body mentions "context deadline exceeded" is now permanently non-retryable, including a genuinely transient upstream timeout. (Netero)
deadlinefeedsretryable: !deadlineat :257. Before this change, deadline text confined to a response body produced a retryable generic 500; now it produces a terminal timeout.
The misconfigured-Bedrock case is the intended win, but the classifier cannot distinguish "credentials can never resolve" from "the gateway's call to a healthy provider timed out once", and both write the same Go string into the body. One transient upstream timeout now fails the turn with no retry.
Counter-argument, stated so it can be judged: wrapper-carried deadline text was already non-retryable, so this extends an existing policy rather than inventing one. It does not dissolve the finding, because the body is a different channel: wrapper deadline text means the local request context expired, while body text can come from the gateway's own upstream call. If the intended discriminator is credential resolution, match on that (resolve aws credentials, failed to refresh cached credentials) rather than on the generic Go deadline string.
🤖
There was a problem hiding this comment.
Only checking for "context deadline exceeded" on lower now.
There was a problem hiding this comment.
Addressed as the reviewer suggested: deadline stays on lower only, so body text mentioning "context deadline exceeded" no longer triggers a terminal timeout. Instead, added "resolve aws credentials" to configPatterns (which does check combinedText), so credential resolution failures classify as a non-retryable config error. This matches the specific permanent failure without making ambiguous body deadline text non-retryable.
There was a problem hiding this comment.
Verified fixed in b927f1f86. deadline reads lower only, and "resolve aws credentials" in configPatterns gives the specific, non-retryable discriminator without making ambiguous body deadline text terminal. This is the better fix of the two options offered.
🤖
| combinedText := lower + "\n" + strings.ToLower(structured.detail) | ||
| deadline := errors.Is(err, context.DeadlineExceeded) || strings.Contains(combinedText, "context deadline exceeded") | ||
| overloadedMatch := statusCode == 529 || containsAny(lower, overloadedPatterns...) | ||
| // Usage limits do not have a dedicated status code, so provider |
There was a problem hiding this comment.
Nit [CRF-4] The comment justifying body-text matching for usage limits is contradicted by the line above it. (Netero)
It reads "Other classes already have status-code signals or transport wrapper text." This PR exists because the deadline class did not.
The comment now asserts as settled fact the assumption the diff disproves. Fold it into the combinedText comment at :182 or drop the second sentence.
🤖
There was a problem hiding this comment.
Fixed. The contradictory comment is replaced with a single consolidated comment above combinedText that accurately describes all body-only signals: "Usage limits, auth, config, and timeout patterns do not have a dedicated status code, so provider response bodies can be the only reliable signal."
There was a problem hiding this comment.
Nit [CRF-8] The replacement comment still makes a false claim, a different one. (Netero)
The new comment says auth and timeout "do not have a dedicated status code" while the adjacent lines check
statusCode == 401,== 403(classify.go:193-194) and the timeout rule checks 408/502/503/504 (classify.go:205-206). It also repeats "Other classes already have status-code signals or transport wrapper text", which the body-only provider-disabled case above disproves.
I read the adjacent lines and confirmed both status-code checks. CRF-4 flagged the false-claim pattern; the reword moved the false claim rather than removing it. State the operative fact instead: aibridge writes these failures as plain-text bodies that never reach the transport wrapper, so the body can be the only signal regardless of the class's nominal status code.
🤖
There was a problem hiding this comment.
Fixed. Rewrote the comment to state the operative fact: "AIBridge writes some failures as plain-text bodies that never reach the transport wrapper, so the body can be the only signal regardless of the class's nominal status code." No more false claims about missing status codes.
5fb3e48 to
2c4da64
Compare
2c4da64 to
b927f1f
Compare
|
Review findings addressed:
|
|
/coder-agents-review |
There was a problem hiding this comment.
First-pass review only, round 2. The full review panel still has not reviewed this PR: a P2 remains on the mechanical floor, so panel selection stays gated. This is the second consecutive first-pass round, so the panel runs next round regardless.
Good progress. Six of seven round-1 findings are fixed and verified: deadline is back on lower only, "resolve aws credentials" is the specific discriminator the finding asked for, the contradictory comment was replaced, the commit scope is now fix(coderd/x/chatd), the em-dash is gone, and the title job is green. Both new tests are non-tautological: reverting classify.go and signals.go to base makes TestClassify_BedrockCredentialResolutionDeadline fail and all four TestClassify_BedrockBodyOnlySignals subtests fail. go build ./... and go test ./coderd/x/chatd/... -count=1 pass locally, including the pg-backed coderd/x/chatd package.
Counts this round: 1 P2 re-raised, 1 new P2, 1 Nit, 1 Note.
The re-raise is on CRF-1's deferred remainder, and it is not a labeling argument. Netero: "Deferring the behavioral half of the bug class to #27918 while shipping the cosmetic half is the wrong split." I reproduced it independently at this head with a throwaway in-package test: a 503 transport wrapper plus a text/plain body carrying only the aibridge provider_disabled sentinel classifies as kind=timeout retryable=true, message "AWS Bedrock is temporarily unavailable." Retryability flips false to true, so a provider an operator deliberately disabled gets retried up to chatretry.MaxAttempts. Details in the reply on that thread. rateLimitMatch staying on lower is fine and I am not asking for it; the ask is one word at classify.go:181 plus a test.
P2 [CRF-9] test-go-pg-17 is red at head b927f1f and the failing test is unnamed. Everything I can run locally passes: go build ./..., the full chaterror, chatloop, chatretry, and chathooks packages, and the pg-backed coderd/x/chatd package. That covers every package this PR touches plus its direct consumers, so the failure is probably elsewhere in the shard, but probably is not proven and a red check is not pre-existing noise. gh returns HTTP 401 in this environment, so run 92622203789's logs are unreadable here. Someone with log access needs to name the failing test before merge.
One process note: #27918 is cited for two deferrals but I could not confirm it exists or that its scope covers CRF-2 and the CRF-1 remainder, for the same auth reason. If the provider-disabled half is not fixed here, that gap needs a human decision rather than an agent-accepted deferral, because neither of us can accept a known behavioral regression as permanent.
🤖 This review was automatically generated with Coder Agents.
|
@coder-agents-review coder/internal#1638 exists and I would like to avoid it creeping into this PR. |
When a Bedrock provider is misconfigured without authentication methods, AWS credential resolution fails and AIBridge writes the error as a plain-text HTTP 500. The fantasy adapter captures the body text in ProviderError.ResponseBody, but Error() returns only the SDK transport wrapper, not the body. Signal patterns in chaterror.Classify checked only err.Error() (the wrapper), missing the useful text in structured.detail (the body). This caused permanent configuration errors to fall through to the generic 500 rule with retryable=true, making the chat worker retry up to 25 times. Introduce combinedText (merging the wrapper with structured.detail) and widen signal checks that have no dedicated status code to use it: overloaded, auth, config, usage limit, and timeout patterns. The deadline signal stays on err.Error() to avoid treating ambiguous body text as a local context deadline. Add a 'resolve aws credentials' config pattern so credential resolution failures classify as config, not generic.
b927f1f to
b1088d6
Compare
There was a problem hiding this comment.
Pull request overview
This PR improves chaterror.Classify in coderd/x/chatd so provider misconfiguration and other body-only error signals (notably from Bedrock via AIBridge/fantasy) are correctly detected and classified as non-retryable when appropriate, preventing wasteful chat worker retries.
Changes:
- Add a
combinedTextsignal source (transport wrapper text + structured response body) and use it for pattern-based classification where the body may be the only reliable signal. - Extend config detection to include the Bedrock/AWS credential resolution failure phrase
"resolve aws credentials". - Add targeted tests covering the original Bedrock credential-resolution scenario and other body-only signal cases.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| coderd/x/chatd/chaterror/signals.go | Adds "resolve aws credentials" to config patterns to classify Bedrock credential-resolution failures as configuration errors. |
| coderd/x/chatd/chaterror/classify.go | Introduces combinedText and broadens several signal checks to include provider response body text when deriving kind/retryability. |
| coderd/x/chatd/chaterror/classify_test.go | Adds regression and table tests to ensure body-only signals classify correctly (config/auth/overloaded/timeout). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
ethanndickson
left a comment
There was a problem hiding this comment.
+1 to cleaning error classification up with a refactor but this seems fine for now
When a Bedrock provider is misconfigured without authentication methods, AWS credential resolution fails and AIBridge writes the error as a plain-text HTTP 500. The fantasy adapter captures the body text in
ProviderError.ResponseBody, butError()returns only the SDK transport wrapper, not the body.Signal patterns in
chaterror.Classifychecked onlyerr.Error()(the wrapper), missing the useful text instructured.detail(the body). This caused permanent configuration errors to fall through to the generic 500 rule withretryable=true, making the chat worker retry up to 25 times.Introduce
combinedText(merging the wrapper withstructured.detail) and widen signal checks that have no dedicated status code to use it: overloaded, auth, config, usage limit, and timeout patterns. The deadline signal stays onerr.Error()to avoid treating ambiguous body text as a local context deadline. Add a "resolve aws credentials" config pattern so credential resolution failures classify as config, not generic.Implementation plan
Root cause
AIBridge writes credential resolution failures as a plain-text HTTP 500. The fantasy adapter captures the body in
ProviderError.ResponseBodybutError()returns onlyTitle: Message(no body text). The classifier signal checks only looked aterr.Error(), missing the body text where the useful signal lives. The precedent for checkingstructured.detailalready existed forusageLimitText; this PR extends it to the remaining body-only signals.Changes
classify.go: introducecombinedText(wrapper + body), widenoverloaded,authStrong,authWeak,config,timeoutPattern, andusageLimitsignal checks to use it.deadlinestays onerr.Error()only.signals.go: add"resolve aws credentials"toconfigPatternsso credential resolution failures classify as config.classify_test.go: addTestClassify_BedrockCredentialResolutionDeadline(the original reported bug) andTestClassify_BedrockBodyOnlySignals(table test covering overloaded, auth, config, and timeout signals arriving via response body only).