🤖 fix: summarize invalid tool input errors and state the advisor question cap - #4227
🤖 fix: summarize invalid tool input errors and state the advisor question cap#4227ibetitsmike wants to merge 3 commits into
Conversation
When the AI SDK rejects a tool call before execution, StreamManager relayed the raw AI_InvalidToolInputError string as the tool result. That string echoes the entire submitted input and buries the single zod issue at the end without stating the received length, so the model retried blind. Add a shared formatToolInputIssues helper that renders zod issues as "<path>: <message> (received N characters)" without echoing the value, use it for the kernel bridge's existing "Invalid arguments" path, and have StreamManager derive the same concise message from the invalid tool-call part's cause chain before the matching tool-error arrives.
Share the 2000-character question limit between the schema and the tool description so the model knows the cap up front. Return an error result when the advisor stream finishes without any text instead of a success-shaped empty advice string, so the caller and the tool card see that the consultation produced nothing.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
@codex review |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b5a7176099
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… retry streamText builds the next step's tool-result from the stream after experimental_transform runs, and prepareStep leaves that content alone, so the fullStream consumer's rewrite only reached the tool card and later turns. Within the same turn the model still received the SDK's input-echoing AI_InvalidToolInputError text on its retry (found in UAT round 2). Move the summary into a streamText transform that records each rejected tool-call's concise message and rewrites the matching tool-error, so the model's same-turn retry, the persisted history, and the card all carry the same text. Drop the consumer-side plumbing that this replaces, and cover the same-turn path with an end-to-end StreamManager test that asserts the second provider request's tool-result.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 44aafafbe2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Summary
Tool input validation failures now reach the model as one concise, actionable sentence instead of the AI SDK's raw
AI_InvalidToolInputErrorstring, theadvisortool description states its 2000-characterquestioncap, and an advisor stream that produces no text is reported as an error instead of a success-shaped empty result.Background
In an RLM-mode session the model called
advisorwith a 3421-characterquestion. The zod schema rejected it before execution and StreamManager relayed the SDK's error verbatim as the tool result: a ~4 KB string that echoes the entire submitted input and only at the very end saysToo big: expected string to have <=2000 characters, without the actual length. The model retried at 2071 characters (same blob), then its two in-limit calls returned{ type: "advice", advice: "" }because the advisor model stream produced nothing, which the tool reported as success. Four calls, zero advice, and the tool description never mentioned the cap.The RLM kernel path (
toolBridge.ts) already rendered zod failures readably; only the top-level AI SDK path relayed the raw string.Implementation
formatToolInputIssues(new,src/common/utils/tools): renders zod v4 issues as<path>: <message>, appending(received N characters)for string size violations by resolving the path inside the parsed input. It never includes the input value. Shared by the kernel bridge and StreamManager so both paths produce the same shape.summarizeInvalidToolInputErrors(new,src/node/utils/messages): astreamTextexperimental_transform. An invalid dynamictool-callpart carries the structured error; its cause chain (InvalidToolInputError->TypeValidationError->ZodError) is walked to buildInvalid input for tool advisor: question: Too big: expected string to have <=2000 characters (received 3421 characters), and the matchingtool-errorpart'serroris rewritten to it. Non-zod causes fall back to the clamped original message. It has to be a transform rather than fullStream-consumer logic: AI SDK 7 builds the next step's tool-result message from the transformed stream (andprepareStepdoes not rewrite it), so a consumer-side rewrite only reached the tool card and later turns while the model's same-turn retry still got the input-echoing text. StreamManager wires the transform increateStreamResult; itstool-errorhandling is unchanged and now simply persists the rewritten string.ADVISOR_QUESTION_MAX_CHARSis shared byAdvisorToolInputSchemaandADVISOR_TOOL_DESCRIPTION.advisor.ts: blank advice returns{ type: "error", message: "Advisor returned no advice (finish reason: ...)" }after usage is still reported.Validation
streamManager.invalidToolInput.test.tsdrives a realstartStreamwith a mock model that emits an over-capadvisorcall and captures the second provider request: its tool-result is the one-line summary (withreceived 2100 characters, no echoed input), and the persisted tool part carries the same text. Without the transform the same test receives the ~2 KBAI_InvalidToolInputErrorblob.Remote dogfood UAT on
b5a71760(Coder Agents, fresh workspace, real models): over-long (2500, 6057 chars), boundary (2000 passes, 2001 fails withreceived 2001), special-character and multibyte payloads all produce the one-line error with no echoed input; a retry within the limit and a short regression call return real advice; the tool description shows the cap; an organic empty advisor response rendered as a failed card with the finish reason. Round 2 (RLM Mode enabled) repeated the over-cap, 2000/2001 boundary, hostile-character, kernel (xum.timeline_event400 chars ->received 400 characters) and short-regression cases, and found the same-turn gap fixed by the third commit. Round 3 on the final head is recorded in the ledger below.Risks
Low. The StreamManager change only alters the
errorstring of tool results the SDK already rejected before execution; successful tool calls and non-validation tool errors are untouched. The advisor change converts a previously silent empty success into a visible error result.Found during UAT, out of scope
Instructing the agent to call
advisor({ question: "" })produced a provider-side stream error from the dev AI gateway (processing error: accumulate event: error converting content block to JSON ... invalid character '}') before any tool-call part reached StreamManager. The stream then auto-retried indefinitely (Stop did not end it) and the app hung on "Loading Xum..." for every browser session until the backend was restarted. Nothing in this diff is on that path (the error arrives as a providererrorpart, not as a tool call), and round 2 reproduced it byte-identically on the merge-base023701d63, so it is pre-existing and tracked separately: see #4228.Round 2 also noted that kernel
code_executionresults echo an oversized argument insidetoolCalls[].argseven though the error string is concise. That is the designed, size-capped args trace (KERNEL_COMPACT_ARGS_CAP_BYTESinsrc/constants/kernelOutput.ts), untouched here.Deferred from review: capping the number of rendered zod issues in
formatToolInputIssues(Codex P1, round 2). Not a regression of this PR; the change is ready and tested on branchmike/bound-tool-input-summary(d65a6739, stacked on this head) and can be opened as a stacked PR on request once this PR lands.Review ledger (delivery record)
@codex review, 01:58Z)b5a71760ADVISOR_QUESTION_MAX_CHARStosrc/constants/src/common/constants/advisor.ts; tool limits already live undersrc/common/constants/toolLimits.ts.b5a7176044aafafbformatToolInputIssuesfor many zod issuesmike/bound-tool-input-summary(d65a6739, stacked on this head) to open as a stacked PR after this one lands.44aafafb44aafafbUAT: round 1 on
b5a71760(remote Coder Agents chat, real models) passed every in-scope scenario; its one Critical finding is the pre-existing gateway/retry hang tracked in #4228. Round 2 onb5a71760(RLM Mode path, kernel path, baseline of the empty-question case on the merge-base): remote verdict FAIL, adjudicated as in-scope PASS; the empty-question hang and the reload hang reproduced identically on the merge-base; one new Medium finding (the model's same-turn retry still received the verbose SDK text) is fixed by44aafafb. Round 3 on44aafafb(final head, API Debug Logs enabled): PASS. Thedevtools.jsonlexcerpt shows the second provider request of the turn carries the rejected call's tool-result as the 119-character one-liner (no echoed question, noType validation failed), the same-turn retry returned real advice, 2000 accepted / 2001 rejected, and an unrelatedfile_readwas unaffected. Review count: 5 of a 6-review budget.Status: held unmerged; BLOCKED on the explicit Codex approval gate. Mike authorized the merge in the delivery chat on 2026-09-13 ("if CI is green and codex is good"); the PR was submitted to the
mainmerge queue at 06:55Z and withdrawn through the normal dequeue path at 07:07Z, because the "Codex is good" condition, as the repo's readiness rule defines it (AGENTS.md "PR readiness",scripts/wait_pr_codex.sh: a thumbs-up reaction or a "Didn't find any major issues" comment), is not met. Codex's round on44aafafb(code review #4227 (review), security clean per the summary comment #4227 (comment)) closed with its one finding answered and resolved, and the PR has no approval reaction, approval comment, or APPROVED review. This delivery runs under a hard six-review ceiling (5 used); a re-request would trigger an automatic code + security pair beyond it, so no further review will be requested and merge consent alone does not clear the gate within this delivery. CI on44aafafbis green (Requiredsuccess). The deferredformatToolInputIssuesbound (d65a6739onmike/bound-tool-input-summary) follows as a stacked PR on request.Generated with
xum• Model:anthropic:claude-fable-5-1• Thinking:xhigh