Thanks to visit codestin.com
Credit goes to github.com

Skip to content

🤖 fix: summarize invalid tool input errors and state the advisor question cap - #4227

Open
ibetitsmike wants to merge 3 commits into
mainfrom
mike/advisor-invalid-input-error
Open

🤖 fix: summarize invalid tool input errors and state the advisor question cap#4227
ibetitsmike wants to merge 3 commits into
mainfrom
mike/advisor-invalid-input-error

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Tool input validation failures now reach the model as one concise, actionable sentence instead of the AI SDK's raw AI_InvalidToolInputError string, the advisor tool description states its 2000-character question cap, and an advisor stream that produces no text is reported as an error instead of a success-shaped empty result.

Background

In an RLM-mode session the model called advisor with a 3421-character question. The zod schema rejected it before execution and StreamManager relayed the SDK's error verbatim as the tool result: a ~4 KB string that echoes the entire submitted input and only at the very end says Too big: expected string to have <=2000 characters, without the actual length. The model retried at 2071 characters (same blob), then its two in-limit calls returned { type: "advice", advice: "" } because the advisor model stream produced nothing, which the tool reported as success. Four calls, zero advice, and the tool description never mentioned the cap.

The RLM kernel path (toolBridge.ts) already rendered zod failures readably; only the top-level AI SDK path relayed the raw string.

Implementation

  • formatToolInputIssues (new, src/common/utils/tools): renders zod v4 issues as <path>: <message>, appending (received N characters) for string size violations by resolving the path inside the parsed input. It never includes the input value. Shared by the kernel bridge and StreamManager so both paths produce the same shape.
  • summarizeInvalidToolInputErrors (new, src/node/utils/messages): a streamText experimental_transform. An invalid dynamic tool-call part carries the structured error; its cause chain (InvalidToolInputError -> TypeValidationError -> ZodError) is walked to build Invalid input for tool advisor: question: Too big: expected string to have <=2000 characters (received 3421 characters), and the matching tool-error part's error is rewritten to it. Non-zod causes fall back to the clamped original message. It has to be a transform rather than fullStream-consumer logic: AI SDK 7 builds the next step's tool-result message from the transformed stream (and prepareStep does not rewrite it), so a consumer-side rewrite only reached the tool card and later turns while the model's same-turn retry still got the input-echoing text. StreamManager wires the transform in createStreamResult; its tool-error handling is unchanged and now simply persists the rewritten string.
  • ADVISOR_QUESTION_MAX_CHARS is shared by AdvisorToolInputSchema and ADVISOR_TOOL_DESCRIPTION.
  • advisor.ts: blank advice returns { type: "error", message: "Advisor returned no advice (finish reason: ...)" } after usage is still reported.

Validation

streamManager.invalidToolInput.test.ts drives a real startStream with a mock model that emits an over-cap advisor call and captures the second provider request: its tool-result is the one-line summary (with received 2100 characters, no echoed input), and the persisted tool part carries the same text. Without the transform the same test receives the ~2 KB AI_InvalidToolInputError blob.

Remote dogfood UAT on b5a71760 (Coder Agents, fresh workspace, real models): over-long (2500, 6057 chars), boundary (2000 passes, 2001 fails with received 2001), special-character and multibyte payloads all produce the one-line error with no echoed input; a retry within the limit and a short regression call return real advice; the tool description shows the cap; an organic empty advisor response rendered as a failed card with the finish reason. Round 2 (RLM Mode enabled) repeated the over-cap, 2000/2001 boundary, hostile-character, kernel (xum.timeline_event 400 chars -> received 400 characters) and short-regression cases, and found the same-turn gap fixed by the third commit. Round 3 on the final head is recorded in the ledger below.

Risks

Low. The StreamManager change only alters the error string of tool results the SDK already rejected before execution; successful tool calls and non-validation tool errors are untouched. The advisor change converts a previously silent empty success into a visible error result.

Found during UAT, out of scope

Instructing the agent to call advisor({ question: "" }) produced a provider-side stream error from the dev AI gateway (processing error: accumulate event: error converting content block to JSON ... invalid character '}') before any tool-call part reached StreamManager. The stream then auto-retried indefinitely (Stop did not end it) and the app hung on "Loading Xum..." for every browser session until the backend was restarted. Nothing in this diff is on that path (the error arrives as a provider error part, not as a tool call), and round 2 reproduced it byte-identically on the merge-base 023701d63, so it is pre-existing and tracked separately: see #4228.

Round 2 also noted that kernel code_execution results echo an oversized argument inside toolCalls[].args even though the error string is concise. That is the designed, size-capped args trace (KERNEL_COMPACT_ARGS_CAP_BYTES in src/constants/kernelOutput.ts), untouched here.

Deferred from review: capping the number of rendered zod issues in formatToolInputIssues (Codex P1, round 2). Not a regression of this PR; the change is ready and tested on branch mike/bound-tool-input-summary (d65a6739, stacked on this head) and can be opened as a stacked PR on request once this PR lands.

Review ledger (delivery record)
# Review Kind Commit Findings Disposition
1 Codex code review (manual @codex review, 01:58Z) code b5a71760 1 P1: move ADVISOR_QUESTION_MAX_CHARS to src/constants/ Reasoned rejection, replied on the thread and resolved: the constant sits with the other advisor bounds and the description that consumes it in src/common/constants/advisor.ts; tool limits already live under src/common/constants/toolLimits.ts.
2 Codex security review (automatic) security b5a71760 none Clean.
3 Codex code review (automatic on push, 03:10Z) code 44aafafb 1 P1: bound formatToolInputIssues for many zod issues Valid hardening, not a regression (the pre-PR SDK text was strictly larger for the same call); deferred with reasoning on the thread and resolved. Follow-up commit ready on mike/bound-tool-input-summary (d65a6739, stacked on this head) to open as a stacked PR after this one lands.
4 Codex security review (automatic on push) security 44aafafb none Clean.
5 Final Advisor (fresh clean-context review, 03:25Z) advisory 44aafafb none blocking VERDICT: READY. Confirmed the transform sits where the same-turn retry reads (AI SDK 7.0.19 event processor), tests are behavioural, both Codex dispositions defensible; rated the deferred issue-count bound Low and recommended folding it in if another review round is affordable, otherwise shipping it as a stacked PR promptly.

UAT: round 1 on b5a71760 (remote Coder Agents chat, real models) passed every in-scope scenario; its one Critical finding is the pre-existing gateway/retry hang tracked in #4228. Round 2 on b5a71760 (RLM Mode path, kernel path, baseline of the empty-question case on the merge-base): remote verdict FAIL, adjudicated as in-scope PASS; the empty-question hang and the reload hang reproduced identically on the merge-base; one new Medium finding (the model's same-turn retry still received the verbose SDK text) is fixed by 44aafafb. Round 3 on 44aafafb (final head, API Debug Logs enabled): PASS. The devtools.jsonl excerpt shows the second provider request of the turn carries the rejected call's tool-result as the 119-character one-liner (no echoed question, no Type validation failed), the same-turn retry returned real advice, 2000 accepted / 2001 rejected, and an unrelated file_read was unaffected. Review count: 5 of a 6-review budget.

Status: held unmerged; BLOCKED on the explicit Codex approval gate. Mike authorized the merge in the delivery chat on 2026-09-13 ("if CI is green and codex is good"); the PR was submitted to the main merge queue at 06:55Z and withdrawn through the normal dequeue path at 07:07Z, because the "Codex is good" condition, as the repo's readiness rule defines it (AGENTS.md "PR readiness", scripts/wait_pr_codex.sh: a thumbs-up reaction or a "Didn't find any major issues" comment), is not met. Codex's round on 44aafafb (code review #4227 (review), security clean per the summary comment #4227 (comment)) closed with its one finding answered and resolved, and the PR has no approval reaction, approval comment, or APPROVED review. This delivery runs under a hard six-review ceiling (5 used); a re-request would trigger an automatic code + security pair beyond it, so no further review will be requested and merge consent alone does not clear the gate within this delivery. CI on 44aafafb is green (Required success). The deferred formatToolInputIssues bound (d65a6739 on mike/bound-tool-input-summary) follows as a stacked PR on request.

Xum, acting for Mike, enqueued and then withdrew this PR from the merge queue; it remains unmerged under the gate above.


Generated with xum • Model: anthropic:claude-fable-5-1 • Thinking: xhigh

When the AI SDK rejects a tool call before execution, StreamManager relayed
the raw AI_InvalidToolInputError string as the tool result. That string
echoes the entire submitted input and buries the single zod issue at the
end without stating the received length, so the model retried blind.

Add a shared formatToolInputIssues helper that renders zod issues as
"<path>: <message> (received N characters)" without echoing the value,
use it for the kernel bridge's existing "Invalid arguments" path, and have
StreamManager derive the same concise message from the invalid tool-call
part's cause chain before the matching tool-error arrives.
Share the 2000-character question limit between the schema and the tool
description so the model knows the cap up front. Return an error result
when the advisor stream finishes without any text instead of a
success-shaped empty advice string, so the caller and the tool card see
that the consultation produced nothing.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 13, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-13T03:10:11.410331Z 44aafaf New commits
🔒 Security Review Completed 2026-09-13T03:09:21.761377Z 44aafaf New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@ibetitsmike

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: b5a7176099

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b5a7176099

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/common/constants/advisor.ts
… retry

streamText builds the next step's tool-result from the stream after
experimental_transform runs, and prepareStep leaves that content alone, so
the fullStream consumer's rewrite only reached the tool card and later turns.
Within the same turn the model still received the SDK's input-echoing
AI_InvalidToolInputError text on its retry (found in UAT round 2).

Move the summary into a streamText transform that records each rejected
tool-call's concise message and rewrites the matching tool-error, so the
model's same-turn retry, the persisted history, and the card all carry the
same text. Drop the consumer-side plumbing that this replaces, and cover the
same-turn path with an end-to-end StreamManager test that asserts the second
provider request's tool-result.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 44aafafbe2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/common/utils/tools/formatToolInputIssues.ts
@ibetitsmike
ibetitsmike added this pull request to the merge queue Sep 13, 2026
@ibetitsmike
ibetitsmike removed this pull request from the merge queue due to a manual request Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant