Thanks to visit codestin.com
Credit goes to github.com

Skip to content

fix(agent): repair Windows-path tool args; gate SD override (#1023)#1027

Merged
kovtcharov-amd merged 4 commits into
mainfrom
fix/sd-windows-path-escape-1023
May 11, 2026
Merged

fix(agent): repair Windows-path tool args; gate SD override (#1023)#1027
kovtcharov-amd merged 4 commits into
mainfrom
fix/sd-windows-path-escape-1023

Conversation

@kovtcharov

Copy link
Copy Markdown
Collaborator

Why this matters

gaia sd <prompt> on Windows would generate an image successfully in Step 1 but then return "Image generation is not available — start GAIA with the --sd flag to enable it" in Step 2 — telling the user to enable a feature that was already on, while silently dropping the story output.

Two stacked bugs:

  • Parse fragility — the native tool_calls parser strictly required JSON-valid backslash escapes, but the LLM probabilistically emits C:\Users\Klaus (single-escape) which JSON rejects as Invalid \escape: \U. Now repaired before re-raise; idempotent and lossless on already-valid input.
  • Overaggressive verbose-failure override — the post-failure guard fired whenever generate_image had been called this turn, even when it succeeded. Now gated on the latest capability call actually erroring.

Test plan

  • pytest tests/unit/agents/test_parse_error_recovery.py — 5 new tests + 6 pre-existing all pass
  • pytest tests/unit/agents/ tests/unit/test_sd_agent.py tests/unit/test_tool_call_priority.py — 190 pass, no regressions
  • Manual on Windows: gaia sd "a forest" --logging-level DEBUG — Step 2 parses the Windows path, story appears in the final answer (not the misleading "--sd" message)

Closes #1023

`gaia sd <prompt>` on Windows would generate an image successfully in
Step 1 but then return *"Image generation is not available — start
GAIA with the `--sd` flag to enable it"* in Step 2 — telling the user
to enable a feature that was already on, while silently dropping the
story output.

Two stacked bugs:

A. Parse fragility: the native tool_calls parser strictly required
   JSON-valid backslash escapes.  Smaller LLMs (Gemma-4-E4B-class)
   probabilistically emit Windows paths with single backslashes
   (`C:\Users\Klaus`), which strict JSON rejects as `Invalid \escape`.
   `_parse_llm_response` now retries once with `\X` (where X is not a
   valid JSON escape char) doubled to `\\X` — idempotent on
   already-valid input.  Truly malformed JSON still raises loudly.

B. Overaggressive verbose-failure override: the post-failure guard at
   `_process_query_impl` fired whenever `generate_image` had been
   *called* this turn — with no check on whether the call actually
   returned an error.  When generate_image succeeded and a *different*
   tool's parse error provoked a verbose apology, the guard clobbered
   the model's reply with the misleading "not available" message.
   Track the latest capability-tool outcome and gate the override on
   `capability_tool_last_succeeded is False`.
Review pass on the #1023 fix surfaced one inconsistency and one missing
regression guard.

The new ``capability_tool_last_succeeded`` tracker matched
``tool_name.startswith(...)`` while the pre-existing
``has_tried_capability_tool`` check uses ``.lower().startswith(...)``.
A model emitting ``Generate_Image`` would slip past the tracker (case
mismatch) but still trigger ``has_tried_capability_tool`` (after lower)
-- so on a *failed* call the gate evaluated ``True and (None is False)``
= False and the override silently failed to fire, leaking the verbose
apology to the user instead of the canonical "not available" message.
Both tracker call sites now apply ``.lower()`` to match.

Tests: added ``test_override_still_fires_when_generate_image_failed``
as a regression guard for the legitimate failure path (without it, a
future refactor could remove the gate entirely and only the
post-success test would catch it), and replaced the previously-useless
mixed-case success test with ``test_override_fires_after_mixed_case_capability_failure``
-- the success variant passed regardless of the bug because both
branches yield "no override" when ``capability_tool_last_succeeded`` is
``None``; only the failure variant pins the ``.lower()`` semantics
down.  Red-green verified by stashing the ``.lower()`` fix and watching
the new test fail with the exact verbose-apology leak.
@itomek

itomek commented May 11, 2026

Copy link
Copy Markdown
Collaborator

Tested end-to-end on Windows with gaia sd "a forest" --logging-level DEBUG from the PR branch.

Fix 1 — Windows-path escape repair

Repair fired on the create_story_from_image call ([PARSE] repaired invalid backslash escape(s)). No JSONDecodeError crash; the agent continued cleanly. One edge case surfaced: \t and \b in path components (\tomas, \busy-shtern) are valid JSON escapes, so the regex leaves them unchanged — the model self-corrected with double backslashes on retry. This is expected behavior, not a regression; the original issue's pattern (\U, \K, \D, etc.) is fully covered.

Fix 2 — SD override gate

generate_image succeeded (16.2s, SDXL-Turbo CPU). The final answer is a rich story/description of the generated image. No "start GAIA with the --sd flag to enable it" message anywhere in the output. Guard correctly stayed silent because capability_tool_last_succeeded was True.

Setup note: Lemonade defaults to rocm-preview for sd-cpp on AMD GPU machines, which crashes silently. Required lemonade config set sdcpp.backend=cpu before the SD pipeline would work. May be worth documenting or detecting upstream.

itomek
itomek previously approved these changes May 11, 2026
Keep main's parallel tool_calls loop (issue #944) and integrate
the Windows-path escape repair from this branch (issue #1023).
@kovtcharov-amd kovtcharov-amd added this pull request to the merge queue May 11, 2026
Merged via the queue into main with commit d99619a May 11, 2026
23 checks passed
@kovtcharov-amd kovtcharov-amd deleted the fix/sd-windows-path-escape-1023 branch May 11, 2026 20:36
itomek added a commit that referenced this pull request May 13, 2026
Three follow-up fixes after PR #1027 for klkr1's post-merge report on
issue #1023.  Before: when `gaia sd <prompt>` step 2 emitted a malformed
envelope or a wrong path, the final answer reported only the successful
image generation and stayed silent about the story-step failure -- the
model lost the canonical `image_path` from step 1 and the user lost
visibility into what went wrong.  After: the envelope parser tolerates
the same two malformed-JSON failure classes already repaired at the
inner-arguments level (single-backslash Windows paths and trailing
commentary), the parse-error recovery prompt names the canonical path
verbatim so the model can retry without retyping, and
`create_story_from_image` surfaces the most-recent PNG path in its
error so the next LLM turn has a clear breadcrumb.

- Envelope `_parse_llm_response` mirrors the inner-arguments recovery:
  retry via `_repair_invalid_json_escapes`, fall back to
  `JSONDecoder.raw_decode` for `Extra data`, validate `__tool_calls__`
  presence so a salvaged non-tool_calls prefix raises `ValueError`
  rather than letting `KeyError` escape the recovery branch's catch.
- Parse-error recovery reads `step_results` for the most-recent
  successful `image_path`; if found, appends a "use VERBATIM" hint to
  the next user-role message AND replaces the give-up fallback with
  "I generated your image at <path>, but couldn't finish the follow-up
  step."  Telemetry marker logs the path so production triage can
  distinguish "model received hint and mangled it" from "model never
  saw the hint."  Legacy single-tool path now mirrors the
  plan-execution path's `step_results.append`, so the recovery branch
  works regardless of which agent flow produced the result.
- SDAgent: system prompt explicitly tells the model to use the literal
  `$PREV.image_path` placeholder rather than retyping the path;
  `create_story_from_image` error scans `self.sd_output_dir` for the
  most-recent `.png` and appends the canonical path as a hint
  (defensive: never raises from inside the error path).

Refs #1023
github-merge-queue Bot pushed a commit that referenced this pull request May 13, 2026
…ry (#1047)

After PR #1027 landed, klkr1 confirmed the Windows-path repair runs but
reported three new failures on Gemma-4-E4B during `generate_image` →
`create_story_from_image`: the model hallucinated extra path segments in
step 2 (causing silent \"Image not found\" errors), and in one case the
outer tool-calls envelope itself failed with `Extra data: line 1 column
368` because the repair was only wired at the inner-arguments level, not
at the envelope level. In all three tries the final answer silently
reported only step 1 success and dropped step 2 entirely.

This PR closes those three gaps: the envelope parser now retries with
backslash repair then falls back to `raw_decode` for trailing-data
failures; the parse-error recovery branch now reads the canonical
`image_path` from `step_results` and surfaces it in both the retry
prompt and the give-up fallback so the model can correct itself; and
`create_story_from_image` now appends the real PNG path to its error
message so the LLM has a verbatim breadcrumb on the next turn. The SD
system prompt is also tightened with an explicit "use
`$PREV.image_path`, do NOT type the path yourself" directive.

## Test plan

- [ ] `PYTHONPATH=<worktree>/src python -m pytest
tests/unit/agents/test_parse_error_recovery.py
tests/unit/test_sd_agent.py -xvs` — 28 new tests pass; 2 pre-existing
failures on `TestProcessQueryRecoversOnContextOverflow` present on base
`main` before this PR
- [ ] `PYTHONPATH=<worktree>/src python -m pytest tests/unit/agents/
tests/unit/test_sd_agent.py -x` — full agent unit suite passes
- [ ] `python util/lint.py --all --fix` — no lint violations

Resolves #1023
@itomek itomek mentioned this pull request May 14, 2026
6 tasks
pull Bot pushed a commit to bhardwajRahul/gaia that referenced this pull request May 15, 2026
## Why this matters

v0.18.0 ships agent memory v2 (hybrid-search second brain with LLM
extraction and observability dashboard), ChatAgent split into three
composable agents (Chat/FileIO/DocumentQA), parallel tool calls, and a
Telegram adapter scaffold — plus fixes the RAG-on-PDF timeout with Gemma
4 that broke document Q&A since v0.17.6 and adds CI gates that enforce
RAG quality baselines on every future PR.

Full notes: `docs/releases/v0.18.0.mdx`.

## What's New

- **Agent memory v2** ([amd#606](amd#606)) —
Hybrid semantic + keyword search, LLM extraction, observability
dashboard via SSE streaming
([amd#1032](amd#1032)). Per-user isolation
enforced; extraction runs async so it doesn't add latency.
- **ChatAgent split** ([amd#979](amd#979)) —
`ChatAgent`, `FileIOAgent`, and `DocumentQAAgent` replace the monolithic
class; each composable via `tools=`. Backward-compatible shim preserved.
- **Parallel tool calls** ([amd#946](amd#946))
— Multiple `tool_calls` from a single LLM turn are executed
concurrently, cutting round-trips for multi-tool workflows.
- **Telegram adapter scaffold, Phase 0**
([amd#951](amd#951)) — `gaia telegram
start|stop|status`, per-user session isolation, `[telegram]` extras.
Phase 1 (message handling + allowed-users gate) tracked in
[amd#889](amd#889).
- **Connectors: per-MCP toggle + single-writer enforcement**
([amd#1018](amd#1018),
[amd#998](amd#998)) — Disable individual MCP
servers without removing them; concurrent writes serialised with
actionable errors on contention.
- **File navigation, web browsing, and write security**
([amd#495](amd#495)) — `FileSearchToolsMixin`,
web browsing tool, and scratchpad mixin in `KNOWN_TOOLS`; write tools
check `allowed_paths` before dispatch.
- **Email UI and policy alerts**
([amd#995](amd#995),
[amd#1039](amd#1039),
[amd#952](amd#952)) — Pre-scan triage card,
in-chat Connect, policy alert cards, and durable receipts for
confirmation-gated actions.

## Bug Fixes

- **RAG-on-PDF timeouts on Gemma 4**
([amd#1034](amd#1034), closes
[amd#1030](amd#1030)) — Prompt-size budget
check added at composition time; CI gates enforce it on every PR
([amd#1040](amd#1040)).
- **Envelope-level parse failure crashed SD recovery**
([amd#1047](amd#1047), closes
[amd#1023](amd#1023)) — Falls through to a
clean recovery path with step-1 context preserved.
- **Windows-path tool args corrupted**
([amd#1027](amd#1027)) — Backslash
normalisation now happens after argument parsing.
- **Blender `send_command` hung**
([amd#1026](amd#1026), closes
[amd#1022](amd#1022)) — Read timeout applied
to persistent-connection servers.
- **`gaia chat init` in post-install banner**
([amd#1029](amd#1029), closes
[amd#1024](amd#1024)) — Replaced with the
correct `gaia init`.
- **Keyring treated as required**
([amd#1028](amd#1028)) — Import guarded;
optional on systems without `keyring`.
- **electron-builder URLs stale**
([amd#953](amd#953)) — Three doc/installer
files updated to current download paths.

## Tooling & Docs

- **RAG eval CI gates** ([amd#1040](amd#1040),
closes [amd#1033](amd#1033)) — RAG quality
baselines + prompt-size budget enforced on every PR.
- **Fork-PR authors now receive Claude review**
([amd#932](amd#932)) —
`allowed_non_write_users: "*"` with prompt-injection mitigations
documented.
- **Eval runs mandated before merging**
([amd#1036](amd#1036)) — `CLAUDE.md` requires
`gaia eval agent` for LLM-affecting changes.
- **GAIA website** ([amd#369](amd#369)) —
[amd-gaia.ai](https://amd-gaia.ai) live.
- **Custom agent guide reorganised**
([amd#997](amd#997)), Lemonade PPA docs
([amd#801](amd#801)), broken Lemonade CLI URL
fixed ([amd#996](amd#996)), WhatsApp adapter
evaluation spec ([amd#950](amd#950)).

## Release checklist

- [x] `util/validate_release_notes.py docs/releases/v0.18.0.mdx --tag
v0.18.0` passes
- [x] `src/gaia/version.py` → `0.18.0`
- [x] `src/gaia/apps/webui/package.json` → `0.18.0`
- [x] Navbar label in `docs/docs.json` → `v0.18.0 · Lemonade 10.2.0`
- [x] All 28 commits in range (v0.17.6..HEAD) are represented in the
notes
- [ ] Review from @kovtcharov-amd addressed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents tests Test changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: tool call parsing fails in SD step

3 participants