fix(agent): repair Windows-path tool args; gate SD override (#1023)#1027
Conversation
`gaia sd <prompt>` on Windows would generate an image successfully in Step 1 but then return *"Image generation is not available — start GAIA with the `--sd` flag to enable it"* in Step 2 — telling the user to enable a feature that was already on, while silently dropping the story output. Two stacked bugs: A. Parse fragility: the native tool_calls parser strictly required JSON-valid backslash escapes. Smaller LLMs (Gemma-4-E4B-class) probabilistically emit Windows paths with single backslashes (`C:\Users\Klaus`), which strict JSON rejects as `Invalid \escape`. `_parse_llm_response` now retries once with `\X` (where X is not a valid JSON escape char) doubled to `\\X` — idempotent on already-valid input. Truly malformed JSON still raises loudly. B. Overaggressive verbose-failure override: the post-failure guard at `_process_query_impl` fired whenever `generate_image` had been *called* this turn — with no check on whether the call actually returned an error. When generate_image succeeded and a *different* tool's parse error provoked a verbose apology, the guard clobbered the model's reply with the misleading "not available" message. Track the latest capability-tool outcome and gate the override on `capability_tool_last_succeeded is False`.
Review pass on the #1023 fix surfaced one inconsistency and one missing regression guard. The new ``capability_tool_last_succeeded`` tracker matched ``tool_name.startswith(...)`` while the pre-existing ``has_tried_capability_tool`` check uses ``.lower().startswith(...)``. A model emitting ``Generate_Image`` would slip past the tracker (case mismatch) but still trigger ``has_tried_capability_tool`` (after lower) -- so on a *failed* call the gate evaluated ``True and (None is False)`` = False and the override silently failed to fire, leaking the verbose apology to the user instead of the canonical "not available" message. Both tracker call sites now apply ``.lower()`` to match. Tests: added ``test_override_still_fires_when_generate_image_failed`` as a regression guard for the legitimate failure path (without it, a future refactor could remove the gate entirely and only the post-success test would catch it), and replaced the previously-useless mixed-case success test with ``test_override_fires_after_mixed_case_capability_failure`` -- the success variant passed regardless of the bug because both branches yield "no override" when ``capability_tool_last_succeeded`` is ``None``; only the failure variant pins the ``.lower()`` semantics down. Red-green verified by stashing the ``.lower()`` fix and watching the new test fail with the exact verbose-apology leak.
|
Tested end-to-end on Windows with Fix 1 — Windows-path escape repair Repair fired on the Fix 2 — SD override gate
Setup note: Lemonade defaults to |
Three follow-up fixes after PR #1027 for klkr1's post-merge report on issue #1023. Before: when `gaia sd <prompt>` step 2 emitted a malformed envelope or a wrong path, the final answer reported only the successful image generation and stayed silent about the story-step failure -- the model lost the canonical `image_path` from step 1 and the user lost visibility into what went wrong. After: the envelope parser tolerates the same two malformed-JSON failure classes already repaired at the inner-arguments level (single-backslash Windows paths and trailing commentary), the parse-error recovery prompt names the canonical path verbatim so the model can retry without retyping, and `create_story_from_image` surfaces the most-recent PNG path in its error so the next LLM turn has a clear breadcrumb. - Envelope `_parse_llm_response` mirrors the inner-arguments recovery: retry via `_repair_invalid_json_escapes`, fall back to `JSONDecoder.raw_decode` for `Extra data`, validate `__tool_calls__` presence so a salvaged non-tool_calls prefix raises `ValueError` rather than letting `KeyError` escape the recovery branch's catch. - Parse-error recovery reads `step_results` for the most-recent successful `image_path`; if found, appends a "use VERBATIM" hint to the next user-role message AND replaces the give-up fallback with "I generated your image at <path>, but couldn't finish the follow-up step." Telemetry marker logs the path so production triage can distinguish "model received hint and mangled it" from "model never saw the hint." Legacy single-tool path now mirrors the plan-execution path's `step_results.append`, so the recovery branch works regardless of which agent flow produced the result. - SDAgent: system prompt explicitly tells the model to use the literal `$PREV.image_path` placeholder rather than retyping the path; `create_story_from_image` error scans `self.sd_output_dir` for the most-recent `.png` and appends the canonical path as a hint (defensive: never raises from inside the error path). Refs #1023
…ry (#1047) After PR #1027 landed, klkr1 confirmed the Windows-path repair runs but reported three new failures on Gemma-4-E4B during `generate_image` → `create_story_from_image`: the model hallucinated extra path segments in step 2 (causing silent \"Image not found\" errors), and in one case the outer tool-calls envelope itself failed with `Extra data: line 1 column 368` because the repair was only wired at the inner-arguments level, not at the envelope level. In all three tries the final answer silently reported only step 1 success and dropped step 2 entirely. This PR closes those three gaps: the envelope parser now retries with backslash repair then falls back to `raw_decode` for trailing-data failures; the parse-error recovery branch now reads the canonical `image_path` from `step_results` and surfaces it in both the retry prompt and the give-up fallback so the model can correct itself; and `create_story_from_image` now appends the real PNG path to its error message so the LLM has a verbatim breadcrumb on the next turn. The SD system prompt is also tightened with an explicit "use `$PREV.image_path`, do NOT type the path yourself" directive. ## Test plan - [ ] `PYTHONPATH=<worktree>/src python -m pytest tests/unit/agents/test_parse_error_recovery.py tests/unit/test_sd_agent.py -xvs` — 28 new tests pass; 2 pre-existing failures on `TestProcessQueryRecoversOnContextOverflow` present on base `main` before this PR - [ ] `PYTHONPATH=<worktree>/src python -m pytest tests/unit/agents/ tests/unit/test_sd_agent.py -x` — full agent unit suite passes - [ ] `python util/lint.py --all --fix` — no lint violations Resolves #1023
## Why this matters v0.18.0 ships agent memory v2 (hybrid-search second brain with LLM extraction and observability dashboard), ChatAgent split into three composable agents (Chat/FileIO/DocumentQA), parallel tool calls, and a Telegram adapter scaffold — plus fixes the RAG-on-PDF timeout with Gemma 4 that broke document Q&A since v0.17.6 and adds CI gates that enforce RAG quality baselines on every future PR. Full notes: `docs/releases/v0.18.0.mdx`. ## What's New - **Agent memory v2** ([amd#606](amd#606)) — Hybrid semantic + keyword search, LLM extraction, observability dashboard via SSE streaming ([amd#1032](amd#1032)). Per-user isolation enforced; extraction runs async so it doesn't add latency. - **ChatAgent split** ([amd#979](amd#979)) — `ChatAgent`, `FileIOAgent`, and `DocumentQAAgent` replace the monolithic class; each composable via `tools=`. Backward-compatible shim preserved. - **Parallel tool calls** ([amd#946](amd#946)) — Multiple `tool_calls` from a single LLM turn are executed concurrently, cutting round-trips for multi-tool workflows. - **Telegram adapter scaffold, Phase 0** ([amd#951](amd#951)) — `gaia telegram start|stop|status`, per-user session isolation, `[telegram]` extras. Phase 1 (message handling + allowed-users gate) tracked in [amd#889](amd#889). - **Connectors: per-MCP toggle + single-writer enforcement** ([amd#1018](amd#1018), [amd#998](amd#998)) — Disable individual MCP servers without removing them; concurrent writes serialised with actionable errors on contention. - **File navigation, web browsing, and write security** ([amd#495](amd#495)) — `FileSearchToolsMixin`, web browsing tool, and scratchpad mixin in `KNOWN_TOOLS`; write tools check `allowed_paths` before dispatch. - **Email UI and policy alerts** ([amd#995](amd#995), [amd#1039](amd#1039), [amd#952](amd#952)) — Pre-scan triage card, in-chat Connect, policy alert cards, and durable receipts for confirmation-gated actions. ## Bug Fixes - **RAG-on-PDF timeouts on Gemma 4** ([amd#1034](amd#1034), closes [amd#1030](amd#1030)) — Prompt-size budget check added at composition time; CI gates enforce it on every PR ([amd#1040](amd#1040)). - **Envelope-level parse failure crashed SD recovery** ([amd#1047](amd#1047), closes [amd#1023](amd#1023)) — Falls through to a clean recovery path with step-1 context preserved. - **Windows-path tool args corrupted** ([amd#1027](amd#1027)) — Backslash normalisation now happens after argument parsing. - **Blender `send_command` hung** ([amd#1026](amd#1026), closes [amd#1022](amd#1022)) — Read timeout applied to persistent-connection servers. - **`gaia chat init` in post-install banner** ([amd#1029](amd#1029), closes [amd#1024](amd#1024)) — Replaced with the correct `gaia init`. - **Keyring treated as required** ([amd#1028](amd#1028)) — Import guarded; optional on systems without `keyring`. - **electron-builder URLs stale** ([amd#953](amd#953)) — Three doc/installer files updated to current download paths. ## Tooling & Docs - **RAG eval CI gates** ([amd#1040](amd#1040), closes [amd#1033](amd#1033)) — RAG quality baselines + prompt-size budget enforced on every PR. - **Fork-PR authors now receive Claude review** ([amd#932](amd#932)) — `allowed_non_write_users: "*"` with prompt-injection mitigations documented. - **Eval runs mandated before merging** ([amd#1036](amd#1036)) — `CLAUDE.md` requires `gaia eval agent` for LLM-affecting changes. - **GAIA website** ([amd#369](amd#369)) — [amd-gaia.ai](https://amd-gaia.ai) live. - **Custom agent guide reorganised** ([amd#997](amd#997)), Lemonade PPA docs ([amd#801](amd#801)), broken Lemonade CLI URL fixed ([amd#996](amd#996)), WhatsApp adapter evaluation spec ([amd#950](amd#950)). ## Release checklist - [x] `util/validate_release_notes.py docs/releases/v0.18.0.mdx --tag v0.18.0` passes - [x] `src/gaia/version.py` → `0.18.0` - [x] `src/gaia/apps/webui/package.json` → `0.18.0` - [x] Navbar label in `docs/docs.json` → `v0.18.0 · Lemonade 10.2.0` - [x] All 28 commits in range (v0.17.6..HEAD) are represented in the notes - [ ] Review from @kovtcharov-amd addressed
Why this matters
gaia sd <prompt>on Windows would generate an image successfully in Step 1 but then return "Image generation is not available — start GAIA with the--sdflag to enable it" in Step 2 — telling the user to enable a feature that was already on, while silently dropping the story output.Two stacked bugs:
C:\Users\Klaus(single-escape) which JSON rejects asInvalid \escape: \U. Now repaired before re-raise; idempotent and lossless on already-valid input.generate_imagehad been called this turn, even when it succeeded. Now gated on the latest capability call actually erroring.Test plan
pytest tests/unit/agents/test_parse_error_recovery.py— 5 new tests + 6 pre-existing all passpytest tests/unit/agents/ tests/unit/test_sd_agent.py tests/unit/test_tool_call_priority.py— 190 pass, no regressionsgaia sd "a forest" --logging-level DEBUG— Step 2 parses the Windows path, story appears in the final answer (not the misleading "--sd" message)Closes #1023