feat(email): daily-driver UI pass — pre-scan card, in-chat Connect, session prefs#995
Merged
Conversation
itomek
added a commit
that referenced
this pull request
May 8, 2026
… mount) Live-testing PR #995 in the Agent UI on Gemma-4-E4B exposed exactly the failure mode three of the six adversarial-reflection agents had predicted: the LLM paraphrased the ``pre_scan_inbox`` envelope into prose ("Here is a summary of your inbox status…") instead of echoing it verbatim inside the ``email_pre_scan`` fence the system prompt told it to emit. The tool fired correctly with real data; the structured envelope was sitting in the agent-step debug panel; the assistant message was prose; the card never mounted. Conditioning a card-rendering contract on LLM compliance is fragile by construction — the data is server-side already, so the LLM doesn't need to relay it. This patch moves the fence emission to the SSE handler: - ``SSEOutputHandler._capture_render_payload`` watches ``pretty_print_json`` for ``pre_scan_inbox`` results and buffers the inner ``data`` envelope on the handler. Handles both dict and JSON-string payloads — the ``@tool`` dispatch hands strings to ``pretty_print_json``, which the initial implementation missed (caught by retest, fixed before commit). - ``print_final_answer`` drains the buffer and prepends each pending payload as a ``\`\`\`<lang>\\n<json>\\n\`\`\``` block before the LLM's prose. The frontend's existing ``pre`` markdown override (PR #995) detects the language tag and mounts ``EmailPreScanCard``. Mount is deterministic — no longer depends on LLM token-level compliance. - The email agent's system prompt drops the brittle "you MUST emit a fenced ``email_pre_scan`` block" instruction. Replaced with a single sentence telling the LLM the user has already been shown the card and to write a brief framing line. Smaller prompt, harder to drift. This is a deliberate hack — the right fix is a tool-use-tuned model handling structured emission while a chat-tuned model handles the prose summary, via Lemonade's multi-model loading. Tracked in a follow-up issue (linked in the buffer's docstring); removing this hack is part of that issue's acceptance criteria. The hack is keyed by a class-level ``_RENDER_TOOL_TO_LANG`` map so future structured-render tools (calendar cards, doc-summary cards) plug into the same pathway by adding one line. Verified live in the Agent UI on Gemma-4-E4B with my real Gmail: card mounted on first try with 5 actionable items (Kalin, Chaked, Maximillian, ObjectWin Accounts), 1 suggested archive (Google Gemini), 17 informational counted in the header, plus the LLM's one-line prose framing below. Tests: 328 unit tests pass total. Added 9 cases in ``test_sse_handler.py::TestStructuredRenderInjection`` covering envelope capture (dict + JSON-string + malformed), fence-prepend ordering, empty-prose path, non-pre-scan tools passthrough, ok=False rejection, wrong-kind rejection, buffer drained between turns. The system-prompt canary test loosened to assert ``pre_scan_inbox`` only (the ``email_pre_scan`` literal is no longer in the prompt — it lives in the SSE hook now).
This was referenced May 8, 2026
7 tasks
kovtcharov
previously approved these changes
May 11, 2026
kovtcharov-amd
previously approved these changes
May 11, 2026
…ession prefs Closes the four UI gaps that turn the shipped EmailTriageAgent (#965) into a tool a user actually opens daily. "Run a pre-scan" now returns a typed ``email_pre_scan`` envelope that the chat surface mounts as a structured triage card with three sections (urgent / actionable / suggested archives) and inline Approve / Reply / Open / Dismiss buttons — no chat-turn required to act on each item. When the agent surfaces an OAuth auth-required error, an inline ``Connect Google`` CTA renders in the same message bubble (no Settings nav). Session-scoped triage preferences (``set_priority_sender``, ``set_low_priority_sender``, ``set_category_default``, ``clear_session_preferences``) are honored by ``triage_inbox`` and ``pre_scan_inbox`` for the lifetime of the agent — wiped on restart by design, clean migration path to the persistent memory subsystem when that lands. Adversarial review caught a showstopper before merge: ``email_pre_scan`` was being stripped by ``stripBogusCodeFences`` in MessageBubble before the markdown renderer saw it, so the card would never have mounted in production. Fixed in the same pass with three other findings: id-only Approve/Reply dispatch (closes a prompt-injection path through email subject text), phishing-aware preference application (a phishing-flagged message bypasses both priority and low-priority overrides), and a single-flight per-card guard against double- click duplicate dispatches. Drafted intentionally — depends on Kalin's memory + 5-agent split PR landing on main first. The in-memory ``_session_preferences`` schema is designed to migrate to that store cleanly without touching the agent or read-tool kwargs. Tests: 132 email unit tests pass (130 baseline + 2 new — phishing-override and a system-prompt canary asserting the ``email_pre_scan`` instruction is still present). Lint clean, TypeScript clean, webui build clean. v0.17.6 installer smoke verified end-to-end on t-nx-strx-halo (Ubuntu 24.04, Strix Halo) for both AppImage and deb paths. Ref: #645
… mount) Live-testing PR #995 in the Agent UI on Gemma-4-E4B exposed exactly the failure mode three of the six adversarial-reflection agents had predicted: the LLM paraphrased the ``pre_scan_inbox`` envelope into prose ("Here is a summary of your inbox status…") instead of echoing it verbatim inside the ``email_pre_scan`` fence the system prompt told it to emit. The tool fired correctly with real data; the structured envelope was sitting in the agent-step debug panel; the assistant message was prose; the card never mounted. Conditioning a card-rendering contract on LLM compliance is fragile by construction — the data is server-side already, so the LLM doesn't need to relay it. This patch moves the fence emission to the SSE handler: - ``SSEOutputHandler._capture_render_payload`` watches ``pretty_print_json`` for ``pre_scan_inbox`` results and buffers the inner ``data`` envelope on the handler. Handles both dict and JSON-string payloads — the ``@tool`` dispatch hands strings to ``pretty_print_json``, which the initial implementation missed (caught by retest, fixed before commit). - ``print_final_answer`` drains the buffer and prepends each pending payload as a ``\`\`\`<lang>\\n<json>\\n\`\`\``` block before the LLM's prose. The frontend's existing ``pre`` markdown override (PR #995) detects the language tag and mounts ``EmailPreScanCard``. Mount is deterministic — no longer depends on LLM token-level compliance. - The email agent's system prompt drops the brittle "you MUST emit a fenced ``email_pre_scan`` block" instruction. Replaced with a single sentence telling the LLM the user has already been shown the card and to write a brief framing line. Smaller prompt, harder to drift. This is a deliberate hack — the right fix is a tool-use-tuned model handling structured emission while a chat-tuned model handles the prose summary, via Lemonade's multi-model loading. Tracked in a follow-up issue (linked in the buffer's docstring); removing this hack is part of that issue's acceptance criteria. The hack is keyed by a class-level ``_RENDER_TOOL_TO_LANG`` map so future structured-render tools (calendar cards, doc-summary cards) plug into the same pathway by adding one line. Verified live in the Agent UI on Gemma-4-E4B with my real Gmail: card mounted on first try with 5 actionable items (Kalin, Chaked, Maximillian, ObjectWin Accounts), 1 suggested archive (Google Gemini), 17 informational counted in the header, plus the LLM's one-line prose framing below. Tests: 328 unit tests pass total. Added 9 cases in ``test_sse_handler.py::TestStructuredRenderInjection`` covering envelope capture (dict + JSON-string + malformed), fence-prepend ordering, empty-prose path, non-pre-scan tools passthrough, ok=False rejection, wrong-kind rejection, buffer drained between turns. The system-prompt canary test loosened to assert ``pre_scan_inbox`` only (the ``email_pre_scan`` literal is no longer in the prompt — it lives in the SSE hook now).
When the LLM emits the pre-scan payload without the fenced-block wrapper,
the triage card never mounted — RenderedContent only triggers on fenced
blocks. This adds a string-aware brace-depth scanner that detects bare
{"kind":"email_pre_scan",...} output and wraps it in the correct fence
before any other content stripping runs.
Also removes a duplicate EmailPreScanCard import that crept in during the
rebase conflict resolution.
eabecbc to
aae8ed0
Compare
itomek-amd
approved these changes
May 11, 2026
pull Bot
pushed a commit
to bhardwajRahul/gaia
that referenced
this pull request
May 15, 2026
## Why this matters v0.18.0 ships agent memory v2 (hybrid-search second brain with LLM extraction and observability dashboard), ChatAgent split into three composable agents (Chat/FileIO/DocumentQA), parallel tool calls, and a Telegram adapter scaffold — plus fixes the RAG-on-PDF timeout with Gemma 4 that broke document Q&A since v0.17.6 and adds CI gates that enforce RAG quality baselines on every future PR. Full notes: `docs/releases/v0.18.0.mdx`. ## What's New - **Agent memory v2** ([amd#606](amd#606)) — Hybrid semantic + keyword search, LLM extraction, observability dashboard via SSE streaming ([amd#1032](amd#1032)). Per-user isolation enforced; extraction runs async so it doesn't add latency. - **ChatAgent split** ([amd#979](amd#979)) — `ChatAgent`, `FileIOAgent`, and `DocumentQAAgent` replace the monolithic class; each composable via `tools=`. Backward-compatible shim preserved. - **Parallel tool calls** ([amd#946](amd#946)) — Multiple `tool_calls` from a single LLM turn are executed concurrently, cutting round-trips for multi-tool workflows. - **Telegram adapter scaffold, Phase 0** ([amd#951](amd#951)) — `gaia telegram start|stop|status`, per-user session isolation, `[telegram]` extras. Phase 1 (message handling + allowed-users gate) tracked in [amd#889](amd#889). - **Connectors: per-MCP toggle + single-writer enforcement** ([amd#1018](amd#1018), [amd#998](amd#998)) — Disable individual MCP servers without removing them; concurrent writes serialised with actionable errors on contention. - **File navigation, web browsing, and write security** ([amd#495](amd#495)) — `FileSearchToolsMixin`, web browsing tool, and scratchpad mixin in `KNOWN_TOOLS`; write tools check `allowed_paths` before dispatch. - **Email UI and policy alerts** ([amd#995](amd#995), [amd#1039](amd#1039), [amd#952](amd#952)) — Pre-scan triage card, in-chat Connect, policy alert cards, and durable receipts for confirmation-gated actions. ## Bug Fixes - **RAG-on-PDF timeouts on Gemma 4** ([amd#1034](amd#1034), closes [amd#1030](amd#1030)) — Prompt-size budget check added at composition time; CI gates enforce it on every PR ([amd#1040](amd#1040)). - **Envelope-level parse failure crashed SD recovery** ([amd#1047](amd#1047), closes [amd#1023](amd#1023)) — Falls through to a clean recovery path with step-1 context preserved. - **Windows-path tool args corrupted** ([amd#1027](amd#1027)) — Backslash normalisation now happens after argument parsing. - **Blender `send_command` hung** ([amd#1026](amd#1026), closes [amd#1022](amd#1022)) — Read timeout applied to persistent-connection servers. - **`gaia chat init` in post-install banner** ([amd#1029](amd#1029), closes [amd#1024](amd#1024)) — Replaced with the correct `gaia init`. - **Keyring treated as required** ([amd#1028](amd#1028)) — Import guarded; optional on systems without `keyring`. - **electron-builder URLs stale** ([amd#953](amd#953)) — Three doc/installer files updated to current download paths. ## Tooling & Docs - **RAG eval CI gates** ([amd#1040](amd#1040), closes [amd#1033](amd#1033)) — RAG quality baselines + prompt-size budget enforced on every PR. - **Fork-PR authors now receive Claude review** ([amd#932](amd#932)) — `allowed_non_write_users: "*"` with prompt-injection mitigations documented. - **Eval runs mandated before merging** ([amd#1036](amd#1036)) — `CLAUDE.md` requires `gaia eval agent` for LLM-affecting changes. - **GAIA website** ([amd#369](amd#369)) — [amd-gaia.ai](https://amd-gaia.ai) live. - **Custom agent guide reorganised** ([amd#997](amd#997)), Lemonade PPA docs ([amd#801](amd#801)), broken Lemonade CLI URL fixed ([amd#996](amd#996)), WhatsApp adapter evaluation spec ([amd#950](amd#950)). ## Release checklist - [x] `util/validate_release_notes.py docs/releases/v0.18.0.mdx --tag v0.18.0` passes - [x] `src/gaia/version.py` → `0.18.0` - [x] `src/gaia/apps/webui/package.json` → `0.18.0` - [x] Navbar label in `docs/docs.json` → `v0.18.0 · Lemonade 10.2.0` - [x] All 28 commits in range (v0.17.6..HEAD) are represented in the notes - [ ] Review from @kovtcharov-amd addressed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the four UI gaps that turn the shipped EmailTriageAgent into something a user opens daily: structured pre-scan card with inline action buttons, in-chat Connect Google CTA, session-scoped triage preferences, and the adversarial-review fixes that emerged in the same pass.
Why
Before: picking Email Triage in Agent UI returns prose from "Triage my inbox" — a CLI experience pasted into chat, no scannable summary, no inline actions, OAuth requires leaving for Settings. After: a typed
email_pre_scanenvelope mounts as a triage card with three sections (urgent / actionable / suggested archives) and per-row Approve / Reply / Open / Dismiss buttons; an inlineConnect GoogleCTA renders when the agent surfaces an auth-required error; in-session sender preferences (set_priority_sender,set_low_priority_sender,set_category_default) are honored by bothtriage_inboxandpre_scan_inbox. Adversarial review caught a P0 rendering bug (stripBogusCodeFenceswas eating theemail_pre_scanfence before the markdown renderer saw it, so the card would never have mounted), plus a subject-text prompt-injection path through Approve dispatch and a phishing-override safety hole — all four fixed in this same pass.Linked issue
Refs #645
Test plan
pytest tests/unit/agents/test_email_agent*.py tests/unit/test_email_cli.py→ 132 passed (130 baseline + 2 new — phishing-override + system-prompt canary).python util/lint.py --allclean.npx tsc --noEmit+npm run buildinsrc/gaia/apps/webuiclean.t-nx-strx-halo(Ubuntu 24.04, Strix Halo): AppImage and deb paths both green end-to-end (shim creation,gaia --version,gaia connectors list, deb upgrade 0.17.1 → 0.17.6,apt remove gaiacleanup).Branch strategy
Drafted intentionally — depends on Kalin's memory + 5-agent split PR landing on main first (per 2026-05-07 scrum). The in-memory
_session_preferencesschema is designed to migrate cleanly to that store:init_session_preferences()becomes a loader, the mutator tools write through, kwarg signatures ontriage_inbox_impl/pre_scan_inbox_impldon't change.Checklist
Refs #645).python util/lint.py --all,pytest tests/unit/).