docs(process): mandate eval runs + layman-led summaries#1036
Merged
Conversation
In plain English: when a contributor changes anything that affects how the LLM behaves (prompts, tools, error handling), they should run the agent eval before claiming the change is done — this is what would have caught #1030 in CI. Also adopts a "layman first, technical second" style for PR descriptions, commit messages, and the auto-review bot, because recent bot reviews trended toward dense framework jargon that non-engineers struggled to act on. CLAUDE.md: - New "Run agent evals when changing LLM-affecting code paths" section, listing the prompt / tool / error-classification surfaces that require an eval run, plus a note that the Claude Code subscription almost always already provides API access — checking $ANTHROPIC_API_KEY is preferred to skipping - "PR Descriptions" section gains a "Layman-first lead" subsection covering both PR bodies and the first body-line of commit messages, plus an anti-pattern for implementation-language openers - "Issue Response Guidelines" tightens length caps, requires a plain-English lead, replaces the Good Bug Report example with a layman-led version, and adds a Bad Excessively-Technical example .github/workflows/claude.yml: - Adds an "Output style: layman-first" section to the pr-review, pr-comment, and issue-handler prompts. Lead with 1-2 sentence plain-English findings; file references go after the finding, not before; hard length caps. Existing review structure (Summary / Issues / Strengths / Verdict) is unchanged
…n rules
Feedback after the first pass: the explicit "In plain English:" labelling and "layman-first" framing reads as formulaic preamble rather than direct prose. The rules now say "lead with the finding" / "concise and direct" without the labelling, and explicitly forbid the "In plain English:" and "TL;DR:" prefixes.
CLAUDE.md:
- Rename "Layman-first lead" → "Lead with the summary"; drop the "non-engineer / AMD product manager / community user" verbosity in favour of "summarize directly in ~3 sentences"
- Issue Response Guidelines "Tone & Style" / "Response Length Guidelines" / Bad-response caption: drop "Layman-first" / "plain-English" / "layman" wording while keeping the substantive rules (length caps, references-after-finding, no opening with a code block)
.github/workflows/claude.yml:
- Rename "Output style: layman-first" → "Output style: concise and direct" across the pr-review, pr-comment, and issue-handler prompts
- Explicit prohibition on labelled prefixes ("In plain English:", "TL;DR:") in each prompt
- Tightened wording in the bullets; length caps and structure unchanged
CLAUDE.md adds a "Code Comments — Short or Skip" section: default to no comments, keep WHY-comments to one short line, never write multi-paragraph history blocks inline. The Bad example is the exact verbose comment from the #1030 fix that motivated the rule; the Good rewrite is one line naming the invariant. PR-description rules: fold the duplicated "Lead with the summary" subsection into the existing "Why this matters" bullet (one place to read, not two saying the same thing), and add a one-line note that the same rule applies to commit messages. TL;DR-prohibition backed out — only "In plain English:" is forbidden as a preamble; `TL;DR:` stays fair game for long reviews that genuinely warrant one. Workflow prompts (pr-review, pr-comment, issue-handler): collapsed each "Output style" section to one paragraph that points at CLAUDE.md and lists only the length caps. The triplicated bullets are gone; CLAUDE.md is now the single source of truth for the style.
itomek
approved these changes
May 11, 2026
pull Bot
pushed a commit
to bhardwajRahul/gaia
that referenced
this pull request
May 15, 2026
## Why this matters v0.18.0 ships agent memory v2 (hybrid-search second brain with LLM extraction and observability dashboard), ChatAgent split into three composable agents (Chat/FileIO/DocumentQA), parallel tool calls, and a Telegram adapter scaffold — plus fixes the RAG-on-PDF timeout with Gemma 4 that broke document Q&A since v0.17.6 and adds CI gates that enforce RAG quality baselines on every future PR. Full notes: `docs/releases/v0.18.0.mdx`. ## What's New - **Agent memory v2** ([amd#606](amd#606)) — Hybrid semantic + keyword search, LLM extraction, observability dashboard via SSE streaming ([amd#1032](amd#1032)). Per-user isolation enforced; extraction runs async so it doesn't add latency. - **ChatAgent split** ([amd#979](amd#979)) — `ChatAgent`, `FileIOAgent`, and `DocumentQAAgent` replace the monolithic class; each composable via `tools=`. Backward-compatible shim preserved. - **Parallel tool calls** ([amd#946](amd#946)) — Multiple `tool_calls` from a single LLM turn are executed concurrently, cutting round-trips for multi-tool workflows. - **Telegram adapter scaffold, Phase 0** ([amd#951](amd#951)) — `gaia telegram start|stop|status`, per-user session isolation, `[telegram]` extras. Phase 1 (message handling + allowed-users gate) tracked in [amd#889](amd#889). - **Connectors: per-MCP toggle + single-writer enforcement** ([amd#1018](amd#1018), [amd#998](amd#998)) — Disable individual MCP servers without removing them; concurrent writes serialised with actionable errors on contention. - **File navigation, web browsing, and write security** ([amd#495](amd#495)) — `FileSearchToolsMixin`, web browsing tool, and scratchpad mixin in `KNOWN_TOOLS`; write tools check `allowed_paths` before dispatch. - **Email UI and policy alerts** ([amd#995](amd#995), [amd#1039](amd#1039), [amd#952](amd#952)) — Pre-scan triage card, in-chat Connect, policy alert cards, and durable receipts for confirmation-gated actions. ## Bug Fixes - **RAG-on-PDF timeouts on Gemma 4** ([amd#1034](amd#1034), closes [amd#1030](amd#1030)) — Prompt-size budget check added at composition time; CI gates enforce it on every PR ([amd#1040](amd#1040)). - **Envelope-level parse failure crashed SD recovery** ([amd#1047](amd#1047), closes [amd#1023](amd#1023)) — Falls through to a clean recovery path with step-1 context preserved. - **Windows-path tool args corrupted** ([amd#1027](amd#1027)) — Backslash normalisation now happens after argument parsing. - **Blender `send_command` hung** ([amd#1026](amd#1026), closes [amd#1022](amd#1022)) — Read timeout applied to persistent-connection servers. - **`gaia chat init` in post-install banner** ([amd#1029](amd#1029), closes [amd#1024](amd#1024)) — Replaced with the correct `gaia init`. - **Keyring treated as required** ([amd#1028](amd#1028)) — Import guarded; optional on systems without `keyring`. - **electron-builder URLs stale** ([amd#953](amd#953)) — Three doc/installer files updated to current download paths. ## Tooling & Docs - **RAG eval CI gates** ([amd#1040](amd#1040), closes [amd#1033](amd#1033)) — RAG quality baselines + prompt-size budget enforced on every PR. - **Fork-PR authors now receive Claude review** ([amd#932](amd#932)) — `allowed_non_write_users: "*"` with prompt-injection mitigations documented. - **Eval runs mandated before merging** ([amd#1036](amd#1036)) — `CLAUDE.md` requires `gaia eval agent` for LLM-affecting changes. - **GAIA website** ([amd#369](amd#369)) — [amd-gaia.ai](https://amd-gaia.ai) live. - **Custom agent guide reorganised** ([amd#997](amd#997)), Lemonade PPA docs ([amd#801](amd#801)), broken Lemonade CLI URL fixed ([amd#996](amd#996)), WhatsApp adapter evaluation spec ([amd#950](amd#950)). ## Release checklist - [x] `util/validate_release_notes.py docs/releases/v0.18.0.mdx --tag v0.18.0` passes - [x] `src/gaia/version.py` → `0.18.0` - [x] `src/gaia/apps/webui/package.json` → `0.18.0` - [x] Navbar label in `docs/docs.json` → `v0.18.0 · Lemonade 10.2.0` - [x] All 28 commits in range (v0.17.6..HEAD) are represented in the notes - [ ] Review from @kovtcharov-amd addressed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this matters
Three process rules from the post-mortem on #1030 (the Gemma 4 RAG-PDF timeout fixed in #1034):
No behaviour change, no code change. CLAUDE.md gets the three rules; the bot prompts in
.github/workflows/claude.ymlnow defer to CLAUDE.md for style and only carry the length caps inline.Test plan
python -c "import yaml; yaml.safe_load(open('.github/workflows/claude.yml'))"— YAML still parsesMerge order
CLAUDE.md's new "eval triggers" bullet references
_extract_lemonade_user_message(introduced by #1034). #1036 should land after #1034 so that symbol resolves on main.Related