Thanks to visit codestin.com
Credit goes to github.com

Skip to content

docs(process): mandate eval runs + layman-led summaries#1036

Merged
kovtcharov-amd merged 3 commits into
mainfrom
chore/meta-claudemd-reviewer-prompts
May 11, 2026
Merged

docs(process): mandate eval runs + layman-led summaries#1036
kovtcharov-amd merged 3 commits into
mainfrom
chore/meta-claudemd-reviewer-prompts

Conversation

@kovtcharov

@kovtcharov kovtcharov commented May 11, 2026

Copy link
Copy Markdown
Collaborator

Why this matters

Three process rules from the post-mortem on #1030 (the Gemma 4 RAG-PDF timeout fixed in #1034):

  1. Run the agent eval when you change anything that affects how the LLM behaves. Don't skip it — the Claude Code subscription typically already provides API access. [Bug]: 1.4MB file indexed no response to queries in the UI #1030 is the canonical example of a bug the eval would have caught in CI.
  2. Lead PR descriptions, commit messages, and bot reviews with the change itself, no labelled preamble. Recent auto-review comments trended toward dense framework jargon non-engineers struggled to act on; the rule now demands the finding up front, then technical detail, with hard length caps.
  3. Keep code comments to one short line or skip them. Multi-paragraph "history of how we got here" blocks belong in the PR / commit message, not inline. The new section quotes the verbose comment from the [Bug]: 1.4MB file indexed no response to queries in the UI #1030 fix as the canonical Bad example.

No behaviour change, no code change. CLAUDE.md gets the three rules; the bot prompts in .github/workflows/claude.yml now defer to CLAUDE.md for style and only carry the length caps inline.

Test plan

  • python -c "import yaml; yaml.safe_load(open('.github/workflows/claude.yml'))" — YAML still parses
  • Read CLAUDE.md end-to-end and confirm the new rules read coherently with the existing rules; no contradictions
  • Confirm the bot prompts still describe the same review structure (Summary → Issues → Strengths → Verdict) — the style rule is additive

Merge order

CLAUDE.md's new "eval triggers" bullet references _extract_lemonade_user_message (introduced by #1034). #1036 should land after #1034 so that symbol resolves on main.

Related

In plain English: when a contributor changes anything that affects how the LLM behaves (prompts, tools, error handling), they should run the agent eval before claiming the change is done — this is what would have caught #1030 in CI. Also adopts a "layman first, technical second" style for PR descriptions, commit messages, and the auto-review bot, because recent bot reviews trended toward dense framework jargon that non-engineers struggled to act on.

CLAUDE.md:
- New "Run agent evals when changing LLM-affecting code paths" section, listing the prompt / tool / error-classification surfaces that require an eval run, plus a note that the Claude Code subscription almost always already provides API access — checking $ANTHROPIC_API_KEY is preferred to skipping
- "PR Descriptions" section gains a "Layman-first lead" subsection covering both PR bodies and the first body-line of commit messages, plus an anti-pattern for implementation-language openers
- "Issue Response Guidelines" tightens length caps, requires a plain-English lead, replaces the Good Bug Report example with a layman-led version, and adds a Bad Excessively-Technical example

.github/workflows/claude.yml:
- Adds an "Output style: layman-first" section to the pr-review, pr-comment, and issue-handler prompts. Lead with 1-2 sentence plain-English findings; file references go after the finding, not before; hard length caps. Existing review structure (Summary / Issues / Strengths / Verdict) is unchanged
…n rules

Feedback after the first pass: the explicit "In plain English:" labelling and "layman-first" framing reads as formulaic preamble rather than direct prose. The rules now say "lead with the finding" / "concise and direct" without the labelling, and explicitly forbid the "In plain English:" and "TL;DR:" prefixes.

CLAUDE.md:
- Rename "Layman-first lead" → "Lead with the summary"; drop the "non-engineer / AMD product manager / community user" verbosity in favour of "summarize directly in ~3 sentences"
- Issue Response Guidelines "Tone & Style" / "Response Length Guidelines" / Bad-response caption: drop "Layman-first" / "plain-English" / "layman" wording while keeping the substantive rules (length caps, references-after-finding, no opening with a code block)

.github/workflows/claude.yml:
- Rename "Output style: layman-first" → "Output style: concise and direct" across the pr-review, pr-comment, and issue-handler prompts
- Explicit prohibition on labelled prefixes ("In plain English:", "TL;DR:") in each prompt
- Tightened wording in the bullets; length caps and structure unchanged
CLAUDE.md adds a "Code Comments — Short or Skip" section: default to no comments, keep WHY-comments to one short line, never write multi-paragraph history blocks inline. The Bad example is the exact verbose comment from the #1030 fix that motivated the rule; the Good rewrite is one line naming the invariant.

PR-description rules: fold the duplicated "Lead with the summary" subsection into the existing "Why this matters" bullet (one place to read, not two saying the same thing), and add a one-line note that the same rule applies to commit messages.

TL;DR-prohibition backed out — only "In plain English:" is forbidden as a preamble; `TL;DR:` stays fair game for long reviews that genuinely warrant one.

Workflow prompts (pr-review, pr-comment, issue-handler): collapsed each "Output style" section to one paragraph that points at CLAUDE.md and lists only the length caps. The triplicated bullets are gone; CLAUDE.md is now the single source of truth for the style.
@kovtcharov-amd kovtcharov-amd added this pull request to the merge queue May 11, 2026
Merged via the queue into main with commit c755cba May 11, 2026
19 checks passed
@kovtcharov-amd kovtcharov-amd deleted the chore/meta-claudemd-reviewer-prompts branch May 11, 2026 20:17
@itomek itomek mentioned this pull request May 14, 2026
6 tasks
pull Bot pushed a commit to bhardwajRahul/gaia that referenced this pull request May 15, 2026
## Why this matters

v0.18.0 ships agent memory v2 (hybrid-search second brain with LLM
extraction and observability dashboard), ChatAgent split into three
composable agents (Chat/FileIO/DocumentQA), parallel tool calls, and a
Telegram adapter scaffold — plus fixes the RAG-on-PDF timeout with Gemma
4 that broke document Q&A since v0.17.6 and adds CI gates that enforce
RAG quality baselines on every future PR.

Full notes: `docs/releases/v0.18.0.mdx`.

## What's New

- **Agent memory v2** ([amd#606](amd#606)) —
Hybrid semantic + keyword search, LLM extraction, observability
dashboard via SSE streaming
([amd#1032](amd#1032)). Per-user isolation
enforced; extraction runs async so it doesn't add latency.
- **ChatAgent split** ([amd#979](amd#979)) —
`ChatAgent`, `FileIOAgent`, and `DocumentQAAgent` replace the monolithic
class; each composable via `tools=`. Backward-compatible shim preserved.
- **Parallel tool calls** ([amd#946](amd#946))
— Multiple `tool_calls` from a single LLM turn are executed
concurrently, cutting round-trips for multi-tool workflows.
- **Telegram adapter scaffold, Phase 0**
([amd#951](amd#951)) — `gaia telegram
start|stop|status`, per-user session isolation, `[telegram]` extras.
Phase 1 (message handling + allowed-users gate) tracked in
[amd#889](amd#889).
- **Connectors: per-MCP toggle + single-writer enforcement**
([amd#1018](amd#1018),
[amd#998](amd#998)) — Disable individual MCP
servers without removing them; concurrent writes serialised with
actionable errors on contention.
- **File navigation, web browsing, and write security**
([amd#495](amd#495)) — `FileSearchToolsMixin`,
web browsing tool, and scratchpad mixin in `KNOWN_TOOLS`; write tools
check `allowed_paths` before dispatch.
- **Email UI and policy alerts**
([amd#995](amd#995),
[amd#1039](amd#1039),
[amd#952](amd#952)) — Pre-scan triage card,
in-chat Connect, policy alert cards, and durable receipts for
confirmation-gated actions.

## Bug Fixes

- **RAG-on-PDF timeouts on Gemma 4**
([amd#1034](amd#1034), closes
[amd#1030](amd#1030)) — Prompt-size budget
check added at composition time; CI gates enforce it on every PR
([amd#1040](amd#1040)).
- **Envelope-level parse failure crashed SD recovery**
([amd#1047](amd#1047), closes
[amd#1023](amd#1023)) — Falls through to a
clean recovery path with step-1 context preserved.
- **Windows-path tool args corrupted**
([amd#1027](amd#1027)) — Backslash
normalisation now happens after argument parsing.
- **Blender `send_command` hung**
([amd#1026](amd#1026), closes
[amd#1022](amd#1022)) — Read timeout applied
to persistent-connection servers.
- **`gaia chat init` in post-install banner**
([amd#1029](amd#1029), closes
[amd#1024](amd#1024)) — Replaced with the
correct `gaia init`.
- **Keyring treated as required**
([amd#1028](amd#1028)) — Import guarded;
optional on systems without `keyring`.
- **electron-builder URLs stale**
([amd#953](amd#953)) — Three doc/installer
files updated to current download paths.

## Tooling & Docs

- **RAG eval CI gates** ([amd#1040](amd#1040),
closes [amd#1033](amd#1033)) — RAG quality
baselines + prompt-size budget enforced on every PR.
- **Fork-PR authors now receive Claude review**
([amd#932](amd#932)) —
`allowed_non_write_users: "*"` with prompt-injection mitigations
documented.
- **Eval runs mandated before merging**
([amd#1036](amd#1036)) — `CLAUDE.md` requires
`gaia eval agent` for LLM-affecting changes.
- **GAIA website** ([amd#369](amd#369)) —
[amd-gaia.ai](https://amd-gaia.ai) live.
- **Custom agent guide reorganised**
([amd#997](amd#997)), Lemonade PPA docs
([amd#801](amd#801)), broken Lemonade CLI URL
fixed ([amd#996](amd#996)), WhatsApp adapter
evaluation spec ([amd#950](amd#950)).

## Release checklist

- [x] `util/validate_release_notes.py docs/releases/v0.18.0.mdx --tag
v0.18.0` passes
- [x] `src/gaia/version.py` → `0.18.0`
- [x] `src/gaia/apps/webui/package.json` → `0.18.0`
- [x] Navbar label in `docs/docs.json` → `v0.18.0 · Lemonade 10.2.0`
- [x] All 28 commits in range (v0.17.6..HEAD) are represented in the
notes
- [ ] Review from @kovtcharov-amd addressed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

devops DevOps/infrastructure changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants