@claude review this PR.

TL;DR: the direction is right (run rag_quality --compare in CI, surface time regressions, add an E2E), but several pieces don't match what the PR description claims and the workflow won't run for fork PRs as written. Recommend changes before merge.

🔴 Description ↔ implementation mismatches

The PR body claims things that aren't in the diff:

• "real-world PDF corpus entry (~1.5 MB, text-heavy, mixed sections)" — eval/corpus/documents/safety_handbook_cc0.txt is 617 bytes / 13 lines and not a PDF. This cannot exercise the [Bug]: 1.4MB file indexed no response to queries in the UI #1030 regression class (52K-char prompt, ~10-min hang on a real PDF). Either land a real fixture or change the prose to match.
• "drives gaia chat --index <small.pdf> --query "..." against a real Lemonade" — tests/integration/test_chat_rag_pdf_e2e.py actually runs gaia eval agent --scenario safety_handbook_water --compare .... Different code path.
• "prompt-size budget tests" is in the title and body, but no test in this diff measures prompt size. The runner-side budget assertion tests/unit/test_chat_system_prompt_budget.py already exists on main; if you're relying on it, say so — otherwise add one.

🔴 Workflow won't run on fork PRs

.github/workflows/test_eval_rag.yml is on: pull_request: and reads secrets.ANTHROPIC_API_KEY. Fork PRs don't get repo secrets, so the judge step will fail with a confusing auth error on every external contribution. Options:

1. Skip the eval cleanly when the secret is empty (preferred — fail loudly with a [skip] no ANTHROPIC_API_KEY available for fork PR message, exit 0).
2. Move the trigger to workflow_dispatch + merge-queue, with a maintainer-gated path for forks.

Also: the runner label [self-hosted, lemonade-eval] — is that label already wired up, or does this PR also need an infra change? Without that label registered, the job sits in queued forever.

🟡 Integration test skip condition is wrong

tests/integration/test_chat_rag_pdf_e2e.py:10-14 skips when shutil.which("claude") is None. But gaia eval agent uses the Anthropic Python SDK + ANTHROPIC_API_KEY for the judge — not the claude binary. So:

• If the runner has claude CLI installed but no API key, the test runs and fails opaquely instead of skipping.
• If the runner has the API key but no claude binary, the test wrongly skips.

Skip on os.getenv("ANTHROPIC_API_KEY") and a Lemonade reachability probe (e.g. HTTP GET to the manager endpoint) instead — the PR description says "skips loudly when runner is unavailable" but right now it only skips on a binary that isn't actually required.

🟡 90s timeout will flake and the assertion order hides real failures

test_chat_rag_pdf_e2e.py:30-37: assert elapsed < 90 runs before assert returncode == 0, so a successful run that takes 91s reports "took too long" instead of "passed slow." Flip the order, and consider raising the cap — a real Gemma-4 RAG turn + Anthropic judge will routinely exceed 90s on a cold-loaded model. The workflow itself allots 90 minutes; 90 seconds for one scenario looks copy-pasted from the wrong scale.

🟡 Silent fallback in compare_scorecards

src/gaia/eval/runner.py:1394-1398:

try:
    if c_elapsed > (b_elapsed * 2):
        time_regress = True
except Exception:
    time_regress = False

This is a bare except Exception: pass in spirit — violates the "fail loudly" rule in CLAUDE.md. Both b_elapsed and c_elapsed are already guarded by or 0 and the isinstance check upstream; the try/except is dead defensive code. Drop it.

Also: surfacing time regressions as score_regressed (line ~1417) conflates two concepts in the report — a fast quality drop and a slow-but-correct run look identical to a reader. Add a time_regressed bucket to the summary instead and have the CLI gate on both.

🟡 CLI exit-code change is a breaking change — call it out

src/gaia/cli.py:3267-3296 flips gaia eval agent --compare from "always returns 0" to sys.exit(2) on regressions. That's the point of the gate, but anyone scripting around --compare today will break. Mention it in the PR description (and ideally in a CHANGELOG/release note) so it's not a surprise. The trailing return on line ~3296 is also now unreachable after the new sys.exit(0).

🟢 Minor

• eval/scenarios/safety_handbook_water.yaml has no trailing newline.
• Workflow step comment says "This runner must have Lemonade + 'claude' CLI installed" — claude CLI isn't a requirement of gaia eval agent; clean up to "Lemonade Server + Gemma model + ANTHROPIC_API_KEY env" to avoid future confusion.
• The base of the PR (fix/ci-rag-eval-gaps-1033) is behind main by 74f637a4 (memory v2). gh pr view still shows mergeable, but worth a rebase for a clean review.

Suggested path forward

1. Fix the description ↔ code mismatches (or update the description to match the actual diff).
2. Decide how this should behave on fork PRs and implement that explicitly.
3. Fix the integration test skip predicates + assertion order.
4. Drop the silent try/except in runner.py.
5. Add a one-liner to the PR description noting the exit-code change.

Happy to re-review once those are addressed. cc @kovtcharov-amd for the fork-PR secret strategy decision (architectural).

@claude you can review the PR again

Re-review: real progress — most of the previous review is addressed. Three items from the prior pass are still open, and the new commits introduce one fresh thing worth a look. Recommend changes before merge.

✅ Resolved since last pass

• Silent try/except: pass in compare_scorecards is gone; b_elapsed / c_elapsed are now guarded by isinstance checks. (src/gaia/eval/runner.py:1389-1399)
• time_regressed now lives in its own bucket (separate from score_regressed) and is surfaced both in the report and the return dict. (src/gaia/eval/runner.py:1346, :1404-1406, :1470-1478, :1548)
• Integration-test skip predicate fixed: uses ANTHROPIC_API_KEY + backend health probe instead of the claude binary. (tests/integration/test_chat_rag_pdf_e2e.py:10-20)
• Assertion order flipped (return-code before elapsed) and the cap raised from 90s → 600s. (tests/integration/test_chat_rag_pdf_e2e.py:48-55)
• CLI exit codes wired up: sys.exit(2) on any regression bucket, sys.exit(1) on misuse, sys.exit(0) on success. (src/gaia/cli.py:3270-3303)
• Workflow comment updated to reference Lemonade + Gemma + ANTHROPIC_API_KEY (no more spurious claude CLI mention). (.github/workflows/test_eval_rag.yml:37-38)

🔴 Still open

1. Fork-PR secret handling — unchanged. .github/workflows/test_eval_rag.yml:20-21 still sets ANTHROPIC_API_KEY from secrets.ANTHROPIC_API_KEY and unconditionally runs the eval. Fork PRs get empty secrets, so the Anthropic judge will fail with an opaque 401 instead of either (a) skipping cleanly with a [skip] no ANTHROPIC_API_KEY available for fork PR message and exit 0, or (b) being gated behind workflow_dispatch / merge-queue. Pick one and implement it — the current behavior fails the rule that errors must be actionable. Self-hosted runner label [self-hosted, lemonade-eval] also still warrants confirmation that the label is registered, otherwise the job sits queued forever.

2. Description ↔ implementation mismatch — partly unresolved. PR body still says the integration test "drives gaia chat --index <small.pdf> --query \"...\" against a real Lemonade." tests/integration/test_chat_rag_pdf_e2e.py:34-42 runs gaia eval agent --scenario safety_handbook_water --compare ... — a different CLI surface. Either update the description, or add a second test that actually exercises gaia chat --index <pdf> --query (which is what the #1030 user hit). The latter is materially more valuable: gaia eval agent and gaia chat --index share the RAG pipeline but assemble prompts differently, and #1030 was specifically a gaia chat failure mode.

3. "Prompt-size budget tests" is in the title and body, but no such test is in the diff. grep -rn "prompt.*budget\|prompt_size" tests/ src/ returns nothing in this branch. If you're relying on a future test or one elsewhere in the tree, drop the phrase from the title — right now the title oversells the diff. If you intend to add one, a 20-line unit test that asserts len(ChatAgent()._get_system_prompt()) stays under e.g. 30k chars would directly guard the #1030 class of regression and would make this PR honest to its title.

🟡 New since last pass

PDF fixture size still off. eval/corpus/documents/safety_handbook_large.pdf is 282 KB (stat -c %s confirms 282087 bytes), not the "~1.5 MB" the description claims. The generator script eval/corpus/scripts/generate_large_pdf.py:12 defaults to target_bytes=1_500_000 but the committed artifact wasn't regenerated at that size — looks like it was capped by the section < 300 loop bound. Either regenerate at the advertised size or update the description. (And while you're there: the script accepts --size but never actually consults it in make_pdf — target_bytes is unused.)

Generated PDF content is generic. The synthetic paragraph (generate_large_pdf.py:6-10) doesn't mention water specifically — it lists "labeling, sanitary guidance, and emergency isolation steps." The scenario asks "what does the document say about water?" The expected phrases (label, sanitary, isolate) will substring-match, but a casual reader of the scenario would expect the doc to discuss water. Either rename the scenario to match the doc ("what does the document say about safety procedures?") or seed water-specific paragraphs into the generator so the test name and document content agree.

🟢 Minor

• eval/scenarios/safety_handbook_water.yaml does end in a newline now — good catch.
• compare_scorecards's result.get(...) if isinstance(result, dict) else [] defence in cli.py:3281-3293 is fine, but compare_scorecards always returns a dict (see runner.py:1544-1553). The isinstance guard is dead code — drop it for clarity.
• The trailing return after sys.exit(0) (cli.py:3303+) is now unreachable; clean it up.

Suggested path forward

1. Decide and implement the fork-PR strategy (skip-cleanly is simplest and matches "fail loudly" — print a one-liner and exit 0).
2. Add the prompt-size budget unit test the title promises, or retitle the PR.
3. Either fix the description's gaia chat --index claim, or add a real gaia chat --index <pdf> --query test alongside the eval-based one.
4. Regenerate the PDF at the size you advertise (or update the prose to "~280 KB").
5. Drop the dead isinstance guard and unreachable return in cli.py.

The core machinery (eval-comparison exit codes, time-regression bucket, integration test skip logic) is in good shape. Once items 1–3 land, this is mergeable. cc @kovtcharov-amd for the fork-PR secret-strategy call (architectural).

@claude check out the new changes

Re-review: good progress on items 1–3 from last pass, but the PDF regeneration didn't actually take, and the new fork-PR gate has a comparison bug that means it almost certainly doesn't behave the way you intended. Recommend one more pass before merge.

✅ Resolved since last pass

• Prompt-size budget unit test added: tests/unit/test_prompt_size_budget.py:1-13 asserts ChatAgent()._get_system_prompt() stays under 30k chars. Title is now honest to the diff. 👍
• Second integration test added that actually drives gaia chat --index <pdf> --query: tests/integration/test_chat_rag_pdf_chat_e2e.py:35-55. This is the real [Bug]: 1.4MB file indexed no response to queries in the UI #1030 reproduction shape — well done.
• Fork-PR skip job added (.github/workflows/test_eval_rag.yml:17-26) with a [SKIP] ANTHROPIC_API_KEY not available... notice.
• Generator now mentions water explicitly (eval/corpus/scripts/generate_large_pdf.py:6-11) so the scenario name and document content agree.
• Dead isinstance guard and unreachable return in cli.py are gone (src/gaia/cli.py:3402-3442).

🔴 Still open

1. PDF fixture is still 282 KB, not 1.5 MB. Commit d8d4b405 claims "regenerate 1.5MB PDF" but stat -c %s eval/corpus/documents/safety_handbook_large.pdf returns 282087. Root cause is a real bug in eval/corpus/scripts/generate_large_pdf.py:43-58:

c.save()                                # writes complete PDF
...
if size >= target_bytes:
    os.replace(temp_path, path)
    break
c = canvas.Canvas(temp_path, pagesize=letter)   # truncates temp_path — starts over
section += 1

reportlab.pdfgen.canvas.Canvas(path) opens the file for writing (truncating). So every loop iteration writes a fresh, one-section PDF on top of the previous file — the file never grows beyond one iteration's content. The committed 282 KB artifact is exactly one section's worth. Fix: accumulate pages on a single long-lived canvas and only call c.save() once, or use PdfMerger/PdfWriter to append. Then regenerate. Also, args.size flows into make_pdf(target_bytes=...) now (good), but as currently coded target_bytes is effectively ignored because of the bug above.

2. Fork-PR gate compares an object to a string — likely always-on or always-off. .github/workflows/test_eval_rag.yml:20,30:

if: ${{ github.event.pull_request.head.repo.owner != github.repository_owner }}

github.event.pull_request.head.repo.owner is an object ({login, id, ...}); github.repository_owner is a string ("amd"). GitHub Actions expressions stringify objects in coercion, so this comparison is unreliable across event payloads — and claude.yml:343 already uses the correct pattern:

github.event.pull_request.head.repo.full_name == github.repository

Use that pattern (or head.repo.fork == true) so the gate behaves predictably. Right now I can't tell whether internal PRs would even reach run-eval.

3. Workflow pip install -e .[dev] runs on the self-hosted runner without a venv. .github/workflows/test_eval_rag.yml:43-46 installs straight into the runner's system Python. On a long-lived self-hosted runner this leaks state across runs and is a known source of flake. Wrap in python -m venv .venv-eval && source .venv-eval/bin/activate && pip install ..., or use uv venv per CLAUDE.md.

🟡 Minor

• tests/integration/test_chat_rag_pdf_chat_e2e.py:55 asserts "water" in proc.stdout.lower() or proc.stderr.lower() — but the document is 282 KB of water-heavy content, so a passing answer should definitely mention water. That assertion is fine; just noting it's load-bearing on the document actually being water-heavy, which is currently true but trivially defeated if anyone tweaks the generator.
• Workflow concurrency: lemonade-eval is correct per CLAUDE.md's "serial-only eval" rule. 👍
• The gaia chat --index E2E uses a 900s cap (test_chat_rag_pdf_chat_e2e.py:53) and the eval-based one uses 600s (test_chat_rag_pdf_e2e.py:54). Both reasonable.

Suggested path forward

1. Fix generate_large_pdf.py (one canvas, append pages until target, then save() once) and regenerate the artifact at 1.5 MB. Verify with stat -c %s.
2. Switch the workflow if: to github.event.pull_request.head.repo.full_name == github.repository (mirroring claude.yml).
3. Wrap the pip install in a venv on the self-hosted runner.

Once those three land, this is mergeable. The core machinery (eval comparison exit codes, time-regression bucket, dual gaia chat + gaia eval agent integration tests, prompt-size budget unit test) is in solid shape. cc @kovtcharov-amd for the self-hosted runner label/venv question.

@claude you can look at it again

Re-review: the two big items from last pass (PDF size, fork-PR gate comparison) are correctly fixed. Two issues remain — one blocking the workflow from actually running, one stray artifact.

✅ Resolved since last pass

• PDF fixture is now 1,518,494 bytes (stat confirms ~1.45 MiB) — matches the description. eval/corpus/scripts/generate_large_pdf.py:27-46 is also cleaned up: single long-lived canvas, one save() at the end. 👍
• Fork-PR gate uses the correct pattern: github.event.pull_request.head.repo.full_name == github.repository (mirrors claude.yml). Skip job and run-eval job have mutually-exclusive if: conditions. 👍
• Self-hosted runner install now uses a dedicated .venv-eval venv. Good direction.
• Generator paragraph now explicitly discusses water (generate_large_pdf.py:6-11), so the scenario name and the document content agree. 👍

🔴 Still open

1. Venv is created but never activated in the eval step. .github/workflows/test_eval_rag.yml:43-56:

- name: Install dependencies in isolated venv
  run: |
    python -m venv .venv-eval
    source .venv-eval/bin/activate     # only lives for THIS step
    pip install -e .[dev]

- name: Run eval rag_quality compare
  run: |
    gaia eval agent --category rag_quality --compare ...   # PATH no longer has the venv

Each GHA run: block is a fresh shell — the source from step 1 doesn't persist into step 2, so gaia won't be on PATH and the workflow will fail with command not found. Two fixes (either works):

• Add the venv bin to GITHUB_PATH at the end of step 1:

  echo "$PWD/.venv-eval/bin" >> "$GITHUB_PATH"

• Or source the venv at the top of step 2:

  run: |
    source .venv-eval/bin/activate
    gaia eval agent --category rag_quality --compare ...

Without this, the gate never actually runs. Worth triggering once via workflow_dispatch after the fix to confirm the runner is wired up.

2. Stray nested PDF committed by accident. eval/corpus/scripts/eval/corpus/documents/safety_handbook_large.pdf (605 KB, 666 pages) is a duplicate that landed when the generator was run with the default --out from somewhere other than the scripts/ dir. The real fixture is at eval/corpus/documents/safety_handbook_large.pdf. Delete the nested copy — it's pure clutter and will confuse anyone grepping for the file. Optionally tighten the generator's --out default to be path-anchored, but git rm is enough for now.

🟢 Minor

• tests/integration/test_chat_rag_pdf_chat_e2e.py:55 — "water" in proc.stdout.lower() or "water" in proc.stderr.lower() is now sensible because the generated doc is water-heavy. 👍
• tests/unit/test_prompt_size_budget.py:1-13 — small, direct, exactly the guard the title promised. 👍
• src/gaia/cli.py:3401-3442 — dead isinstance guard and unreachable return are gone. Clean.

Suggested path forward

1. Wire the venv into the eval step (either GITHUB_PATH or re-source).
2. git rm eval/corpus/scripts/eval/corpus/documents/safety_handbook_large.pdf (plus the now-empty intermediate dirs).
3. Trigger the workflow via workflow_dispatch once to confirm the runner picks the job up and the venv path is correct.

Once those land, this is mergeable. cc @kovtcharov-amd for sign-off on the self-hosted runner label.

@claude recheck it I just made some changes

@kovtcharov and @kovtcharov-amd you can check out this PR when you're free

Re-review: both blocking items from the last pass are correctly fixed. This is mergeable from my side.

✅ Resolved since last pass

• Venv wired into eval step. .github/workflows/test_eval_rag.yml:50 now appends $PWD/.venv-eval/bin to GITHUB_PATH, so gaia is on PATH in the subsequent Run eval rag_quality compare step. This was the one thing that would have kept the workflow from actually running.
• Stray nested PDF removed. Commit a930387f deletes eval/corpus/scripts/eval/corpus/documents/safety_handbook_large.pdf (the 12,769-line duplicate). The real fixture at eval/corpus/documents/safety_handbook_large.pdf is 1,518,494 bytes (~1.45 MiB), matching the description.

🟢 Optional follow-up (non-blocking)

• Last review suggested validating the runner pickup via workflow_dispatch once. The workflow is still on: pull_request: only, so there's no manual-trigger path. Adding workflow_dispatch: alongside pull_request: is a one-line change that would let a maintainer fire the job on-demand and confirm the self-hosted runner + venv path are wired up end-to-end — useful since this PR itself doesn't touch a path that triggers the workflow on its own merge. Not a blocker.

Final state of the gate

The core machinery is solid: dual integration tests (gaia chat --index + gaia eval agent --scenario), prompt-size budget unit test (tests/unit/test_prompt_size_budget.py), score/time regression buckets surfaced separately, CLI exits 2 on regression, fork-PR gate uses head.repo.full_name == github.repository, and concurrency: lemonade-eval keeps eval runs serial per CLAUDE.md.

cc @kovtcharov-amd for sign-off on the self-hosted runner label and the optional workflow_dispatch addition.