@claude review this PR

Claude finished @itomek's task —— View job

PR Review: MCP Tool Calling Reliability Test Framework

• Read PR diff and changed files
• Review CLI changes (src/gaia/cli.py)
• Review dispatch module (src/gaia/ui/dispatch.py)
• Review test scenarios (eval/scenarios/mcp_reliability/)
• Review unit tests
• Review frontend changes
• Provide comprehensive feedback

Summary

This is a well-structured PR that adds a comprehensive MCP reliability evaluation framework along with a boot-time initialization system for the Agent UI. The code is clean, well-documented, and includes thorough test coverage.

Review Feedback

🟢 Strengths

1. Clean Architecture - The DispatchQueue in src/gaia/ui/dispatch.py is well-designed with:

• Clear job lifecycle states (PENDING → RUNNING → DONE/FAILED)
• Dependency handling with timeout protection (60s)
• Automatic pruning of old jobs
• Thread-safe design using asyncio event loop

2. Comprehensive Test Coverage - 16 new tests covering:

• Job lifecycle transitions
• Dependency handling edge cases
• Concurrency verification
• CLI validation (--iterations 0, --fix --iterations N)

3. Well-Designed Eval Scenarios - The 10 MCP reliability scenarios cover a good range:

• Simple: basic tool calls, no-tool-needed restraint, introspection
• Moderate: structured params, error handling, search context
• Complex: multi-step chains, conditional selection, multi-document

4. Good UX - Frontend changes properly:

• Block input during initialization (ChatView.tsx:503-506)
• Show initialization progress with animated banner (ConnectionBanner.tsx:66-82)
• Adaptive polling (3s during init, 15s after)

🟡 Suggestions (Non-blocking)

1. Consider using asyncio.Event instead of polling for dependencies

src/gaia/ui/dispatch.py:126-138 - The current implementation polls every 0.2s:

while dep.status not in (JobStatus.DONE, JobStatus.FAILED):
    if time.monotonic() >= deadline:
        ...
    await asyncio.sleep(_DEPENDENCY_POLL_INTERVAL)

An asyncio.Event per job would be more efficient for CPU and latency, though the current approach is perfectly acceptable for boot-time use.

2. CLI table formatting edge case

src/gaia/cli.py:904 - If a scenario ID exceeds 40 characters, the table formatting may break:

print(f"  {sid:<40} {rate_str:>12} {result_str}")

Consider truncating long IDs or using dynamic width.

3. Test isolation with subprocess

tests/unit/eval/test_iterations.py:112-146 - The subprocess tests call the full CLI. While this tests the real behavior, consider mocking AgentEvalRunner for faster unit tests and reserving subprocess tests for integration.

🔴 Issues (Should Address)

1. Unused import in test file

tests/unit/eval/test_iterations.py:6:

import io

This import is not used in the test file.

2. model_load_lock visibility change should be documented

src/gaia/ui/_chat_helpers.py:62-63 - The lock was renamed from _model_load_lock to model_load_lock (public) for cross-module access. The comment explains why, but this is a public API change that could affect external code importing this module. Consider adding to the module docstring.

📋 Test Plan Verification

Based on code review, the test plan items should work:

• ✅ python -m pytest tests/unit/eval/ -xvs — Tests are well-structured
• ✅ gaia eval agent --help — --iterations flag is registered at line 2106-2112
• ✅ gaia eval agent --iterations 0 — Validation at line 3972-3974
• ✅ gaia eval agent --fix --iterations 3 — Incompatibility check at line 3976-3982

Overall Assessment

Approve ✅ - This is high-quality code with good design patterns, comprehensive tests, and proper documentation. The minor issues noted above are non-blocking. The PR successfully adds:

1. A robust MCP reliability evaluation framework with 10 scenarios across 3 complexity tiers
2. A --iterations flag for measuring consistency across multiple runs
3. A boot-time initialization system with frontend progress tracking
4. GO/NO_GO readiness signals for reliability assessment

Nice work! 🎉

Addressed the blockers + important items from the review (force-pushed to 2257e86b8):

Fixed

• 🔴 Spacer → Sptool regression in src/gaia/eval/pdf_document_generator.py (3 sites — import + 2 call sites). Was caused by an over-broad substring substitution during an earlier vendor-name scrub; reverted to Spacer.
• 🟡 Removed **_ignored_kwargs from Agent.__init__ — the python_factory already filters kwargs via _accepted_init_params; the base class no longer silently swallows unknowns. This fixes the 4 failing test_registry_python_factory.py cases.
• 🟡 _post_process_tool_result return-type annotations updated to Optional[List[Dict[str, Any]]] in both chat/agent.py and blender/agent.py to match the base-class contract.
• 🟡 uv.lock requires-python reverted from >=3.14 back to >=3.13 (matched origin/main). The bump was an accidental side-effect of running uv lock on a 3.14 env.

Not addressed in this push (deferring for your call)

• 🟡 Hardcoded EDT timezone in analyze_failures.py — agreed this should be a CLI flag (--tz-offset). Worth a separate change; this PR is already large.
• 🟢 mcp_tool_limit default raised 10 → 100 — happy to lower to ~50 if you'd prefer that as the OEM-realistic ceiling; flag it and I'll change.
• 🟢 DEFAULT_OEM_LOG_DIR Windows-only default — agreed; will move to None + only-warn-when-explicitly-set in a follow-up if needed.

Verified locally:

• pytest tests/unit/agents/ tests/unit/mcp/ → 504 passed, 2 skipped, 3 warnings
• python util/lint.py --black --isort → clean

Re-running CI now.

Five concrete fixes for real eval-discovered regressions, each paired with a regression test — this is exactly how eval-driven development should work. The bulk of the diff is deletion of session logs that should never have shipped; the actual code changes are focused and well-reasoned.

Summary

The lie-on-loop fix (_build_loop_break_summary) and the model_id-ordering fix are the headline wins. Both were caught by the new eval harness, both are deterministic regressions that unit tests missed, and both now have targeted tests that guard against reintroduction. Server-name sanitisation is idempotent, documented, and has a thorough parametric test matrix. The MCP subprocess cleanup on cache eviction (_disconnect_cached_agent) plugs a real resource leak.

The one thing to eyeball before merge: the MCP tool limit silently went from 10 → 100, which is a meaningful behaviour change for local-model users.

Issues

🟡 MCP tool-limit default raised 10 → 100 without changelog note (chat/agent.py:1539)

The original hard-coded _MCP_TOOL_LIMIT = 10 was intentional — the comment read "skip if MCP would add >10 tools (context bloat guard)". The new default is 100 (10×). For local models on NPU/iGPU with tight context budgets, registering 49+ MCP tools against what was previously a 10-tool ceiling is a silent prompt-size regression.

The change is correct for the OEM MCP use case (the --mcp-tool-limit help text says ~49 tools), but it needs a line in the PR description so future maintainers understand why the conservative guard was lifted. The previous default should also be mentioned in the commit message so bisects land here.

No code change required; this is a docs/comms gap.

🟢 sanitise_server_name warns on pure case differences (mcp_client.py:5576)

The warning predicate if n != raw fires whenever the final normalised form differs from the raw input — including for names like "Filesystem" or "MyServer" that contain no mcp token, only an uppercase letter. The warning says "Consider renaming the server in mcp_servers.json," which is misleading when the only difference is capitalisation.

The test test_sanitise_does_not_warn_on_clean_input only exercises lowercase inputs, so this slips by.

    baseline = (raw or "").strip().lower().replace("-", "_").replace(" ", "_")
    if n != baseline and raw not in _logged_sanitisations:

This compares against the case/separator-normalised form rather than the raw string, so only genuine mcp-token stripping triggers the warning.

🟢 _single_tool_done relies on getattr fallback instead of __init__ initialisation (base/agent.py:1085, 1997)

getattr(self, "_single_tool_done", False) is called in _parse_response and _build_tool_message, but the attribute is not set in Agent.__init__. It's reset at the top of _process_query_impl (line 2228) and set to True inside _post_process_tool_result. Any call to those two helpers outside the standard query lifecycle (e.g., from a subclass or a unit test that drives the parsing path directly) silently falls back to False, which could mask a misconfigured single_tool_per_turn=True agent. One line in __init__ is cheaper than the mental overhead of tracking the implicit initialization:

        self._single_tool_done: bool = False

(Place immediately after the self.model_id = model_id line at base/agent.py:~323.)

🟢 Multi-paragraph docstrings violate CLAUDE.md one-line rule

_build_loop_break_summary and sanitise_server_name both have 10+ line docstrings. CLAUDE.md: "Never write multi-paragraph docstrings or multi-line comment blocks — one short line max." The sanitise_server_name docstring is especially long (≈40 lines including the KNOWN LIMITATION block). That context belongs in the commit message, not the code.

Example fix for _build_loop_break_summary (base/agent.py:4046):

    def _build_loop_break_summary(
        self,
        tool_name: str,
        consecutive_count: int,
        step_results: list,
    ) -> str:
        """Final-answer text when the loop breaks on repeats; honest on errors."""

🟢 except Exception: pass in two new helpers (mcp/mixin.py:5739, _chat_helpers.py:5756)

Both usages (__del__ cleanup and cache-eviction MCP disconnect) are genuinely best-effort cleanup — the __del__ case is especially well-justified since raising from __del__ produces a noisy interpreter traceback without propagating anywhere useful. CLAUDE.md calls bare except Exception: pass tech debt and asks new contributors not to cite existing violations as precedent. These two are justified exceptions; a short inline # __del__ must not raise or # best-effort on eviction makes the intent explicit rather than leaving the reviewers to infer it.

🟢 _LoopDummyAgent alias adds confusion (test_loop_break_truthful.py:6081)

# Need _LoopDummyAgent in scope for the second test.
_LoopDummyAgent = _DummyAgent

test_helper_rejects_wrapper_dicts_silently is the only caller. The alias exists solely because the test was written with a different class name in mind. Just use _DummyAgent directly in that test and remove the alias.

    with patch("gaia.agents.base.agent.AgentSDK"):
        a = _DummyAgent(silent_mode=True, skip_lemonade=True)

Strengths

• Lie-on-loop fix is clean and complete. _build_loop_break_summary is a single, centralized helper that eliminates the two in-line "Task completed with…" literals that were the bug. The AST invariant test (test_agent_source_invariants.py) is a smart defence: if anyone re-introduces a second copy of that literal the test breaks immediately, before any eval run.

• model_id ordering fix is subtle and important. The system-prompt cache was being populated during _register_tools() with model_id=None, causing is_tool_calling_model() to return False and the JSON envelope to leak into every tool-calling agent's prompt. Moving the assignment one line earlier, then backing it with a direct assertion test, is the right approach.

• Register/unregister symmetry test (test_mcp_client.py:TestRegisterUnregisterSymmetry) catches the exact class of bug the PR fixes — sanitised prefix at register time, raw name at unregister time. End-to-end tests like this are much more valuable than mocking each side separately.

Verdict

Approve with suggestions. The core fixes are correct, well-tested, and address real eval-discovered regressions. The 🟡 issue (MCP tool-limit default) needs a line in the PR description before merge; the 🟢 items can be applied as one-click suggestions or deferred to follow-up.

Addressed the second-round review (force-pushed):

CI-blocking lint fixes

• 🔴 analyze_failures.py: removed f prefix on 2 non-interpolating f-strings (W1309)
• 🔴 chat/agent.py: added missing from gaia.agents.base.tools import _TOOL_REGISTRY (E0602 / F821)
• 🔴 test_iterations.py: removed unused import io (F401)
• 🔴 test_mcp_reliability_scenarios.py: removed unused from pathlib import Path (F401)
• 🔴 agent.py: 3 × # pylint: disable-next=assignment-from-none on _post_process_tool_result call sites (subclasses override to return plans)

Review nits addressed

• 🟢 _single_tool_done initialised in Agent.__init__ (no more lazy getattr fallback)
• 🟢 sanitise_server_name: warning now compares against case/separator-normalised baseline so pure-case differences don't nag the user
• 🟢 _build_loop_break_summary + sanitise_server_name docstrings trimmed to one-liner per CLAUDE.md
• 🟢 __del__ and cache-eviction except Exception: pass blocks now carry inline rationale comments
• 🟢 _LoopDummyAgent alias removed in test_loop_break_truthful.py; second test uses _DummyAgent directly

Still deferred (worth a separate PR, called out in PR description gaps)

• 🟡 mcp_tool_limit 10 → 100 — needs a changelog line; happy to lower to ~50 if preferred
• 🟡 Hardcoded EDT timezone in analyze_failures.py — should become --tz-offset flag
• 🟢 DEFAULT_OEM_LOG_DIR Windows-only path — should default None + warn-when-explicitly-set

Verified locally:

• pytest tests/unit/agents/ tests/unit/mcp/ tests/unit/eval/ → 520 passed
• python util/lint.py --black --isort → clean

This PR ships four well-diagnosed framework fixes, 10 generic MCP reliability scenarios, and 113 new tests — the eval infrastructure is doing exactly what it was built to do. One shipped file needs cleanup before merge; the rest is ready.

Issues Found

🟡 Important — OEM-specific internals shipped in the public package (src/gaia/eval/analyze_failures.py)

The scenarios the analyzer was built to process (eval/scenarios/mcp_tool_reliability/) are correctly gitignored. The analyzer itself is not, so it ships as part of the installable package with hardcoded internal details that will confuse or break for any external user:

DEFAULT_OEM_LOG_DIR = Path("C:/ProgramData/OEM/OEMService")   # line 51
_OEM_LOG_TZ_OFFSET = timedelta(hours=-4)  # EDT               # line 278

The module docstring also names 41-tool rollup, OEMService*.log, and C:/ProgramData/OEM/OEMService as usage examples. None of this is meaningful outside the original development environment.

Options (pick one):

1. Move the file to the gitignored eval/ tree (outside src/) so it stays on disk but doesn't ship.
2. Genericize: replace DEFAULT_OEM_LOG_DIR with a None default and remove the OEM-named constant and log-parsing regexes, keeping only the generic YAML/JSON analysis logic.
3. Add src/gaia/eval/analyze_failures.py to .gitignore (acceptable since it has no public callers and setup.py packages by find_packages, not by explicit listing).

DEFAULT_LOG_DIR = None   # supply via --tool-log-dir

🟡 Important — mcp_tool_limit default raised 10 → 100 without per-model guidance (src/gaia/agents/chat/agent.py:101, src/gaia/cli.py:1342)

The old limit of 10 was a conservative context-bloat guard. Raising the default to 100 is needed for the OEM MCP service (~49 tools), but it now affects every gaia chat user who has MCP servers configured — including users running 4B-class models whose context windows can't absorb 100 extra tool descriptions. The CLI help text names the OEM MCP specifically, which is an internal detail.

    chat_parser.add_argument(
        "--mcp-tool-limit",
        type=int,
        default=100,
        help="Maximum MCP tools to register (default: 100). "
        "Reduce for small models with limited context windows.",
    )

And in ChatAgentConfig:

    mcp_tool_limit: int = 100  # Raise to accommodate large MCP servers; reduce for small models

Consider documenting the tradeoff (more tools = larger system prompt = context pressure) so users can tune it.

🟢 Minor — Module-level _logged_sanitisations is mutable global state (src/gaia/mcp/client/mcp_client.py:30)

_logged_sanitisations: set = set()

This accumulates entries for the lifetime of the process and is never cleared. Fine for production (warn-once is the goal), but it bleeds between tests — the second test that constructs an MCPClient("tool_mcp", ...) won't see the warning. The existing test suite accounts for this with the warn-once semantics test, but future tests relying on warning presence can silently pass. A pytest fixture that clears the set between tests would make this airtight. Not blocking, but worth noting.

🟢 Minor — json and defaultdict imported inside function body (src/gaia/cli.py:4246-4249)

def _print_reliability_summary(scorecards, pass_threshold=0.90):
    import json
    from collections import defaultdict
    ...

Both are stdlib and have no side effects. They belong at the top of cli.py with the other stdlib imports.

    from collections import defaultdict
    import json

(Move to the top-level import block.)

Strengths

• Root-cause accuracy is excellent. All four problems (bare-prefix emission, model_id ordering, native-path unwrap, rubric conflation) are diagnosed at the exact failure point, not papered over with prompt rules. The eval transcripts → code fix pipeline is the system working as designed.
• The _build_loop_break_summary extraction is clean and the "kept failing" vs "Task completed" branching is the right abstraction boundary. The AST-based invariant test (test_agent_source_invariants.py) guarding against the lie literal reappearing is unusually thorough engineering.
• Test suite quality is high. The test_helper_rejects_wrapper_dicts_silently test documents the exact call-site contract (the native path must unwrap before calling the helper) as a test rather than a comment — future authors can't accidentally re-introduce the bug without breaking the suite.

Verdict

Request changes on analyze_failures.py (OEM internals in a shipped package module). The mcp_tool_limit default change is worth a discussion comment but not a hard blocker if the team is intentionally raising it. Everything else is clean; the loop-fix and model_id ordering fix are especially good and should go in.

This PR ships solid, root-cause-driven fixes for five concrete agent-loop failures, backed by 113 unit tests and a clean deletion of stale AI-generated run logs. The single most important thing: the model_id ordering fix (P2) is subtle but high-impact — the regression test that asserts the JSON envelope is NOT in tool-calling agents' prompts is exactly the right guard.

Issues

🟡 Important — _post_process_tool_result doesn't call super() in BlenderAgent/ChatAgent

Both overrides were updated to return Optional[List[Dict]] but neither calls super()._post_process_tool_result(). The base class is now where _single_tool_done = True gets set for single_tool_per_turn agents:

# base agent.py:4085
if self.single_tool_per_turn and not self._is_error_result(_tool_result):
    self._single_tool_done = True

No existing agent uses single_tool_per_turn=True today, so this is not a current bug. But the first subclass that does will need to call super() — and there's nothing enforcing that. The docstring on _post_process_tool_result should say so explicitly:

src/gaia/agents/blender/agent.py:435, src/gaia/agents/chat/agent.py:382

        """
        Post-process the tool result for Blender-specific handling.

        Returns ``None`` (no recovery plan); the framework's default
        success/error gating applies.

        Subclasses that override this and intend to support
        ``single_tool_per_turn=True`` MUST call ``super()._post_process_tool_result(...)``
        so the base class can set ``_single_tool_done``.

        Args:

🟡 Important — MCP tool limit default jump (10 → 100) needs eval note

src/gaia/agents/chat/agent.py:184: _MCP_TOOL_LIMIT was hardcoded to 10 and is now configurable at a default of 100 — a 10× increase that lands in the system prompt. The PR description doesn't include a baseline eval comparison showing existing RAG/chat scenario scores are unaffected by the larger prompt. CLAUDE.md requires an eval run for prompt-mass changes. The fix is correct (10 was too aggressive), but the PR should reference a --compare scorecard or explicitly note that existing categories weren't re-evaluated.

🟢 Minor — ANSI escape codes without TTY check

src/gaia/cli.py:4294–4296: _print_reliability_summary writes \033[32m / \033[31m directly. On CI, piped output, or Windows terminals, these appear as literal ^[[32m escape sequences. GAIA's AgentConsole uses explicit color guards:

        if passed:
            if sys.stdout.isatty():
                result_str = f"\033[32m{'PASS':>8}\033[0m"
            else:
                result_str = f"{'PASS':>8}"
        else:
            if sys.stdout.isatty():
                result_str = f"\033[31m{'FAIL':>8}\033[0m"
            else:
                result_str = f"{'FAIL':>8}"

Same pattern applies to the GO/NO_GO lines at cli.py:4303–4309.

🟢 Minor — Duplicate word in eval/prompts/judge_turn.md

eval/prompts/judge_turn.md:11: "the the OEM MCP service's tools"

service layer (~38–60% of the OEM MCP service's tools currently report

🟢 Minor — test_task_completed_with_appears_in_exactly_one_string_literal is fragile

tests/unit/agents/test_agent_source_invariants.py:76: The AST walk catches every string literal, including test strings, docstrings, and f-string constant parts. Any future test that asserts "Task completed with" (e.g., assert "Task completed" not in summary written as assert "Task completed with" not in ...) will break this invariant without touching the guarded code. The intent is right; consider scoping the walk to only _build_loop_break_summary's function body via ast.get_source_segment or a narrower node filter.

Strengths

• P2 (model_id ordering) + regression test is the cleanest fix in the PR: a one-line reorder in Agent.__init__, a minimal reproduction path, and a test that fails with the old ordering and passes with the new one. Textbook.
• sanitise_server_name is properly idempotent — the while-loop strips cascading _mcp_ tokens and the 12-case parametrized matrix including "foo_mcp_bar_mcp" → "foo_bar" and the empty-string fallback covers all expected real-world server naming variants.
• _disconnect_cached_agent is applied consistently to all three eviction paths (model change, agent_type change, LRU eviction) in _chat_helpers.py. This is the kind of symmetric fix that prevents subprocess leaks from accumulating silently.
• Deletion of 2,500+ lines of AI-generated run logs (eval_run_report.md, ARCHITECTURE_ANALYSIS.md, monitor_log.md) keeps the repo signal-to-noise ratio healthy.

Verdict

Approve with suggestions. The two Important items are design notes, not blocking bugs (no current agent uses single_tool_per_turn and the 10→100 limit change is directionally correct). Apply the typo fix and ANSI-guard before merge if easy; the super() docstring note is a low-priority follow-up.

Four real bugs fixed, all with targeted regression tests and clear root-cause documentation — this is exactly what the eval framework should surface. No blocking issues; two latent correctness concerns worth addressing before the next agent ships with single_tool_per_turn=True.

Summary

P1–P4 fixes are correct and well-tested: the model_id ordering bug (JSON envelope leak), the native-path loop-break lie, the bare-prefix tool-name truncation, and the server-name doubling pattern. The 10 generic mcp_reliability scenarios are well-structured and the --iterations flag is a clean addition. Cleanup of ~1500 lines of session-specific working docs is appropriate housekeeping.

Main concern: both BlenderAgent and ChatAgent override _post_process_tool_result without calling super(), which silently breaks single_tool_per_turn for those two agents (latent, not active yet). Secondary: a type annotation nit in the refactored to_gaia_format.

Issues Found

🟡 Important — BlenderAgent / ChatAgent don't call super()._post_process_tool_result (src/gaia/agents/blender/agent.py:433, src/gaia/agents/chat/agent.py:380)

The base class Agent._post_process_tool_result now owns _single_tool_done management:

# agent.py (base)
def _post_process_tool_result(self, ...):
    if self.single_tool_per_turn and not self._is_error_result(_tool_result):
        self._single_tool_done = True
    return None

Both BlenderAgent and ChatAgent override this method and return None without calling super(). If either agent (or any existing subclass) sets single_tool_per_turn = True, the _single_tool_done flag will never be set — the agent will loop indefinitely instead of stopping after one tool. The docstrings warn future subclasses to call super(), but the warning is on the wrong class — BlenderAgent and ChatAgent are already the offending overrides.

Since neither currently sets single_tool_per_turn = True this is latent, but the next OEM agent that does will silently break. Fix both to delegate through:

    def _post_process_tool_result(
        self,
        tool_name: str,
        tool_args: Dict[str, Any],
        tool_result: Dict[str, Any],
    ) -> Optional[List[Dict[str, Any]]]:
        """..."""
        # (existing Blender-specific handling here)
        ...
        return super()._post_process_tool_result(tool_name, tool_args, tool_result)

Same pattern needed in ChatAgent.

🟡 Important — MCPTool.to_gaia_format parameter type: str = None (src/gaia/mcp/client/mcp_client.py:151)

raw_server_name: str = None is not valid — None is not a str and pylint/mypy will flag it.

    def to_gaia_format(
        self, prefix: str, raw_server_name: Optional[str] = None
    ) -> Dict[str, Any]:

Optional is already imported at the top of the file.

🟢 Minor — _disconnect_cached_agent swallows exceptions silently (src/gaia/ui/_chat_helpers.py:5817)

The __del__ case in mcp/mixin.py is a legitimate exception to the fail-loudly rule (Python contracts: __del__ must not raise). The cache-eviction case in _disconnect_cached_agent has no such constraint — a failed disconnect during eviction is observable and useful for debugging. At minimum, log at DEBUG:

        try:
            mcp_manager.disconnect_all()
        except Exception as exc:  # best-effort on cache eviction
            logger.debug("MCP disconnect failed during cache eviction: %s", exc)

🟢 Minor — analyze_failures.py default scenarios dir points to gitignored path (src/gaia/eval/analyze_failures.py:4689)

DEFAULT_SCENARIOS_DIR = REPO_ROOT / "eval" / "scenarios" / "mcp_tool_reliability"

This default points to the private/gitignored vendor-coupled directory, not the public mcp_reliability one. On a fresh checkout load_scenarios() will log a WARNING: Scenarios dir missing and return an empty map — silently producing an empty report. The module docstring shows the private path in its --scenarios-dir example, so the intent is clear for internal use, but a contributor running analyze_failures will get a confusing silent-no-results experience. Consider defaulting to the public dir and noting in the module docstring that --scenarios-dir should point at the private scenarios for vendor-coupled analysis.

Strengths

• AST-based regression guard (test_agent_source_invariants.py): using ast.walk to assert the "Task completed with" literal lives in exactly one place in _build_loop_break_summary is creative, runs in milliseconds, and would have caught the original lie-on-loop bug at CI time.

• Native-path regression test is precisely documented (test_loop_break_truthful.py:6088): the test comment explicitly captures the call-site contract — unwrapped results at the native path — making it impossible to misread what the test is verifying.

• model_id ordering fix is surgical and self-contained: one line moved earlier in __init__, one regression test (test_json_envelope_suppressed_for_tool_calling_models) that exercises the exact failure mode. No side effects.

• Warn-once sanitisation (_logged_sanitisations set): the pattern correctly prevents log spam in long-running server processes while still surfacing the server-name misconfiguration on first contact.

• Cleanup: removing ~1500 lines of session-specific working docs (ARCHITECTURE_ANALYSIS.md, eval_run_report.md, monitor_log.md) is the right call. Those files read like in-progress notes, not permanent reference material, and were cluttering the public repo.

Verdict

Approve with suggestions. The core fixes are correct and thoroughly tested. The two 🟡 issues are latent (not currently active) but the BlenderAgent/ChatAgent super-call omission will catch the next developer who tries single_tool_per_turn=True on those agents, so it's worth fixing in this PR while the context is fresh.

Five real framework bugs fixed, solid test coverage, clear PR description. Two items need resolution before merge.

Summary

This PR ships an MCP tool-calling reliability eval framework together with four framework-level fixes that the eval directly exposed: JSON-envelope leak (P2), truthful loop-break summary (P1/P3), server-name sanitisation (P1), and model_id ordering in __init__ (P2). The fixes are well-scoped and each is covered by a targeted regression test. The three deleted files (ARCHITECTURE_ANALYSIS.md, eval_run_report.md, monitor_log.md) are correctly removed — they were AI session logs that should never have been committed. One hardcoded vendor constant and one missing eval gate are the blockers.

Issues Found

🟡 analyze_failures.py — hard-coded vendor constant 41 will spam every public user (src/gaia/eval/analyze_failures.py:5447)

_self_check warns if the per-tool rollup has anything other than 41 unique tools. 41 is the internal OEM tool count; the public repo ships 10 generic scenarios. Every run of the public analyzer will always emit the warning, making it noise from day one. Either remove the guard entirely or accept it via a configurable flag.

    # Tool-count guard is vendor-specific; skip for public/generic scenarios.
    # Pass --expected-tool-count N to add it back when running against a
    # full OEM scenario set.

Affect: remove or gate the check at analyze_failures.py lines 5447–5450.

🟡 mcp_tool_limit default raised 10 → 100 without an eval run (src/gaia/agents/chat/agent.py:1233, src/gaia/cli.py)

Raising the MCP tool registration cap changes how many tools appear in the system prompt, which directly affects tool-selection accuracy for small local models (Gemma-4, Qwen3 series). CLAUDE.md requires a gaia eval agent comparison to baseline whenever tool registration changes. The PR description doesn't include one for this specific change.

The direction is likely correct — 10 was probably too restrictive for real-world MCP configs — but it needs the eval gate cleared. Run:

# Terminal 1
python -m gaia.ui.server --port 4200 --host 127.0.0.1

# Terminal 2
gaia eval agent --category mcp_reliability --compare eval/results/<baseline>/scorecard.json

and include the result in the PR description. If no regression, this is a one-line comment addition.

🟡 mcp/mixin.py:__del__ — bare except Exception: pass (src/gaia/mcp/mixin.py:395–402)

CLAUDE.md prohibits bare exception swallowing. The __del__ constraint is real (Python prints and discards __del__ exceptions anyway), but the fix should at minimum log at DEBUG — exactly what the equivalent change in _chat_helpers.py already does correctly:

            try:
                manager.disconnect_all()
            except Exception as exc:
                logger.debug("MCP disconnect failed in __del__: %s", exc)

🟢 simple_tool_call.yaml requires a tool that doesn't exist in default GAIA (eval/scenarios/mcp_reliability/simple_tool_call.yaml)

The scenario asks the agent to call system_status, get_status, or similar — none of which exist in a clean GAIA install. Running this in CI or a vanilla setup will always produce FAIL, making the "generic public scenarios" claim misleading. Update the success criteria to use a tool that actually exists by default (e.g., any RAG tool, browse_files) or add a tags: [requires_mcp_setup] marker and skip it in default runs.

🟢 test_helper_rejects_wrapper_dicts_silently asserts the wrong behavior by design (tests/unit/agents/test_loop_break_truthful.py:6142)

The test name and internal comment explain this is intentionally documenting the bug shape — the helper returns "Task completed" when fed wrapped dicts because the unwrap must happen at the call site. But assert "Task completed" in summary looks like an accidental success assertion to any future reader who hasn't read the 20-line comment block. Rename the test to test_caller_must_unwrap_previous_outputs_before_passing so the intent is self-describing without requiring the comment. The comment block itself violates CLAUDE.md's "one short line max" guidance for WHY comments.

Strengths

• P1–P4 fixes are each pinned by a regression test — the AST-level invariant (test_task_completed_with_appears_in_exactly_one_string_literal) is genuinely creative and will catch future refactors that reintroduce the lie-on-loop pattern.
• sanitise_server_name is idempotent, warns once, and is tested with a 12-case matrix. The _reset_logged_sanitisations autouse fixture correctly isolates the module global between test runs.
• MCP resource cleanup is now consistent: _disconnect_cached_agent is called in all three eviction paths in _chat_helpers.py (model change, agent-type change, LRU eviction), and evict_session_agent also cleans up. The helper correctly catches exceptions and logs at DEBUG rather than swallowing them.
• _post_process_tool_result return-type change is non-breaking: existing subclasses that return None (implicit or explicit) stay unaffected; the new Optional[List[dict]] path is opt-in. BlenderAgent and ChatAgent both correctly call super().

Verdict

Request changes — the analyze_failures.py constant and the missing eval run for mcp_tool_limit are the blockers. The __del__ fix is a one-liner. Address those three and this is ready to merge.

This PR delivers real value — four well-diagnosed agent-loop bugs fixed with targeted tests, a clean eval framework extension, and long-overdue session-artifact cleanup. The model_id-ordering fix and the truthful loop-break helper are production-quality fixes that would have been invisible without the eval framework that caught them.

Issues

🟡 --reset-between-scenarios flags advertised but not wired (src/gaia/cli.py)

The CLI registers three new args (--reset-between-scenarios, --lemonade-model, --lemonade-ctx-size) and validates them (e.g. full mode requires --lemonade-model), but the validated values are never passed to AgentEvalRunner and AgentEvalRunner has no parameters to accept them. A user who runs:

gaia eval agent --reset-between-scenarios fast --iterations 3

will get the banner-and-loop logic but zero resets — silent no-op. The --help output will list flags that don't do anything.

Either remove the flags from this PR and add them when the implementation lands, or add a # Not yet implemented — stub note and gate the --reset-between-scenarios validation with a NotImplementedError / early-exit so users get an explicit error rather than a silent no-op.

🟡 analyze_failures.py has no unit tests (src/gaia/eval/analyze_failures.py)

This is 995 lines of new code (timestamp-alignment logic, log parsing, per-tool rollup, severity classification, markdown rendering) with zero test coverage. All the other new modules in this PR ship with direct unit tests. Even a smoke test exercising the FailureAnalyzer with a synthetic trace JSON would prevent the most likely regressions.

🟡 MCP tool limit quietly jumped from 10 → 100 (src/gaia/agents/chat/agent.py)

# old
_MCP_TOOL_LIMIT = 10
# new
_MCP_TOOL_LIMIT = self.config.mcp_tool_limit  # default 100

For any user with 11–99 MCP tools configured, the old code silently skipped MCP loading; the new code loads all of them. This is a correct fix (the old limit was too aggressive), but a behavior change at this scope belongs in the PR description so reviewers can evaluate the prompt-mass tradeoff deliberately. The validation evidence (49-tool server, Gemma-4-E4B) is buried in a config comment — move it to the PR description or at minimum to a changelog entry.

Nits

🟢 _disconnect_cached_agent logs at DEBUG — upgrade to WARNING (src/gaia/ui/_chat_helpers.py:466)

logger.debug("MCP disconnect failed during cache eviction: %s", exc)

MCP subprocess cleanup failures on cache eviction are silent in production (DEBUG is off by default). logger.warning matches the __del__ pattern in mixin.py and the GAIA "fail loudly" convention.

            logger.warning("MCP disconnect failed during cache eviction: %s", exc)

🟢 [SYSTEM: Tool call complete...] string leaks into multi-turn history (src/gaia/agents/base/agent.py:1919–1927)

block["text"] += (
    "\n\n[SYSTEM: Tool call complete. "
    "Write your one-sentence response to the user now. "
    "Do not call any more tools.]"
)

This is appended to the tool-result message that goes into messages. For single_tool_per_turn=True agents this is a single-turn-only concern, but the decorated string will appear in the role-history that subsequent audit/replay tools read. A named constant would also make it easier to grep for and strip in test fixtures.

🟢 test_caller_must_unwrap_previous_outputs_before_passing asserts success on wrapped input (tests/unit/agents/test_loop_break_truthful.py:163)

The test name describes a caller contract (must unwrap), but the assertion proves the WRONG behaviour: assert "Task completed" in summary confirms that passing wrapped dicts silently produces a false-success message. Any future call-site that forgets to unwrap will produce a lie and this test will pass. Consider renaming to test_wrapped_dicts_fall_through_to_success_illustrating_required_unwrap and adding a comment that this is a "trap test" documenting the wrong behaviour, not endorsing it.

Strengths

• P2 fix (model_id ordering) is textbook: one-line move, comment explains the exact race, regression test asserts the prompt invariant. The AST-based source invariant (test_agent_source_invariants.py) is a creative addition that prevents the lie-on-loop literal from drifting back without anyone noticing.
• P1 fix (bare-prefix + silent-success lie) consolidates two identical hard-coded strings into _build_loop_break_summary and properly branches on error vs success. The "no-echo" invariant — bad tool name must not be re-quoted in the error message — is a thoughtful UX detail that directly breaks the model's reinforcement loop.
• MCP cleanup (_disconnect_cached_agent) plumbs disconnect through all four eviction paths (model change, agent-type change, LRU, explicit eviction) — previously leaked subprocess handles on every cache miss.
• Deleted artifacts (eval_run_report.md, monitor_log.md, ARCHITECTURE_ANALYSIS.md) — the net −413 lines is a genuine improvement to repo navigability. The .gitignore entries for vendor-coupled scenarios are exactly the right boundary between public and internal eval infrastructure.
• sanitise_server_name idempotency loop is concise and correct; the _logged_sanitisations warn-once guard is the right default for a configuration-level advisory.

Verdict

Request changes. The --reset-between-scenarios stub and the untested analyze_failures.py are 🟡-blocking. Both are straightforward fixes: stub → remove or gate with NotImplementedError; analyze_failures → smoke test with a synthetic trace dict. Everything else is polished and the core fixes (P1–P4) are merge-ready.

This is a well-engineered PR that ships real framework fixes (P1–P3 in the author's taxonomy) alongside the eval harness that found them. The five bugs are clearly described, the fixes are targeted, and the 113-unit-test suite is unusually thorough. Two items need attention before merge.

Issues

🟡 mcp_tool_limit default 10 → 100 is a silent behavioral change (src/gaia/agents/chat/agent.py)

The old limit was _MCP_TOOL_LIMIT = 10, hardcoded with the comment "context bloat guard." This PR bumps the default to 100. That's a 10× increase that affects every user who has an MCP server registered in ~/.gaia/mcp_servers.json, without any opt-in — servers with 11–100 tools that were previously silently skipped now load in full.

The PR comment correctly notes "For larger tool sets the prompt bloat can hurt small-model accuracy" and "keep this as low as the workflow allows" — but then ships 100 as the default. The validation cited (a 49-tool server on Gemma-4-E4B) is one data point on one model; existing users on Qwen3.5-35B, Gemma-4-E4B, or smaller models may see prompt-mass regressions.

This warrants either: (a) a smaller validated default (e.g. 50) that's still an improvement over 10 and leaves room to tune up, or (b) an explicit statement in the PR body that existing MCP users should re-run evals with their tool sets, so the reviewer can make an informed decision.

🟡 _inject_recovery_plan is called from non-planning loop paths with no test coverage (src/gaia/agents/base/agent.py)

_inject_recovery_plan sets execution_state = STATE_EXECUTING_PLAN and resets current_step = 0. It is wired into all three _post_process_tool_result call sites — including the native-tool-call path and the native-fanout path — which are NOT inside an existing STATE_EXECUTING_PLAN execution context.

Today this is safe because no subclass returns a non-None plan from _post_process_tool_result (both BlenderAgent and ChatAgent just call super() which returns None). But when the first subclass uses this API, if it triggers mid-fanout, the state machine transition is untested. There's no test that exercises _inject_recovery_plan via the hook at any of the three call sites.

Suggested fix: add a guard in _inject_recovery_plan that logs a warning (or raises) if called while execution_state is already STATE_EXECUTING_PLAN or in the fanout path — or add a note to the docstring that callers must only return a plan from the sequential _post_process_tool_result context.

🟢 Three stale pylint: disable-next=assignment-from-none comments (src/gaia/agents/base/agent.py)

                    _next_plan = self._post_process_tool_result(
                        tool_name, tool_args, tool_result
                    )

The base class now returns Optional[List[Dict[str, Any]]] instead of None, so pylint will no longer flag assignment-from-none on these three call sites. The # pylint: disable-next=assignment-from-none comments can be removed (lines ~2397, ~3395, ~3613).

🟢 .gitignore missing newline at EOF (lines 19–20 in the diff)

eval/scenarios/mcp_tool_reliability/
tests/unit/eval/test_mcp_tool_reliability_scenarios.py

The added block ends without a trailing newline. Both existing gitignore entries and the eval/monitor_log.md deletion have the same issue — worth a one-line fix to keep git diff clean.

🟢 Stub CLI flags raise NotImplementedError on use (src/gaia/cli.py)

--reset-between-scenarios, --lemonade-model, and --lemonade-ctx-size are added to the parser but raise NotImplementedError when passed any value. The error is loud (consistent with CLAUDE.md) and well-documented. However, CLAUDE.md also says "no half-finished implementations." The approach is defensible as API reservation — just flag it so the reviewer makes a conscious decision to accept it.

Strengths

• model_id-before-_register_tools() fix (P2) is clean and correctly diagnosed. The regression test in test_response_format.py (test_json_envelope_suppressed_for_tool_calling_models) is a model example of how to pin a subtle ordering bug — it will prevent the exact same regression from re-entering silently.

• _build_loop_break_summary + AST invariant test is a creative guard. Writing a structural test that parses the source AST to assert the literal "Task completed with" appears in exactly one place inside the helper — rather than relying on behavioural tests that the literal is in the right branch — is the right call for this class of lie-on-loop bug. Keeps the guarantee live even if the method is refactored.

• MCP subprocess cleanup on cache eviction (_disconnect_cached_agent) closes a real resource leak. The three _agent_cache.pop() + _disconnect_cached_agent() call sites in _chat_helpers.py are consistent and the __del__ hardening in MCPClientMixin correctly swallows errors without suppressing them.

• sanitise_server_name test matrix (12 parametrized cases + idempotency + warn-once isolation via autouse fixture) is thorough. The _reset_logged_sanitisations fixture clearing the module-level set before/after each test correctly solves the only shared-state concern.

Verdict

Approve with suggestions. The two 🟡 items (default limit change and untested _inject_recovery_plan call sites) are not blocking if the author can address the mcp_tool_limit concern in a comment or the PR description and acknowledges the _inject_recovery_plan limitation in the docstring. The rest of the fixes are solid.