Summary

Before this PR, #800's scratchpad/memory tool collision was resolved in code (by the #688 dynamic tool loader, landed via #1449/#1450/#1451) but nothing pinned it — no test asserted that the structured-data tool and memory recall can't crowd each other out of the prompt, so the fix could silently regress. This adds a deterministic regression test, a live eval scenario, and an in-repo note that together lock the resolution and let #800 close with evidence. No runtime code changes — this is a closeout + regression, not a feature.

Why

#800 is a coordination tracker: it exists to prove the collision is fixed, not leave it as a known gap. The decisive finding (verified on main) is that the literal pair in the title — scratchpad.query_data vs memory.recall — cannot occur in the loaded profile: scratchpad tools are registered only for the ChatAgent data/full profiles, never doc, which is the only profile the loader is wired to. The real doc-profile arbitration is analyze_data_file (structured-data, conditional) vs recall (CORE, always-on). That asymmetry is the resolution — and until now no test encoded it.

Linked issue

Closes #800

Changes

• Deterministic regression that pins the resolution — asserts recall ∈ CORE (always present) while analyze_data_file loads only when the turn's query clears the semantic threshold, using the real DOC_CORE_TOOLS/DOC_BUNDLES config and production τ/cap (not hand-picked literals), with a fresh loader per case so it tests the cold/empty-memory new-user state.
• Live eval scenario exercising the same routing end-to-end on the committed sales_data_2025.csv corpus, so the disambiguation is checked against a real model, not just unit logic.
• In-repo closeout note in the tool-loader plan doc explaining how Tool-registry scaling: resolve scratchpad/memory collision via #688 dynamic loading #800 is resolved (design asymmetry), cross-linked to the test and scenario.

Deviations from the approved sketch (#800's body)

Flagged per CLAUDE.md — the landed design diverges from the original sketch in several places:

Test plan

• python -m pytest tests/unit/test_tool_loader_disambiguation.py -v → 4 passed (the AC Update Documentation #5 gate)
• python -m pytest tests/unit/test_tool_loader_selection.py tests/unit/test_chat_tool_bundles.py tests/unit/test_chat_dynamic_tools.py -q → 67 passed (no loader regression; confirms the new test's registry/embedder assumptions match shipped config)
• python -m pytest tests/test_eval.py -k scenarios -q → 27 passed (validates the new scenario YAML: required fields, sequential turns, existing corpus path)
• python util/lint.py --all → clean
• (optional, needs a running Lemonade backend + UI server) gaia eval agent --category tool_selection --agent-type doc → the 4 pre-existing scenarios match the Tool loader — Part 3: skill-driven signal (gated on #887) #1451 Part-3 proof (no selection regression)

Checklist

• I have linked a GitHub issue above (Closes #800).
• I have described why this change is being made, not just what changed.
• I have run linting and tests locally (python util/lint.py --all, pytest tests/unit/).
• I have updated documentation if user-visible behavior changed (in-repo closeout note; no user-facing behavior changed).

#800 Acceptance Criteria — Proof

Verdict: all 7 ACs satisfied on main + this PR. The collision is resolved structurally by the #688 dynamic tool loader (landed via #1449/#1450/#1451); this PR pins it. Three ACs are satisfied with the documented reframing above.

Structural finding: the literal scratchpad.query_data vs memory.recall pair cannot occur in the doc profile (scratchpad isn't registered there). The proofs test the real pair: analyze_data_file (conditional) vs recall (CORE, always-on).

AC #1 — Tool-prompt token reduction

Reframed: the ~12K figure assumed the text path; the real cost (and gate) is the native tool-schema path / TTFT. The 38-tool doc profile is capped to 14 (DEFAULT_MAX_TOOLS).

$ pytest tests/unit/test_tool_loader_token_budget.py -q     →  10 passed

test_core_only_is_the_reduction_best_case pins the always-on CORE floor at ≤45% of the native baseline (~50–60% reduction).

AC #2 — query_data and recall not both unless justified

recall ∈ DOC_CORE_TOOLS; analyze_data_file ∈ the conditional data bundle (tool_bundles.py). The conditional tool loads only when the query clears τ:

$ pytest tests/unit/test_tool_loader_disambiguation.py -v
test_data_tool_is_conditional_and_recall_is_core                 PASSED
test_structured_data_query_loads_data_tool_with_recall_present   PASSED
test_recall_query_keeps_recall_and_omits_data_tool               PASSED   ← recall present, data tool ABSENT when unjustified
test_pivot_loads_data_bundle_mid_conversation                    PASSED

AC #3 — Core tools always available despite heuristic failure

CORE is admitted unconditionally and is cap-/eviction-exempt; on embedder failure the loader disables for the session and falls back to the full registry, logging loudly (tool_loader.py).

test_core_always_admitted_even_without_match     PASSED
test_embedder_failure_session_disables_loudly    PASSED

AC #4 — Decisions logged for eval/tune

Satisfied with a deliberate deviation: decisions are emitted as structured TOOL_LOADER {json} INFO lines (not the SQL tool_history table — avoids a UI-DB migration). Consumed by src/gaia/eval/tool_recall.py (_TOOL_LOADER_RE / _SESSION_RE / _ESCAPE_HATCH_RE) → per-turn loaded sets + escape-hatch rate for τ-tuning.

AC #5 — End-to-end regression test (data-query vs recall) — new in this PR

• Deterministic gate: tests/unit/test_tool_loader_disambiguation.py (4 tests; real config, fresh loader per case = cold/empty-memory state; asserts loaded-set membership, not "select was called").
• Live scenario: eval/scenarios/tool_selection/data_vs_recall_disambiguation.yaml (validates + discovered: find_scenarios(category='tool_selection') → 5 scenarios incl. the new one).

AC #6 — Eval suite passes, no selection regression

Live serial run on Gemma-4-E4B corroborates the #1451 Part-3 success-criteria proof — no regression:

* hardware artifact of the local Apple-Silicon box (Metal llama.cpp ~14–19 tok/s), not a behavior change. The new scenario records FAIL 6.58 for reasons unrelated to #800: the disambiguation works (agent routed both aggregates to analyze_data_file, never misused recall; Turn 2 returned the exact answer), but Turn 1's correctness failed on analyze_data_file's date-filter handling of the monthly-summary CSV plus an agent hallucination — flagged as a separate follow-up. Committed baseline refresh deferred to a clean run on target hardware.

AC #7 — Mid-conversation re-evaluation works

Proven for in-profile pivots (the sketch's "file→web" example is out of the doc profile — browser tools aren't registered there):

test_pivot_loads_data_bundle_mid_conversation   PASSED   ← turn 1 omits data tool; turn 2 adds it
test_monotonic_growth_no_pruning_on_score_drop  PASSED
test_lru_evicts_oldest_last_call                PASSED
test_evicted_tool_can_be_readmitted             PASSED