Review: tool-prompt cost & TTFT baseline harness (#1448 Part 0)

Summary

Approve with minor suggestions — this is a clean, well-scoped, measure-only addition that changes no loader/agent/selection logic, and the one thing worth knowing is that the whole thing is correctly built to run model-free in CI. The harness defers all heavy imports, stubs missing optional deps, and isolates the global tool registry behind a try/finally restore, so the deterministic cost/slope/distribution numbers (and the TTFT parser) are pinned without a Lemonade backend. I verified the code against the actual Agent internals it leans on (_tools_registry, _instance_tools, _TOOL_REGISTRY, the two render methods) and they all match, and the committed scorecard fixture produces exactly the values the test hardcodes (first_turn [0.231, 0.856, 0.122, None], max_needed_set == 2). No security surface: the only network/exec path is the opt-in --live-ttft bench, which re-raises loudly with actionable guidance and never falls back silently.

One caveat on verification: I could not execute the suite in this environment (no gaia install, pytest/tiktoken absent, read-only FS), so the "7 passed" claim is taken on the static evidence above rather than re-run here.

Issues Found

🟢 Unbounded tiktoken pin while its neighbours are capped (setup.py:195, setup.py:205)
The surrounding pins in both extras carry upper bounds (httpx<0.29.0, respx<0.24.0, keyring<26.0.0, numpy<2.3.0). tiktoken is left open-ended, so a future major could land unpinned in CI. Suggest capping it consistently — apply to both the dev and eval entries:

            "tiktoken>=0.7.0,<1.0.0",

🟢 Model-free test lives under tests/integration/ (tests/integration/test_tool_loader_token_budget.py)
The file's own docstring stresses the checks are "model-free (no Lemonade backend)", and the TTFT case reads a committed fixture rather than a live server. The integration/ convention in this repo is real-service tests gated on require_lemonade; this one needs no backend and would sit more naturally in tests/unit/. Not blocking — it does exercise cross-component wiring (chat agent + registry + renderers), so the placement is defensible. Flagging only for consistency.

🟢 Tight, deliberate coupling to Agent internals (src/gaia/eval/tool_cost.py)
build_doc_agent_skeleton patches Agent.__init__, hand-sets instance attributes, and reaches into _TOOL_REGISTRY / _instance_tools. That's a reasonable tradeoff for a measurement tool, and the pinned test is the right guard — but it means a rename of any of those internals breaks the harness silently rather than at a type boundary. Worth a one-line note in the module docstring pointing future maintainers at the pinned test as the canary. No change required.

Strengths

• Genuinely CI-safe by construction — deferred imports, optional-dep stubbing, and _isolated_registry() restoring the global _TOOL_REGISTRY exactly on exit mean measurement never leaks state into other tests in the process. This is the hard part done right.
• Fail-loudly done correctly — the --live-ttft path catches, then re-raises with from exc and a three-part actionable message (what failed, what to do, that it never loads/evicts models itself). Matches the repo's no-silent-fallbacks rule.
• Honest baseline discipline — the pinned numbers carry an explicit "update in the same commit and note it in the PR" instruction mirroring the [Bug]: 1.4MB file indexed no response to queries in the UI #1030 system-prompt budget test, and the TTFT parser counts null timings instead of silently treating them as zero. The PR also corrects Tool loader — Part 0: measure tool-prompt cost & TTFT baseline #1448's original "~12K / ~400-per-tool" premise with measured reality rather than restating the assumption.

Verdict

Approve with suggestions. No blocking issues. The tiktoken cap is a one-click suggestion; the other two notes are non-blocking consistency observations. Nice, disciplined groundwork for the Part-1 loader.

Latest commit addresses all three review suggestions:

• Capped tiktoken to >=0.7.0,<1.0.0 in the dev/eval extras (matches the surrounding pins).
• Added a "Coupling note" to tool_cost.py's module docstring listing the Agent/ChatAgent internals it relies on.
• Moved the (model-free) test to tests/unit/test_tool_loader_token_budget.py.

The review above still showing the original three items is expected, not stale CI: per .github/workflows/claude.yml, the full review runs on opened/ready_for_review, while pushes trigger a lightweight regression-only re-review that stays silent unless it finds something new and doesn't edit the original comment. Happy to request a fresh full review (@claude review) if useful.

On the 4 failing checks — they're Lemonade-server startup timeouts on the self-hosted runners, not caused by this PR:

• API Tests — "Start Lemonade Server and Run Tests" exited 1 with no server logs.
• Test Agent SDK on Windows — "Start Lemonade Server…" timed out after 15 min.
• Example Agents Integration Tests (stx) — "Start Lemonade Server…" exceeded 30 min, canceled.
• Lemonade Server Smoke Test (stx) — "Start Lemonade Server, Load Model, and Test" timed out after 10 min.

All four fail at the Lemonade-server-start step, before any test executes — the smoke test (which runs no GAIA tests, just starts the server and loads a model) timing out shows the runner pool can't start Lemonade at the moment. None of these jobs import the new module; they only ran because the PR touches setup.py, which is in their paths: filters. This PR changes only a dependency cap, a docstring, and a test-file location — nothing that affects server startup. Could a maintainer re-run these once the runners are healthy?

Confirmed — all three review items landed in f72abef4, and the red checks are infra, not this PR.

Fixes verified:

• tiktoken capped to >=0.7.0,<1.0.0 in the dev/eval extras
• "Coupling note" added to tool_cost.py's module docstring
• Test moved to tests/unit/test_tool_loader_token_budget.py

On the failing jobs: agreed they're environmental — this PR's three-file diff (a dependency cap, a docstring, a test relocation) can't affect server startup, and none of those jobs import the new module; they only ran because setup.py is in their paths: filters. One small correction to the framing: the API Tests job actually got Lemonade up fine ([OK] Lemonade server is ready) and then failed because the GAIA API server didn't come up within 30 s (API server failed to start) — a different startup step than the others, but still infra/runner flakiness rather than anything in this change. The stx smoke/integration timeouts point the same way.

These need a maintainer to re-run once the runner pool is healthy — I can't re-trigger checks. @kovtcharov-amd, could you kick off a re-run? I'll hold off on a fresh full review unless you'd like one (@claude review).

Thank you for working on this! Could you kindly share the baseline results from the tool?

Thanks @kovtcharov-amd — I've posted the full baseline results from the tool as a comment on #1448 (comment).

Quick summary of what's there:

• Deterministic cost (model-free): native tool schemas = 5,128 tok (37-tool doc set), ~4.5× the text path, ~117 native tok/added-tool slope.
• Live prompt-prefill TTFT (Gemma-4-E4B resident, no RAG): full 37 tools 16.1 s vs 5-tool subset 0.33 s → ~15.8 s saved on the first turn.
• Repro: python -m gaia.eval.tool_cost --profile doc (no backend) and --live-ttft (model resident).

Full reasoning, the premise correction, recommended max_tools (~8–12), and the Part-1 target are in that #1448 comment.

Re-confirming the 5 red checks are infra, not this PR. I just merged the latest main into this branch and pushed — that re-triggered the full CI run, and the same checks failed again the same way, at the same step. So this is a fresh run on top of current main, not stale results.

Every job ran the same prefix successfully (checkout → Python setup → Install Lemonade Server ✓ → Verify GAIA Installation ✓), then failed at the first step that starts Lemonade and loads a model — before any test executed (test steps show skipped):

The clincher: the Lemonade Server Smoke Test itself failed. That job only starts Lemonade, loads a model, and pings it — it imports none of this PR's code. The failures also span two runners (sjlab-stx-1 and sjlab-stx-3), so it's pool-wide. These jobs only triggered because setup.py is in their path filters; none import the changed module.

Could a maintainer re-run these once the runners are healthy? If they keep timing out at Lemonade start, the runners likely have stale lemonade-server/llama-server processes holding the port/model slot — a cleanup or reboot on sjlab-stx-1/stx-3 should clear it. @kovtcharov-amd

Should be fixed now, rerunning CI/CD.

Looks like one of the workflows is still failing. I'm going to complete this PR since it's unrelated to the failure.