test(mcp): add tool selection eval harness - #61

Summary

• add a reproducible MCP tool-selection eval runner that derives the baseline catalog from the real RegisterDeveloperTools MCP registration path
• add 54 synthetic, privacy-safe scenarios informed by persisted PostgreSQL MCP/command activity aggregates
• compare a compact remote-target design (32 tools) with a specialized remote_* design (52 tools), including safety-annotation trade-offs
• score tool choice, argument subset, schema validity, forbidden fallbacks, category breakdowns, confusion pairs, token usage, and latency
• never execute model-selected tools; the runner only sends prompt + function schemas to an OpenAI Responses-compatible endpoint

Production-history basis

The checked-in scenarios are derived from aggregate shapes only. No raw MCP inputs, command arguments/output, workspace identifiers, host names, or credentials are committed. The observed snapshot used for prioritization contained 20,286 persisted MCP requests across six workspaces and 23 tools. In particular, SSH work was frequent and failure-prone enough to justify first-class remote-tool evaluation.

Validation

• go test ./scripts/mcp-tool-eval
• go test -race ./scripts/mcp-tool-eval
• golangci-lint run ./scripts/mcp-tool-eval/...
• git show --check HEAD
• go run ./scripts/mcp-tool-eval --list
• privacy scan for deployment host/IP/token patterns in scripts/mcp-tool-eval

Full scripts/githooks/pre-commit was attempted. Its generated/golden test phase cannot complete in this Developer Workspace because /var/run/docker.sock is absent; existing cli and coderd DB golden tests fail while trying to start PostgreSQL containers. The hook also formatted one new import group; that change was kept and the targeted checks above were rerun on exact HEAD.

A live model A/B is intentionally not fabricated. This workspace currently has no injected MCP_TOOL_EVAL_MODEL or MCP_TOOL_EVAL_API_KEY; the runner is ready to execute against a configured Responses-compatible model endpoint once those are supplied.