fix(builder): MCP-only flow with reliable persona authoring (#1532)#1533
Conversation
Scaffolded agents always shipped the built-in zookeeper persona, zoo conversation starters, and a TODO-placeholder docstring regardless of the requested purpose. The create_agent tool exposed no system_prompt or conversation_starters parameter, so _create_agent_impl always passed the hardcoded zoo constants to the generator. - Add system_prompt and conversation_starters parameters to create_agent and thread them through _create_agent_impl into the source generator. - Fill the class docstring from the description; drop the TODO placeholder. - No silent zoo: when the model omits a prompt/starters, derive a generic purpose-based persona instead of the zookeeper. - Guard _split_camel_case to skip splitting when the name already has a space, preserving intentional internal caps (arXiv, iOS). - Update BUILDER_SYSTEM_PROMPT to author a tailored persona and starters. Closes #1532
Remove the tools parameter from the create_agent tool and _create_agent_impl. The Builder now asks only about MCP (yes/no); tool-mixin selection is deferred to a future iteration so Qwen3.5-35B has fewer things to mis-wire. Rewrite BUILDER_SYSTEM_PROMPT to the MCP-only flow: greet → name → purpose → restate what will be built + single MCP question → author tailored persona → call tool. Examples show only name/description/enable_mcp/system_prompt/ conversation_starters (no tools field). If asked about other capabilities the model tells the user to add them by editing the code (custom-agent guide link). generate_agent_source/KNOWN_TOOLS/_render_with_tools remain untouched — they are used by cli_agent.py and tests, and will be re-exposed in a future iteration.
Replace the "To customize it, open agent.py and: ..." hints with a clear
"This is a starter setup" + docs link so users know the agent is working
but has no special capabilities yet. Both the MCP and non-MCP confirmation
strings now:
- keep the {py_path} line (tests assert agent-id substrings appear)
- frame the result as a simple starter agent
- link https://amd-gaia.ai/docs/guides/custom-agent for adding tools/RAG
- MCP variant retains the mcp_servers.json edit hint
- contain none of the fabrication-guard markers (Agent Created / ✅ / File location)
… turn On a successful create_agent call, short-circuit _process_query_impl by returning the tool's confirmation string directly instead of looping for an LLM summary. This guarantees the starter-framing and docs link reach the user verbatim, and removes the extra model turn that produced degenerate output in issue #1382. Also extend error detection to cover dict results with status=="error" (e.g. from _execute_tool wrapping errors in a dict), in addition to the existing "Error:"-prefix string check. Remove the now-unused created_ok variable and the post-tool LLM-summary path. The fabricated-success guard (hallucination → corrective turn) is kept intact.
Move the 6 TestCreateAgentImplTools tests and the acceptance scenario from _create_agent_impl(tools=...) to generate_agent_source(tools=...) directly, since tool-mixin composition now lives only at that layer (Builder surface no longer accepts a tools param). Coverage parity maintained: mixin imports, MRO order, MCP-last ordering, invalid-tool ValueError, all-KNOWN_TOOLS importable. Drop the mcp_json.exists() assertion in test_tools_combined_with_mcp (file creation already covered by TestCreateAgentImplMCP.test_mcp_enabled_creates_json_file); assert the class signature class OpsBotAgent(Agent, FileIOToolsMixin, MCPClientMixin) instead. Add TestBuilderSurface: - test_create_agent_tool_has_no_tools_param: introspect the registered tool's schema/signature and assert tools is absent - test_stray_tools_kwarg_surfaces_error: feed a tools=[...] call through _process_query_impl and assert honest failure (no crash, no fabricated success) - test_confirmation_mentions_starter_and_docs_link: assert result contains starter framing and amd-gaia.ai/docs/guides/custom-agent Update fenced integration + fail-loudly tests for the short-circuit: - test_fenced_call_result_is_deterministic_confirmation: assert answer == tool_result and send_messages.call_count == 1 - test_bare_call_still_fires: assert answer == tool_result - test_real_tool_call_and_file_present_returns_success: assert answer == tool_result, call_count == 1 - test_create_agent_error_result_returns_honest_failure: unchanged (still green)
- test_hotreload_skipped_gracefully_when_no_registry: the input "NoReg Agent"
already contains a space, so _split_camel_case short-circuits on the space
(not because "NoReg" is a deliberate single token), leaving agent-id "noreg".
- test_stray_tools_kwarg_surfaces_error: _execute_tool does not propagate the
TypeError — it catches it (agent.py:1741) and returns a {"status": "error"}
dict, which the dict-error check in _process_query_impl then surfaces.
Comment-only changes, no behavior change.
Code Review — #1533
|
Pull request was converted to draft
- test_stray_tools_kwarg_surfaces_error: drop the hardcoded agent.py line number from the comment; name the base Agent._execute_tool method instead so the reference doesn't rot. - cli_agent._scaffold_python: add a one-line WHY comment above the generate_agent_source(...) call making explicit that the dev scaffold intentionally seeds the playful demo persona, while the conversational UI Builder authors a purpose-matched one. Comment-only changes, no behavior change.
|
Thanks — addressed in 610037e.
|
Review —
|
Close a real coverage gap: existing tests prove the Builder *writes* a correct agent.py, but none *runs* a built agent. test_built_agent_runs_with_authored_persona builds a non-MCP agent, dynamically imports the generated file, instantiates the class with skip_lemonade=True + a mocked send_messages (offline, CI-safe), and drives process_query through the base Agent loop. Asserts the run succeeds (status == success, error_count == 0), returns the mocked answer, and — the load-bearing check — runs on ITS authored persona (system_prompt == the authored prompt), proving the built agent uses its own persona rather than a placeholder.
…ated-agent caveat, confirmation, UI badge) The Builder previously read as if it produced a turnkey agent that performs the described task. It actually scaffolds a starter template the user extends in code. Set honest expectations on every surface: - Builder intro (system_prompt.py): opens by stating it's an alpha feature whose sole job is to scaffold a starter template; first reply sets this expectation before asking for a name. - Scaffolded agent persona (template.py STARTER_CAVEAT, appended in _create_agent_impl): the built agent itself tells the user it's a starting point with no real tools yet, linking the custom-agent guide. - Generated file header gains an "Alpha: scaffolded by the Gaia Builder" line. - Description tagged "(alpha template)" so the Hub card reads honestly. - Confirmation strings (MCP + non-MCP): replace "starter setup" wording with an explicit alpha caveat — the agent can chat but won't fetch data or perform the task on its own until tools/MCP are added. No fabrication-guard markers. - Agent UI: small "Alpha" pill on the Build-a-Custom-Agent card. Tests cover the caveat in the generated source, the alpha/template framing in the Builder prompt, and the alpha + "won't/own" framing + docs link in both confirmation strings.
|
🟡
'Summarizes arXiv papers daily. (alpha template)'as its class docstring — visible in IDE tooltips and Fix: separate the Hub-display description from the docstring description: hub_desc = f"{desc} (alpha template)" # Hub card only
source = generate_agent_source(
...
description=hub_desc, # → AGENT_DESCRIPTION
docstring_description=desc, # → class docstring (new param, or keep desc clean)
...
)Or, simpler: apply the tag only to |
|
Pushed a framing pass + refreshed all screenshots (commits through
On the prior review's two 🟢 notes: both are intentional and unchanged — the |
|
Thanks for testing on Windows, @amd-pworfolk — glad the zookeeper's retired! 🎉 And good catch. The file on disk was always correct; this was display-only. The chat renders messages with Fixed in |
Review:
|
…verbatim The MCP confirmation emitted its mcp_servers.json example as a plain 2-space-indented block. react-markdown/remark-gfm collapses its newlines (and ["mcp-server-time"] can be mis-parsed as a link), so it rendered as a mangled one-liner in the chat. Same display-only markdown-safety class as the path bug. Render the example as a fenced ```json block (no leading indent so the fence is recognized) and wrap the inline mcp_servers.json filename in backticks. Path line, docs link, and the non-MCP confirmation are unchanged.
Review:
|
|
Addressed in
|
…ters
1. _process_query_impl create_agent error path: extract a clean detail from the
error dict (key "error", falling back to "error_brief" for the exception path)
instead of interpolating the raw dict, so the chat no longer shows
{'status': 'error', ...}.
2. Add a standing note where STARTER_CAVEAT is appended: the caveat is
unconditional because the Builder attaches no tools today; gate it if
tool-mixin re-exposure is added so a tool-equipped agent doesn't keep
disclaiming "no tools yet".
3. default_conversation_starters now returns three starters to honor the "2-3"
contract in the tool docstring and Builder system prompt.
Strengthen test_stray_tools_kwarg_surfaces_error to assert no raw dict repr
leaks into the user-facing answer.
|
All three addressed in
116 passed, lint clean. |
Code Review —
|
1. _process_query_impl create_agent error branch: replace the ternary detail-extraction one-liner with an explicit if/else for readability. Same behavior, same final_answer + break. 2. The starter-caveat comment in all three _render_* System-Prompt blocks said the caveat is "appended after the '---' below", implying a separate code block; _get_system_prompt emits the prompt as a single-line repr(). Reword to "appended to the end of the returned prompt (after the '---' marker)" so it's accurate. Cosmetic only — no behavior change. Test assertion updated to the new wording.
|
Both polish items addressed in
116 passed, lint clean. |
SummarySolid, well-tested refactor that makes the Builder honest about what it produces. It drops the capability menu (MCP-only surface), authors a purpose-matched Issues Found🟢 Unnecessary 🟢 Builder system prompt is a large LLM-affecting change not covered by the eval harness ( Strengths
VerdictApprove. No blocking issues — only one trivial cosmetic nit. The behavior change (MCP-only Builder surface) is intentional, documented, and the low-level tool-composition path is preserved for the deferred re-exposure work. |
kovtcharov-amd
left a comment
There was a problem hiding this comment.
Re-approving after the latest polish commits (my earlier approval was auto-dismissed by the new pushes). The refinements are all solid:
- Clean error-detail extraction (
error/error_brief) so a raw dict never leaks into the user-facing message — fail-loud, not silent. - Fenced the MCP confirmation JSON so it renders as a code block, plus 3 fallback starters honoring the 2–3 contract. Well-commented (the caveat-gating note is a nice breadcrumb) and covered by new tests.
All substantive CI is green (Code Quality, Unit Tests 3.10/3.11/3.12 + macOS, Security Linux/Windows).
behavior-e2e is red due to a CI infra bug, not your code — the Windows runner's PowerShell rejects uv venv && uv pip install -e ".[dev]" (The token '&&' is not a valid statement separator in this version) at the Setup step, before any test runs. That Windows job needs its run-step split onto separate lines (or shell: bash); worth a separate CI fix and shouldn't block this merge.
Review —
|
Closes #1532
Why this matters
Before: the Builder created an agent successfully but ignored what you asked for — every scaffolded
agent.pyshipped the built-in zookeeper persona, zoo conversation-starter chips, and aTODO: Replace this docstringplaceholder, and it offered a capability menu (RAG, file search, shell, browser…) the local model frequently mis-wired (likely a driver of the "~40% of runs fail" pattern in #1428).After: the Builder reliably scaffolds a purpose-matched starter template and is honest about what that means. It authors a tailored
system_prompt+conversation_startersfrom the described purpose, asks one question (MCP), and — framed as an alpha feature — states up front that it creates a template you enhance yourself, not a turnkey agent that autonomously fetches data or runs complex logic. The generated agent says the same on its first reply, so it never pretends to already do its job.Reviewer threads (each evaluable on its own):
toolsparam + capability menu); persona authored from the purpose with a description-derived fallback (never the zoo); deterministic confirmation (returns the tool result directly instead of a second LLM turn — also removes the exact turn that produced [Bug]: Custom Agent not working #1382's degenerate output); dict-error detection so a stray kwarg fails loud instead of fabricating success.(alpha template)._split_camel_casepreserves intentional caps (arXiv,iOS) when the name already has spaces.Out of scope (deferred, not lost): re-exposing tool mixins (RAG, file search, …) in generated agents via a tested flow — the low-level
generate_agent_source(tools=…)is retained for it.Test plan
pytestbuilder suite +test_amd_gaia_urls.py— 111 passed (run against the modified src; import resolution verified)python util/lint.py --all— cleanprocess_querythrough the real agent loop with a mocked LLM, asserting it executes on its authored persona (offline / CI-safe); verified it fails cleanly on a reverted persona (no false pass)gaia chat --ui: MCP-only flow, persona match, alpha/template framing, and the built agent runs in-character (citedarXiv:2006.11239) while honestly disclaiming it can't fetch live data until tools are addedgaia chat --uiScreenshots — alpha/template framing (real UI, Qwen3.5-35B)
1. "Alpha" badge on the entry card:
2. Builder intro sets expectations on start — alpha feature, scaffolds a template you extend (not a turnkey agent):
3. Build confirmation — "starter template", alpha, won't perform the task on its own + custom-agent docs link:
4. The built agent's FIRST reply is honest — it does not pretend to fetch live arXiv: "…running as a starter template scaffolded by GAIA Agent Builder (alpha feature). At this moment, I don't have direct access to fetch live arXiv papers…" then it explains how to enable it via MCP/tools.
5. The created agent in the selector, tagged
(alpha template):6. MCP-flow confirmation — the
mcpServersexample renders as a verbatim code block (not a markdown-collapsed one-liner), and the path is shown verbatim:Images on the non-merging
assets/pr-1533-screenshotsbranch (safe to delete after merge); session sidebar cropped from captures.