Mark failed PR tool calls as isError instead of successful text - #3163
Mark failed PR tool calls as isError instead of successful text#3163noQbot wants to merge 1 commit into
Conversation
list_prs, triage_prs, and get_pr_impact returned their failure messages ('gh
CLI not found or not authenticated', 'PR #N not found') as ordinary text with
isError unset, so an MCP client that only checks isError read a genuine failure
as success (ADR-0001 finding 4).
Add a ToolError exception a handler raises to signal an error result. The PR
tools raise it for the gh-missing / unresolved-PR cases. call_tool re-raises it
(the mcp 1.x decorator wraps a raised exception into an error result) and the
2.x _on_call_tool catches it and returns CallToolResult(..., isError=True).
Normal not-found text (missing node/community) and get_pr_impact's 'no changed
files' empty result are unchanged, so only genuine failures flip isError.
Tests: a PR tool with gh mocked to fail returns isError:true; a normal tool
result is not isError.
Co-Authored-By: Vinv-AI <[email protected]>
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 1 advisory finding(s) below merit a look before merge.
Formal verification. No changes could be formally verified in this run.
Graphify review — findings
Adds a ToolError exception that PR tool handlers raise when gh is missing/unauthenticated or a PR can't be resolved, so those cases surface as tool results with isError: true instead of being returned as ordinary successful text; the _on_call_tool (mcp 2.x) path catches it into a CallToolResult with isError=True, while the mcp 1.x path re-raises so its decorator marks the error. Normal string returns (including benign "no changed files" cases) stay successful results. Tests cover both the failing PR path setting isError and a successful tool result staying unmarked.
Worth a look
- Test monkeypatches the wrong fetch_prs reference —
tests/test_serve_http.py:371· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 491 functions depend on the 160 functions this change touches.
Health — this change adds coupling hotspots:
- new:
dispatch_command()— 2 callers, 122 callees - new:
_query_graph_text()— 20 callers, 9 callees - new:
_score_query()— 15 callers, 5 callees - new:
_query_terms()— 20 callers, 3 callees - new:
run_benchmark()— 16 callers, 3 callees - new:
_build_server()— 2 callers, 16 callees - new:
_load_graph()— 9 callers, 3 callees - new:
_query_subgraph_tokens()— 7 callers, 3 callees - …and 9 more — each is listed as a finding
Verification — 491 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 324 function(s) in the blast radius were not formally verified this run
Formal verification
Could not verify: Could not verify \_build\_server.
The verifier did not have enough to check \_build\_server, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous
· 1 grounded finding(s) anchored inline below; 16 more finding(s) on lines outside this diff (see the check run).
| """ | ||
|
|
||
|
|
||
| def _load_graph(graph_path: str) -> nx.Graph: |
There was a problem hiding this comment.
_load_graph()
9 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
|
Shipped in v0.9.52 via authorship-preserving cherry-pick so you keep contributor-graph credit. Thanks @noQbot! Release: https://github.com/Graphify-Labs/graphify/releases/tag/v0.9.52 |
Summary
Mark failed MCP PR tool calls as errors instead of returning the failure message as a successful text result.
list_prs,triage_prs, andget_pr_impactreturned "gh CLI not found or not authenticated" / "PR #N not found" as ordinary text withisErrorunset, so a client that checksisErrorread a genuine failure as success.Closes #3162.
What changed
ToolErrorexception that a tool handler raises to signal an error result.list_prs/triage_prson thegh-missingRuntimeError, andget_pr_impactwhen the PR can't be resolved (pr_data is None).call_toolre-raisesToolError(the mcp 1.x decorator wraps a raised exception into an error result), and the 2.x_on_call_toolcatches it and returnsCallToolResult(content=…, isError=True).Normal not-found text (missing node/community) and
get_pr_impact's "no changed files" empty result are unchanged, so only genuine failures flipisError.Why
The PR tools were inconsistent with the rest of the server: input-validation errors already surface as
isError: true, but a runtime failure inside these handlers (gh missing / unresolved PR) was swallowed into an ordinary text result.A client relying on
isErrorto branch on failure would treat those as success.Testing
list_prswithfetch_prsmocked to fail returnsisError:trueand the gh message.graph_statsresult is notisError.test_serve.py,test_serve_http.py,test_prs.py(240 tests). ruff clean.Notes
triage_prsas well — it had the identicalreturn f"Error: {e}"bug — so all three PR tools behave consistently. Happy to narrow tolist_prs/get_pr_impactif you'd prefer.