feat(coderd/x/chatd): skip MCP servers with recently timed-out connects - #28406
Closed
ibetitsmike wants to merge 1 commit into
Closed
feat(coderd/x/chatd): skip MCP servers with recently timed-out connects#28406ibetitsmike wants to merge 1 commit into
ibetitsmike wants to merge 1 commit into
Conversation
chatd reconnects to every configured MCP server on every generation step, so a black-holed server (even with the connect budget enforced) still costs the full budget per step of every chat on the pod. Cache connect timeouts per process and skip the server for 60 seconds, keyed by config ID and updated_at so config edits retry immediately. Skips are logged and recorded in the debug run's mcp_connect summary with a "skipped" outcome. Fast failures (auth, DNS) are not cached: they are cheap to retry and may be fixed mid-conversation.
ibetitsmike
force-pushed
the
mike/chatd-mcp-negative-cache
branch
from
August 20, 2026 17:20
76bb414 to
7afa2c9
Compare
Collaborator
Author
|
Dropping this PR: the negative cache is disproportionate to the residual problem. With #28400's connect budget in place, a wedged MCP server costs a bounded ~10s per generation step instead of hanging indefinitely; caching that failure for 60s only shaves the repeat cost during an active incident and is not worth the extra machinery. Closing without merge; the rest of the stack (#28400, #28402, #28405) stands on its own.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack context
Top of the MCP connect-stall incident stack: #28400 (connect budget), #28402 (observability), #28405 (stream silence timeout). This layer stops a bad server from taxing every generation step.
Why
chatd reconnects to every configured MCP server on every generation step. Even with #28400 capping each connect at 10s, a black-holed server (as registry.coder.com was during the incident) still costs 10 seconds per step of every chat on the pod until it recovers.
Changes
mcpclient.NegativeCache: after a connect timeout, the server is skipped for 60s (NegativeCacheTTL). Entries are keyed by config ID and bound to the config'supdated_at, so editing the server config retries immediately. Fast failures (auth, DNS) are deliberately not cached; they are cheap to retry and may be fixed mid-conversation.skipping MCP server due to recent connect timeout) and surfaced in the chat debug run'smcp_connectsummary with askippedoutcome, riding the observability from feat: add MCP connect and generation-prep observability #28402.prepareGeneration: configs are filtered through the cache beforeConnectAll, and timeout outcomes are recorded after.Tests
TestGeneration_MCPNegativeCacheSkipsTimedOutServer): a real chatd worker with a black-holed MCP server records outcometimeouton turn 1 andskippedon turn 2, read from the debug run summaries. Red-green verified: unwiring the cache fromprepareGenerationfails only this test (both turns time out).