Danny suggested adding CI testing to catch tool changes that would break MCP server functionality:
we could have a CI job which runs Claude Code & Codex against our MCP server and do something like what is my coder username with a regex check. that should be pretty reliable.
This would help catch issues like the one identified in #19825 where tool schema changes could break integration with OpenAI/Anthropic.
Possible approaches:
- Run Claude/Codex against MCP server in CI
- Use test queries with regex validation
- Add e2e tests against upstream services
- Validate tools against JSON schemas
@ThomasK33
Danny suggested adding CI testing to catch tool changes that would break MCP server functionality:
This would help catch issues like the one identified in #19825 where tool schema changes could break integration with OpenAI/Anthropic.
Possible approaches:
@ThomasK33