fix(pydantic-ai): Stop capturing internal exceptions#5237
fix(pydantic-ai): Stop capturing internal exceptions#5237alexander-alderman-webb merged 3 commits intomasterfrom
Conversation
| **kwargs, | ||
| ) | ||
| update_execute_tool_span(span, result) | ||
| return result |
There was a problem hiding this comment.
I wonder if removing the exception capturing altogether is the way to go. If a user has errors happening in their tool, even if they don't necessarily bubble all the way up, they'll lose visibility into them. If e.g. a tool is consistently providing erroneous output, the dev might want to know about it and fix it, even if the whole LLM chain manages to recover.
What we could consider:
- still capturing other exceptions, just not
ToolRetryError(that wouldn't fix the example I talk about above, but it'd at least make sure we don't stop capturing other potentially important exceptions) - see which of the exceptions captured here will be handled by Pydantic AI internally and capture those with
handled=True - add an integration-level option to turn tool exceptions on/off
Since I have little experience in this type of development though, might be this is also fine as is and no one cares about recoverable errors in their tools. 😅
There was a problem hiding this comment.
good points, I did some more investigation.
- Other errors bubble all the way up, so if we catch them here we're catching them multiple times. At least on the code path in
_agent_graph.process_tool_calls()
-> _agent_graph._call_tools()
-> _agent_graph._call_tool()
-> ToolManager.handle_call()
-> ToolManager._call_tool()`
A few places call process_tool_calls(), but only ToolRetryError is handled, and experimentally, other errors bubble up to the top-level run methods.
- I'm open to this. Tool authors can raise
ModelRetryto trigger aToolRetryError, orToolRetryErrorcan result from a validation failure. These are documented, and the only two ways to raiseToolRetryErrorinToolManager._call_tool(). See https://ai.pydantic.dev/tools-advanced/#tool-retries. - We should add the option if we capture
ToolRetryError.
There was a problem hiding this comment.
Thanks for investigating!
I'm leaning towards the following:
- removing the broad
except Exceptionand replacing it withexcept ToolRetryError, to prevent double capture of other errors - gating the actual
ToolRetryErrorcapture behind an option (unsure about the default) - if the option is on and we end up capturing a
ToolRetryError, capturing it withhandled=True
WDYT?
There was a problem hiding this comment.
I agree, was already working on this approach.
I set the option to True, since this is useful information in my view.
|
Please also add to https://docs.sentry.io/platforms/python/integrations/pydantic-ai/#options 🙏🏻 |
Description
Stop capturing broad exceptions, that bubble up to top-level run functions, at the tool call level for Pydantic AI. Only capture
ToolRetryErrorat the tool call level, and mark these exceptions as handled.Issues
Closes #5232
Reminders
tox -e linters.feat:,fix:,ref:,meta:)