Codestin Search App

theomonnom · 2026-06-12T17:52:03Z

A run with an output_type ends with final_output=None whenever the model finishes its turn in prose instead of calling the task's completion tool — common with chatty models, and currently surfaced as a generic RuntimeError that callers can't distinguish or recover from.

Following pydantic-ai's output-tool semantics:

New output_options on run() (an options TypedDict in the style of keyterm_options/expressiveness): when the run ends without its output_type, the session re-prompts in the same context as a per-turn system message (max_retries, default 2) before raising; retry_instructions overrides the built-in retry prompt.
```
result = await sess.run(
    user_input=...,
    output_type=SummarizeOutput,
    output_options={"max_retries": 2, "retry_instructions": "Call submit_analysis, nothing else."},
)
```
A distinct UnexpectedModelBehavior (exported from livekit.agents, same name as pydantic-ai's) replaces the generic RuntimeError once the budget is exhausted, so callers can catch the failure specifically.

Defaults convert the dominant failure (model summarizes in prose) into a recovered run. Unit tests cover recovery, the prompt override, and exhaustion via FakeLLM.

🤖 Generated with Claude Code

…Error when exhausted

devin-ai-integration

Devin Review found 1 new potential issue.

devin-ai-integration · 2026-07-07T05:58:37Z

+    if session is not None:
+        _SessionMockTools.setdefault(session, {})[agent] = dict(mocks)
+        return None


🔴 Session-scoped mock tools are silently ignored during tool execution

Mock tools registered via the session-scoped path are stored (_SessionMockTools.setdefault(session, {})[agent] = dict(mocks) at livekit-agents/livekit/agents/voice/run_result.py:1154) but never consulted when tools are actually executed, so they have no effect.

Impact: Users calling mock_tools(MyAgent, {...}, session=session) will see no error but their mocks will never intercept tool calls.

Tool execution only reads from the context-variable store, not the session store

The tool execution code in livekit-agents/livekit/agents/voice/generation.py:856 only reads from _MockToolsContextVar:

mock_tools: dict[str, Callable] = _MockToolsContextVar.get({}).get( type(session.current_agent), {} )

It never reads from _SessionMockTools. The _SessionMockTools dictionary is only written to (at run_result.py:1154) and declared (at run_result.py:1114), but has zero read sites anywhere in the codebase. The docstring at run_result.py:1150-1151 even describes the intended precedence ("the context-manager mocks take precedence over the session ones"), but this fallback lookup was never implemented.

To fix this, generation.py:856 needs to also check _SessionMockTools.get(session, {}).get(type(session.current_agent), {}) as a fallback when _MockToolsContextVar has no mocks for the agent type.

Was this helpful? React with 👍 or 👎 to provide feedback.

# Conflicts: # livekit-agents/livekit/agents/voice/__init__.py

devin-ai-integration

Devin Review found 1 new potential issue.

devin-ai-integration · 2026-07-07T06:07:45Z

+        except Exception:
+            # an unhandled exception here would leave the run future
+            # unresolved; fall through to UnexpectedModelBehavior instead
+            return False


🔍 Exception swallowing in _maybe_retry_output could hide root causes

The broad except Exception at run_result.py:293 catches all errors from generate_reply and returns False, causing the caller to fall through to UnexpectedModelBehavior. While the comment explains the rationale (preventing an unresolved future), this means errors like RuntimeError('AgentSession isn't running') or RuntimeError('AgentSession is closing') from agent_session.py:1297-1310 are silently swallowed. The user sees UnexpectedModelBehavior with no indication that a retry was attempted and failed. Consider logging the caught exception at warning/debug level before returning False.

Was this helpful? React with 👍 or 👎 to provide feedback.

voice: retry a run that ends without its output_type, raise RunOutput…

972a8bd

…Error when exhausted

theomonnom requested a review from a team as a code owner June 12, 2026 17:52

theomonnom added 2 commits June 12, 2026 10:57

voice: overridable output retry instructions

51fe116

voice: fold retry config into output_retries

ef284b8

This comment was marked as resolved.

Sign in to view

theomonnom added 2 commits June 12, 2026 11:08

voice: the task owns its output retry instructions

b71f13f

voice: output retry prompt configured on the session

f006d67

This comment was marked as resolved.

Sign in to view

theomonnom added 4 commits June 12, 2026 12:01

voice: group structured-output behavior into output_options

b7389bf

voice: rename RunOutputError to UnexpectedModelBehavior

9238266

voice: output_options moves to run()

50561cc

voice: retry only the no-output case, fold retry tests

e712c0d

This comment was marked as resolved.

Sign in to view

theomonnom added 4 commits June 12, 2026 12:54

fix formatting

5038667

voice: retry via per-turn system instructions, broaden retry guard

3a9cf15

voice: rename retries to max_retries

4538b33

voice: default output max_retries to 2

d9d91c9

This comment was marked as resolved.

Sign in to view

theomonnom added 2 commits June 12, 2026 15:58

drop the retry test file

0d4fd62

remove unrelated local changes

86acdca

theomonnom force-pushed the theo/output-retries branch from c63462a to 86acdca Compare June 12, 2026 23:00

Bobronium approved these changes Jun 12, 2026

View reviewed changes

theomonnom added 3 commits June 12, 2026 16:12

voice: UnexpectedModelBehavior extends RuntimeError

a6b7fa5

voice: rename RunOutputOptions (class only)

85b21fd

sort imports

6fce0bf

chenghao-mou reviewed Jun 14, 2026

View reviewed changes

Comment thread livekit-agents/livekit/agents/voice/agent_session.py Outdated

Comment thread livekit-agents/livekit/agents/voice/agent_session.py Outdated

Comment thread livekit-agents/livekit/agents/voice/run_result.py Outdated

theomonnom added 2 commits July 2, 2026 10:47

voice: run() output_options accepts None to disable retries

d162cdc

voice: resolve output_options defaults in one place (RunResult)

758de45

devin-ai-integration Bot reviewed Jul 7, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into theo/output-retries

0a1ab63

# Conflicts: # livekit-agents/livekit/agents/voice/__init__.py

voice: soften default output retry prompt

00c1c7d

devin-ai-integration Bot reviewed Jul 7, 2026

View reviewed changes

theomonnom merged commit bb62c33 into main Jul 7, 2026
25 of 26 checks passed

theomonnom deleted the theo/output-retries branch July 7, 2026 06:11

rosetta-livekit-bot Bot mentioned this pull request Jul 7, 2026

voice: retry missing run outputs livekit/agents-js#1978

Merged

toubatbrian mentioned this pull request Jul 7, 2026

feat(voice): add session-scoped mockTools for testing livekit/agents-js#1980

Open

cursor Bot mentioned this pull request Jul 8, 2026

docs: daily engineering digest 2026-07-08 Seventhen/agents#24

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

voice: output retries for run(output_type=...)#6080

voice: output retries for run(output_type=...)#6080
theomonnom merged 22 commits into
mainfrom
theo/output-retries

theomonnom commented Jun 12, 2026 •

edited

Loading

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Jul 7, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

theomonnom commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

theomonnom commented Jun 12, 2026 •

edited

Loading

devin-ai-integration Bot Jul 7, 2026 •

edited

Loading