perf: send only the current round's output at tool-call boundaries - #29854
Open
Classic298 wants to merge 1 commit into
Open
perf: send only the current round's output at tool-call boundaries#29854Classic298 wants to merge 1 commit into
Classic298 wants to merge 1 commit into
Conversation
Every tool-call round re-sent the whole assistant message so far, all earlier rounds' text, reasoning, calls and tool results, twice over the websocket as a chat:completion snapshot, so agentic runs grew with the square of the round count: 45 MB Redis publishes stalling the instance for 400 ms (open-webui#29833). The two boundary emits now send only the items of the response that just ended, one response.output_item.done event each with its absolute index, which the browser and the channel emitter already apply. Earlier rounds never change after their own boundary, so they are never re-sent; the final done snapshot, the approval pauses and the code-interpreter loop keep their snapshots. A continuation round's response.completed drops its round-only output, which wiped the earlier rounds from the channel emitter state, and the channel emitter now writes tool items before the first text token, as the snapshots used to give it. Boundary payload, 3 tool rounds with 4 calls, chat-completions upstream: | tool result | before | after | | --- | --- | --- | | 4 KiB | 51.6 KiB | 18.7 KiB | | 400 KiB | 4.30 MiB | 1.18 MiB | Fixes open-webui#29833
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every tool-call round re-sent the whole assistant message so far, all earlier rounds' text, reasoning, calls and tool results, twice over the websocket as a chat:completion snapshot, so agentic runs grew with the square of the round count: 45 MB Redis publishes stalling the instance for 400 ms (#29833).
The two boundary emits now send only the items of the response that just ended, one response.output_item.done event each with its absolute index, which the browser and the channel emitter already apply. Earlier rounds never change after their own boundary, so they are never re-sent; the final done snapshot, the approval pauses and the code-interpreter loop keep their snapshots. A continuation round's response.completed drops its round-only output, which wiped the earlier rounds from the channel emitter state, and the channel emitter now writes tool items before the first text token, as the snapshots used to give it.
Boundary payload, 3 tool rounds with 4 calls, chat-completions upstream:
Fixes #29833
Contributor License Agreement