fix: store tool result images as file references - #29765
Conversation
A tool that returned an image wrote the entire base64 data URI into the chat, and stored it twice: once on the message row and again in the chat's own history map. A single 2 MB image cost 5.6 MB of database, and the same payload was sent to the browser with every chat load, so tab memory grew by roughly the image size per image and only came back when the tab was closed. Long image-heavy chats reached tens of megabytes of JSON. The image is now uploaded once and the chat keeps a short file reference, the same way an uploaded image is already stored. The base64 is rebuilt only while a provider request is being assembled, so what the model receives is unchanged. Measured on a 2 MB tool image, with an identical request reaching the provider in both cases: | | stored bytes | base64 copies | | --- | --- | --- | | before | 5,594,322 | 2 | | after | 1,962 | 0 | Fixes open-webui#29761
|
Test result: fix confirmed working — built from fix/tool-image-file-refs (PR head 7c7a14e, base dev) and tested end-to-end. Setup: isolated test instance — fresh SQLite volume (no Postgres/Redis), official ghcr.io/open-webui/open-terminal container, port-isolated from my production stack. Vision-capable model via SGLang (450,560-token KV pool), compaction enabled at token_threshold = 350,000 in the Admin panel. Control run: the exact same 0.11.3 image digest as my production instance, on a fresh SQLite volume, sharing nothing with production except the engine — which rules the backend out as a factor. Procedure: fresh chat → agent views a 3,092,378-byte JPG via Open Terminal read_file → multiple follow-up turns (ping/pong ×5). Results (same 3MB-image repro in all three environments): Production incident (0.11.3, Postgres): chat JSON 4,201,936 bytes; auto-compaction fired (dropped=2, kept=3); meter ~1M "tokens" estimate; vision worked. The unfixed builds compact identically on Postgres and on SQLite (same compact_messages_for_request log line, same dropped=2 kept=3 signature) — the bug is backend-independent, and the PR eliminates it on both paths. |
A tool that returned an image wrote the entire base64 data URI into the chat, and stored it twice: once on the message row and again in the chat's own history map. A single 2 MB image cost 5.6 MB of database, and the same payload was sent to the browser with every chat load, so tab memory grew by roughly the image size per image and only came back when the tab was closed. Long image-heavy chats reached tens of megabytes of JSON.
The image is now uploaded once and the chat keeps a short file reference, the same way an uploaded image is already stored. The base64 is rebuilt only while a provider request is being assembled, so what the model receives is unchanged.
Measured on a 2 MB tool image, with an identical request reaching the provider in both cases:
Fixes #29761
Contributor License Agreement