fix(rag): scope embedder unload to keep chat model resident (#1544)#1586
Conversation
ReviewSummaryClean, well-scoped fix that does exactly what the title says — Issues🟡 Cold-start scoped unload depends on Lemonade On a cold start the embedder slot is empty, yet Can you confirm the M2 Max eval actually exercised the empty-embedder-slot path (first RAG call after a fresh server start, nothing pre-loaded)? If Lemonade 4xx's on "unload a non-resident model" on any supported version, this turns a previously-tolerated no-op into a hard cold-start failure. If you've already seen it return success there, a one-line note in the PR is enough — no code change needed. Strengths
VerdictApprove with suggestions — no blocking code changes; just confirm the cold-start (embedder-not-yet-loaded) unload was exercised on real hardware, since the mocked test can't cover it. Nice, surgical fix. |
|
Good catch — confirmed empirically. On a running Lemonade server, a scoped unload of a non-resident model returns Pushed a fix (
This stays within the no-silent-fallbacks rule — it's the explicit opt-in exception (one named benign condition), not a blanket swallow. |
itomek
left a comment
There was a problem hiding this comment.
Scoped unload is correct; cold-start 404 handled via opt-in ignore_if_not_loaded. LGTM.
Generated by Claude Code
Summary
_load_embedder()now issues a model-scoped Lemonade/unload(embedder only) instead of a global one, so the chat model is no longer evicted when RAG first loads its embedder.Why
When the doc agent did its first RAG work,
_load_embedder()called a global/unloadto force a fresh embedder load. That global unload also evicted the chat model, so the next generation turn cold-reloaded it (~100–150 s on the reporter's M2 Max) — slow enough to hit Lemonade's 120 s read-timeout and time out an entire eval scenario (smart_discoveryproduced 0 judged turns). Lemonade can hold the chat model and embedder at once; the code only ever meant to refresh the embedder. The chat-model reload was pure collateral damage.Linked issue
Closes #1544
Changes
lemonade_client.py—unload_model()gains an optionalmodel_name; when given it sends{"model_name": ...}so Lemonade unloads only that model, andself.modelis cleared only on a global unload or when it matches. No arg = global, preserving every existing caller (chat preflight, init, UI eviction).base_client.py/providers/lemonade.py— widen theunload_modelsignature to match and pass through.rag/sdk.py—_load_embedder()does a scoped unload of the embedder then reloads it; removed the two broadexcept Exception: … continueswallows so a real embedder failure surfaces instead of silently pretending RAG is ready (no-silent-fallbacks). Stale docstring corrected./unloadpayload test (test_lemonade_client.py); RAG cold-start regression asserting the embedder unload is scoped and the chat model is never globally unloaded (test_rag.py).Test plan
python util/lint.py --all— black + isort clean.pytest tests/test_rag.py tests/test_lemonade_client.py tests/unit/test_chat_preflight.py -k "not Integration"— 118 passed. (TheTestLemonadeClientIntegrationcases require a Windows/NPU Hybrid model and only run on that hardware.)test_load_embedder_scoped_unload_keeps_chat_model_resident(test_rag.py),test_unload_model_scoped_vs_global(test_lemonade_client.py)._load_embedderlogs"chat model left resident";smart_discoveryis now judged instead of 0-turn timeout; per-turn TTFT stays ≤60 s (e.g. a 3.3 s TTFT immediately after an embedder load) with no ~100 s chat-model cold reload. Headline scorecard vs the committed baseline is not comparable on this hardware — the baseline was generated on ~6.5× faster (NPU) hardware (122.9 tok/s / 0.31 s TTFT vs 18.9 tok/s / 20.8 s TTFT).Checklist
Closes #1544).python util/lint.py --all;pytest tests/test_rag.py tests/test_lemonade_client.py tests/unit/test_chat_preflight.py -k "not Integration"). Fulltests/unit/is not run becausetests/unit/test_file_tools.pyhangs — see feat(connectors): per-agent MCP tool-visibility activations (#1005) #1219.