Thanks to visit codestin.com
Credit goes to github.com

Skip to content

fix(rag): scope embedder unload to keep chat model resident (#1544)#1586

Merged
itomek merged 5 commits into
amd:mainfrom
alexey-tyurin:fix/1544-rag-embedder-scoped-unload
Jun 11, 2026
Merged

fix(rag): scope embedder unload to keep chat model resident (#1544)#1586
itomek merged 5 commits into
amd:mainfrom
alexey-tyurin:fix/1544-rag-embedder-scoped-unload

Conversation

@alexey-tyurin

Copy link
Copy Markdown
Contributor

Summary

_load_embedder() now issues a model-scoped Lemonade /unload (embedder only) instead of a global one, so the chat model is no longer evicted when RAG first loads its embedder.

Why

When the doc agent did its first RAG work, _load_embedder() called a global /unload to force a fresh embedder load. That global unload also evicted the chat model, so the next generation turn cold-reloaded it (~100–150 s on the reporter's M2 Max) — slow enough to hit Lemonade's 120 s read-timeout and time out an entire eval scenario (smart_discovery produced 0 judged turns). Lemonade can hold the chat model and embedder at once; the code only ever meant to refresh the embedder. The chat-model reload was pure collateral damage.

Linked issue

Closes #1544

Changes

  • lemonade_client.pyunload_model() gains an optional model_name; when given it sends {"model_name": ...} so Lemonade unloads only that model, and self.model is cleared only on a global unload or when it matches. No arg = global, preserving every existing caller (chat preflight, init, UI eviction).
  • base_client.py / providers/lemonade.py — widen the unload_model signature to match and pass through.
  • rag/sdk.py_load_embedder() does a scoped unload of the embedder then reloads it; removed the two broad except Exception: … continue swallows so a real embedder failure surfaces instead of silently pretending RAG is ready (no-silent-fallbacks). Stale docstring corrected.
  • Tests — scoped-vs-global /unload payload test (test_lemonade_client.py); RAG cold-start regression asserting the embedder unload is scoped and the chat model is never globally unloaded (test_rag.py).

Test plan

  • python util/lint.py --all — black + isort clean.
  • pytest tests/test_rag.py tests/test_lemonade_client.py tests/unit/test_chat_preflight.py -k "not Integration"118 passed. (The TestLemonadeClientIntegration cases require a Windows/NPU Hybrid model and only run on that hardware.)
  • New regression tests pass: test_load_embedder_scoped_unload_keeps_chat_model_resident (test_rag.py), test_unload_model_scoped_vs_global (test_lemonade_client.py).
  • On-machine eval (M2 Max): _load_embedder logs "chat model left resident"; smart_discovery is now judged instead of 0-turn timeout; per-turn TTFT stays ≤60 s (e.g. a 3.3 s TTFT immediately after an embedder load) with no ~100 s chat-model cold reload. Headline scorecard vs the committed baseline is not comparable on this hardware — the baseline was generated on ~6.5× faster (NPU) hardware (122.9 tok/s / 0.31 s TTFT vs 18.9 tok/s / 20.8 s TTFT).

Checklist

  • I have linked a GitHub issue above (Closes #1544).
  • I have described why this change is being made, not just what changed.
  • I have run linting and selected unit tests locally (python util/lint.py --all; pytest tests/test_rag.py tests/test_lemonade_client.py tests/unit/test_chat_preflight.py -k "not Integration"). Full tests/unit/ is not run because tests/unit/test_file_tools.py hangs — see feat(connectors): per-agent MCP tool-visibility activations (#1005) #1219.
  • I have updated documentation if user-visible behavior changed — N/A, internal model-lifecycle fix with no user-facing surface change.

@github-actions github-actions Bot added rag RAG system changes llm LLM backend changes tests Test changes performance Performance-critical changes labels Jun 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Review

Summary

Clean, well-scoped fix that does exactly what the title says — unload_model() gains an optional model_name so RAG's embedder refresh stops evicting the co-resident chat model. The signature is widened consistently across the ABC, the Lemonade provider, and the concrete client; the no-arg global form is preserved so every existing caller (init, chat preflight, UI eviction) is untouched. Removing the two except Exception: … continue swallows in _load_embedder is the right call under the no-silent-fallbacks rule, and tests cover both the scoped-vs-global payload and the cold-start regression. The one thing worth confirming before merge is below.

Issues

🟡 Cold-start scoped unload depends on Lemonade /unload being idempotent for a non-resident model (src/gaia/rag/sdk.py:492)

On a cold start the embedder slot is empty, yet _load_embedder() now issues unload_model(self.config.embedding_model) against a model that isn't loaded — and with the old try/except … continue gone, any error from that call propagates and breaks RAG entirely. The new regression test mocks unload_model, so it proves the call is scoped but can't prove a real Lemonade returns success when asked to unload a model it isn't holding. Your on-machine eval is the only evidence this case is safe.

Can you confirm the M2 Max eval actually exercised the empty-embedder-slot path (first RAG call after a fresh server start, nothing pre-loaded)? If Lemonade 4xx's on "unload a non-resident model" on any supported version, this turns a previously-tolerated no-op into a hard cold-start failure. If you've already seen it return success there, a one-line note in the PR is enough — no code change needed.

Strengths

  • Signature widened everywhere it needs to bebase_client.py, providers/lemonade.py, and lemonade_client.py all match, and the model is None or self.model == model_name guard correctly scopes the self.model clear so a scoped unload of the embedder doesn't wrongly null out the chat model handle.
  • Removing the swallows is correct, not just convenient — the old code set self.embedder = self.llm_client even after a load failure, deferring the error to encode time with a worse message; failing fast at load time is the better behavior and aligns with CLAUDE.md's no-silent-fallbacks rule.
  • Tests pin the actual regressiontest_unload_model_scoped_vs_global asserts the request body shape directly, and assert_called_once_with(config.embedding_model) doubly guards that the bare global form is never used on the RAG path. Docstrings and the corrected _load_embedder comment explain the why ([Bug][P0]: RAG embedder load issues a global /unload that evicts the chat model → 100s+ cold reloads & eval timeout #1544) without rotting.

Verdict

Approve with suggestions — no blocking code changes; just confirm the cold-start (embedder-not-yet-loaded) unload was exercised on real hardware, since the mocked test can't cover it. Nice, surgical fix.

kovtcharov-amd
kovtcharov-amd previously approved these changes Jun 11, 2026
@alexey-tyurin

Copy link
Copy Markdown
Contributor Author

Good catch — confirmed empirically. On a running Lemonade server, a scoped unload of a non-resident model returns 404 {"error":"Model not loaded: ..."}, so with the try/except gone this would have hard-failed RAG on a cold start. You also correctly guessed my eval never exercised it: nomic was already co-resident before the run, so every scoped unload hit a loaded model (200), never the cold 404.

Pushed a fix (fix(rag): don't break RAG cold start on empty embedder slot):

  • unload_model() gains a keyword-only ignore_if_not_loaded=False. When set, it treats only Lemonade's 404 "Model not loaded" as a no-op ({"status": "not_loaded"}); a global unload, auth error, 500, or outage still raises. Default stays strict, so init / chat-preflight / UI-eviction callers are untouched.
  • _load_embedder() opts in with ignore_if_not_loaded=True — it reloads on the next line, so an empty slot is benign here.
  • Added test_unload_model_ignore_if_not_loaded covering all three branches: 404-tolerated (flag on), 404-raises (flag off / default), and 500-still-raises (flag on). RAG cold-start regression updated for the new kwarg.

This stays within the no-silent-fallbacks rule — it's the explicit opt-in exception (one named benign condition), not a blanket swallow.

@itomek itomek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scoped unload is correct; cold-start 404 handled via opt-in ignore_if_not_loaded. LGTM.


Generated by Claude Code

@itomek itomek added this pull request to the merge queue Jun 11, 2026
Merged via the queue into amd:main with commit f06763c Jun 11, 2026
27 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

llm LLM backend changes performance Performance-critical changes rag RAG system changes tests Test changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][P0]: RAG embedder load issues a global /unload that evicts the chat model → 100s+ cold reloads & eval timeout

3 participants