Thanks to visit codestin.com
Credit goes to github.com

Skip to content

fix(installer): detect and auto-start modern Lemonade in gaia init#1940

Merged
kovtcharov-amd merged 12 commits into
mainfrom
fix/issue-316-lemonade-upgrade-modern-cli
Jul 8, 2026
Merged

fix(installer): detect and auto-start modern Lemonade in gaia init#1940
kovtcharov-amd merged 12 commits into
mainfrom
fix/issue-316-lemonade-upgrade-modern-cli

Conversation

@itomek

@itomek itomek commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Closes #316

Modern Lemonade (10.7+, including the 10.8.1 GAIA installs) removed the lemonade-server CLI — the server is now LemonadeServer.exe --silent on Windows (ctx size via the LEMONADE_CTX_SIZE env var) and a systemd lemond service on Linux. GAIA still hard-coded lemonade-server serve everywhere, so on a modern install gaia init reported a working Lemonade as not installed, and after any install it told the user to start the server by hand (#316's "never started it") — interactive mode could never auto-start because of an indentation bug that made the manual prompt unconditional. Now detection finds modern and legacy installs, and gaia init auto-starts the server in both interactive and CI modes, falling back to a prompt (showing the correct per-tooling command) only if auto-start fails. (#316's uninstall failure was previously fixed by #478; verified still working — the registry ProductCode lookup finds modern installs.)

All changes route through one new stdlib-only primitive, gaia.llm.lemonade_launcher (resolve_lemonade / get_installed_version / build_start_command), consumed by the installer and LemonadeClient — legacy lemonade-server behavior is pinned byte-identical by tests, and CI's LEMONADE_SERVER_PATH override is honored first.

Evidence (real-world, Windows 11 + Ryzen AI box, Lemonade 10.7.0)

Tested by deploying this branch to the machine's editable GAIA install and running the real CLI over SSH.

Detection — before: main's check is shutil.which("lemonade-server"), which is NOT ON PATH on this modern install → "not installed" → forced reinstall over a working setup. After:

resolve_lemonade: found=True kind=modern
  client_path     = C:\Users\...\AppData\Local\lemonade_server\bin\lemonade.exe
  server_launcher = C:\Users\...\AppData\Local\lemonade_server\bin\LemonadeServer.exe
check_installation: installed=True version=10.7.0
build_start_command: argv=['C:\\Users\\...\\LemonadeServer.exe', '--silent'] env={'LEMONADE_CTX_SIZE': '32768'}

Auto-start — LemonadeServer.exe stopped, then interactive gaia init --profile minimal --skip-models (stdin-driven, yes not set):

Step 1/4: Checking Lemonade Server installation...
   ✓ Lemonade Server found: v10.7.0
   Upgrade to v10.8.1? [y/N]:    ✓ Continuing with v10.7.0

Step 2/4: Checking Lemonade Server...
   Lemonade Server is not running — starting it...
   | INFO | ..._auto_start_server | Starting Lemonade Server: C:\Users\...\LemonadeServer.exe --silent
   ✓ Server started and ready (waited 2s)

Step 4/4: Verifying setup...
   ✓ Server health: OK
   ✓ Context size verified: 32768 tokens
GAIA initialization complete!   (exit code 0)

No "Please start Lemonade Server" prompt appeared; LemonadeServer.exe alive (pid captured) and /api/v1/health = 200 afterwards. The ctx-size env path is proven end-to-end by the Step-4 32768-token model preload.

Test plan

  • python -m pytest tests/unit/test_init_command.py tests/unit/installer/ tests/unit/test_lemonade_launcher.py tests/test_lemonade_client.py -q → 245 passed; the 4 failures are pre-existing environment artifacts (3 tests short-circuited by a live local Lemonade on :13305 — fail identically on main — plus 1 live-server integration test), none in changed code
  • python util/lint.py --all → exit 0
  • Real-world Windows validation (above): modern detection + interactive auto-start + health + ctx preload
  • CI green (note: CI workflows start Lemonade externally, so the auto-start path itself is exercised by the real-world run above, not CI)
Reviewer notes — bundled behavior deltas & rationale
  • Tests were written first as a fixed contract (28 new/updated across 4 files), including falsifying assertions: input() never called on auto-start success, Popen env must retain the parent environment (merge, not replace), legacy argv pinned byte-identical, LEMONADE_SERVER_PATH honored before any PATH probe with modern-beats-stale-legacy precedence.
  • launch_server() now skips kill_process_on_port when a healthy server is already listening (never restarts a tray/systemd-managed server the user didn't ask to restart), raises an actionable error when no tooling is found, and its background="terminal" branch is argv-only (CREATE_NEW_CONSOLE) instead of a shell=True string.
  • check_installation() no longer returns installed=False on a version-probe timeout (that could trigger a pointless reinstall); a found install with a failed probe stays installed=True with a diagnostic error. LemonadeInfo shape and find_product_code() are unchanged (all five callers unaffected).
  • Windows legacy launch_server now passes --no-tray (aligning with what gaia init always did); --log-level is appended for legacy only (the modern launcher has no such flag — logged and ignored, not silently dropped).
  • User-facing "start/restart with:" hints render the resolved tooling's real command (env prefix + argv) instead of the removed lemonade-server CLI.
  • _find_lemonade_server() retained as a compat surface (delegates to the primitive) though it has no remaining callers.
  • Modern-Linux auto-start (systemctl --user start lemond) is best-effort and unit-tested only — the daemon is normally already running under systemd.
  • Follow-up (tracked separately): the PowerShell-side detection in .github/actions/install-lemonade/action.yml and installer/scripts/*.ps1 duplicates this logic and has one stale search path; consolidating PS/Python detection is out of scope here.

Tomasz Iniewicz added 9 commits July 6, 2026 14:36
Add failing tests for the not-yet-implemented modern-Lemonade
detection primitive ahead of the implementation.
Rewire check_installation tests onto resolve_lemonade/get_installed_version
(AC1/AC2), and pin the corrected _ensure_server_running behavior where
auto-start is attempted before the interactive "Please start Lemonade
Server" prompt in both CI and interactive modes, with a fail-fast
guarantee that a failed auto-start never hangs on input() (AC5/AC6).

Also extend the #839 ctx-size regression guard to cover the modern
LEMONADE_CTX_SIZE env-var path alongside the existing legacy --ctx-size
argv path, so a future change can't silently drop ctx-size propagation
for modern tooling while the legacy assertions keep passing.

3 pre-existing test failures unrelated to this change (already broken
on main, not introduced here):
TestEnsureLemonadeInstalledSkipsWhenPresent::test_older_version_below_minimum_triggers_install_in_ci,
TestLegacyFallback::test_not_installed_on_linux_proceeds_to_install_via_ppa,
TestLegacyFallback::test_not_installed_on_windows_proceeds_to_download_and_install
Pin that launch_server() builds its Popen argv/env from
resolve_lemonade()/build_start_command() instead of the hardcoded
["lemonade-server", "serve"] literal, with a byte-identical regression
guard for the legacy argv shape and a guard that kill_process_on_port
is skipped when health_check() already reports the server healthy at
entry.
Pin at the Popen call sites that spec.env is MERGED into the parent
environment ({**os.environ, **spec.env}) rather than replacing it — an
implementation passing env=spec.env alone would drop PATH/LOCALAPPDATA
and break LemonadeServer.exe while still passing the previous
assertions. Sentinel env var seeded before the call and asserted
retained, alongside PATH, in the three modern-path launch tests.

Also add the direct build_start_command-level Windows-legacy test
asserting --no-tray in argv, pin platform in the non-Windows legacy
test for cross-platform determinism, and replace a no-op tuple
assertion with a real assertTrue.
Modern Lemonade (10.7/10.8) removed the lemonade-server CLI; the server
is LemonadeServer.exe --silent on Windows (ctx via LEMONADE_CTX_SIZE
env) and the lemond systemd unit on Linux. Add the stdlib-only shared
primitive (resolve_lemonade / get_installed_version /
build_start_command) that the installer and runtime client will adopt
instead of hard-coding lemonade-server.

Precedence: LEMONADE_SERVER_PATH env override (verbatim, no PATH
search) -> modern by canonical path (wins over a stale legacy binary on
PATH) -> legacy which(lemonade-server|lemonade-server-dev). Legacy argv
stays byte-identical (serve --ctx-size, + --no-tray on Windows).

All 12 contract tests in tests/unit/test_lemonade_launcher.py pass;
includes black formatting fixes to the contract test files.
check_installation() previously probed only shutil.which("lemonade-server"),
so a modern Lemonade install (10.7+, no lemonade-server CLI) reported
installed=False and gaia init tried to reinstall over a working setup.
Delegate detection to the shared resolve_lemonade()/get_installed_version()
primitive: modern installs are found at their canonical path, legacy
installs keep working unchanged, and a found install whose version probe
fails still reports installed=True with a diagnostic error (mirroring the
old failed-to-get-version branch). LemonadeInfo shape and find_product_code
are untouched — all five callers are unaffected.
…316)

Interactive `gaia init` previously never auto-started the server — an
indentation bug made the "Please start Lemonade Server" prompt block run
unconditionally, blocking on input() even in flows that could have
started the server themselves. Restructure _ensure_server_running():
auto-start is attempted first in BOTH CI (yes=True) and interactive
modes, success returns before any prompt, and the manual prompt is the
only remaining fall-through (interactive auto-start failure). CI mode
returns False on a failed auto-start without ever touching input().

Auto-start now routes through resolve_lemonade()/build_start_command()
in a new _auto_start_server() helper — the raw
[path, "serve", "--no-tray", --ctx-size] literal is gone, so modern
tooling (LemonadeServer.exe --silent + LEMONADE_CTX_SIZE env, merged
into the parent environment) starts correctly, including under CI's
LEMONADE_SERVER_PATH override which resolve_lemonade() now honors in
one place instead of a parallel os.environ.get. The interactive
fallback prompt renders the exact start command for the resolved
tooling; _find_lemonade_server() delegates to the shared primitive.
)

launch_server() hardcoded ["lemonade-server", "serve"], which no longer
exists in modern Lemonade (10.7+) — auto-start was broken for every
modern install. Route argv/env through
resolve_lemonade()/build_start_command(): modern gets
LemonadeServer.exe --silent with LEMONADE_CTX_SIZE merged into the
parent environment, legacy keeps its exact argv. A missing install now
raises an actionable LemonadeClientError instead of a bare
FileNotFoundError from Popen.

Also: skip kill_process_on_port() when health_check() already reports
OK at entry (never restart a healthy tray/systemd-managed server the
user didn't ask to restart); convert the "terminal" branch from a
shell=True f-string to argv-only Popen with CREATE_NEW_CONSOLE so a
resolved path never passes through a shell string; log_level appends
--log-level only for legacy tooling (modern launcher has no such flag —
logged at debug and ignored); _check_lemonade_installed() and
get_lemonade_version() use the primitive; user-facing "restart with"
hints render the resolved tooling's real start command instead of the
removed lemonade-server CLI (test assertion widened to accept the
modern LEMONADE_CTX_SIZE form).
…nned

TestLaunchServerCtxSize asserted the removed hardcoded lemonade-server
argv and hit the real network (unmocked health_check), so results
flipped depending on whether a local server happened to be running and
which tooling the host had. Pin resolve_lemonade to legacy (the argv
path under test — modern env-var ctx is covered by the #316 contract
tests) and stub health_check to fail so the already-healthy launch
guard never short-circuits the assertion.
@github-actions github-actions Bot added llm LLM backend changes tests Test changes performance Performance-critical changes labels Jul 6, 2026
@itomek itomek self-assigned this Jul 6, 2026
@itomek itomek marked this pull request as ready for review July 6, 2026 20:17
@itomek itomek requested a review from kovtcharov-amd as a code owner July 6, 2026 20:17
@itomek itomek enabled auto-merge July 6, 2026 20:17
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Verdict: Approve with suggestions — no blocking issues.

This fixes the real #316 breakage: modern Lemonade (10.7+) dropped the lemonade-server CLI, so gaia init reported a working install as "not installed" and never auto-started it. The change routes all detection/launch through one new stdlib primitive and makes gaia init auto-start the server in both CI and interactive modes, with a corrected manual-prompt fallback. It's test-first with genuinely strong coverage — precedence, both tooling kinds, env-merge, the CI no-hang case, and the don't-kill-a-healthy-server guard are all pinned.

The only things worth a look are minor edge-cases, none of which affect the tested path: an env-override to a modern Linux binary gets silently ignored in favor of systemctl, and the old finder method is now dead code kept for compat. Details below.

🔍 Technical details

🟢 Minor

1. LEMONADE_SERVER_PATH override silently dropped for a modern non-.exe path (src/gaia/llm/lemonade_launcher.py:809)
resolve_lemonade() honors the override and sets server_launcher to it, but build_start_command()'s modern branch dispatches on the .exe suffix — so a modern Linux override (e.g. /usr/bin/lemonade) falls through to systemctl --user start lemond, ignoring the path the caller explicitly set. CI is unaffected (it uses the legacy dev CLI, classified legacy, which does honor the path), so this is edge-only — but it's the kind of silent fallback CLAUDE.md discourages. Worth either honoring the override path directly or raising when a modern override can't be launched as given.

2. _find_lemonade_server() is now dead code (src/gaia/installer/init_command.py:1043)
No remaining callers in src/ or tests/ (tests that used to patch it now patch resolve_lemonade). The reviewer notes call this out as a retained compat surface — fine to keep, but a one-line docstring note that it's compat-only (or removal) would save the next reader a grep.

3. Manual-prompt hint uses POSIX VAR=val cmd env-prefix syntax (src/gaia/installer/init_command.py:1232)
Harmless because this branch only renders on non-Windows, but the env_prefix string wouldn't be copy-pasteable on Windows if that guard ever changes. Not worth a change now; just noting the coupling.

Strengths

  • Test-first, high-value coverage. The falsifying assertions are the good kind: input() never called on auto-start success, env must be {**os.environ, **spec.env} (PATH/sentinel retained), legacy argv pinned byte-identical, and the CI must-never-hang timeout path. This is exactly the "assert the shape of the outgoing call" discipline CLAUDE.md asks for.
  • launch_server Windows path hardened — the old background="terminal" branch built a shell=True command string; it's now argv-only with CREATE_NEW_CONSOLE, removing a shell-injection surface on a resolved path.
  • check_installation no longer flips to installed=False on a version-probe timeout — a found install with a failed probe stays installed=True, avoiding a pointless reinstall over a working setup.
  • Clean single-primitive design — stdlib-only, no import cycle (installer → llm), CI override checked before any PATH probe.

An explicit LEMONADE_SERVER_PATH override classified as modern but not a
Windows .exe (e.g. a Linux daemon path) was silently rerouted to
'systemctl --user start lemond', ignoring the binary the caller named.
Track resolution provenance on LemonadeTooling (source: env|probe) and
launch an env-override binary verbatim; probe-resolved modern Linux keeps
the best-effort systemctl start. Also note _find_lemonade_server() is a
compat-only surface with no in-tree callers.
@itomek

itomek commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed the review's minor items in 807b3b5:

  • Update installer and workflows/actions for CI/CD #1 (env-override silently dropped): LemonadeTooling now carries resolution provenance (source: env|probe); a modern-classified LEMONADE_SERVER_PATH override that isn't a Windows .exe is launched verbatim instead of being rerouted to systemctl. Two new contract tests pin the override fast-path and guard that probe-resolved modern Linux still uses the best-effort systemctl start.
  • Use public Lemonade hybrid installer #2 (dead finder): _find_lemonade_server() docstring now states it's a compat-only surface with no in-tree callers.
  • Update Driver Check #3 (POSIX env-prefix in the hint): no change — that branch only renders on non-Windows, as the review noted.

Also for the record: the earlier "Example Agents Unit Tests" red was infrastructure (a 503 from download.pytorch.org during env setup — no tests ran), and the Merge branch 'main' commit on this branch was unintended tooling noise — happy to drop it with a force-push if a clean history is preferred; content-wise it's inert and the squash on merge makes it moot.

@itomek

itomek commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

📸 Real-world proof (Windows 11 and macOS · AMD Ryzen AI · Lemonade 10.7.0)

Live over SSH against the current PR tip `807b3b53`. Terminal output rendered from the actual run (raw text is in the PR description as verifiable source).

issue #316 real-world proof: modern detection + interactive auto-start

Top panel — detection: where.exe lemonade-servernot found (the CLI main probes for is gone on modern Lemonade), yet check_installation() now returns installed=True version=10.7.0 — on main this reports "not installed" and forces a pointless reinstall.

Bottom panel — auto-start (#316 obs. 2): with LemonadeServer.exe stopped, interactive gaia init auto-started it (LemonadeServer.exe --silent), reached health OK and 32768-token ctx (via LEMONADE_CTX_SIZE env), and left the process alive (pid=15772, /api/v1/health → 200) — no "Please start Lemonade Server" prompt, which is exactly what main blocks on.

Image lives on the disposable assets/issue-316-proof branch — safe to delete after merge.

@itomek

itomek commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

CI note — the two red integration checks are a known runner-infra flake, not this change

RAG Integration Tests and Test Lemonade Embeddings API both fail in CI's own "Start Lemonade Server" setup step (before any test runs) with:

model_load_error: Failed to load model 'nomic-embed-text-v2-moe-GGUF': llama-server failed to start

This is the documented runner-side llama-server failed to start condition (stray llama-server holding a lock / port conflict — see the install-lemonade action's own comments; previously chased via #1869/#1867/#1870), needing runner cleanup. Evidence it is not this PR:

  • RAG Unit Tests pass on the same run — only the live-embedding-model integration job fails.
  • The diff touches only Lemonade detection/launch (resolve_lemonade, check_installation, launch_server, _ensure_server_running); these jobs start Lemonade via their own PowerShell and load models through the server API — paths this change never enters.
  • Reran once; it recurred identically → persistent runner state, not transient.

The gates that actually exercise the change are green (Unit Tests py3.10/3.11/3.12 + macOS, Code Quality, CodeQL, Linux CLI integration, pr-review bot) plus the Windows real-world proof above. Not re-running further to avoid churn — flagging for a maintainer to clear the runner.

@itomek itomek disabled auto-merge July 6, 2026 21:27

@kovtcharov-amd kovtcharov-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — this is a clean, well-scoped fix for #316 with real-world evidence on a modern (10.7) Windows box.

What makes it solid:

  • Detection/launch is consolidated into one stdlib-only primitive (resolve_lemonade / build_start_command) with clear precedence (env override → modern canonical path → legacy PATH), and the legacy lemonade-server serve argv is pinned byte-identical by tests.
  • Fails loud, not silent: build_start_command and the client raise actionable errors ("Run gaia init … or set LEMONADE_SERVER_PATH") when no tooling is found; auto-start failure surfaces the error and prints the correct per-tooling start command rather than degrading quietly.
  • Nice safety bonus: the runtime launch drops shell=True in favor of argv-only Popen with env={**os.environ, **spec.env}.
  • Tests added for the launcher, the modern/legacy dispatch, and the installer detection path.

On the red checks (API Tests / Test Lemonade Embeddings / behavior-e2e): these are the known Lemonade cold-start infra flake — they fail in the workflow's own lemonade-server serve start step (Server health check failed after 60 seconds), before any test code runs, and reproduce on unrelated PRs. Not a regression from this change. A rebase onto current main (which now has #1946's stray-Lemonade-process cleanup) should reduce them.

@kovtcharov-amd kovtcharov-amd added this pull request to the merge queue Jul 8, 2026
Merged via the queue into main with commit d643954 Jul 8, 2026
37 of 41 checks passed
@kovtcharov-amd kovtcharov-amd deleted the fix/issue-316-lemonade-upgrade-modern-cli branch July 8, 2026 00:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

llm LLM backend changes performance Performance-critical changes tests Test changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GAIA mismanages lemonade upgrade

3 participants