fix(installer): detect and auto-start modern Lemonade in gaia init#1940
Conversation
Add failing tests for the not-yet-implemented modern-Lemonade detection primitive ahead of the implementation.
Rewire check_installation tests onto resolve_lemonade/get_installed_version (AC1/AC2), and pin the corrected _ensure_server_running behavior where auto-start is attempted before the interactive "Please start Lemonade Server" prompt in both CI and interactive modes, with a fail-fast guarantee that a failed auto-start never hangs on input() (AC5/AC6). Also extend the #839 ctx-size regression guard to cover the modern LEMONADE_CTX_SIZE env-var path alongside the existing legacy --ctx-size argv path, so a future change can't silently drop ctx-size propagation for modern tooling while the legacy assertions keep passing. 3 pre-existing test failures unrelated to this change (already broken on main, not introduced here): TestEnsureLemonadeInstalledSkipsWhenPresent::test_older_version_below_minimum_triggers_install_in_ci, TestLegacyFallback::test_not_installed_on_linux_proceeds_to_install_via_ppa, TestLegacyFallback::test_not_installed_on_windows_proceeds_to_download_and_install
Pin that launch_server() builds its Popen argv/env from resolve_lemonade()/build_start_command() instead of the hardcoded ["lemonade-server", "serve"] literal, with a byte-identical regression guard for the legacy argv shape and a guard that kill_process_on_port is skipped when health_check() already reports the server healthy at entry.
Pin at the Popen call sites that spec.env is MERGED into the parent
environment ({**os.environ, **spec.env}) rather than replacing it — an
implementation passing env=spec.env alone would drop PATH/LOCALAPPDATA
and break LemonadeServer.exe while still passing the previous
assertions. Sentinel env var seeded before the call and asserted
retained, alongside PATH, in the three modern-path launch tests.
Also add the direct build_start_command-level Windows-legacy test
asserting --no-tray in argv, pin platform in the non-Windows legacy
test for cross-platform determinism, and replace a no-op tuple
assertion with a real assertTrue.
Modern Lemonade (10.7/10.8) removed the lemonade-server CLI; the server is LemonadeServer.exe --silent on Windows (ctx via LEMONADE_CTX_SIZE env) and the lemond systemd unit on Linux. Add the stdlib-only shared primitive (resolve_lemonade / get_installed_version / build_start_command) that the installer and runtime client will adopt instead of hard-coding lemonade-server. Precedence: LEMONADE_SERVER_PATH env override (verbatim, no PATH search) -> modern by canonical path (wins over a stale legacy binary on PATH) -> legacy which(lemonade-server|lemonade-server-dev). Legacy argv stays byte-identical (serve --ctx-size, + --no-tray on Windows). All 12 contract tests in tests/unit/test_lemonade_launcher.py pass; includes black formatting fixes to the contract test files.
check_installation() previously probed only shutil.which("lemonade-server"),
so a modern Lemonade install (10.7+, no lemonade-server CLI) reported
installed=False and gaia init tried to reinstall over a working setup.
Delegate detection to the shared resolve_lemonade()/get_installed_version()
primitive: modern installs are found at their canonical path, legacy
installs keep working unchanged, and a found install whose version probe
fails still reports installed=True with a diagnostic error (mirroring the
old failed-to-get-version branch). LemonadeInfo shape and find_product_code
are untouched — all five callers are unaffected.
…316) Interactive `gaia init` previously never auto-started the server — an indentation bug made the "Please start Lemonade Server" prompt block run unconditionally, blocking on input() even in flows that could have started the server themselves. Restructure _ensure_server_running(): auto-start is attempted first in BOTH CI (yes=True) and interactive modes, success returns before any prompt, and the manual prompt is the only remaining fall-through (interactive auto-start failure). CI mode returns False on a failed auto-start without ever touching input(). Auto-start now routes through resolve_lemonade()/build_start_command() in a new _auto_start_server() helper — the raw [path, "serve", "--no-tray", --ctx-size] literal is gone, so modern tooling (LemonadeServer.exe --silent + LEMONADE_CTX_SIZE env, merged into the parent environment) starts correctly, including under CI's LEMONADE_SERVER_PATH override which resolve_lemonade() now honors in one place instead of a parallel os.environ.get. The interactive fallback prompt renders the exact start command for the resolved tooling; _find_lemonade_server() delegates to the shared primitive.
) launch_server() hardcoded ["lemonade-server", "serve"], which no longer exists in modern Lemonade (10.7+) — auto-start was broken for every modern install. Route argv/env through resolve_lemonade()/build_start_command(): modern gets LemonadeServer.exe --silent with LEMONADE_CTX_SIZE merged into the parent environment, legacy keeps its exact argv. A missing install now raises an actionable LemonadeClientError instead of a bare FileNotFoundError from Popen. Also: skip kill_process_on_port() when health_check() already reports OK at entry (never restart a healthy tray/systemd-managed server the user didn't ask to restart); convert the "terminal" branch from a shell=True f-string to argv-only Popen with CREATE_NEW_CONSOLE so a resolved path never passes through a shell string; log_level appends --log-level only for legacy tooling (modern launcher has no such flag — logged at debug and ignored); _check_lemonade_installed() and get_lemonade_version() use the primitive; user-facing "restart with" hints render the resolved tooling's real start command instead of the removed lemonade-server CLI (test assertion widened to accept the modern LEMONADE_CTX_SIZE form).
…nned TestLaunchServerCtxSize asserted the removed hardcoded lemonade-server argv and hit the real network (unmocked health_check), so results flipped depending on whether a local server happened to be running and which tooling the host had. Pin resolve_lemonade to legacy (the argv path under test — modern env-var ctx is covered by the #316 contract tests) and stub health_check to fail so the already-healthy launch guard never short-circuits the assertion.
|
Verdict: Approve with suggestions — no blocking issues. This fixes the real #316 breakage: modern Lemonade (10.7+) dropped the The only things worth a look are minor edge-cases, none of which affect the tested path: an env-override to a modern Linux binary gets silently ignored in favor of 🔍 Technical details🟢 Minor1. 2. 3. Manual-prompt hint uses POSIX Strengths
|
An explicit LEMONADE_SERVER_PATH override classified as modern but not a Windows .exe (e.g. a Linux daemon path) was silently rerouted to 'systemctl --user start lemond', ignoring the binary the caller named. Track resolution provenance on LemonadeTooling (source: env|probe) and launch an env-override binary verbatim; probe-resolved modern Linux keeps the best-effort systemctl start. Also note _find_lemonade_server() is a compat-only surface with no in-tree callers.
|
Addressed the review's minor items in 807b3b5:
Also for the record: the earlier "Example Agents Unit Tests" red was infrastructure (a 503 from download.pytorch.org during env setup — no tests ran), and the |
📸 Real-world proof (Windows 11 and macOS · AMD Ryzen AI · Lemonade 10.7.0)Live over SSH against the current PR tip `807b3b53`. Terminal output rendered from the actual run (raw text is in the PR description as verifiable source). Top panel — detection: Bottom panel — auto-start (#316 obs. 2): with Image lives on the disposable |
CI note — the two red integration checks are a known runner-infra flake, not this changeRAG Integration Tests and Test Lemonade Embeddings API both fail in CI's own "Start Lemonade Server" setup step (before any test runs) with: This is the documented runner-side
The gates that actually exercise the change are green (Unit Tests py3.10/3.11/3.12 + macOS, Code Quality, CodeQL, Linux CLI integration, pr-review bot) plus the Windows real-world proof above. Not re-running further to avoid churn — flagging for a maintainer to clear the runner. |
kovtcharov-amd
left a comment
There was a problem hiding this comment.
Approving — this is a clean, well-scoped fix for #316 with real-world evidence on a modern (10.7) Windows box.
What makes it solid:
- Detection/launch is consolidated into one stdlib-only primitive (
resolve_lemonade/build_start_command) with clear precedence (env override → modern canonical path → legacy PATH), and the legacylemonade-server serveargv is pinned byte-identical by tests. - Fails loud, not silent:
build_start_commandand the client raise actionable errors ("Rungaia init… or set LEMONADE_SERVER_PATH") when no tooling is found; auto-start failure surfaces the error and prints the correct per-tooling start command rather than degrading quietly. - Nice safety bonus: the runtime launch drops
shell=Truein favor of argv-onlyPopenwithenv={**os.environ, **spec.env}. - Tests added for the launcher, the modern/legacy dispatch, and the installer detection path.
On the red checks (API Tests / Test Lemonade Embeddings / behavior-e2e): these are the known Lemonade cold-start infra flake — they fail in the workflow's own lemonade-server serve start step (Server health check failed after 60 seconds), before any test code runs, and reproduce on unrelated PRs. Not a regression from this change. A rebase onto current main (which now has #1946's stray-Lemonade-process cleanup) should reduce them.

Closes #316
Modern Lemonade (10.7+, including the 10.8.1 GAIA installs) removed the
lemonade-serverCLI — the server is nowLemonadeServer.exe --silenton Windows (ctx size via theLEMONADE_CTX_SIZEenv var) and a systemdlemondservice on Linux. GAIA still hard-codedlemonade-server serveeverywhere, so on a modern installgaia initreported a working Lemonade as not installed, and after any install it told the user to start the server by hand (#316's "never started it") — interactive mode could never auto-start because of an indentation bug that made the manual prompt unconditional. Now detection finds modern and legacy installs, andgaia initauto-starts the server in both interactive and CI modes, falling back to a prompt (showing the correct per-tooling command) only if auto-start fails. (#316's uninstall failure was previously fixed by #478; verified still working — the registry ProductCode lookup finds modern installs.)All changes route through one new stdlib-only primitive,
gaia.llm.lemonade_launcher(resolve_lemonade/get_installed_version/build_start_command), consumed by the installer andLemonadeClient— legacylemonade-serverbehavior is pinned byte-identical by tests, and CI'sLEMONADE_SERVER_PATHoverride is honored first.Evidence (real-world, Windows 11 + Ryzen AI box, Lemonade 10.7.0)
Tested by deploying this branch to the machine's editable GAIA install and running the real CLI over SSH.
Detection — before:
main's check isshutil.which("lemonade-server"), which isNOT ON PATHon this modern install → "not installed" → forced reinstall over a working setup. After:Auto-start —
LemonadeServer.exestopped, then interactivegaia init --profile minimal --skip-models(stdin-driven,yesnot set):No "Please start Lemonade Server" prompt appeared;
LemonadeServer.exealive (pid captured) and/api/v1/health= 200 afterwards. The ctx-size env path is proven end-to-end by the Step-4 32768-token model preload.Test plan
python -m pytest tests/unit/test_init_command.py tests/unit/installer/ tests/unit/test_lemonade_launcher.py tests/test_lemonade_client.py -q→ 245 passed; the 4 failures are pre-existing environment artifacts (3 tests short-circuited by a live local Lemonade on :13305 — fail identically onmain— plus 1 live-server integration test), none in changed codepython util/lint.py --all→ exit 0Reviewer notes — bundled behavior deltas & rationale
input()never called on auto-start success, Popen env must retain the parent environment (merge, not replace), legacy argv pinned byte-identical,LEMONADE_SERVER_PATHhonored before any PATH probe with modern-beats-stale-legacy precedence.launch_server()now skipskill_process_on_portwhen a healthy server is already listening (never restarts a tray/systemd-managed server the user didn't ask to restart), raises an actionable error when no tooling is found, and itsbackground="terminal"branch is argv-only (CREATE_NEW_CONSOLE) instead of ashell=Truestring.check_installation()no longer returns installed=False on a version-probe timeout (that could trigger a pointless reinstall); a found install with a failed probe stays installed=True with a diagnostic error.LemonadeInfoshape andfind_product_code()are unchanged (all five callers unaffected).launch_servernow passes--no-tray(aligning with whatgaia initalways did);--log-levelis appended for legacy only (the modern launcher has no such flag — logged and ignored, not silently dropped).lemonade-serverCLI._find_lemonade_server()retained as a compat surface (delegates to the primitive) though it has no remaining callers.systemctl --user start lemond) is best-effort and unit-tested only — the daemon is normally already running under systemd..github/actions/install-lemonade/action.ymlandinstaller/scripts/*.ps1duplicates this logic and has one stale search path; consolidating PS/Python detection is out of scope here.