ci(windows): run Gemma on a persistent Lemonade server (fix Windows integration jobs)#1733
Conversation
GitHub Actions kills any server a job starts, and the 10.7.0 GUI server is flaky to launch headless per-job. ensure-lemonade-running.ps1 launches it as a Scheduled Task (owned by Task Scheduler, not the job), reuses it if already healthy, retries a wedged start, and warms Gemma-4.
|
Both Windows integration jobs verified green on this branch head, on
The second run also confirms the persistence path: |
Review: ci(windows) — persistent Lemonade server for Windows integration jobsSummarySolid infra fix that targets a real, well-diagnosed problem: GitHub Actions kills any server a job spawns, so the inline Issues Found🟡 Warm-pull failure is swallowed; the helper exits 0 anyway ( The removed inline code in both workflows } catch { Write-Host "WARN: warm pull failed: $($_.Exception.Message)" }
...
exit 0The pull does double duty here — it both downloads/registers the model and warms it. If it's purely a warm-up and the model is already present, a } catch { Write-Host "ERROR: warm pull failed: $($_.Exception.Message)"; exit 1 }If you intend warm-up to be best-effort, a one-line comment saying so (and noting the model presence is guaranteed elsewhere) would make the intent explicit. 🟢 PATH-strip filter is a broad substring regex (
🟢 No integrity check on the downloaded MSI ( The MSI is fetched over HTTPS from the pinned official release and installed silently with admin rights. A SHA-256 check against a known hash would harden the reinstall path. Acceptable as-is given the pinned official source; noting for future hardening. 🟢 Note (not a blocker): persistent task runs the server as SYSTEM/Highest at boot ( This is the intended design for a self-hosted runner, and the exe is resolved from standard install locations, so it's not a code-level vulnerability. Flagging only so the runner owner is aware the task survives reboots and runs elevated indefinitely until explicitly removed. Strengths
VerdictApprove with suggestions. No blocking issues. The warm-pull swallow (🟡) is the one item worth resolving before merge — either make it fatal to match the prior |
…md#1867) ## Why this matters The two Windows integration jobs — **Test Agent SDK on Windows** and **Test GAIA CLI on Windows** — went intermittently red on `main` starting 06-24, failing at Lemonade startup (`Lemonade server not ready`) before any test ran. Root cause is **runner state, not a code regression**: the pinned `LemonadeServer.exe` binary is correctly v10.7.0 (CI verifies "no reconcile needed"), but a **stale `config.json`** in the runner's SYSTEM profile pins the server to the old default port **8000**, while the health check polls **13305**. It only looked like "flaky infra a re-run clears" because a previous run's healthy 13305 server was sometimes still alive. No PR caused it — the startup script has been unchanged since amd#1733 (06-18); the trigger was a v10.8.x install touching the runner's config. The fix makes the server bind 13305 deterministically: launch `LemonadeServer.exe --port 13305` (in v10.x the port is a bare flag — there's no `serve` subcommand, and the old `lemonade-server` shim was removed in v10.5) **and** clear the stale `config.json` so the documented 13305 default regenerates. Also broadens the pre-start process cleanup to the full Lemonade/`llama-server` set so a wedged child can't survive a retry. A second change adds `installer/scripts/**` and these workflows' own files to their `paths` triggers, so changes to the Lemonade startup script actually re-run the jobs they govern (previously they only triggered on `src/**`/`tests/**`, so this very fix couldn't validate itself). ## Test plan These jobs run only on the self-hosted Windows runner, so CI is the verification: - [x] **Test GAIA CLI on Windows** passes — server binds 13305 (validated on an earlier push of this branch). - [ ] **Test Agent SDK on Windows** passes with the same `Server binary: …LemonadeServer.exe` → `Healthy after attempt 1` signature. - [ ] Re-run from a cold runner (no pre-existing 13305 server) self-heals to 13305 instead of getting stuck on 8000.
The Windows Agent SDK and GAIA CLI integration jobs were failing because the runner couldn't actually serve
Gemma-4-E4B-it-GGUF(GAIA's default model). Two root causes: a stale Lemonade install in another profile shadowed the current one onPATH(old catalog, no Gemma), and the 10.7.0 server is flaky to start headless per-job — and GitHub Actions kills any server a job spawns when the job ends. After cleaning the runner to a single 10.7.0 install, both jobs now run reliably on Gemma by talking to a persistent Lemonade server.What this does:
ensure-lemonade-running.ps1launches the version-matched server as a Windows Scheduled Task (owned by Task Scheduler, not the job, so it survives across jobs and reboots), reuses it when already healthy, retries a wedged start, and warms Gemma-4. Both Windows workflows now call it instead of starting the server inline.reset-lemonade.ps1is the one-shot runner-cleanup tool (wipe every Lemonade install incl. stale copies → reinstall the pinned version → warm the backend). Already run on thestxrunner to bring it to a single clean 10.7.0.Verified on the runner (two consecutive
workflow_dispatchruns oftest_agent_sdk.yml):Ran 8 tests in 140s / OKLemonade already healthy ... reusing persistent server→Ran 8 tests in 109s / OKTest plan
Test Agent SDK on Windows (Lemonade Integration)is green on Gemma-4-E4B-it-GGUFTest GAIA CLI on Windows (Full Integration)is green on Gemma-4-E4B-it-GGUF