Two bugs, one symptom: how a stage that never reports looked like a platform difference #65
Replies: 2 comments
Update: a Windows machine works, and the "two bugs" picture was still too simpleA team member ran the pipeline on a Windows machine (Cursor + claude-code) and it completed: Four mechanisms, not twoWindows, 64 runs, by what the handbook stage did:
And orthogonal to all of the above: The method that separated themThe discriminator was stage-event accounting, per window, in time order. Averages hid everything: the mean handbook failure duration on Windows was 49 s, which suggested "the agent runs and fails". The distribution was bimodal — 19 at 0 ms, a handful at 6–10 minutes — and those are different bugs. Likewise One user's timeline is the whole story in miniature: ten days of What's still openThe long-run failures. User C (VS Code + claude-code) gets Why installs end up partial. Correction to my own earlier claims here
|
|
The split holds up, and the accounting behind it was the right call — the two mechanisms really are unrelated, and the handbook stage was the one carrying both. Acting on the half that was still open: enginesSynced was one existsSync on tracelens standing in for all six Python engines, so a uv sync that installed tracelens and stopped reported READY while every stage needing one of the other five bailed in milliseconds. Readiness now requires all six, and missingPythonEngines names the absent ones so a caller can say which. The Rust index stays out of that list deliberately — it isn't a console script and has its own resolver. ENGINE_NAMES is derived from the same constant now, so an engine can't be added to one list and missed by the check reading the other. Only the checkout route changes; enginesReady consults the wheel first, and a wheel install has no venv to be partial about. (#62) The "invisible twice over" point generalised further than the two bugs here: both discovery stages that gate on the index binary returned a bare false outside the telemetry wrapper too, so "engines never installed" and "the stage ran and broke" were the same 0 ms event. Both now record engines.not_found. (#66) One thing this doesn't cover: the picker still has no ignoreFocusOut. The uncommitted work in harnessPicker.ts fixes a different defect there — selecting an agent that isn't installed — so #64 as filed is still open. |
Uh oh!
There was an error while loading. Please reload this page.
Following on from #62 / discussion #63, I went back through the discovery telemetry looking for why runs end with
services_count: 0, expecting to confirm the Windows engines story. Instead the data splits cleanly into two unrelated defects, and the second one had been hiding behind the first.Filed as #64. Writing up the analysis here because the method is the interesting part: the bug was found by noticing that some stages don't report at all.
The observation
Every discovery run emits
discovery_stageevents per stage. So the number of stage events should track the number of runs. It doesn't:indexfires on all 62 Windows runs.handbookfires on 32. Thirty runs executed discovery and never emitted a handbook event in either direction — not success, not failure.The macOS column makes it unambiguous:
macOS has zero handbook errors, yet 12 of its 29 runs report
handbook_ok: false. A stage cannot fail without recording a failure. Those runs never ran it.The cause
discovery.tsgates both LLM stages on a harness:ensureHarnessChosenreturns null when the picker is dismissed — and the picker's QuickPick never setsignoreFocusOut, which defaults tofalse. VS Code hides it on any focus loss, not just Esc. Since discovery auto-runs on startup, the picker appears unprompted while you're reading code; one click elsewhere and the LLM stages are gone.Seven other prompts in the extension set
ignoreFocusOut: true. The one gating the whole pipeline doesn't.Why this was worth separating
The two defects produce the same symptom and want opposite fixes:
enginesSyncedtests 1 of 7 enginesENGINE_NAMESentryignoreFocusOut+ wrap the skipWindows carries both. Because the engines bug is loud, it absorbed attention and the quieter one — which loses more runs across the fleet — read as "macOS is fine."
The general lesson
The instrumentation was good enough to find this, but only because
indexwas there as a control. A stage that returns a barefalseoutside the telemetry wrapper is invisible twice over: the user sees a run that looks busy and healthy, and analytics sees a failure with no attributable stage.Worth a broader sweep for the pattern — any
Promise.resolve(false)or early return that bypasses the stage wrapper is a candidate. The fix in #64 proposes an explicitskippedoutcome with a reason, so "we chose not to run this" stops being indistinguishable from "this never happened."Happy to be told I've misread any of it — particularly whether the auto-discovery picker is intended to be dismissible at all, or whether it should block until answered.
All reactions