Thanks to visit codestin.com
Credit goes to github.com

Skip to content

v1.5.0 — multi-step cockpit orchestration + BSS_REPL_LLM_AUTONOMY#43

Merged
chiam-ck merged 14 commits into
mainfrom
feat/v1.5-multi-step-cockpit
May 26, 2026
Merged

v1.5.0 — multi-step cockpit orchestration + BSS_REPL_LLM_AUTONOMY#43
chiam-ck merged 14 commits into
mainfrom
feat/v1.5-multi-step-cockpit

Conversation

@chiam-ck

Copy link
Copy Markdown
Owner

Summary

v1.5 unlocks multi-step orchestration in the cockpit — operators can ask compound questions ("investigate CASE-…", "register CUST + create order", "show customer and their subscriptions") and the agent chains the work. Adds BSS_REPL_LLM_AUTONOMY={granular,batched} to control how many /confirms a compound destructive action needs. No new container, no schema, no migration (head stays at 0030).

Design lifted from the proven loyalty-cli v0.11 pattern; adapted for bss-cli surfaces (REPL + browser veneer).

Five pieces

  1. BSS_REPL_LLM_AUTONOMY env var (orchestrator/bss_orchestrator/autonomy.py). granular (default — re-gates after each destructive) / batched (one /confirm authorises the loop). Fail-closed at boot with AutonomyMisconfigured — same shape as BSS_API_TOKEN=changeme. Per-process scope; per-session override deferred to v1.5.1.

  2. Autonomy-aware destructive gating (safety.py + graph.py + session.py). wrap_destructive grows autonomy_mode + loop_state kwargs; build_tools creates one LoopState per build_graph invocation and shares it across every destructive wrapper. Granular re-blocks after the first destructive fires; batched authorises the whole loop. DESTRUCTIVE_TOOLS unchanged — autonomy controls how many /confirms, not which tools.

  3. ITERATIVE FLOW prompt block + softened v0.19 "Done." rule (bss_cockpit/prompts.py). Agent may chain another tool call instead of the mandatory terminating sentence — anti-duplication rule unchanged. Three bss-shaped worked examples (read chain, case investigation, compound write). Doctrine guard pins ITERATIVE FLOW to operator surface only — must not leak into customer_chat_prompt.py.

  4. 3-strike loop bail (session.py). MAX_CONSECUTIVE_TOOL_FAILURES=3 terminates astream_once with a structured AgentEventError when the LLM keeps re-firing failure-shaped tool calls (real exceptions OR POLICY_VIOLATION / DESTRUCTIVE_OPERATION_BLOCKED / CLIENT_ERROR). Counter resets on any success.

  5. Cockpit chrome filter (bss_cockpit/chrome_filter.py). Strips cockpit-emitted chrome from history rehydration AND live emits — including the narrated-tool.name(args) mimicry shape Gemma produces. Plus inline post-propose / post-execute bubble overrides so /confirm-shaped bubbles say what they mean ("Proposed X(args). Type /confirm to authorise." / "Executed X(args).") instead of Gemma's misleading "Done.".

Real-world bug fixes

Two pre-existing bugs surfaced + fixed during live REPL testing:

  • Route + REPL pending-destructive staging derived from STARTED-side capture + not allow_destructive_this_turn heuristic — silently lost the granular re-gate signal on /confirm-resumed turns. Rewired to derive strictly from COMPLETED-with-BLOCKED in both surfaces.
  • REPL /confirm slash command was a no-op marker (returned a never-consumed string). Now drives a real _drive_turn with the synthetic prompt the cockpit browser veneer always used.

Doctrine

  • CLAUDE.md gains a new Cockpit (v1.5 — multi-step autonomy) anti-pattern block: autonomy module ownership, ITERATIVE FLOW scope (operator-only), MAX_CONSECUTIVE_TOOL_FAILURES contract, _ASSISTANT_CHROME_PREFIXES inventory lock.
  • ARCHITECTURE.md § Operator cockpit gains a v1.5 paragraph covering both autonomy modes + the 3-strike bail + the chrome filter.
  • HANDBOOK.md gains §8.20 "Multi-step cockpit actions (v1.5)" with worked examples + recovery semantics + when-to-flip-to-batched guidance.
  • DECISIONS.md 2026-05-26 entry covers the five pieces + three rejected alternatives (plan-emit-then-confirm-once, composite read tools, per-tool autonomy annotations).

Test plan

  • make testorchestrator 403→420 (+17 new unit tests across autonomy/safety/loop_bailout/iterative_flow_scope), bss-cockpit 66→95 (+29 chrome filter tests pinning every observed mimicry shape), cli 127 unchanged. All other suites unchanged. No regressions.
  • make e2e (granular default) — 12 passed, 1 skipped (batched gated on env), 0 failed. New specs test_v15_compound_action_granular + test_v15_case_investigation green. All v1.4.1 specs unaffected.
  • make e2e-batched (env-flipped) — 12 passed, 1 skipped (granular gated), 0 failed. New test_v15_compound_action_batched spec green; granular spec auto-skips with a clear reason.
  • Live REPL retest — full terminate subscription → propose → /confirm → execute → verify state=TERMINATED workflow end-to-end. Bubble overrides ("Proposed X(args)…" / "Executed X(args)…") render correctly. /confirm-into-the-void shows the mimicry-stall warning. Anti-mimicry runtime strip catches Gemma's "I propose to terminate ... \subscription.terminate(...)`"` shape.
  • Soak v3 re-run on v1.5 baseline (deferred — operator's call whether to gate the merge on it; the v0.12 soak corpus is autonomy-agnostic so this is a smoke-check more than a real GA gate)

🤖 Generated with Claude Code

Ck and others added 14 commits May 26, 2026 10:51
…iring

New orchestrator/bss_orchestrator/autonomy.py module mirroring loyalty-cli's
read_autonomy_mode + AutonomyMisconfigured. Single env var BSS_REPL_LLM_AUTONOMY,
two valid values (granular default / batched), fail-closed validation at
process boot. Case + whitespace normalised before validation; empty/unset
returns the default; anything else raises AutonomyMisconfigured with a
message that names the env var, the bad value, and the valid set.

Wired into both cockpit boot paths:
- portals/csr/bss_csr/main.py lifespan reads + caches on app.state and
  logs the resolved mode at "cockpit.starting"
- cli/bss_cli/repl.py _bootstrap_store_and_config() reads + stashes in
  a module-level cache so _drive_turn doesn't have to re-parse env per
  turn (and per-session override in v1.5.1 has an obvious touch point)

Both paths let AutonomyMisconfigured propagate — same fail-closed shape
as the v0.9 named-token sentinel rejection. No silent default-on-typo.

20 new unit tests in orchestrator/tests/test_autonomy.py cover: unset
+ empty + whitespace-only default to "granular"; both valid modes load;
case/whitespace normalisation (GRANULAR / Batched / "  granular  "); 10
unknown-value rejection cases including typos (granuler, batch); error
message must name env var + bad value + valid set; canonical-set guard.

Behaviour-preserving: nothing reads the cached mode yet. Phase B wires
it into the destructive-tool gating in safety.py + session.py.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
safety.py — wrap_destructive grows two new kwargs:
  - autonomy_mode: "granular" or "batched" (default "batched" to preserve
    pre-v1.5 behaviour for the dozens of test/scenario callers that don't
    pass it; production cockpit paths in v1.5+ pass it explicitly)
  - loop_state: per-graph mutable LoopState dict, created fresh by
    build_tools per build_graph invocation and shared across every
    destructive wrapper in that graph

In granular mode + allow_destructive=True: the FIRST destructive call
fires; subsequent destructive calls in the same graph block again with
the same structured DESTRUCTIVE_OPERATION_BLOCKED response, forcing the
LLM to re-propose so the operator can /confirm each one. Per-step
operator control even after the first authorisation.

In batched mode + allow_destructive=True: every destructive in the
graph fires — historical pre-v1.5 behaviour, preserved for callers
that want one /confirm to authorise the loop.

graph.py — build_tools + build_graph + _as_structured_tool plumb
autonomy_mode + a freshly-created LoopState through to every wrapped
destructive tool in the graph. Each astream_once invocation builds its
own graph, so LoopState naturally resets between turns.

session.py — astream_once + ask_once grow an autonomy_mode kwarg
(default "batched" for backward compat) and surface it on the OTel
span as bss.ask.autonomy_mode for trace-level observability.

portals/csr cockpit route + cli REPL — read the cached autonomy mode
(app.state.autonomy_mode / module-level _AUTONOMY_MODE respectively)
and pass it into astream_once. Both paths set the cached value at
boot from read_autonomy_mode().

6 new safety tests + the existing 4 still pass:
  - granular blocks second destructive even when allow_destructive=True
  - batched runs every destructive once authorised
  - granular doesn't relax the disallowed case
  - non-destructive calls don't consume the granular budget
  - fresh LoopState resets between graph builds (= between turns)
  - default autonomy is "batched" for the primitive (back-compat guard)

make test: orchestrator 403→409, cli 127, bss-cockpit 66 — all green.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Catches the Gemma thrash pattern where the model fires the same broken
call over and over instead of adapting. After 3 consecutive failure-
shaped tool results, the stream terminates with a structured
AgentEventError so the cockpit can render a "couldn't recover" panel
instead of letting the loop spin forever. Counter resets on any success.

"Failure" = ToolMessage.status="error" (LangGraph exception path) OR
the result body contains one of three structured-error markers:
DESTRUCTIVE_OPERATION_BLOCKED, POLICY_VIOLATION, CLIENT_ERROR. The
marker check is conservative — false negatives just delay or skip the
bail; false positives could end a healthy investigation loop early,
so the marker must be a literal "key: value" substring match (not a
loose "error" word check).

MAX_CONSECUTIVE_TOOL_FAILURES = 3 — direct lift from loyalty-cli.

8 new unit tests cover the classifier surface:
  - status="error" trips regardless of body
  - each structured-error marker trips
  - a customer email containing "error" does NOT trip (false-positive guard)
  - empty/normal results don't trip
  - threshold constant guard so the test surfaces if someone changes it

End-to-end bail behaviour will be exercised in Phase E's e2e suite —
testing the loop integration here would require either a real graph
(slow + non-deterministic) or a heavy mock. The classifier is the
small surface most likely to drift.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
New packages/bss-cockpit/bss_cockpit/chrome_filter.py with:
  - is_cockpit_chrome(content): True for cockpit-emitted assistant
    strings (route error fallback "Sorry — something went wrong...",
    Gemma empty-final recovery "(The model called X but did not
    synthesise...)", total-empty fallback "(no reply)", citation-
    guard fallback "I don't have a citation for that..."). Empty /
    whitespace-only content also classified as chrome (the cockpit
    never persists empty real replies — they get replaced before
    persistence).
  - strip_fake_propose(text): runtime backstop for LLMs that mimic
    the cockpit's "⚠ PROPOSE: ..." banner shape as plain text
    (the anti-mimicry prompt rule is the first line of defence).

Wired into Conversation.transcript_text() so the cockpit's own
emit chrome is filtered out before astream_once(transcript=...) sees
it. Without the filter, the LLM reads its own past placeholder
bubbles as prior reasoning and three failure modes follow: mimicry
(emits new turns in the same chrome shape), state confusion (sees
"did not synthesise" and re-tries the same broken call), and
citation thrash (learns the fallback as the default).

Conservative prefix-match design — false negatives just leave benign
noise; false positives would strip real LLM output. Pattern lifted
from loyalty-cli's _is_cockpit_chrome / _strip_fake_propose; bss-cli
chrome strings differ but the design is identical.

22 new chrome-filter unit tests:
  - every known cockpit prefix recognised (parametrised)
  - empty/whitespace classified as chrome
  - 9 legitimate LLM replies NOT misclassified (false-positive guard)
  - fake-PROPOSE stripper handles step variants, case-insensitive,
    Unicode warning glyph, leaves real prose alone
  - inventory lock: _ASSISTANT_CHROME_PREFIXES contents pinned so a
    new cockpit fallback added without a matching prefix surfaces
    in code review

make test: bss-cockpit 66→87 — all green.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…E FLOW

Two edits to _COCKPIT_INVARIANTS in packages/bss-cockpit/bss_cockpit/
prompts.py:

1. Soften the v0.19 "after renderer-backed call → one short sentence,
   STOP" rule. The TEXT reply that TERMINATES a compound action is
   still bound by the one-sentence + no-re-render rule. But the agent
   MAY now emit ANOTHER TOOL CALL instead of a text reply when the
   prompt clearly requires more steps — the one-sentence rule kicks in
   only on the terminating reply. Anti-duplication contract unchanged.

2. Add the ITERATIVE FLOW block as a new (v1.5+) rule, with three
   bss-shaped worked examples:
   - "show me CUST-001 and then list their subscriptions" → reads
     chain in one turn, terminate with a brief one-sentence summary
   - "investigate CASE-042" → case.get → customer.get →
     subscription.list_for_customer → ticket.list (any order), end
     with situational summary + proposed next operator action; do NOT
     propose destructive tools unless the operator's prompt explicitly
     asks for one (key doctrine — agent doesn't escalate on its own)
   - "register CUST-XYZ then issue PLAN_M" → compound write, gets
     re-gated under granular autonomy and runs inline under batched
   Plus explicit guidance on tool-error adaptation (the 3-strike bail
   is the safety rail; ask for help on attempt #2 rather than letting
   it fire) and a reminder to terminate with a brief text message.

Doctrine guard in orchestrator/tests/test_iterative_flow_scope.py:
  - customer_chat_prompt.py MUST NOT contain "ITERATIVE FLOW"
    (compound actions are an operator capability, not a customer-
    chat one — the v0.12 chat caps + ownership trip-wire were never
    stress-tested against agent-driven write chains)
  - bss_cockpit/prompts.py MUST contain "ITERATIVE FLOW" (guard
    against accidental removal regressing v1.5's main unlock)
  - The block must be tagged with the v1.5 marker for archaeology

make test: orchestrator 417→420 — all green.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…tonomy

CLAUDE.md anti-patterns gain a new Cockpit (v1.5 — multi-step autonomy)
block with four entries: autonomy module ownership (don't read
BSS_REPL_LLM_AUTONOMY outside bss_orchestrator.autonomy); ITERATIVE
FLOW scope (must not leak into customer_chat_prompt.py, guarded by
orchestrator/tests/test_iterative_flow_scope.py); MAX_CONSECUTIVE_TOOL_
FAILURES contract (3, lifted from loyalty-cli); _ASSISTANT_CHROME_
PREFIXES inventory lock (new cockpit fallback bubble = new prefix entry).

ARCHITECTURE.md § Operator cockpit gains a v1.5 paragraph covering the
two autonomy modes + their /confirm semantics, the read_autonomy_mode
fail-closed boot contract, the 3-strike loop bail, and the cockpit-
chrome filter. Notes that DESTRUCTIVE_TOOLS is unchanged — autonomy
controls how many /confirms a compound action needs, not which tools
require one.

HANDBOOK.md:
  - env var table gains BSS_REPL_LLM_AUTONOMY row pointing at §8.20
  - new §8.20 "Multi-step cockpit actions (v1.5)" with granular vs
    batched worked examples, abort semantics, recovery + 3-strike
    bail rule, chrome-filter contract, and an opinion on when to flip
    to batched (mirrors loyalty-cli HANDBOOK §8.16 shape)

README.md gains a "Multi-step cockpit actions (BSS_REPL_LLM_AUTONOMY,
v1.5)" subsection between the e2e block and the Documentation map.

.env.example BSS_REPL_LLM_AUTONOMY=granular entry with explanation of
both modes and the fail-closed boot contract.

DECISIONS.md 2026-05-26 v1.5.0 entry covers the five-piece scope
(autonomy env, autonomy-aware gating, ITERATIVE FLOW prompt + softened
v0.19 rule, 3-strike bail, chrome filter), the three rejected
alternatives (plan-emit-then-confirm-once, composite read tools,
per-tool autonomy annotations), and the consequences (compound actions
work in one prompt under batched, doctrine/handbook updates, deferred
v1.5.1/v1.5.2 follow-ups).

make test still green: orchestrator 420, cli 127, bss-cockpit 87.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Three new Playwright specs in packages/bss-e2e/tests/test_v15_multi_step.py
exercise the v1.5 multi-step unlock end-to-end through the cockpit
browser veneer:

1. test_v15_compound_action_granular — operator types a compound
   prompt, turn 1 stages pending_destructive(A), /confirm POSTs,
   turn 2 executes A and the granular wrapper re-blocks B,
   cockpit stages pending_destructive(B). The "second pending row
   appears after /confirm" is the v1.5 signal.
2. test_v15_compound_action_batched — same flow, but under
   BSS_REPL_LLM_AUTONOMY=batched the loop executes BOTH destructives
   autonomously and zero pending rows remain. Skipped by default
   (granular is the v1.5 doctrine default); run via the new
   `make e2e-batched` target.
3. test_v15_case_investigation — chained reads in ONE turn
   (case.list → customer.list → catalog.list_active_offerings →
   one-sentence summary). Tests that v0.19's "Done." rule is
   softened (the agent chains the three reads in one turn) AND
   that the anti-duplication contract is intact (no markdown
   pipe-table re-render of the data the cockpit already showed).

Required fixture entries in packages/bss-e2e/fixtures/cockpit_e2e.json:
  - v15-compound-action-turn1-propose-first-destructive (matches
    "plan compound")
  - v15-compound-action-turn2-after-confirm-tries-two-destructives
    (matches "operator typed /confirm" — the synthetic prompt the
    cockpit injects on /confirm-resumed turns)
  - v15-case-investigation-chained-reads (matches "investigate the
    demo state")
All three use provisioning.set_fault_injection / read-only tools
with safe args so the suite can run repeatedly without dependent
fixtures.

Route + REPL fix — pending_destructive STAGING now derives from
COMPLETED-with-BLOCKED, not STARTED. The pre-v1.5 path set
last_proposal on STARTED for every destructive when
allow_destructive=False; that heuristic missed the v1.5 granular
re-gate (which fires on a /confirm-resumed turn where
allow_destructive=True yet the wrapper still BLOCKS the second
destructive). The new path observes the tool RESULT (BLOCKED
sub-string in event.result_full), captures the matching args via
call_id round-trip on captured_tool_calls, and unconditionally
stages whenever a destructive's result is BLOCKED. Covers both
paths cleanly:
  • pre-v1.5 (allow=False blocks at the gate) → BLOCKED → stage
  • v1.5 granular (second destructive in /confirm-resumed loop
    re-gates with allow=True) → BLOCKED → stage
  • v1.5 batched (both destructives execute) → no BLOCKED → no stage
The staging guard at the bottom of each handler also drops the
`not allow_destructive_this_turn` condition that would have
silently swallowed case 2.

docker-compose.e2e.yml propagates BSS_REPL_LLM_AUTONOMY into
portal-csr with a granular default (Compose env-var interpolation).
Makefile gains a thin `e2e-batched` target that pre-exports the
env before invoking the existing `e2e` recipe.

make test still green: orchestrator 420, cli 127, bss-cockpit 87.
E2E specs themselves require docker compose + Playwright — run
`make e2e` (or `make e2e-batched` for the batched spec) to exercise.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…ched green

Two real bugs surfaced when running the v1.5 specs against a live stack:

1. Fixture args for provisioning.set_fault_injection were wrong against
   the actual tool signature. LangChain pydantic-validates structured
   tool args BEFORE the destructive wrapper runs, so the gate never
   even fired — the tool produced "Error invoking tool ... task_type:
   Input should be 'HLR_PROVISION' | ..." validation errors and the
   COMPLETED-with-BLOCKED detector (correctly) didn't trigger. Fixed
   by using the real enum values:
     - task_type: HLR_PROVISION / ESIM_PROFILE_PREPARE (not the made-up
       create_msisdn / create_esim_profile)
     - fault_type: fail_first_attempt (not fault_kind: fail)
     - enabled: false (was missing — required field)
   This is also why the pre-v1.5 test_cockpit_propose_then_confirm
   spec "passed" before my route handler change: it only asserted on
   the STARTED-side last_proposal capture, which fired regardless of
   whether the wrapper was reached. Now that staging derives from
   COMPLETED-BLOCKED, the fixture args have to actually validate so
   the wrapper actually runs and BLOCKs.

2. The granular + batched specs used POST /cockpit/<id>/confirm to
   simulate /confirm, but that endpoint is a no-op marker (per its
   own docstring) — it doesn't add a synthetic user message or
   trigger a new turn. The actual /confirm-resumed turn fires only
   when the operator TYPES /confirm via the compose form (which the
   cockpit's slash-command interceptor at routes/cockpit.py:755
   catches and rewrites). Switched both specs to use _send_message
   (page, "/confirm") + _wait_for_turn_done — same path a real
   operator takes in the cockpit.

Live verification:
  - make e2e (granular default): 12 passed, 1 skipped (batched gated
    on env), 0 failed in 37.76s. All 10 v1.4.1 specs + the new
    granular + case-investigation specs green.
  - make e2e-batched: 12 passed, 1 skipped (granular gated when
    env=batched), 0 failed in 38.61s. Batched spec exercised the
    autonomous-second-destructive path successfully.

Both autonomy modes now have real end-to-end coverage on the
cockpit browser veneer.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
… warning

Live REPL session caught Gemma narrating the proposal as prose instead of
emitting a real tool_call:

  > terminate subscription SUB-0005
  ╭─ bss ai ───────────────────────────────────────────────────────╮
  │ I propose to terminate subscription SUB-0005.                  │
  │ subscription.terminate(subscription_id="SUB-0005")             │
  │                                                                │
  │ Please type /confirm to proceed.                               │
  ╰────────────────────────────────────────────────────────────────╯
  > /confirm
  >    <-- silence; no pending row was ever staged

The cockpit had three holes, all fixed here:

1. **Prompt rule.** Loyalty-cli's anti-mimicry doctrine ported into
   bss_cockpit/prompts._COCKPIT_INVARIANTS as a (v1.5+) CALL TOOLS —
   DO NOT NARRATE THEM block. Names the failure mode explicitly with
   bad/good examples, drawn from the actual Gemma output above.
   Bad: "I propose ... subscription.terminate(...)", "PROPOSE: ...",
   "Proposed action: ... Type /confirm to proceed." Good: emit the
   structured tool_call (cockpit owns the banner formatting), OR ask
   cleanly in plain prose when more info is needed.

2. **Runtime strip — broader.** chrome_filter.strip_fake_propose was
   pre-this-commit only catching banner-shape mimicry ("PROPOSE:" /
   "⚠ PROPOSE:" line prefixes). Now it also catches narrated-call
   shape: any \b<lower>.<lower>(...)\b pattern in prose, plus the
   "Please type /confirm" boilerplate that wraps it. Conservative
   trade-off — a legitimate "subscription.state field" prose mention
   survives (no parens), but "subscription.terminate(SUB-0001)" in
   prose gets stripped. False-negative cost (chrome leaks through) >
   false-positive cost (one fragment of legitimate prose lost).
   Wired into the live-emit path in BOTH portals/csr cockpit route
   AND cli/bss_cli/repl turn driver; previously only used on history
   rehydration. When strip leaves the bubble empty AND no real
   tool_call fired this turn → surface an explicit "stalled" warning
   instead of an empty bubble that hides the doctrine failure.

3. **/confirm slash handler peek-and-warn.** cli/bss_cli/repl
   /confirm slash handler used to be a no-op marker (returned a
   "_confirmed_marker" string that was never consumed). Now peeks
   cockpit.pending_destructive via the new
   Conversation.peek_pending_destructive (read without consuming —
   _drive_turn still owns the actual pop on the next message). If
   a row exists, reports what's staged + how to drive it. If NO row
   exists, warns loudly with the most common cause (mimicry-narrated
   propose). Catches the operator-typed-/confirm-into-the-void
   silent-bug UX.

5 new chrome_filter unit tests pin the v1.5 narrated-call detector
against verbatim Gemma output + edge cases (multiple tools in one
prose burst, dotted prose without parens left alone, prose around
the call survives the strip).

Verified live:
  - make test: orchestrator 420, cli 127, bss-cockpit 87 → 91, all
    other suites unchanged. All green.
  - make e2e (with rebuilt portal-csr): 12 passed, 1 skipped, 0
    failed in 39.78s. Existing specs unaffected by the broader strip.

REPL test: the operator's exact session shape now produces (a) the
strip removes the narrated call from the bubble, (b) the operator's
follow-up /confirm reports "Nothing pending /confirm" with the
mimicry-explanation hint.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Second pass on the v1.5 anti-mimicry defences after live REPL retest
still showed a half-finished propose bubble:

  > terminate subscription SUB-0005
  ╭─ bss ai ─────────────────────────────────────────────╮
  │ I propose to terminate subscription SUB-0005. ``     │
  ╰──────────────────────────────────────────────────────╯
  >    <-- silence; operator confused; /confirm into the void

Three issues this commit addresses:

1. **Strip didn't go far enough.** The narrated-call regex caught
   `subscription.terminate(subscription_id="SUB-0005")` but the LLM
   had wrapped it in backticks; only the call body got stripped, the
   surrounding backticks survived as ``. The leading
   "I propose to terminate subscription SUB-0005." narration also
   survived. Half-stripped propose still misleads.
   Fixes:
   - Broaden _NARRATED_CALL_RE to consume optional surrounding
     backticks (1-3 ticks, matches inline `call(...)` and triple-
     backtick code fences).
   - New _NARRATION_LEAD_RE strips "I [propose|will|would like|
     intend|am going] to [terminate|cancel|close|remove|refund|
     revoke|propose|call|invoke] ..." sentences when adjacent to
     a stripped call (only runs if a call was actually removed —
     keeps false-positive risk low).
   - New _EMPTY_BACKTICK_RE sweeps any leftover empty `` / ``` pairs.

2. **Warning only fired on empty bubbles.** The previous condition
   was `mimicry_stripped AND not final_text.strip() AND not
   captured_tool_calls`. The user's bubble was non-empty after the
   first-pass strip, so the warning never fired and the misleading
   text reached the operator. New condition:
     no tool_call this turn AND (mimicry_stripped OR mentions /confirm)
   → fire the warning AND replace the bubble with the canonical
   explanation. Independent of whether prose remains — a half-
   finished propose with /confirm intent is the failure shape we're
   catching. Replacement (not append) so the persisted transcript
   doesn't feed Gemma its own half-finished narration on the next
   turn's rehydration.

3. **Prompt rule was too soft.** Rewrote the destructive contract +
   anti-mimicry block in _COCKPIT_INVARIANTS:
   - Destructive contract opener now leads with "MUST be EMITTED AS
     STRUCTURED tool_calls — never written as prose." (was: "must be
     PROPOSED first with a one-line summary and the exact tool name +
     arguments", which Gemma read as "write a one-line text summary").
   - Anti-mimicry block opens with "**This is the most common doctrine
     failure on small models.**" and gives a verbatim BAD example
     (the exact "I propose ... subscription.terminate(...) Please
     type /confirm" shape Gemma produced) followed by the explicit
     failure mode ("the cockpit reports 'no pending action — the
     model narrated...'").

4 new chrome_filter tests pin the v1.5 follow-up shapes:
  - backtick-wrapped call from live REPL → strip removes call + ticks
    + "I propose" narration; bubble ends empty (caller's warning fires)
  - triple-backtick code fence variant
  - narration regex requires destructive canon AND "to" (false-positive
    guard — "I'm going to think" survives intact)
  - narration only fires when a call was stripped this turn

Verified:
  - make test: orchestrator 420, cli 127, bss-cockpit 91→95, all
    other suites unchanged. All green.
  - make e2e (rebuilt portal-csr): 12 passed, 1 skipped, 0 failed
    in 39.55s. Existing specs unaffected by the broader strip + new
    warning trigger.

REPL retest expectation: same input "terminate subscription SUB-0005"
now produces either (a) a real tool_call and a Pending /confirm
yellow line OR (b) the explicit "no pending action — model narrated
proposal in prose" warning + canonical replacement bubble. No more
silent stalls.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…r BLOCK

Live REPL retest revealed two more UX bugs:

1. **REPL /confirm was informational-only.** My previous fix made
   /confirm "peek and report" but never DROVE a turn — so after
   /confirm the operator had to type a second prompt to actually
   re-evoke the destructive. That broke the workflow because Gemma
   rarely re-emits a destructive without explicit prompting; the
   pending row sat unconsumed and the next normal message wiped it
   silently.

   Fix: /confirm slash handler now returns a new "drive_confirm_turn"
   action. run_repl detects that action and drives a real _drive_turn
   with the synthetic "(operator typed /confirm — proceed with the
   prior destructive proposal now; call the tool)" prompt. Same shape
   as the browser cockpit's slash-command interceptor in
   routes/cockpit.py around line 755. The /confirm in REPL now
   actually fires the action.

2. **Gemma's "Done." wrap-up after a destructive BLOCK was misleading.**
   When the LLM emitted a real tool_call for subscription.terminate,
   the wrapper BLOCKED, the cockpit staged pending_destructive (and
   printed "Pending /confirm for ..."), but Gemma's terminal AIMessage
   was "Done." (v0.19 one-short-sentence rule picked the wrong words).
   Operator sees "Pending /confirm for subscription.terminate" AND
   "Done." in the bubble — looks like the action already ran.

   Fix: when last_tool_proposal is set (= a destructive returned
   BLOCKED this turn) AND the bubble is short (< 40 chars, i.e. a
   wrap-up rather than a real explanation), replace the bubble with
   an explicit "Proposed <tool>(args). Type /confirm to authorise."
   The persisted transcript carries the right state; the next turn's
   rehydration won't feed Gemma its own misleading "Done." Same fix
   landed in both portals/csr cockpit route AND cli/bss_cli/repl.

Both fixes preserve pre-v1.5 behaviour for non-destructive turns —
the override only fires when last_proposal is set (= a destructive
BLOCKED was observed), and the new /confirm flow only triggers when
there's actually a pending row.

Verified:
  - make test: orchestrator 420, cli 127, bss-cockpit 95, all green
  - make e2e (rebuilt portal-csr): 12 passed, 1 skipped, 0 failed
    in 40.04s. v15 specs unaffected.

REPL retest now: typing "terminate subscription SUB-0005" stages
pending, bubble reads "Proposed subscription.terminate(subscription_id=
'SUB-0005'). Type /confirm to authorise." Typing /confirm drives a
real turn — terminate actually executes — and the next "show
subscription SUB-0005" shows state=terminated.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…e post-BLOCK

Third-pass observation: Gemma can produce a 79-char "I propose to
terminate SUB-0005. Please type /confirm to proceed." narration that
PASSES the < 40 wrap-up gate (long enough to look like real prose)
AND survives the mimicry-stall warning (real tool_call did fire this
turn, so captured_tool_calls is non-empty). Result: the operator sees
honest "Pending /confirm" status above + misleading narrated propose
below, then types /confirm into a freshly-rebuilt REPL and... the
old code was loaded by their existing `bss` process anyway. Two
separate problems, same UX outcome.

Threshold removal:
  Whenever last_tool_proposal is set (= a destructive returned
  BLOCKED this turn), the bubble is replaced unconditionally with
  the canonical "Proposed X(args). Type /confirm to authorise."
  The pending_destructive row IS the truth; the bubble must match.
  Trade-off: lose any legitimate prose Gemma might have added
  ("This will stop all data charges immediately"). The cost of
  occasionally dropping that vs. the cost of misleading the
  operator about what state the system is in is one-sided. Same
  fix in both REPL and cockpit route.

The REPL-process-cache issue is unrelated to my code — Python
imports cache per-process; the operator was running a `bss` that
loaded the prior commit before /confirm-drives-turn landed. Restart
of `bss` is required between code changes, not a server-rebuild
issue. Document this in the operator-retest instructions next time.

make test green: orchestrator 420, cli 127, bss-cockpit 95.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…of "Done."

Live retest showed the workflow WORKING end-to-end (state went to
TERMINATED after /confirm) but the post-execute bubble was the same
"Done." word Gemma uses for every successful tool call. After the
operator just SAW "Done." on a stalled-propose path (where it was a
lie), seeing "Done." again — even though THIS time the action really
fired — is exactly what teaches them not to trust the cockpit.

The propose-side already overrides Gemma's wrap-up with the canonical
"Proposed X(args). Type /confirm to authorise." This commit lands the
symmetric override for the execute side.

Track destructive tool_calls that ACTUALLY RAN this turn (wrapper
let them through, result wasn't BLOCKED) as the new
executed_destructive list, parallel to last_tool_proposal /
last_proposal. After the turn's FinalMessage:
  - If last_proposal set (propose path): "Proposed X(args). Type /confirm ..."
  - Elif executed_destructive (execute path): "Executed X(args)."
    (or for compound under batched, "Executed N destructive actions,
    last was X(args).")
  - Else: Gemma's wrap-up (the v0.19 anti-duplication rule keeps it short)

So a /confirm-resumed turn now produces a bubble that names what
actually happened:

  > /confirm
  /confirm staged for subscription.terminate args={'subscription_id': 'SUB-0005'}
  ╭─ bss ai ──────────────────────────────────────────────────────────────╮
  │ Executed subscription.terminate(subscription_id='SUB-0005').          │
  ╰───────────────────────────────────────────────────────────────────────╯

Same fix landed in both REPL and cockpit route.

make test green: orchestrator 420, cli 127, bss-cockpit 95.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Single-line release version bump — picked up by every surface that
reads the constant (REPL banner, self-serve portal brand-tag, CSR
cockpit health, every service /health endpoint).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@chiam-ck chiam-ck merged commit e0aa64b into main May 26, 2026
@chiam-ck chiam-ck deleted the feat/v1.5-multi-step-cockpit branch May 26, 2026 06:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant