fix: read loop watch-events with a stream-global cursor - #356
Conversation
loop_run_events.seq restarts at 1 for every loop run, but the watch-events loop-stream cursor treated it as a workspace-global stream position: arming snapshotted MAX(seq) across all runs and wake re-derivation read seq > cursor. Any event from a run whose per-run seq stayed below the historical maximum was permanently invisible, so a parked watch-events Loop enqueued its wake (the hook doorbell matched), re-read the ledger, found nothing, and re-parked forever without delivering the batch. Use the table-global rowid as the loop-stream cursor for both the cursor snapshot and the match read, mirroring the observe stream. Per-run seq keeps serving the SSE resume contract unchanged. Repro (isolated daemon): run a loop that accumulates more events than a fresh run ever reaches, arm a watch-events Loop on loop.terminal, then terminate a fresh matching run — the wake fired but delivered no batch and the cursor never advanced. With this change the wake delivers the terminal event and advances the cursor past it. Co-Authored-By: Claude Fable 5 <[email protected]>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
WalkthroughLoop watch-event cursors now use durable workspace-scoped ChangesLoop watch-event cursor
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant ParkedLoop
participant WatchCoordinator
participant GlobalDB
participant LoopRunEvents
ParkedLoop->>WatchCoordinator: resume with cursor_version 1
WatchCoordinator->>GlobalDB: read eligible events after watch_seq
GlobalDB->>LoopRunEvents: query workspace-scoped ordered events
LoopRunEvents-->>GlobalDB: return matching events
GlobalDB-->>WatchCoordinator: return matches and next cursor
WatchCoordinator-->>ParkedLoop: deliver events or persist updated cursor
🚥 Pre-merge checks | ✅ 2 | ❌ 3❌ Failed checks (3 warnings)
✅ Passed checks (2 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
| Filename | Overview |
|---|---|
| internal/store/globaldb/schema/migrations/00061_schema.sql | Rebuilds the loop-event ledger with durable stream positions and converts parked watch-event cursor state to the new namespace. |
| internal/store/globaldb/global_db_watch_events_loop.go | Uses one shared eligibility predicate and monotonic watch_seq ordering for loop-stream cursor and replay reads. |
| internal/loop/watch/output.go | Versions persisted watch-event cursor values and rejects unsupported cursor namespaces. |
| internal/store/globaldb/global_db_watch_events_test.go | Adds regression, filtering, pagination, migration, and replay coverage for the durable loop-event cursor. |
| internal/daemon/loop_run_events_e2e_integration_test.go | Expands end-to-end coverage for parked-state parity, workspace isolation, cursor progress, and restart recovery. |
Sequence Diagram
sequenceDiagram
participant Watcher as Parked loop watcher
participant Repo as Watch-events repository
participant Ledger as loop_run_events
participant Coordinator as Loop coordinator
Watcher->>Repo: Arm with subscriptions
Repo->>Ledger: Read latest eligible watch_seq
Ledger-->>Repo: Stream cursor
Repo-->>Watcher: Persist versioned cursor
Ledger->>Ledger: Append event with next watch_seq
Coordinator->>Repo: Check for cursor gap
Repo->>Ledger: "Read eligible rows where watch_seq > cursor"
Ledger-->>Repo: Ordered event batch
Repo-->>Coordinator: Matches and advanced cursor
Coordinator->>Watcher: Wake and continue loop
Reviews (8): Last reviewed commit: "fix: resolve loop watch cursor review fi..." | Re-trigger Greptile
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@internal/store/globaldb/global_db_watch_events_test.go`:
- Around line 270-277: Update the test around the terminalSeen check to query
loop_run_events for the persisted per-run sequence values, identify the fresh
terminal event and noisy run maximum, and assert the fresh sequence is lower
before calling ReadMatches. Keep the existing event.Seq > armedCursor assertion,
while ensuring the setup explicitly preserves the lower-sequence precondition
that would fail with the old cursor implementation.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: a1ae70cf-01a7-44fb-ad84-c396de1c3f8c
📒 Files selected for processing (3)
internal/store/globaldb/global_db_watch_events.gointernal/store/globaldb/global_db_watch_events_loop.gointernal/store/globaldb/global_db_watch_events_test.go
The regression only discriminates against the old per-run cursor while the fresh run's persisted seq stays below the noisy run's maximum. Assert that precondition from the durable ledger so future setup drift cannot silently hollow out the test. Co-Authored-By: Claude Fable 5 <[email protected]>
|
@franciscpd is attempting to deploy a commit to the Compozy Team on Vercel. A member of the Team first needs to authorize it. |
Co-Authored-By: Claude Fable 5 <[email protected]>
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (3)
internal/loop/watch/output.go (1)
110-119: 🩺 Stability & Availability | 🔵 Trivial | 🏗️ Heavy liftConsider a rearm path instead of a permanent error for an unsupported cursor version.
An unsupported
cursor_versioncurrently produces an error on every decode. Both known callers propagate it:evaluateWatchEventsNodeininternal/loop/coordinator_watch_events.goreturns it as a coordinator failure, andloopWatchEventsReadModelininternal/daemon/loop_api_watch_events.gofails the read model. A parked Loop in that state has no recovery path short of manual database repair.An alternative is to treat an unknown version as "cursors are not usable", discard the stored cursors, and let
recoverWatchEventsStaterearm from the current stream position. That degrades to a hard cut for one run instead of blocking it. The current fail-closed behavior is defensible; record the operational recovery procedure if you keep it.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/loop/watch/output.go` around lines 110 - 119, The unsupported cursor-version branch in the pending-state decode should recover instead of returning a permanent error: discard the stored cursors, retain the subscriptions, and return state that allows recoverWatchEventsState to rearm from the current stream position. Update the logic around EventsCursorVersion and ensure both evaluateWatchEventsNode and loopWatchEventsReadModel can continue through this recovery path.internal/daemon/loop_run_events_e2e_integration_test.go (1)
965-971: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winSplit workspace creation out of the parity assertion helper.
assertWatchEventsReadModelParityasserts read-model parity across four transports, then creates a foreign workspace and returns its ID. The name states one responsibility; the function performs two. The caller at Line 256 has to write_ = assertWatchEventsReadModelParity(...)to discard a value it does not need.Move the foreign-workspace creation and the 404 check into their own helper. The parity helper then returns nothing, and Line 256 reads cleanly.
♻️ Proposed split
-) string { +) { t.Helper() ... - foreignWorkspaceID := createWorkspaceViaUDS(t, ctx, harness, t.TempDir(), "foreign-workspace") - foreignPath := ... - ... - return foreignWorkspaceID } + +func assertForeignWorkspaceLoopRunNotFound( + t testing.TB, + ctx context.Context, + harness *e2etest.RuntimeHarness, + runID string, +) string { + t.Helper() + foreignWorkspaceID := createWorkspaceViaUDS(t, ctx, harness, t.TempDir(), "foreign-workspace") + // existing 404 assertion + return foreignWorkspaceID +}Then update Line 179 to call both, and Line 256 to call only the parity helper.
Also applies to: 1034-1051
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/daemon/loop_run_events_e2e_integration_test.go` around lines 965 - 971, Split the foreign-workspace creation and 404 verification out of assertWatchEventsReadModelParity into a separate helper that returns the workspace ID. Change assertWatchEventsReadModelParity to return nothing while retaining only the four-transport parity assertions, then update the callers so the setup path invokes both helpers and the later call invokes only the parity helper without discarding a return value.internal/store/globaldb/global_db_watch_events_loop.go (1)
22-27: 🚀 Performance & Scalability | 🔵 Trivial | ⚖️ Poor tradeoffConsider the scan cost of the cursor aggregate on large workspaces.
idx_loop_run_events_watch_streamcovers(workspace_id, watch_seq), but the eligibility predicate filters onkindand on ajson_extractofpayload_json. SQLite cannot satisfyMAX(watch_seq)from the index under those filters, so this aggregate scans the full workspace partition ofloop_run_events. The read path bounds its work withwatch_seq > ?andLIMIT; the cursor path does not.
ReadCursorsruns at arming time and during gap reconciliation, so the cost is bounded today. If the loop event ledger grows without pruning, add a descending scan bound or a retention sweep.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/store/globaldb/global_db_watch_events_loop.go` around lines 22 - 27, Bound the cursor aggregate in ReadCursors so it does not scan an unbounded loop_run_events workspace partition. Prefer reusing or adding a retention sweep for loop_run_events; alternatively apply a correct descending scan bound while preserving the maximum eligible watch_seq result from loopWatchEventsEligibilitySQL.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@internal/daemon/loop_run_events_e2e_integration_test.go`:
- Around line 186-194: Make the negative assertion deterministic instead of
relying on the fixed 300 ms timeout in assertLoopRunRemainsParked. Instrument or
reuse a daemon signal that confirms the non-matching event was evaluated and
discarded, or measure the matching wake latency via waitForLoopRunStatus and set
both quiet windows above the observed latency while preserving the parked-state
assertion.
In `@internal/store/globaldb/global_db_watch_events_test.go`:
- Around line 1221-1228: Refactor TestGlobalDBWatchEventsCursorMigration to use
t.Run subtests named with the “Should...” pattern, splitting the scenario into
focused cases for identity preservation, cursor rearming at the migration fence,
gap detection, post-migration replay, and replay after reopen. Keep the existing
migration setup and assertions appropriate to each behavior within its
corresponding subtest.
In `@internal/store/globaldb/schema/migrations/00060_schema.sql`:
- Around line 30-45: Restrict the migration update guarded by json_valid and the
watch_events_pending checks to lgo rows whose owning loop_runs record exists for
lgo.loop_run_id. Add an EXISTS-based guard using the existing loop_runs
relationship, so missing runs are excluded and their legacy cursor remains
unchanged instead of being set to 0.
---
Nitpick comments:
In `@internal/daemon/loop_run_events_e2e_integration_test.go`:
- Around line 965-971: Split the foreign-workspace creation and 404 verification
out of assertWatchEventsReadModelParity into a separate helper that returns the
workspace ID. Change assertWatchEventsReadModelParity to return nothing while
retaining only the four-transport parity assertions, then update the callers so
the setup path invokes both helpers and the later call invokes only the parity
helper without discarding a return value.
In `@internal/loop/watch/output.go`:
- Around line 110-119: The unsupported cursor-version branch in the
pending-state decode should recover instead of returning a permanent error:
discard the stored cursors, retain the subscriptions, and return state that
allows recoverWatchEventsState to rearm from the current stream position. Update
the logic around EventsCursorVersion and ensure both evaluateWatchEventsNode and
loopWatchEventsReadModel can continue through this recovery path.
In `@internal/store/globaldb/global_db_watch_events_loop.go`:
- Around line 22-27: Bound the cursor aggregate in ReadCursors so it does not
scan an unbounded loop_run_events workspace partition. Prefer reusing or adding
a retention sweep for loop_run_events; alternatively apply a correct descending
scan bound while preserving the maximum eligible watch_seq result from
loopWatchEventsEligibilitySQL.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 9efd3115-007a-4dc4-8c27-4a7cbd50272d
⛔ Files ignored due to path filters (9)
docs/qa/reports/2026-08-12-pr-356-loop-watch-cursor.mdis excluded by!**/*.mddocs/qa/scenarios/LP-040.mdis excluded by!**/*.mddocs/qa/scenarios/LP-041.mdis excluded by!**/*.mddocs/qa/scenarios/LP-042.mdis excluded by!**/*.mddocs/qa/scenarios/LP-044.mdis excluded by!**/*.mdinternal/store/globaldb/schema/migrations/atlas.sumis excluded by!**/*.sum,!**/*.sumpackages/site/content/docs/loops/reference-grammar.mdxis excluded by!**/*.mdxskills/compozy/references/loops.mdis excluded by!**/*.mdweb/e2e/__tests__/loops.spec.tsis excluded by!web/e2e/**
📒 Files selected for processing (12)
internal/daemon/loop_run_events_e2e_integration_test.gointernal/loop/watch/adapter_test.gointernal/loop/watch/output.gointernal/store/globaldb/global_db_loop_schema_integration_test.gointernal/store/globaldb/global_db_watch_events.gointernal/store/globaldb/global_db_watch_events_loop.gointernal/store/globaldb/global_db_watch_events_test.gointernal/store/globaldb/queries/loop_core.sqlinternal/store/globaldb/schema/definitions/50_loops.sqlinternal/store/globaldb/schema/migrations/00060_schema.sqlinternal/store/globaldb/sqlcgen/loop_core.sql.gointernal/store/globaldb/sqlcgen/models.go
|
CodeRabbit follow-up for 3ea8a1e:
Focused GlobalDB tests, Go compilation, test-convention checks, and make codegen-check passed. Full gates are delegated to CI. |
Closes #355.
Problem
loop_run_events.seqrestarts at 1 for every loop run, but the watch-events loop-stream cursor treated it as a workspace-global stream position: arming snapshottedMAX(seq)across all runs, and wake re-derivation readseq > cursor. Any event from a run whose per-run seq stayed below the historical maximum was permanently invisible — the hook doorbell matched and enqueued the wake, the coordinator woke, read nothing, and re-parked forever without delivering the batch.Live-daemon evidence and the deterministic repro (transform-only loops, no agents) are in #355. The failure signature is fully silent:
matched+wake_enqueuedinevent_summaries, wake runcompletedwith no error, cursor frozen, Loop dormant.Change
Both loop-stream queries in the watch-events repo now cursor on the table-global
rowidinstead of the per-runseq:ReadCursors:MAX(lre.rowid)(wasMAX(lre.seq));ReadMatches:WHERE lre.rowid > ?ordered byrowid(wasseq), withrowidprojected as the event's cursor value.This mirrors the observe stream, which already cursors
event_summariesonMAX(rowid). The per-runseqcolumn is untouched and keeps serving the per-run SSE resume contract. The gap-recovery reconcile (EnqueueWatchEventsGapWakes*) goes through the same two functions, so boot/backstop recovery becomes consistent as well. No schema change and no migration, per the repo's hard-cut policy.Design note:
rowid(implicit, TEXT PK table) is renumberable byVACUUM, but the global DB is never vacuumed (only sessiondb is) — the same exposure the observe stream already accepts. If an explicitAUTOINCREMENTcolumn (the automation-stream pattern) is preferred for the loop ledger, happy to rework in that direction.Coverage
global_db_watch_events_test.go: a run inflates the ledger, a watcher arms (cursor snapshot), a fresh run terminates at a lower per-run seq — its terminal must appear strictly after the armed cursor and advance it. Red on the old queries, green now.internal/store,internal/loop,internal/daemonsuites pass; watch-events tests pass under-race.🤖 Generated with Claude Code
Summary by CodeRabbit
Bug Fixes
New Features
Tests