Thanks to visit codestin.com
Credit goes to github.com

Skip to content

fix: read loop watch-events with a stream-global cursor - #356

Merged
pedronauck merged 8 commits into
compozy:mainfrom
franciscpd:fix/watch-events-loop-stream-cursor
Aug 12, 2026
Merged

fix: read loop watch-events with a stream-global cursor#356
pedronauck merged 8 commits into
compozy:mainfrom
franciscpd:fix/watch-events-loop-stream-cursor

Conversation

@franciscpd

@franciscpd franciscpd commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Closes #355.

Problem

loop_run_events.seq restarts at 1 for every loop run, but the watch-events loop-stream cursor treated it as a workspace-global stream position: arming snapshotted MAX(seq) across all runs, and wake re-derivation read seq > cursor. Any event from a run whose per-run seq stayed below the historical maximum was permanently invisible — the hook doorbell matched and enqueued the wake, the coordinator woke, read nothing, and re-parked forever without delivering the batch.

Live-daemon evidence and the deterministic repro (transform-only loops, no agents) are in #355. The failure signature is fully silent: matched + wake_enqueued in event_summaries, wake run completed with no error, cursor frozen, Loop dormant.

Change

Both loop-stream queries in the watch-events repo now cursor on the table-global rowid instead of the per-run seq:

  • ReadCursors: MAX(lre.rowid) (was MAX(lre.seq));
  • ReadMatches: WHERE lre.rowid > ? ordered by rowid (was seq), with rowid projected as the event's cursor value.

This mirrors the observe stream, which already cursors event_summaries on MAX(rowid). The per-run seq column is untouched and keeps serving the per-run SSE resume contract. The gap-recovery reconcile (EnqueueWatchEventsGapWakes*) goes through the same two functions, so boot/backstop recovery becomes consistent as well. No schema change and no migration, per the repo's hard-cut policy.

Design note: rowid (implicit, TEXT PK table) is renumberable by VACUUM, but the global DB is never vacuumed (only sessiondb is) — the same exposure the observe stream already accepts. If an explicit AUTOINCREMENT column (the automation-stream pattern) is preferred for the loop ledger, happy to rework in that direction.

Coverage

  • New regression in global_db_watch_events_test.go: a run inflates the ledger, a watcher arms (cursor snapshot), a fresh run terminates at a lower per-run seq — its terminal must appear strictly after the armed cursor and advance it. Red on the old queries, green now.
  • Full internal/store, internal/loop, internal/daemon suites pass; watch-events tests pass under -race.
  • Validated end-to-end on an isolated daemon: armed cursor 129, terminal at 138 delivered, batch landed at the downstream node, cursor advanced; a previously-stuck parked run from the failing build recovered on restart via the gap reconcile and delivered its missed event.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes

    • Fixed loop event pagination with globally unique positions, preventing missed or duplicated events across runs.
    • Improved detection of fresh terminal events after cursor advancement.
    • Strengthened workspace isolation and handling of deleted, orphaned, and nonterminal events.
    • Preserved watch-event history and cursor state during migrations and restarts.
  • New Features

    • Added cursor versioning for safely restoring pending watch-event output.
  • Tests

    • Expanded coverage for pagination, migration recovery, replay, durability, and cross-interface consistency.

loop_run_events.seq restarts at 1 for every loop run, but the watch-events
loop-stream cursor treated it as a workspace-global stream position: arming
snapshotted MAX(seq) across all runs and wake re-derivation read seq > cursor.
Any event from a run whose per-run seq stayed below the historical maximum
was permanently invisible, so a parked watch-events Loop enqueued its wake
(the hook doorbell matched), re-read the ledger, found nothing, and re-parked
forever without delivering the batch.

Use the table-global rowid as the loop-stream cursor for both the cursor
snapshot and the match read, mirroring the observe stream. Per-run seq keeps
serving the SSE resume contract unchanged.

Repro (isolated daemon): run a loop that accumulates more events than a
fresh run ever reaches, arm a watch-events Loop on loop.terminal, then
terminate a fresh matching run — the wake fired but delivered no batch and
the cursor never advanced. With this change the wake delivers the terminal
event and advances the cursor past it.

Co-Authored-By: Claude Fable 5 <[email protected]>
@franciscpd
franciscpd requested a review from pedronauck as a code owner August 12, 2026 09:53
@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: e95b02da-fb96-4dad-bf77-c46b6ecd9a8a

📥 Commits

Reviewing files that changed from the base of the PR and between d1795a5 and a9c1563.

📒 Files selected for processing (1)
  • internal/store/globaldb/global_db_watch_events_test.go

Walkthrough

Loop watch-event cursors now use durable workspace-scoped watch_seq values instead of per-run sequence values. The migration preserves existing events and initializes parked cursors. Cursor versioning and integration tests cover recovery, replay, and workspace isolation.

Changes

Loop watch-event cursor

Layer / File(s) Summary
Durable watch sequence storage
internal/store/globaldb/schema/..., internal/store/globaldb/queries/loop_core.sql, internal/store/globaldb/sqlcgen/*, internal/store/globaldb/global_db_loop_schema_integration_test.go
loop_run_events now stores an autoincrementing watch_seq and a unique event ID. The migration preserves event data and parked cursor state. Generated queries and schema tests expose and validate the new sequence.
Cursor arming and eligible event reads
internal/store/globaldb/global_db_watch_events.go, internal/store/globaldb/global_db_watch_events_loop.go, internal/store/globaldb/global_db_watch_events_test.go
Loop cursors use the maximum eligible watch_seq. Reads apply shared workspace, event-kind, and terminal-status eligibility rules, then filter and order by watch_seq. Regression tests cover pagination, deletion, multiple runs, orphan events, and migration replay.
Versioned cursor recovery and integration validation
internal/loop/watch/output.go, internal/loop/watch/adapter_test.go, internal/daemon/loop_run_events_e2e_integration_test.go
Persisted watch-event references include cursor version 1. Recovery rejects unsupported versions. Integration tests validate read-model parity, restart durability, cross-workspace isolation, and parked-loop stability.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

  • compozy/compozy#300: Both changes modify loop watch-event infrastructure and lifecycle-event handling.
  • compozy/compozy#301: Both changes modify loop watch-event eligibility queries and status filtering.
  • compozy/compozy#290: Both changes modify durable loop watch-event behavior and persistence.

Suggested reviewers: pedronauck

Sequence Diagram(s)

sequenceDiagram
  participant ParkedLoop
  participant WatchCoordinator
  participant GlobalDB
  participant LoopRunEvents
  ParkedLoop->>WatchCoordinator: resume with cursor_version 1
  WatchCoordinator->>GlobalDB: read eligible events after watch_seq
  GlobalDB->>LoopRunEvents: query workspace-scoped ordered events
  LoopRunEvents-->>GlobalDB: return matching events
  GlobalDB-->>WatchCoordinator: return matches and next cursor
  WatchCoordinator-->>ParkedLoop: deliver events or persist updated cursor
Loading
🚥 Pre-merge checks | ✅ 2 | ❌ 3

❌ Failed checks (3 warnings)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning The implementation uses a new watch_seq column and migration instead of the required table-global rowid, violating issue #355's no-schema-change constraint. Use existing table-global rowid for cursor snapshots, matching, ordering, and gap recovery; keep per-run seq for SSE and remove the schema migration.
Out of Scope Changes check ⚠️ Warning The PR adds schema migration, cursor-version persistence, and migration coverage outside issue #355's rowid-only fix. Remove schema and cursor-version changes and limit the patch to rowid-based cursor snapshots, reads, ordering, and gap-recovery reconciliation.
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the main change: reading loop watch-events with a stream-global cursor.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@greptile-apps

greptile-apps Bot commented Aug 12, 2026

Copy link
Copy Markdown

Greptile Summary

The PR replaces per-run loop-event cursors with a durable, stream-wide sequence and migrates parked watcher state to the new cursor namespace.

  • Adds an auto-incrementing watch_seq to the loop-event ledger and uses it consistently for cursor snapshots and ordered replay.
  • Aligns cursor and match eligibility for workspace, event kind, and terminal status.
  • Versions persisted watch-event cursors and migrates existing parked outputs.
  • Expands storage, migration, daemon, interface-parity, and browser coverage.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains in the eligible follow-up review scope.

No blocking failure remains.

Important Files Changed

Filename Overview
internal/store/globaldb/schema/migrations/00061_schema.sql Rebuilds the loop-event ledger with durable stream positions and converts parked watch-event cursor state to the new namespace.
internal/store/globaldb/global_db_watch_events_loop.go Uses one shared eligibility predicate and monotonic watch_seq ordering for loop-stream cursor and replay reads.
internal/loop/watch/output.go Versions persisted watch-event cursor values and rejects unsupported cursor namespaces.
internal/store/globaldb/global_db_watch_events_test.go Adds regression, filtering, pagination, migration, and replay coverage for the durable loop-event cursor.
internal/daemon/loop_run_events_e2e_integration_test.go Expands end-to-end coverage for parked-state parity, workspace isolation, cursor progress, and restart recovery.

Sequence Diagram

sequenceDiagram
  participant Watcher as Parked loop watcher
  participant Repo as Watch-events repository
  participant Ledger as loop_run_events
  participant Coordinator as Loop coordinator

  Watcher->>Repo: Arm with subscriptions
  Repo->>Ledger: Read latest eligible watch_seq
  Ledger-->>Repo: Stream cursor
  Repo-->>Watcher: Persist versioned cursor
  Ledger->>Ledger: Append event with next watch_seq
  Coordinator->>Repo: Check for cursor gap
  Repo->>Ledger: "Read eligible rows where watch_seq > cursor"
  Ledger-->>Repo: Ordered event batch
  Repo-->>Coordinator: Matches and advanced cursor
  Coordinator->>Watcher: Wake and continue loop
Loading

Reviews (8): Last reviewed commit: "fix: resolve loop watch cursor review fi..." | Re-trigger Greptile

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/store/globaldb/global_db_watch_events_test.go`:
- Around line 270-277: Update the test around the terminalSeen check to query
loop_run_events for the persisted per-run sequence values, identify the fresh
terminal event and noisy run maximum, and assert the fresh sequence is lower
before calling ReadMatches. Keep the existing event.Seq > armedCursor assertion,
while ensuring the setup explicitly preserves the lower-sequence precondition
that would fail with the old cursor implementation.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: a1ae70cf-01a7-44fb-ad84-c396de1c3f8c

📥 Commits

Reviewing files that changed from the base of the PR and between 8947b9b and 9d5ff08.

📒 Files selected for processing (3)
  • internal/store/globaldb/global_db_watch_events.go
  • internal/store/globaldb/global_db_watch_events_loop.go
  • internal/store/globaldb/global_db_watch_events_test.go

Comment thread internal/store/globaldb/global_db_watch_events_test.go
The regression only discriminates against the old per-run cursor while the
fresh run's persisted seq stays below the noisy run's maximum. Assert that
precondition from the durable ledger so future setup drift cannot silently
hollow out the test.

Co-Authored-By: Claude Fable 5 <[email protected]>
@vercel

vercel Bot commented Aug 12, 2026

Copy link
Copy Markdown

@franciscpd is attempting to deploy a commit to the Compozy Team on Vercel.

A member of the Team first needs to authorize it.

coderabbitai[bot]
coderabbitai Bot previously approved these changes Aug 12, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (3)
internal/loop/watch/output.go (1)

110-119: 🩺 Stability & Availability | 🔵 Trivial | 🏗️ Heavy lift

Consider a rearm path instead of a permanent error for an unsupported cursor version.

An unsupported cursor_version currently produces an error on every decode. Both known callers propagate it: evaluateWatchEventsNode in internal/loop/coordinator_watch_events.go returns it as a coordinator failure, and loopWatchEventsReadModel in internal/daemon/loop_api_watch_events.go fails the read model. A parked Loop in that state has no recovery path short of manual database repair.

An alternative is to treat an unknown version as "cursors are not usable", discard the stored cursors, and let recoverWatchEventsState rearm from the current stream position. That degrades to a hard cut for one run instead of blocking it. The current fail-closed behavior is defensible; record the operational recovery procedure if you keep it.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/loop/watch/output.go` around lines 110 - 119, The unsupported
cursor-version branch in the pending-state decode should recover instead of
returning a permanent error: discard the stored cursors, retain the
subscriptions, and return state that allows recoverWatchEventsState to rearm
from the current stream position. Update the logic around EventsCursorVersion
and ensure both evaluateWatchEventsNode and loopWatchEventsReadModel can
continue through this recovery path.
internal/daemon/loop_run_events_e2e_integration_test.go (1)

965-971: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Split workspace creation out of the parity assertion helper.

assertWatchEventsReadModelParity asserts read-model parity across four transports, then creates a foreign workspace and returns its ID. The name states one responsibility; the function performs two. The caller at Line 256 has to write _ = assertWatchEventsReadModelParity(...) to discard a value it does not need.

Move the foreign-workspace creation and the 404 check into their own helper. The parity helper then returns nothing, and Line 256 reads cleanly.

♻️ Proposed split
-) string {
+) {
 	t.Helper()
 	...
-	foreignWorkspaceID := createWorkspaceViaUDS(t, ctx, harness, t.TempDir(), "foreign-workspace")
-	foreignPath := ...
-	...
-	return foreignWorkspaceID
 }
+
+func assertForeignWorkspaceLoopRunNotFound(
+	t testing.TB,
+	ctx context.Context,
+	harness *e2etest.RuntimeHarness,
+	runID string,
+) string {
+	t.Helper()
+	foreignWorkspaceID := createWorkspaceViaUDS(t, ctx, harness, t.TempDir(), "foreign-workspace")
+	// existing 404 assertion
+	return foreignWorkspaceID
+}

Then update Line 179 to call both, and Line 256 to call only the parity helper.

Also applies to: 1034-1051

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/daemon/loop_run_events_e2e_integration_test.go` around lines 965 -
971, Split the foreign-workspace creation and 404 verification out of
assertWatchEventsReadModelParity into a separate helper that returns the
workspace ID. Change assertWatchEventsReadModelParity to return nothing while
retaining only the four-transport parity assertions, then update the callers so
the setup path invokes both helpers and the later call invokes only the parity
helper without discarding a return value.
internal/store/globaldb/global_db_watch_events_loop.go (1)

22-27: 🚀 Performance & Scalability | 🔵 Trivial | ⚖️ Poor tradeoff

Consider the scan cost of the cursor aggregate on large workspaces.

idx_loop_run_events_watch_stream covers (workspace_id, watch_seq), but the eligibility predicate filters on kind and on a json_extract of payload_json. SQLite cannot satisfy MAX(watch_seq) from the index under those filters, so this aggregate scans the full workspace partition of loop_run_events. The read path bounds its work with watch_seq > ? and LIMIT; the cursor path does not.

ReadCursors runs at arming time and during gap reconciliation, so the cost is bounded today. If the loop event ledger grows without pruning, add a descending scan bound or a retention sweep.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/store/globaldb/global_db_watch_events_loop.go` around lines 22 - 27,
Bound the cursor aggregate in ReadCursors so it does not scan an unbounded
loop_run_events workspace partition. Prefer reusing or adding a retention sweep
for loop_run_events; alternatively apply a correct descending scan bound while
preserving the maximum eligible watch_seq result from
loopWatchEventsEligibilitySQL.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/daemon/loop_run_events_e2e_integration_test.go`:
- Around line 186-194: Make the negative assertion deterministic instead of
relying on the fixed 300 ms timeout in assertLoopRunRemainsParked. Instrument or
reuse a daemon signal that confirms the non-matching event was evaluated and
discarded, or measure the matching wake latency via waitForLoopRunStatus and set
both quiet windows above the observed latency while preserving the parked-state
assertion.

In `@internal/store/globaldb/global_db_watch_events_test.go`:
- Around line 1221-1228: Refactor TestGlobalDBWatchEventsCursorMigration to use
t.Run subtests named with the “Should...” pattern, splitting the scenario into
focused cases for identity preservation, cursor rearming at the migration fence,
gap detection, post-migration replay, and replay after reopen. Keep the existing
migration setup and assertions appropriate to each behavior within its
corresponding subtest.

In `@internal/store/globaldb/schema/migrations/00060_schema.sql`:
- Around line 30-45: Restrict the migration update guarded by json_valid and the
watch_events_pending checks to lgo rows whose owning loop_runs record exists for
lgo.loop_run_id. Add an EXISTS-based guard using the existing loop_runs
relationship, so missing runs are excluded and their legacy cursor remains
unchanged instead of being set to 0.

---

Nitpick comments:
In `@internal/daemon/loop_run_events_e2e_integration_test.go`:
- Around line 965-971: Split the foreign-workspace creation and 404 verification
out of assertWatchEventsReadModelParity into a separate helper that returns the
workspace ID. Change assertWatchEventsReadModelParity to return nothing while
retaining only the four-transport parity assertions, then update the callers so
the setup path invokes both helpers and the later call invokes only the parity
helper without discarding a return value.

In `@internal/loop/watch/output.go`:
- Around line 110-119: The unsupported cursor-version branch in the
pending-state decode should recover instead of returning a permanent error:
discard the stored cursors, retain the subscriptions, and return state that
allows recoverWatchEventsState to rearm from the current stream position. Update
the logic around EventsCursorVersion and ensure both evaluateWatchEventsNode and
loopWatchEventsReadModel can continue through this recovery path.

In `@internal/store/globaldb/global_db_watch_events_loop.go`:
- Around line 22-27: Bound the cursor aggregate in ReadCursors so it does not
scan an unbounded loop_run_events workspace partition. Prefer reusing or adding
a retention sweep for loop_run_events; alternatively apply a correct descending
scan bound while preserving the maximum eligible watch_seq result from
loopWatchEventsEligibilitySQL.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 9efd3115-007a-4dc4-8c27-4a7cbd50272d

📥 Commits

Reviewing files that changed from the base of the PR and between 89a51fe and d1795a5.

⛔ Files ignored due to path filters (9)
  • docs/qa/reports/2026-08-12-pr-356-loop-watch-cursor.md is excluded by !**/*.md
  • docs/qa/scenarios/LP-040.md is excluded by !**/*.md
  • docs/qa/scenarios/LP-041.md is excluded by !**/*.md
  • docs/qa/scenarios/LP-042.md is excluded by !**/*.md
  • docs/qa/scenarios/LP-044.md is excluded by !**/*.md
  • internal/store/globaldb/schema/migrations/atlas.sum is excluded by !**/*.sum, !**/*.sum
  • packages/site/content/docs/loops/reference-grammar.mdx is excluded by !**/*.mdx
  • skills/compozy/references/loops.md is excluded by !**/*.md
  • web/e2e/__tests__/loops.spec.ts is excluded by !web/e2e/**
📒 Files selected for processing (12)
  • internal/daemon/loop_run_events_e2e_integration_test.go
  • internal/loop/watch/adapter_test.go
  • internal/loop/watch/output.go
  • internal/store/globaldb/global_db_loop_schema_integration_test.go
  • internal/store/globaldb/global_db_watch_events.go
  • internal/store/globaldb/global_db_watch_events_loop.go
  • internal/store/globaldb/global_db_watch_events_test.go
  • internal/store/globaldb/queries/loop_core.sql
  • internal/store/globaldb/schema/definitions/50_loops.sql
  • internal/store/globaldb/schema/migrations/00060_schema.sql
  • internal/store/globaldb/sqlcgen/loop_core.sql.go
  • internal/store/globaldb/sqlcgen/models.go

Comment thread internal/daemon/loop_run_events_e2e_integration_test.go Outdated
Comment thread internal/store/globaldb/global_db_watch_events_test.go Outdated
Comment thread internal/store/globaldb/schema/migrations/00060_schema.sql Outdated
@pedronauck

Copy link
Copy Markdown
Member

CodeRabbit follow-up for 3ea8a1e:

  • Replaced the fixed 300 ms negative wait with durable cursor/gap conditions.
  • Split foreign-workspace setup from the transport parity helper.
  • Renumbered the watch cursor migration to 00061 after fix: reject automation trigger events no producer emits #358 and guarded missing owning runs.
  • Replaced the unbounded MAX aggregate with a descending indexed cursor lookup.
  • Kept unsupported cursor versions fail-closed intentionally: a newer cursor namespace must not be silently discarded or rearmed by an older binary, matching the project schema-ahead policy.

Focused GlobalDB tests, Go compilation, test-convention checks, and make codegen-check passed. Full gates are delegated to CI.

@pedronauck
pedronauck merged commit 765eba1 into compozy:main Aug 12, 2026
22 of 24 checks passed
@franciscpd
franciscpd deleted the fix/watch-events-loop-stream-cursor branch August 12, 2026 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

watch-events: loop-stream cursor uses per-run seq as a global stream position, so parked Loops never see events from fresh runs

2 participants