feat(anomaly): detect stalled BGSAVE/AOF persistence forks#294
Conversation
Add a state-based persistence-child detector that flags a fork child whose key progress freezes, whose elapsed time exceeds a ceiling, or whose last bgsave/bgrewrite status flips to error. Emits a new PERSISTENCE_CHILD metric and PERSISTENCE_STALL correlation pattern with remediation guidance. Thresholds are env-configurable.
Add PERSISTENCE_STALL pattern, persistence_child metric, and the MONITOR_PERSISTENCE_* threshold env vars to the anomaly detection guide.
Per-episode persistence tracking was reset only when a poll observed in_progress=0. A BGSAVE/AOF rewrite that finished and restarted between polls (tight save cadence, slow interval, or a failed poll) reused the prior PersistenceChildTrack, causing a false CRITICAL stall from a stale lastAdvanceTs or suppressing alerts for a genuinely stuck new child via a carried-over reportedStall/warnedLong. A running child's elapsed time and processed-key count are monotonic within an episode and reset when a new fork starts. Detect a regression in either signal and re-baseline the track (clearing the reported flags), mirroring the first-observation branch. Covers RDB (elapsed or processed) and AOF (elapsed only, no per-key progress).
jamby77
left a comment
There was a problem hiding this comment.
Reviewed the detector at the current head — the re-baseline fix for between-poll fork restarts looks sound (regression in either elapsed or processed re-arms the track cleanly). Two things bugbot hasn't flagged, one correctness, one minor.
| const status = signals.lastStatus; | ||
| if (status !== undefined && status !== '') { | ||
| const prevStatus = kind === 'rdb' ? state.rdbLastStatus : state.aofLastStatus; | ||
| if (prevStatus !== undefined && prevStatus !== status && status === 'err') { |
There was a problem hiding this comment.
A server already in err status is never reported.
The error anomaly only fires on an observed transition:
if (prevStatus !== undefined && prevStatus !== status && status === 'err') {prevStatus comes from the in-memory lastPersistenceState, which starts empty and is dropped in onConnectionRemoved. Consequences:
- If
rdb_last_bgsave_status/aof_last_bgrewrite_statusis alreadyerrwhen monitoring begins (monitor start, API restart, connection add/re-add), the first poll swallows theerras baseline and the guard can never pass while the status stayserr. - Repeated failures with no intervening successful save also never re-fire — only an
oksample re-arms the check.
That's precisely the stop-writes-on-bgsave-error scenario the comment above cites (#2322): the instance can be rejecting writes while the anomaly feed stays silent, and nothing else in the codebase reads these status fields. The spec only covers the ok→err transition with an explicit ok baseline poll first.
Suggestion: make this level-triggered rather than edge-triggered — fire when status === 'err' and there's no unresolved persistence-error anomaly for this connection (or use a dedup/cooldown), so a pre-existing failure state is caught on the first poll.
| const n = Number(value); | ||
| return Number.isFinite(n) && n > 0 ? n : fallback; | ||
| }; | ||
| this.persistenceStallSec = positiveNumber(this.configService.get('MONITOR_PERSISTENCE_STALL_SEC'), 60); |
There was a problem hiding this comment.
Minor: the three MONITOR_PERSISTENCE_*_SEC vars are parsed with a local positiveNumber fallback here, while the sibling MONITOR_* thresholds (MONITOR_RECENT_OOM_WINDOW_MS, MONITOR_MEMORY_PCT_THRESHOLD, MONITOR_REPLICATION_LAG_BYTES, …) are declared in the Zod schema in apps/api/src/config/env.schema.ts. As written, a typo like MONITOR_PERSISTENCE_STALL_SEC=6O silently falls back to 60 instead of failing startup validation, and the defaults live in a different place than every other monitor threshold. Worth adding the three vars to the schema and reading them plainly.
Address PR review feedback on the BGSAVE/AOF rewrite stall detector: - Move the three MONITOR_PERSISTENCE_*_SEC thresholds into the Zod env schema (env.schema.ts) and read them plainly, so a typo fails startup validation instead of silently falling back, matching every other MONITOR_* threshold. - Report a pre-existing err status. Persistence-child error detection is now level-triggered with a latch: an err already present at monitor or connection start fires on the first poll, a persisting err fires once, and a later ok re-arms it for the next failure. - Distinguish the elapsed-time ceiling from a frozen-progress stall. When a child crosses persistenceCritSec its event now carries threshold=crit and a "time ceiling" message instead of the stall shape (threshold=60, "no progress"), which was misleading when keys were still advancing. Adds tests for the pre-existing err, the latch dedup/re-arm, and the time-ceiling reason; updates the ConfigService mock to return the schema-validated numeric thresholds.
jamby77
left a comment
There was a problem hiding this comment.
Approving — solid, well-tested detector.
One non-blocking correctness nit not yet raised in the thread: the frozen-stall check (frozenStall) has no processed < total guard, so a large RDB whose keys are all serialized but still fflush/fsync/rename-ing (in_progress stays 1) can trip a false CRITICAL … appears stuck (processed N/N keys) once the tail exceeds MONITOR_PERSISTENCE_STALL_SEC. A signals.total === null || signals.processed < signals.total guard closes it. Fine to address in a follow-up.
…tall Addresses review feedback on #294. The frozen-progress stall check fired whenever current_save_keys_processed stayed flat for persistenceStallSec while rdb_bgsave_in_progress was still 1 — but a BGSAVE stays in_progress through the RDB flush/fsync/rename tail after every key is serialized, during which processed sits frozen at N/N. On a large save over a slow disk that tail can exceed the stall window and trip a false "CRITICAL: appears stuck (processed N/N keys)". Only treat frozen progress as a stall while keys remain (processed < total); when total is unknown (older servers without current_save_keys_total) keep the prior elapsed-only behavior. A genuine hang in the tail is still caught by the elapsed-time ceiling, which reports the (accurate) duration message instead of "stuck / no progress". Tests: a completed save lingering in its tail past the stall window fires nothing; a tail hang past the crit ceiling still fires 'exceeded'.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 2c7b27f. Configure here.
…known Bugbot follow-up on #294. The progressIncomplete guard was `total === null || processed < total`, so a missing current_save_keys_total made it true and frozen-progress CRITICALs still fired — contradicting the adjacent comment, which said unknown totals should rely only on the elapsed thresholds. Without a total we can't distinguish the completion tail (processed frozen at N/N) from a genuine stall, so raising a "stuck" CRITICAL is unsubstantiated. Require the total to be known (`total !== null && processed < total`); when it's absent, frozen-progress detection is skipped and the elapsed-time ceiling still covers a real hang. Adds a test: frozen processed past the stall window with no total reports nothing.
…-stall-detector # Conflicts: # proprietary/anomaly-detection/__tests__/anomaly.service.spec.ts # proprietary/anomaly-detection/anomaly.service.ts # proprietary/anomaly-detection/types.ts

Summary
Adds a persistence-stall detector to the anomaly-detection module. Valkey issue valkey-io/valkey#2322 asks for visibility when a background save gets wedged. Monitor now flags a BGSAVE / AOF-rewrite fork child that stops making progress, runs past a time ceiling, or whose last save status flips to error, so operators find out before the fork silently blocks writes or exhausts memory.
Changes
MetricType.PERSISTENCE_CHILDandAnomalyPattern.PERSISTENCE_STALL(types.ts).anomaly.service.tsthat tracks RDB and AOF fork children per connection from the INFO persistence section:MONITOR_PERSISTENCE_STALL_SEC(default 60s).MONITOR_PERSISTENCE_CRIT_SEC(default 600s).oktoerrlast-save/last-rewrite status transition.MONITOR_PERSISTENCE_WARN_SEC(default 120s) while still advancing.PERSISTENCE_STALLcorrelation rule incorrelator.tswith diagnosis and remediation guidance (memory headroom / COW growth, server log,BGSAVE CANCEL, disk space and I/O).Checklist
roborev review --branchor/roborev-review-branchin Claude Code (internal)Notes: 45/45 anomaly-detection tests pass,
tsc --noEmitclean. Thresholds are env-configurable with safe fallbacks (the ConfigService mock returns non-numeric values, so parsing guards against NaN). No dedicated webhook event in this PR (deferred).Note
Medium Risk
New monitor path on every anomaly poll reads persistence INFO and maintains per-connection state; mis-tuned thresholds or edge-case logic could cause false positives/negatives on production saves, though behavior is covered by tests and defaults are conservative.
Overview
Adds state-based monitoring for wedged or failing RDB/AOF fork children so operators get alerted before persistence problems block writes or blow up memory.
Anomaly pipeline: Introduces
MetricType.PERSISTENCE_CHILDandAnomalyPattern.PERSISTENCE_STALL.AnomalyServicepolls INFO persistence per connection, tracks RDB and AOF episodes (progress, elapsed time, last status), and emits WARNING/CRITICAL events for long runs, frozen key progress (RDB), time ceilings, and error status—with once-per-episode dedupe, re-baseline when a new child starts between polls, and guards for N/N flush tail and missing key totals. A correlator rule surfaces diagnosis and remediation for PERSISTENCE_STALL.Config & docs: Three env vars (
MONITOR_PERSISTENCE_STALL_SEC,WARN,CRIT) are validated inenv.schema.ts(defaults 60/120/600).docs/anomaly-detection.mddocuments the pattern, metric, and variables.Tests: Large
anomaly.service.specsuite covers stall, warn, error latching, episode boundaries, and cleanup;ConfigServicemock returns numeric threshold defaults.Reviewed by Cursor Bugbot for commit 1ea9773. Bugbot is set up for automated code reviews on this repo. Configure here.