Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat(anomaly): detect stalled BGSAVE/AOF persistence forks#294

Merged
KIvanow merged 7 commits into
masterfrom
feature/persistence-stall-detector
Jul 8, 2026
Merged

feat(anomaly): detect stalled BGSAVE/AOF persistence forks#294
KIvanow merged 7 commits into
masterfrom
feature/persistence-stall-detector

Conversation

@KIvanow

@KIvanow KIvanow commented Jul 1, 2026

Copy link
Copy Markdown
Member

Summary

Adds a persistence-stall detector to the anomaly-detection module. Valkey issue valkey-io/valkey#2322 asks for visibility when a background save gets wedged. Monitor now flags a BGSAVE / AOF-rewrite fork child that stops making progress, runs past a time ceiling, or whose last save status flips to error, so operators find out before the fork silently blocks writes or exhausts memory.

Changes

  • Added MetricType.PERSISTENCE_CHILD and AnomalyPattern.PERSISTENCE_STALL (types.ts).
  • New state-based detector in anomaly.service.ts that tracks RDB and AOF fork children per connection from the INFO persistence section:
    • CRITICAL when key progress freezes for MONITOR_PERSISTENCE_STALL_SEC (default 60s).
    • CRITICAL when elapsed time exceeds MONITOR_PERSISTENCE_CRIT_SEC (default 600s).
    • CRITICAL on an ok to err last-save/last-rewrite status transition.
    • WARNING when elapsed exceeds MONITOR_PERSISTENCE_WARN_SEC (default 120s) while still advancing.
    • Fires once per episode and clears state when the child finishes or the connection is removed. AOF has no per-key counter, so it relies on the time-based ceilings only.
  • Added a PERSISTENCE_STALL correlation rule in correlator.ts with diagnosis and remediation guidance (memory headroom / COW growth, server log, BGSAVE CANCEL, disk space and I/O).

Checklist

  • Unit / integration tests added
  • Docs added / updated
  • Roborev review passed - run roborev review --branch or /roborev-review-branch in Claude Code (internal)
  • Competitive analysis done / discussed (internal)
  • Blog post about it discussed (internal)

Notes: 45/45 anomaly-detection tests pass, tsc --noEmit clean. Thresholds are env-configurable with safe fallbacks (the ConfigService mock returns non-numeric values, so parsing guards against NaN). No dedicated webhook event in this PR (deferred).


Note

Medium Risk
New monitor path on every anomaly poll reads persistence INFO and maintains per-connection state; mis-tuned thresholds or edge-case logic could cause false positives/negatives on production saves, though behavior is covered by tests and defaults are conservative.

Overview
Adds state-based monitoring for wedged or failing RDB/AOF fork children so operators get alerted before persistence problems block writes or blow up memory.

Anomaly pipeline: Introduces MetricType.PERSISTENCE_CHILD and AnomalyPattern.PERSISTENCE_STALL. AnomalyService polls INFO persistence per connection, tracks RDB and AOF episodes (progress, elapsed time, last status), and emits WARNING/CRITICAL events for long runs, frozen key progress (RDB), time ceilings, and error status—with once-per-episode dedupe, re-baseline when a new child starts between polls, and guards for N/N flush tail and missing key totals. A correlator rule surfaces diagnosis and remediation for PERSISTENCE_STALL.

Config & docs: Three env vars (MONITOR_PERSISTENCE_STALL_SEC, WARN, CRIT) are validated in env.schema.ts (defaults 60/120/600). docs/anomaly-detection.md documents the pattern, metric, and variables.

Tests: Large anomaly.service.spec suite covers stall, warn, error latching, episode boundaries, and cleanup; ConfigService mock returns numeric threshold defaults.

Reviewed by Cursor Bugbot for commit 1ea9773. Bugbot is set up for automated code reviews on this repo. Configure here.

Add a state-based persistence-child detector that flags a fork child
whose key progress freezes, whose elapsed time exceeds a ceiling, or
whose last bgsave/bgrewrite status flips to error. Emits a new
PERSISTENCE_CHILD metric and PERSISTENCE_STALL correlation pattern
with remediation guidance. Thresholds are env-configurable.
@KIvanow KIvanow requested a review from jamby77 July 2, 2026 06:27
Add PERSISTENCE_STALL pattern, persistence_child metric, and the
MONITOR_PERSISTENCE_* threshold env vars to the anomaly detection guide.
Comment thread proprietary/anomaly-detection/anomaly.service.ts
Per-episode persistence tracking was reset only when a poll observed
in_progress=0. A BGSAVE/AOF rewrite that finished and restarted between
polls (tight save cadence, slow interval, or a failed poll) reused the
prior PersistenceChildTrack, causing a false CRITICAL stall from a stale
lastAdvanceTs or suppressing alerts for a genuinely stuck new child via a
carried-over reportedStall/warnedLong.

A running child's elapsed time and processed-key count are monotonic
within an episode and reset when a new fork starts. Detect a regression in
either signal and re-baseline the track (clearing the reported flags),
mirroring the first-observation branch. Covers RDB (elapsed or processed)
and AOF (elapsed only, no per-key progress).
Comment thread proprietary/anomaly-detection/anomaly.service.ts

@jamby77 jamby77 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the detector at the current head — the re-baseline fix for between-poll fork restarts looks sound (regression in either elapsed or processed re-arms the track cleanly). Two things bugbot hasn't flagged, one correctness, one minor.

const status = signals.lastStatus;
if (status !== undefined && status !== '') {
const prevStatus = kind === 'rdb' ? state.rdbLastStatus : state.aofLastStatus;
if (prevStatus !== undefined && prevStatus !== status && status === 'err') {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A server already in err status is never reported.

The error anomaly only fires on an observed transition:

if (prevStatus !== undefined && prevStatus !== status && status === 'err') {

prevStatus comes from the in-memory lastPersistenceState, which starts empty and is dropped in onConnectionRemoved. Consequences:

  • If rdb_last_bgsave_status / aof_last_bgrewrite_status is already err when monitoring begins (monitor start, API restart, connection add/re-add), the first poll swallows the err as baseline and the guard can never pass while the status stays err.
  • Repeated failures with no intervening successful save also never re-fire — only an ok sample re-arms the check.

That's precisely the stop-writes-on-bgsave-error scenario the comment above cites (#2322): the instance can be rejecting writes while the anomaly feed stays silent, and nothing else in the codebase reads these status fields. The spec only covers the ok→err transition with an explicit ok baseline poll first.

Suggestion: make this level-triggered rather than edge-triggered — fire when status === 'err' and there's no unresolved persistence-error anomaly for this connection (or use a dedup/cooldown), so a pre-existing failure state is caught on the first poll.

const n = Number(value);
return Number.isFinite(n) && n > 0 ? n : fallback;
};
this.persistenceStallSec = positiveNumber(this.configService.get('MONITOR_PERSISTENCE_STALL_SEC'), 60);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: the three MONITOR_PERSISTENCE_*_SEC vars are parsed with a local positiveNumber fallback here, while the sibling MONITOR_* thresholds (MONITOR_RECENT_OOM_WINDOW_MS, MONITOR_MEMORY_PCT_THRESHOLD, MONITOR_REPLICATION_LAG_BYTES, …) are declared in the Zod schema in apps/api/src/config/env.schema.ts. As written, a typo like MONITOR_PERSISTENCE_STALL_SEC=6O silently falls back to 60 instead of failing startup validation, and the defaults live in a different place than every other monitor threshold. Worth adding the three vars to the schema and reading them plainly.

Address PR review feedback on the BGSAVE/AOF rewrite stall detector:

- Move the three MONITOR_PERSISTENCE_*_SEC thresholds into the Zod env
  schema (env.schema.ts) and read them plainly, so a typo fails startup
  validation instead of silently falling back, matching every other
  MONITOR_* threshold.
- Report a pre-existing err status. Persistence-child error detection is
  now level-triggered with a latch: an err already present at monitor or
  connection start fires on the first poll, a persisting err fires once,
  and a later ok re-arms it for the next failure.
- Distinguish the elapsed-time ceiling from a frozen-progress stall. When
  a child crosses persistenceCritSec its event now carries threshold=crit
  and a "time ceiling" message instead of the stall shape (threshold=60,
  "no progress"), which was misleading when keys were still advancing.

Adds tests for the pre-existing err, the latch dedup/re-arm, and the
time-ceiling reason; updates the ConfigService mock to return the
schema-validated numeric thresholds.
Comment thread proprietary/anomaly-detection/anomaly.service.ts

@jamby77 jamby77 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — solid, well-tested detector.

One non-blocking correctness nit not yet raised in the thread: the frozen-stall check (frozenStall) has no processed < total guard, so a large RDB whose keys are all serialized but still fflush/fsync/rename-ing (in_progress stays 1) can trip a false CRITICAL … appears stuck (processed N/N keys) once the tail exceeds MONITOR_PERSISTENCE_STALL_SEC. A signals.total === null || signals.processed < signals.total guard closes it. Fine to address in a follow-up.

…tall

Addresses review feedback on #294. The frozen-progress stall check fired
whenever current_save_keys_processed stayed flat for persistenceStallSec while
rdb_bgsave_in_progress was still 1 — but a BGSAVE stays in_progress through the
RDB flush/fsync/rename tail after every key is serialized, during which
processed sits frozen at N/N. On a large save over a slow disk that tail can
exceed the stall window and trip a false "CRITICAL: appears stuck (processed
N/N keys)".

Only treat frozen progress as a stall while keys remain (processed < total);
when total is unknown (older servers without current_save_keys_total) keep the
prior elapsed-only behavior. A genuine hang in the tail is still caught by the
elapsed-time ceiling, which reports the (accurate) duration message instead of
"stuck / no progress".

Tests: a completed save lingering in its tail past the stall window fires
nothing; a tail hang past the crit ceiling still fires 'exceeded'.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 2c7b27f. Configure here.

Comment thread proprietary/anomaly-detection/anomaly.service.ts
KIvanow added 2 commits July 8, 2026 09:57
…known

Bugbot follow-up on #294. The progressIncomplete guard was
`total === null || processed < total`, so a missing current_save_keys_total
made it true and frozen-progress CRITICALs still fired — contradicting the
adjacent comment, which said unknown totals should rely only on the elapsed
thresholds.

Without a total we can't distinguish the completion tail (processed frozen at
N/N) from a genuine stall, so raising a "stuck" CRITICAL is unsubstantiated.
Require the total to be known (`total !== null && processed < total`); when it's
absent, frozen-progress detection is skipped and the elapsed-time ceiling still
covers a real hang.

Adds a test: frozen processed past the stall window with no total reports
nothing.
…-stall-detector

# Conflicts:
#	proprietary/anomaly-detection/__tests__/anomaly.service.spec.ts
#	proprietary/anomaly-detection/anomaly.service.ts
#	proprietary/anomaly-detection/types.ts
@KIvanow KIvanow merged commit 39460d5 into master Jul 8, 2026
3 checks passed
@KIvanow KIvanow deleted the feature/persistence-stall-detector branch July 8, 2026 07:35
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 8, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants