feat(anomaly): data-loss guard for empty-primary full resync (valkey#579)#302
Conversation
…y#579) When a primary without persistence restarts empty, replicas full-resync and silently wipe their own data. Add two state-transition detectors to the anomaly poller: Rule A fires when a primary comes back empty with restart evidence (replid change / uptime reset / offset regression), and Rule B fires when a replica's dataset collapses after a replid change. Both emit a CRITICAL dataset_keys anomaly, dispatch a new Pro-tier data.loss.detected webhook, and surface a remediation-runbook banner on the anomaly dashboard.
Resolving an anomaly only flipped the in-memory cache entry, so data-loss
dismissals were lost on restart or storage-backed queries and the critical
banner reappeared. Now resolveAnomaly/resolveGroup also persist via
storage.resolveAnomaly. The banner additionally keeps showing unless the
API confirms { success: true }, instead of hiding on failure.
…r visible across date filter Rule A treated a restarted primary with zero keys as an empty-primary wipe even while INFO reported loading in progress, so a normal RDB/AOF restart could emit a critical data-loss alert before the snapshot loads. Gate detection on !isLoading (loading / async_loading) and skip recording the transient empty snapshot as baseline. The DataLossAlertBanner shared the date-filtered events list, so changing the date range could hide an active unresolved dataset_keys incident. Feed it a dedicated unfiltered dataset_keys poll instead. Adds regression tests for the loading and async_loading cases. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_013ntFKmeHP3LBzsTMxswrxH
resolveAnomaly flipped the in-memory resolved flag and returned success even when storage persistence threw, so the data-loss banner would add the id to its dismissed set and stay hidden while storage-backed polls still returned the incident as unresolved. Persist first, return false on failure or when storage did not resolve the event, and only sync the cached copy on a durable resolution.
| let total = 0; | ||
| for (const [key, value] of Object.entries(info)) { | ||
| if (!/^db\d+$/.test(key)) continue; | ||
| const match = /keys=(\d+)/.exec(value); |
There was a problem hiding this comment.
sumKeyspaceKeys never returns nonzero in production, so Rule A/B can't fire.
It regexes keys=(\d+) out of each db<N> value expecting a raw string, but the real INFO parser stores each db as an object ({keys, expires, avg_ttl} — see info.parser.ts parseKvLine and the consumer at metrics.service.ts:199-201). convertInfoToRecord then stringifies it to "[object Object]", so the regex misses → total stays 0 every poll → prev.totalKeys > 0 is never true.
Tests pass only because the mock injects db0: 'keys=150,...' as a string, a shape the parser never emits.
Fix: read .keys off the typed keyspace object (mirror metrics.service.ts) and switch the test mock to the real object shape.
There was a problem hiding this comment.
Dug into this and the premise is actually inverted — sumKeyspaceKeys does fire in production. The parser does not store keyspace dbs as objects: InfoParser.parse (info.parser.ts) writes each section value as the raw substring after the colon, and parseKvLine is never applied to the keyspace section. So info.keyspace.db0 is the string "keys=150,expires=5,avg_ttl=0", and the regex matches. Verified against the real parser:
InfoParser.parse("# Keyspace\r\ndb0:keys=150,expires=5,avg_ttl=0\r\n").keyspace.db0
// => "keys=150,expires=5,avg_ttl=0" (typeof "string")
migration.service.ts:113 relies on this same string shape (val.match(/keys=(\d+)/)) and would throw at runtime if val were an object — so it is definitively a string in prod. And the existing test mock (db0: 'keys=150,...') is faithful, not fictional.
The metrics.service.ts:199 reference is actually the mis-shaped one: its typeof val === 'object' && 'keys' in val check is false against the real string, so its keyspaceSize is silently always 0. The root cause is a type-vs-runtime mismatch: the KeyspaceInfo type declares an object but the parser emits strings. (Worth a separate fix for metrics.service — flagging it.)
Addressed the underlying fragility in 182d7b4: sumKeyspaceKeys now reads the typed keyspace section off the INFO response (before the lossy String() flatten that would produce "[object Object]") and handles both the string and object shapes, so it is correct today and cannot silently break if the parser is later aligned with the type. Added tests for both shapes.
There was a problem hiding this comment.
You're right and I was wrong here — apologies. The anomaly service uses getInfo (→ InfoParser.parse, which keeps each section value as a raw string), not the typed getInfoParsed, so keyspace.db0 is "keys=150,..." and the regex matches; sumKeyspaceKeys was firing correctly in prod. My proof cited metrics.service.ts:199, which is actually the mis-shaped consumer (silently 0 against the real string) — I inverted it. Ironically this contradicts what I'd verified on #304 (that InfoParser.parse keeps strings) and I failed to carry it back here.
182d7b4e is a good hardening regardless — reading the typed keyspace before the lossy flatten and handling both shapes future-proofs it, and the separate metrics.service keyspaceSize === 0 bug is a real catch. Withdrawing this as a blocker.
…h shapes
Review hardening for sumKeyspaceKeys. The count is now read from the typed
`keyspace` section of the parsed INFO response rather than the flattened,
stringified record — the flatten runs String() on each value, which would
collapse an object-shaped db to "[object Object]" and silently zero the total.
InfoParser currently emits each db as a raw string ("keys=123,expires=5,
avg_ttl=0"), which is what anomaly.service (and migration.service) already
regex against and what the Rule A/B tests exercise, so detection works today.
But the KeyspaceInfo type declares an object ({ keys, expires, avg_ttl }), so
the function now accepts both shapes and can't silently break if the parser is
ever aligned with the type. Adds focused tests for both shapes plus the
empty/missing/non-db cases.
The banner poll passed only metricType and relied on the events endpoint, but GET /anomaly/events applies a default 24h startTime when none is given. An unresolved data-loss incident older than 24h was filtered out, so the banner could go silent while storage still held an open incident. Add an explicit `activeOnly` feed end to end: - getRecentAnomalies(activeOnly) queries durable storage with resolved:false and no startTime floor (the in-memory cache is capped and lost on restart, so an old open incident must come from storage). AnomalyQueryOptions.resolved is already supported by every adapter (postgres has a partial unresolved index). - Controller exposes ?activeOnly=true, skips the 24h default for it, and switches the default from `||` to `??` so an explicit startTime=0 is honored. - API client forwards activeOnly; the data-loss banner poll sets it. Adds service tests: activeOnly reads storage with resolved:false and no startTime, and bypasses the in-memory cache branch.
| } | ||
|
|
||
| resolveGroup(correlationId: string): boolean { | ||
| async resolveGroup(correlationId: string): Promise<boolean> { |
There was a problem hiding this comment.
The storage-first fix landed on resolveAnomaly but not here — resolveGroup is now inconsistent with its sibling. It sets anomaly.resolved = true before the storage call, swallows every per-item error in the try/catch, and unconditionally return true.
So if storage is briefly down, POST /anomaly/groups/:id/resolve returns {success:true}, the UI hides the group, but the next storage-backed poll returns them still unresolved (banner reappears) and the in-memory copies are wrongly marked resolved — the exact durability gap resolveAnomaly was just fixed to avoid.
Mirror the resolveAnomaly shape: persist first, only flip the cached copy on success, and return false if any member fails to persist (or aggregate the results).
…unpersisted incidents Addresses two bugbot findings on #302. - resolveGroup marked members resolved in memory and returned true even when storage.resolveAnomaly threw/failed, so a client could dismiss a group while storage-backed polls still returned members unresolved. Now mirror resolveAnomaly: only flip the cached copy for durably-resolved members and return true only when every member persisted. - The activeOnly (data-loss banner) feed read durable storage only, so an incident whose saveAnomalyEvent failed — while the Pro webhook still fired — had no matching banner until a later poll persisted it. The feed now unions storage with in-memory unresolved events (dedup by id, storage wins; same filters), so a dispatched incident is always visible. Tests: resolveGroup success/partial-failure/no-row-updated; activeOnly union, dedupe, and resolved-exclusion.
# Conflicts: # proprietary/anomaly-detection/__tests__/anomaly.service.spec.ts # proprietary/anomaly-detection/anomaly.service.ts # proprietary/anomaly-detection/types.ts
…es closed members Bugbot follow-up on #302. resolveAnomaly returned false for an already-resolved row (WHERE resolved = false matched nothing), indistinguishable from a missing row. resolveGroup then reported { success: false } when any member was already resolved — via a retry, a partial resolve, or resolving one anomaly through the data-loss banner before the group — even though every event was closed in storage. resolveAnomaly is now idempotent across all three adapters: it returns true when the anomaly is resolved (newly or already), and false only when no anomaly with that id exists. resolveGroup's all-persisted check is now correct as-is. Adds a describe.each (memory + sqlite) test: first resolve and re-resolve both return true; an unknown id returns false.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 12bbbfe. Configure here.
|
Resolve regression on Postgres: Fix: fall back to the in-memory flip when storage reports no row (safe — a never-stored event can't be resurfaced by a storage-backed poll). Prefer this over giving everything Nit: new |
On Postgres, anomaly_events.id is a UUID PK, but failover/promotion/
cluster/persistence/dup-primary events use deterministic string ids
(${connectionId}-failover-${ts} etc). Their inserts throw and are
swallowed in addAnomaly, so they never persist; the persist-first
resolveAnomaly/resolveGroup then fail for them and they surface from the
in-memory cache as undismissable banners in prod.
Track durability per cached event (persisted flag, set when
saveAnomalyEvent succeeds) and fall back to an in-memory flip when the
event was never stored — safe, since a storage-backed poll can't
resurface a row that was never written. Durable events keep the
storage-first path so a transient resolve failure still can't dismiss a
banner a later poll re-surfaces. Deterministic ids are preserved (they
drive the idempotent ON CONFLICT upsert on sqlite/memory).
Also type sumKeyspaceKeys off a structural INFO shape instead of any.
@jamby77 this should be fixed now. When requesting changes, please use the |
jamby77
left a comment
There was a problem hiding this comment.
Approved. Solid data-loss guard — persisted-flag durability model is sound, loading guard and FLUSHALL discrimination are correct, strong test coverage. Non-blocking: a few one-line ifs (sumKeyspaceKeys + the 3 adapters + resolveGroup guard) should be braced per CLAUDE.md; Postgres resolveAnomaly idempotency path is untested (describe.each covers Memory+Sqlite only). CI red is the unrelated pre-existing cache-proposals-sqlite failure.
Summary
Detects the silent data-loss scenario from valkey/valkey#579: a primary without persistence restarts empty, replicas see a new
master_replid, full-resync, and wipe their own copies. The monitor now alerts the moment the primary comes back empty (Rule A), confirms when a replica has been wiped (Rule B), dispatches a new Pro-tierdata.loss.detectedwebhook, and shows a critical banner with a remediation runbook on the Anomaly dashboard.Changes
proprietary/anomaly-detection: newMetricType.DATASET_KEYS+ per-connection replication snapshot (replid, offset, uptime, keyspace key count) inAnomalyService.pollConnection(); Rule A (primary restarted empty with restart evidence: replid change / uptime reset / offset regression) and Rule B (replica dataset collapsed ≥90% after replid change) emit CRITICAL anomalies. Same-replid FLUSHALL is deliberately not flagged.packages/shared+proprietary/webhook-pro: newdata.loss.detectedwebhook event (Pro tier) withdispatchDataLossDetected()payload (kind, previous/current keys, replids, role, instance).apps/web: newDataLossAlertBanner(critical alert + REPLICAOF NO ONE runbook, resolve action) on the Anomaly dashboard;dataset_keysmetric label; webhook event label;resolveAnomalyEventAPI helper.randomUUID()so they persist on the Postgres storage adapter (uuid column).Checklist
roborev review --branchor/roborev-review-branchin Claude Code (internal)Note
High Risk
Changes anomaly detection and resolution paths for replication/data-loss incidents; incorrect rules or resolution semantics could miss real data loss or let critical banners dismiss without durable state.
Overview
Adds critical data-loss detection for the empty-primary / full-resync wipe scenario (valkey#579):
AnomalyServicetracks replication snapshots and emitsdataset_keysCRITICAL anomalies (primary restarted empty vs replica wiped), with guards for loading/FLUSHALL and a new Prodata.loss.detectedwebhook.The Anomaly dashboard gets a
DataLossAlertBannerfed byactiveOnlyanomaly queries (no 24h window) plusresolveAnomalyEvent; dismiss only succeeds after the server persists resolution.Resolution behavior is reworked:
resolveAnomaly/resolveGroupare async and storage-first for durable events; memory-only anomalies use apersistedflag; storageresolveAnomalyis idempotent (true if already resolved) across memory/SQLite/Postgres.API/client tweaks:
activeOnlyon anomaly events, async resolve endpoints, and webhook form label for the new event type.Reviewed by Cursor Bugbot for commit 620991b. Bugbot is set up for automated code reviews on this repo. Configure here.