feat(latency): P99 latency regression guard for version upgrades (valkey#3527)#304
Conversation
…y#579) When a primary without persistence restarts empty, replicas full-resync and silently wipe their own data. Add two state-transition detectors to the anomaly poller: Rule A fires when a primary comes back empty with restart evidence (replid change / uptime reset / offset regression), and Rule B fires when a replica's dataset collapses after a replid change. Both emit a CRITICAL dataset_keys anomaly, dispatch a new Pro-tier data.loss.detected webhook, and surface a remediation-runbook banner on the anomaly dashboard.
Resolving an anomaly only flipped the in-memory cache entry, so data-loss
dismissals were lost on restart or storage-backed queries and the critical
banner reappeared. Now resolveAnomaly/resolveGroup also persist via
storage.resolveAnomaly. The banner additionally keeps showing unless the
API confirms { success: true }, instead of hiding on failure.
…r visible across date filter Rule A treated a restarted primary with zero keys as an empty-primary wipe even while INFO reported loading in progress, so a normal RDB/AOF restart could emit a critical data-loss alert before the snapshot loads. Gate detection on !isLoading (loading / async_loading) and skip recording the transient empty snapshot as baseline. The DataLossAlertBanner shared the date-filtered events list, so changing the date range could hide an active unresolved dataset_keys incident. Feed it a dedicated unfiltered dataset_keys poll instead. Adds regression tests for the loading and async_loading cases. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_013ntFKmeHP3LBzsTMxswrxH
resolveAnomaly flipped the in-memory resolved flag and returned success even when storage persistence threw, so the data-loss banner would add the id to its dismissed set and stay hidden while storage-backed polls still returned the incident as unresolved. Persist first, return false on failure or when storage did not resolve the event, and only sync the cached copy on a durable resolution.
…key#3527) Detects per-command P99 latency regressions after a Valkey/Redis version upgrade (valkey/valkey#3527) and sustained degradations, with cluster topology-refresh correlation. - apps/api metrics: INFO latencystats parser + poller (60s, 7d retention) and GET /metrics/latencystats summary+history (Community). - storage: latency_stats_samples table + StoragePort methods across sqlite/postgres/memory adapters. - proprietary/latency-regression: pure RegressionDetector (upgrade-window and sustained rules, top-K volume gating, topology correlation) wired into a MultiConnectionPoller that emits command_p99 anomalies and the new Pro latency.regression.detected webhook with a remediation runbook (current prefetch-batch-max-size, valkey PR #2092). - packages/shared + webhook-pro: latency.regression.detected event (Pro) and dispatchLatencyRegressionDetected payload. - apps/web: LatencyRegressionBanner on the Anomaly dashboard + labels. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- getLatencyStatsHistory kept the OLDEST rows within the 10k cap (ORDER BY captured_at ASC + LIMIT), so busy instances dropped the newest samples the regression guard depends on for freshness/version/consecutive logic. Now return the most-recent `limit` rows (inner DESC + LIMIT, then ascending) in the sqlite/postgres/memory adapters. (High) - Sustained-degradation rule was only suppressed while the upgrade window was UNFIRED, so after the one-shot upgrade event fired it could re-report the same post-upgrade regression as a second anomaly/webhook. Suppress sustained for the entire open upgrade window; it resumes only after the 24h window expires. (Medium) Adds tests: cap keeps newest samples; sustained does not re-fire after an upgrade event within the window.
- Empty upgrade window no longer blocks sustained detection: suppress the sustained rule only for commands the open upgrade window actually owns (has a baseline for), instead of gating on the window globally. A version change where no command had enough pre-change samples now leaves sustained active rather than silencing it for up to 24h. Still prevents the one-shot upgrade event from being re-reported via sustained. (Medium) - AnomalyDashboard formatValue now renders command_p99 in ms/µs (values are stored in microseconds) instead of the generic K/M scale. (Low) Adds a test that sustained still fires when a version change opens an upgrade window with no baselines.
…latency-regression-guard # Conflicts: # apps/web/src/pages/AnomalyDashboard.tsx
- Duplicate webhooks per regression (Medium): the anomaly->webhook integration poller fired a generic anomaly.detected for every saved command_p99 event, on top of the dedicated latency.regression.detected, producing two Pro webhooks for one incident. Skip command_p99 in the generic dispatcher. - Missing anomaly instance fields (Low): populate sourceHost/sourcePort on the saved command_p99 anomaly (from the connection context), consistent with the other anomaly producers. - Stale summary after section loss (Low): the latencystats poller returned early when the INFO section disappeared, leaving the in-memory snapshot intact so GET /metrics/latencystats/summary served outdated percentiles. Clear the snapshot when the section is absent. Adds tests for each: command_p99 skipped by the generic dispatcher, the saved event carries instance metadata, and the snapshot is cleared on section loss.
| latest.capturedAt >= nowMs - STALE_SAMPLE_MS && | ||
| latest.serverVersion === upgrade.toVersion; | ||
| const degraded = fresh && isDegraded(latest.p99Us, baseline, UPGRADE_DEGRADATION_FACTOR); | ||
| upgrade.consecutive.set(command, degraded ? (upgrade.consecutive.get(command) ?? 0) + 1 : 0); |
There was a problem hiding this comment.
The consecutive counter tracks poll ticks, not distinct samples. Freshness here is only latest.capturedAt >= nowMs - STALE_SAMPLE_MS (150s ≈ 2.5 poll intervals) with no memory of the last-counted capturedAt, so the same physical sample can be counted more than once.
If a couple of storage writes fail (or a poll is skipped), getLatencyStatsHistory keeps returning the same prior sample; it's still <150s old and still degraded, so consecutive goes 1→2→3 off one physical sample — firing before 5 genuinely-distinct samples. Conversely a real sample gap never resets the streak. Same pattern in the sustained branch (state.consecutive += 1).
Suggest remembering the last-counted capturedAt per command and only incrementing when a strictly newer sample arrives (reset when it doesn't advance).
| consecutive: new Map(), | ||
| fired: false, | ||
| }; | ||
| this.sustained.clear(); |
There was a problem hiding this comment.
The upgrade window opens on exact version-string inequality (currentVersion === this.lastVersion at :184), so ANY change — including a patch bump or build-metadata flip like 7.4.0→7.4.1 — reaches this.sustained.clear() here. That drops every command's lastFiredAt (forgetting the 30-min cooldown) and wipes all in-progress consecutive streaks.
Failure: a command fires a sustained finding at t0 (cooldown to t0+30m); at t0+10m the reported version flips a patch level; sustained.clear() erases its lastFiredAt; still degraded, it re-fires at ~t0+15m, inside the intended cooldown.
parseMajorVersion already exists but is only used for the runbook/config gating — gating the window on a major-version change instead of raw string diff fixes this, and also stops downgrades/patch-bumps from being labeled upgrade_regression.
…stinct samples Addresses two review comments on #304. 1. Version gating (regression-detector.ts:~184). The upgrade window opened on any version-string change, so a patch/build flip (7.4.0 -> 7.4.1) or a downgrade reached sustained.clear() — dropping every command's 30-min cooldown and consecutive streaks — and got labeled upgrade_regression. Now only a major-version *increase* opens a window (parseMajorVersion already existed); other changes just update lastVersion, leaving sustained state intact. 2. Consecutive counting (regression-detector.ts:~244, both rules). The streak advanced once per poll tick, keyed only on freshness (<150s), so a repeated sample (skipped poll or a failed storage write re-returning the prior row) inflated the count and could fire before 5 genuinely-distinct samples. Track the last-counted capturedAt per command and only advance on a strictly newer sample; a stale/non-degraded newer sample still resets. Tests: repeated samples don't fire early (needs 5 distinct); a patch bump opens no upgrade window while a major bump still fires upgrade_regression.
…h shapes
Review hardening for sumKeyspaceKeys. The count is now read from the typed
`keyspace` section of the parsed INFO response rather than the flattened,
stringified record — the flatten runs String() on each value, which would
collapse an object-shaped db to "[object Object]" and silently zero the total.
InfoParser currently emits each db as a raw string ("keys=123,expires=5,
avg_ttl=0"), which is what anomaly.service (and migration.service) already
regex against and what the Rule A/B tests exercise, so detection works today.
But the KeyspaceInfo type declares an object ({ keys, expires, avg_ttl }), so
the function now accepts both shapes and can't silently break if the parser is
ever aligned with the type. Adds focused tests for both shapes plus the
empty/missing/non-db cases.
The banner poll passed only metricType and relied on the events endpoint, but GET /anomaly/events applies a default 24h startTime when none is given. An unresolved data-loss incident older than 24h was filtered out, so the banner could go silent while storage still held an open incident. Add an explicit `activeOnly` feed end to end: - getRecentAnomalies(activeOnly) queries durable storage with resolved:false and no startTime floor (the in-memory cache is capped and lost on restart, so an old open incident must come from storage). AnomalyQueryOptions.resolved is already supported by every adapter (postgres has a partial unresolved index). - Controller exposes ?activeOnly=true, skips the 24h default for it, and switches the default from `||` to `??` so an explicit startTime=0 is honored. - API client forwards activeOnly; the data-loss banner poll sets it. Adds service tests: activeOnly reads storage with resolved:false and no startTime, and bypasses the in-memory cache branch.
Two Bugbot findings on the upgrade-regression rule: 1. Upgrade streak required each fresh sample's serverVersion to equal the window's exact toVersion string, so a patch/minor bump during the 24h window (9.0.0 -> 9.0.1) reset the consecutive streak and a genuine post-major regression could never reach CONSECUTIVE_REQUIRED. Compare major versions instead (>= toVersion major), consistent with the rest of the detector. A rollback below the target major still fails the check and resets, as intended. 2. A rollback/downgrade left the open upgrade window (and its baselines) in place, so evaluateSustained kept suppressing those commands for up to 24h and real regressions on the reverted version went unreported. trackVersion now closes the window when the current major drops below the window's target major. Adds tests for both: a patch bump mid-window still fires the upgrade rule, and a rollback re-enables sustained detection on the reverted version. Both fail without the fixes.
…unpersisted incidents Addresses two bugbot findings on #302. - resolveGroup marked members resolved in memory and returned true even when storage.resolveAnomaly threw/failed, so a client could dismiss a group while storage-backed polls still returned members unresolved. Now mirror resolveAnomaly: only flip the cached copy for durably-resolved members and return true only when every member persisted. - The activeOnly (data-loss banner) feed read durable storage only, so an incident whose saveAnomalyEvent failed — while the Pro webhook still fired — had no matching banner until a later poll persisted it. The feed now unions storage with in-memory unresolved events (dedup by id, storage wins; same filters), so a dispatched incident is always visible. Tests: resolveGroup success/partial-failure/no-row-updated; activeOnly union, dedupe, and resolved-exclusion.
Two Bugbot findings: 1. Volume gate under-reported call rate. callsPerMin divided summed callsDelta by the fixed VOLUME_WINDOW_MS (10 min), so before the buffer fills the rate was scaled down (5 samples at 100/min read as 50) and active commands were wrongly excluded by the >=MIN_CALLS_PER_MIN gate, skipping their P99 regression. Each stored sample is one ~60s commandstats poll, so divide by the number of samples that actually exist (totalCalls / history.length). 2. latencystats summary published before durable save. The in-memory snapshot served by GET /metrics/latencystats/summary was set before saveLatencyStatsSamples; a failed write left /summary showing fresh percentiles while history and the regression poller kept the last stored row. Persist first, then publish the snapshot. Tests: partly-filled buffer still passes the volume gate (fires); snapshot is not updated when the save fails. Both fail without the fixes.
# Conflicts: # proprietary/anomaly-detection/__tests__/anomaly.service.spec.ts # proprietary/anomaly-detection/anomaly.service.ts # proprietary/anomaly-detection/types.ts
…es closed members Bugbot follow-up on #302. resolveAnomaly returned false for an already-resolved row (WHERE resolved = false matched nothing), indistinguishable from a missing row. resolveGroup then reported { success: false } when any member was already resolved — via a retry, a partial resolve, or resolving one anomaly through the data-loss banner before the group — even though every event was closed in storage. resolveAnomaly is now idempotent across all three adapters: it returns true when the anomaly is resolved (newly or already), and false only when no anomaly with that id exists. resolveGroup's all-persisted check is now correct as-is. Adds a describe.each (memory + sqlite) test: first resolve and re-resolve both return true; an unknown id returns false.
getLatencyStatsHistory applied a single limit (default 10,000) across ALL commands when no command filter was given. LatencyRegressionService loads a 24h window that way, so on busy instances with many tracked commands the newest 10,000 rows were a short shared slice — each command kept only a fraction of its window. Upgrade baselines (6h, >=5 samples) and sustained baselines (24h, >=30) could be missing or skewed despite older rows still in storage. Apply the cap PER COMMAND: postgres and sqlite via ROW_NUMBER() partitioned by command, memory via a per-command slice. With a command filter the partition is that one command, so behaviour is unchanged. Documented the per-command semantics on LatencyStatsHistoryQueryOptions.limit. Adds a Memory+Sqlite test: two commands with more rows than the limit each keep their own most-recent `limit` rows (fails under a global cap).
…lure Two Bugbot findings: 1. Patch bump shrinks upgrade baseline. The pre-upgrade baseline included only samples whose serverVersion exactly equalled fromVersion. After a patch/minor bump just before a major upgrade (8.1.0 -> 8.1.1) fromVersion is the patch label, so hours of 8.1.0 samples in the 6h lookback were dropped, leaving too few rows or a skewed median. Match the pre-upgrade MAJOR line instead. 2. Detector fires before persistence. evaluate() latches the one-shot upgrade guard and per-command sustained cooldown when it returns a finding, but the service persisted afterwards and swallowed save errors while still dispatching the webhook. A transient saveAnomalyEvent failure therefore dropped the record yet suppressed re-emit (and still notified). Add detector.revert(finding), which the service calls only on a save failure to re-arm the finding for the next poll; the webhook now fires only after a durable save. Tests: baseline built from the whole pre-upgrade major line; revert re-fires the upgrade one-shot and sustained cooldown; the service does not notify on a failed save and re-emits+persists+notifies once storage recovers.
…stalled The revert() added for persistence-failure retries restored a sustained finding's consecutive streak and cleared its cooldown but left lastCountedAt on the sample that already fired. evaluateSustained's "count physical samples, not poll ticks" guard then skipped the command until a NEWER latencystats row existed, so a failed persist would not re-emit on the same stored sample — unlike upgrade retries, whose fire loop is separate from that guard. Reset lastCountedAt to -1 in the sustained revert branch so the unchanged sample is re-counted and the finding re-emits on the next poll. Adds a repoll-based test (same sample, no new row) that fails without the reset.
…sion-guard # Conflicts: # proprietary/anomaly-detection/__tests__/anomaly.service.spec.ts # proprietary/anomaly-detection/anomaly.service.ts # proprietary/anomaly-detection/types.ts
…latency-regression-guard # Conflicts: # proprietary/anomaly-detection/__tests__/anomaly.service.spec.ts # proprietary/anomaly-detection/anomaly.service.ts # proprietary/anomaly-detection/types.ts
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 3917ed6. Configure here.
… no rows Two bugbot findings on #304. - The P99 regression banner polled getAnomalyEvents with only metricType=command_p99, so the API's default 24h startTime hid unresolved command_p99 incidents older than a day — unlike the data-loss banner. Pass activeOnly:true so open incidents of any age reach the banner. - MemoryAdapter.getLatencyStatsHistory used `options.limit ?? 10_000`, so a limit of 0 survived (0 is not nullish) and `slice(-0)` returned every row instead of none. Guard limit <= 0 to return [], matching SQL `LIMIT 0`. Adds a describe.each (memory + sqlite) test that limit:0 returns no rows.
…sion-guard # Conflicts: # apps/web/src/components/webhooks/WebhookForm.tsx # apps/web/src/pages/AnomalyDashboard.tsx # packages/shared/src/webhooks/types.ts # proprietary/anomaly-detection/__tests__/anomaly.service.spec.ts # proprietary/anomaly-detection/anomaly.service.ts # proprietary/anomaly-detection/types.ts # proprietary/webhook-pro/webhook-events-pro.service.ts

Summary
Detects per-command P99 latency regressions after a version upgrade (and sustained degradations) from valkey/valkey#3527, using
INFO latencystats. A version change opens a 24h window; when an eligible command's P99 stays ≥1.5× its pre-upgrade baseline (and ≥1ms above it) for 5 consecutive samples, the monitor fires one aggregatedcommand_p99anomaly, dispatches a new Pro-tierlatency.regression.detectedwebhook with a remediation runbook, and shows a banner on the Anomaly dashboard. Hourly spikes that coincide withcluster|slots/cluster|shardsbursts are annotated as topology-refresh-correlated.Changes
apps/apimetrics:latencystatsparser + poller (60s poll, 7d retention) andGET /metrics/latencystats/summary+/history(Community).latency_stats_samplestable +StoragePortmethods across the sqlite / postgres / memory adapters.proprietary/latency-regression: pureRegressionDetector— upgrade-window rule (per-command median baseline over last 6h old-version samples) and sustained-degradation rule (rolling median, 30-min cooldown), top-K-by-volume gating (≥60 calls/min), and topology-refresh correlation. Wired into aMultiConnectionPollerthat emitscommand_p99anomalies and the Pro webhook with a runbook citing the liveprefetch-batch-max-size(valkey PR #2092).packages/shared+proprietary/webhook-pro:latency.regression.detectedevent (Pro tier) +dispatchLatencyRegressionDetected()payload.apps/web:LatencyRegressionBanneron the Anomaly dashboard (runbook + Mark resolved),command_p99metric label, webhook event label.randomUUID()so they persist on the Postgres adapter (uuid column).Live end-to-end verification
Exercised against real Valkey containers (8.1.8 → 9.0.4 on the same port), Pro tier via a mock entitlement server, full detection → anomaly → webhook chain:
valkey/valkey:8.1.8, Pro tier, pipelinedhmgetload;/metrics/latencystats/summary+/historyhmgetsamples,serverVersion 8.1.8, p99 ~5µs ✅9.0.4, forced genuine server-side p99 (6000-field HMGET + 0.5 CPU)command_p99anomaly (critical, 1032µs vs 4.5µs baseline, 228×) +latency.regression.detectedwebhook withprefetchBatchMaxSize: 16,callsPerMin, and the 5-step runbook ✅CONFIG SET prefetch-batch-max-size 4, +3 min loadcluster slotsbaseline with one timed spiketopologyRefreshCorrelated: true; message + runbook switch to the correlated variant; anomalyrelatedMetrics: [cluster_state, command_p99]✅Notes / observations
cluster|slotsbaseline with an occasional spike (spike must exceed2× hourly median). A burst-only pattern pushes the median up so no single burst clears the threshold — it correctly staysfalse. With a steady trickle + one spike it correlates. This matches the design intent (periodic client topology refresh + a burst), so the behavior is correct — not a bug.latency_percentiles_usec_*is cumulative (t-digest). Under CPU throttle the p99 climbed into the tens-of-ms range across polls, so the regression fired reliably every time, well past the1.5×and+1msfloor.lastVersion); an API restart mid-upgrade loses that (expected). Not a concern in production where the monitor runs continuously across the upgrade.Checklist
RegressionDetectorrules, service wiring, webhook payload, banner) + full live e2e above.roborev review --branchor/roborev-review-branchin Claude Code (internal)🤖 Generated with Claude Code
Note
Medium Risk
Touches persistence schema, continuous polling against live Valkey/Redis, and proprietary detection/webhook paths; logic is heavily tested but mis-tuned thresholds or storage/query limits could miss or delay regressions on busy clusters.
Overview
Adds end-to-end per-command P99 latency regression detection from
INFO latencystats, aimed at post-upgrade regressions (valkey#3527).Community API path: a 60s poller parses
latencystats, persists p50/p99/p99.9 plusserverVersion(7d retention), and exposesGET /metrics/latencystats/summaryand/history. Storage gainslatency_stats_samplesandStoragePortmethods on memory, SQLite, and Postgres, with history limits applied per command so multi-command instances keep enough baseline data.Pro proprietary guard:
LatencyRegressionModuleruns aRegressionDetectoron stored samples—upgrade (24h window after a major bump, 6h pre-upgrade median baseline, 5 consecutive degraded samples, one-shot) and sustained (rolling baseline, 30m cooldown)—with volume gating, topology-refresh correlation viacluster|slots/cluster|shards, and persist-before-notify (re-arm on failedsaveAnomalyEvent). Findings becomecommand_p99anomalies pluslatency.regression.detectedwebhooks; genericanomaly.detectedis skipped for that metric.UI: Anomaly dashboard banner (unfiltered
command_p99feed),command_p99labels/formatting, and webhook form event for latency regression.Reviewed by Cursor Bugbot for commit 96618ab. Bugbot is set up for automated code reviews on this repo. Configure here.