feat(anomaly): detect duplicate primaries in a shard (split-brain)#305
Conversation
Adds topology-level detection for the fault behind valkey-io/valkey#2261: two primaries owning the same slots in one shard. From a single node's CLUSTER NODES view, a slot range claimed by more than one master-flagged node is the unambiguous symptom; the master with the lower configEpoch is the stale/phantom primary — the node Valkey's own proposed fix demotes. - New pure detector `duplicate-primary-detector.ts` (fully unit-tested): finds overlapping-slot primary pairs and identifies the phantom. - Wire it into AnomalyService.pollConnection, inside the existing cluster-mode block, emitting a CRITICAL CLUSTER_TOPOLOGY anomaly. Alerts are deduped per conflict signature and re-arm once the conflict resolves. - Add MetricType.CLUSTER_TOPOLOGY and AnomalyPattern.SPLIT_BRAIN. - Surface configEpoch on DiscoveredNode for future UI highlighting. Detection only — the server-side consensus bug is Valkey's to fix; this alerts operators the moment a cluster enters the invalid state.
A failed CLUSTER NODES poll yields no observation of the topology, so a previously-stored conflict signature must not survive it. Otherwise a missed heal (cluster resolves and re-splits while polls are failing) would suppress a legitimately-new CRITICAL split-brain alert on the next successful poll. Clear the connection's active-conflict set on poll error so the next observation re-alerts. Adds a regression test.
…rom build The streaming wrapStream work (#262) added `import ... from '@ai-sdk/provider'` to src/adapters/ai.ts but never declared the package, so a clean `tsc` build failed with TS2307 — it only resolved transitively via `ai` on machines where pnpm happened to hoist it. - Declare @ai-sdk/[email protected] (the version ai@6 resolves) as a devDependency so the type-only import resolves under pnpm's strict isolation. - Broaden the build tsconfig exclude from `src/__tests__` to `src/**/__tests__` so the nested src/adapters/__tests__ dir is excluded like every other package's tests. The build was typechecking (and emitting) test files, which is why the vitest-typed mocks tripped tsc. Tests still run under vitest. agent-cache build is green and all 262 tests pass.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 35dc854. Configure here.
CLUSTER_TOPOLOGY is a state-based signal emitted from CLUSTER NODES, not an INFO extractor, so it must be skipped in the buffer-initialization loop like CLUSTER_STATE and REPLICATION_ROLE. Otherwise a MetricBuffer was created that never received samples, leaving isReady() false forever and preventing getWarmupStatus() from ever reporting isReady: true. Extends the exclusion test coverage accordingly.
| nodes: ClusterNode[], | ||
| ): DuplicatePrimaryConflict[] { | ||
| const masters = nodes.filter( | ||
| (n) => n.flags.includes('master') && n.slots.length > 0, |
There was a problem hiding this comment.
This filter matches fail/handshake nodes as live masters, which can fire a false CRITICAL during a normal failover: from one node's CLUSTER NODES view the old primary transiently shows as master,fail still listing its slots while the just-promoted replica already lists the same slots as master — both pass flags.includes('master'), overlap is found, and split-brain alerts on healthy recovery.
Suggest excluding transient/unhealthy states, e.g. also require the node isn't flagged fail/handshake/noaddr before treating it as a live primary.
There was a problem hiding this comment.
Nit addressed and resolved :)
jamby77
left a comment
There was a problem hiding this comment.
Approving — cleanest of the anomaly detectors: well-separated pure module, correct data-shape assumptions, both prior Bugbot comments already addressed.
One robustness nit left inline (non-blocking): the master filter can count a master,fail node during a normal failover and fire a false CRITICAL split-brain. A one-line flag exclusion closes it. Minor follow-ups: equal-configEpoch ties name an arbitrary phantom (soften the message), a couple of single-line conditionals vs CLAUDE.md, and as any in the detector spec.
…ction Addresses review feedback on #305. The live-primary filter matched any master-flagged node with slots, so during a normal failover the old primary — briefly `master,fail` but still listing its slots — and the just-promoted replica (same slots, `master`) both counted as live primaries, overlapped, and fired a false CRITICAL split-brain on healthy recovery. A node is now a live primary only if it is master-flagged, owns slots, and carries none of the transient/unhealthy flags fail / fail? / handshake / noaddr. A genuine split-brain (two healthy masters claiming the same slots) still fires. Tests: normal failover (master,fail overlap) and fail?/handshake/noaddr transients report nothing; a healthy two-master overlap still reports one.
…ary-detector # Conflicts: # packages/agent-cache/package.json # pnpm-lock.yaml
Summary
Adds topology-level anomaly detection for the fault described in valkey-io/valkey#2261: a shard ending up with two primaries (split-brain). We can't fix the server-side consensus bug, but we can alert operators the moment a cluster enters the invalid state.
From any single node's
CLUSTER NODESview, the unambiguous symptom is a hash-slot range claimed by more than onemaster-flagged node. The master with the lowerconfigEpochis the stale/phantom primary — exactly the node Valkey's own proposed fix demotes to a replica — so we name it in the alert.Changes
proprietary/anomaly-detection/duplicate-primary-detector.ts— finds overlapping-slot primary pairs and identifies the phantom (lowerconfigEpoch). Fully unit-tested in isolation.AnomalyService.pollConnection, inside the existing cluster-mode block, emitting aCRITICALCLUSTER_TOPOLOGYanomaly. Alerts are deduped per conflict signature and re-arm once the conflict resolves.MetricType.CLUSTER_TOPOLOGYandAnomalyPattern.SPLIT_BRAIN.configEpochonDiscoveredNodefor future UI highlighting of the phantom primary.CLUSTER NODEScall never breaks the poll loop (debug-logged, like the adjacent cluster-info block).Checklist
AnomalyServicepoll cases (emit, healthy, dedupe, resolve-and-recur, error-safe). 63 tests pass; typecheck clean.roborev review --branchor/roborev-review-branchin Claude Code (internal)Note
Medium Risk
New cluster-mode poll path affects alerting on production Valkey clusters; logic is guarded for failovers and poll failures but misclassification could cause alert noise or missed incidents.
Overview
Adds topology-level split-brain alerting when two live primaries claim overlapping hash slots (Valkey #2261), using
CLUSTER NODESon each cluster poll.A new pure
duplicate-primary-detectorfinds overlapping-slot primary pairs, treats the lowerconfigEpochnode as the stale primary, and skips transient flags (fail,handshake, etc.) so normal failovers do not false-positive.AnomalyServiceemits CRITICALCLUSTER_TOPOLOGYevents with per-conflict dedupe, re-alert after heal/recur, and clears dedupe whengetClusterNodesfails so missed heals cannot silence alerts.Also adds
MetricType.CLUSTER_TOPOLOGY,AnomalyPattern.SPLIT_BRAIN,configEpochonDiscoveredNode, broadensagent-cachetsconfig test excludes, and adds unit/integration tests for the detector and poll wiring.Reviewed by Cursor Bugbot for commit 24caaf5. Bugbot is set up for automated code reviews on this repo. Configure here.