dream(memory): #3231 wire embedding near-dup detection into MemoryConsolidator.dedup() (evaluated, ACCEPT) - #3232
dream(memory): #3231 wire embedding near-dup detection into MemoryConsolidator.dedup() (evaluated, ACCEPT)#3232ruvnet wants to merge 6 commits into
Conversation
…solidator.dedup() MemoryConsolidator.dedup() only ever deduplicated entries via byte-exact SHA-256 content hashing, even though every entry already carries a computed .embedding and the adapter's HNSWIndex (already held in scope for removal bookkeeping) is incrementally kept in sync. Paraphrases and reformattings of the same memory were never caught. Adds a second pass: for hash-pass survivors with an embedding, query the already-populated HNSWIndex for cosine-similarity neighbors above a configurable similarityThreshold (default 0.95, matching the unwired domain-layer consolidator's own constant), and merge using the same keeper-selection strategies (now factored into shared selectKeeper/ mergeGroup helpers). Guarded to cosine-metric indexes only; disabled via similarityThreshold >= 1. Evaluation: 471/472 passing in @claude-flow/memory (1 pre-existing, unrelated, environmental failure); baseline (git-stash-isolated source) fails exactly the 2 new discriminating tests and passes the rest, confirming the fix is additive and non-regressive. tsc --noEmit clean. Co-Authored-By: RuFlo <[email protected]> Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG
…in one dedup() call Independent adversarial critique (STEP 10) reproduced a real, bounded gap in the prior commit: a near-duplicate cluster larger than NEAR_DUP_SEARCH_K (8) split into multiple leftover sub-group survivors within a single dedup() call, because each round's `consumed` bookkeeping permanently excluded a group's keeper from further matching even though it was still fully present in the index. 15 pairwise-identical-embedding entries collapsed to 2 survivors (merged: 13) instead of 1 (merged: 14). It self-healed across repeated runAll() calls (background timer / nightlyLearner), so this was never a permanent-data-loss bug, but a single manual dedup() call under-converged relative to its documented "collapse duplicates" contract. Fix: loop pass 2 to a fixed point (re-scan until a full round produces zero merges) instead of a single scan. Always terminates — entries strictly decrease each round that merges anything. `groups` now counts merge operations across all rounds, which can exceed the number of underlying duplicate clusters when one needed more than one round; documented in the method's doc comment. Added a discriminating regression test reproducing the critic's exact 15-entry scenario; confirmed it fails against the pre-fix single-round code (merged: 13) and passes against this fix (merged: 14, single survivor). Full package suite: 472/473 passing (1 pre-existing, unrelated, environmental failure, unchanged from before this commit). tsc --noEmit clean. Co-Authored-By: RuFlo <[email protected]> Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG
…ation Full SOTA report: 5-role parallel research fan-out, ledger check with GitHub-verified fates for the last 14 rows (3 merged since 09-03, one PR now crossing the 14-day stale threshold for the first time this cycle), competitor comparison (CrewAI/Mem0/Zep/LangMem/Qdrant/SemDedup), plugins and automation scan findings, adversarial critique with a real bug found and fixed, and witness stamp. No gist-creation MCP tool available in this session (GitHub MCP tools cover issues/PRs/repos, not gists) — committing to docs/dream-cycle/ instead, matching every dream-cycle night since 2026-08-14. Co-Authored-By: RuFlo <[email protected]> Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG
Co-Authored-By: RuFlo <[email protected]> Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG
|
CI red on This is not this PR's failure — this PR touches only Re-ran the failed job once to confirm (the allowed one re-run for a suspected flake) — will report back once it completes. No fix exists to port (this isn't caused by any change in this PR's diff, and I found no related open PR). If the re-run is still red, I'll dig further rather than assume flake. Generated by Claude Code Generated by Claude Code |
|
Update: the re-run came back green ( Generated by Claude Code Generated by Claude Code |
ruvnet
left a comment
There was a problem hiding this comment.
Dream exact-head review — INCONCLUSIVE
The implementation is deterministic and all six observed workflows pass, including CVE Audit, CodeQL, Cross-Agent Integration, Verification, and both CI pipelines. The focused tests also exercise fixed-point convergence and preserve explicitly distinct vectors.
The frozen acceptance gate is not yet met. The semantic-positive fixtures use identical cloned Float32Arrays rather than embeddings produced by a pinned model, and there is no representative labeled corpus measuring false merges, missed merges, latency, namespace isolation, or threshold sensitivity. Because the pass is default-on for MemoryConsolidator consumers, the 0.95 threshold needs an empirical contract rather than a smoke fixture.
Please retain issue #3231 and add:
- a committed or reproducibly generated, contamination-checked embedding corpus with pinned model/version;
- precision/recall (or false-merge/false-split) thresholds, including hard negatives and cross-namespace cases;
- baseline versus candidate latency/memory at realistic cardinalities;
- deterministic replay and unchanged exact-memory behavior.
This review does not reject the code path; it withholds the semantic-dedup claim until those receipts exist.
…sponse to review) Response to PR #3232's INCONCLUSIVE review, which asked for a threshold contract rather than a smoke fixture: a contamination-checked embedding corpus with pinned model, precision/recall including hard negatives and cross-namespace cases, baseline vs. candidate latency/memory at realistic cardinality, and deterministic replay + unchanged exact-dedup behavior. Real pinned-model corpus: attempted directly against both ONNX backends this monorepo ships. Both fail for reasons unrelated to this candidate, reproduced on an unmodified checkout before writing anything: - @huggingface/[email protected]: transformers.node.mjs does `import { Tensor } from "onnxruntime-common"` against a CJS module — a real ESM/CJS interop break under this sandbox's Node version. - @xenova/[email protected]: pulls in [email protected], whose native binding (sharp-linux-x64.node) isn't present for this platform — the same failure independently reproduced moments later by this package's own nightlyLearner test, which falls back to mock embeddings. Neither is fixable within this PR's scope without touching unrelated, pre-existing native/module-resolution infrastructure. Documented in detail at the top of the new test file rather than silently worked around or omitted. What's delivered instead, in v3/@claude-flow/memory/src/consolidator-embedding-benchmark.test.ts: - A deterministic synthetic corpus where pairwise cosine similarity is constructed EXACTLY via vectorAtSimilarity() (Gram-Schmidt against a random orthogonal vector), not measured after the fact — a rigorous instrument for threshold mechanics specifically, disclosed as not validating real-world semantic accuracy. - Precision/recall/false-merge-rate across a 5-point threshold sweep, every point at a deliberate margin from every corpus similarity value after an exact-boundary collision (sweep value == corpus value) proved float32-jitter-sensitive while writing this — documented as a finding, not silently patched around. - Namespace isolation: proved cross-namespace near-dup merging matches cross-namespace hash-exact merging exactly (both pre-existing, unscoped-by-namespace behavior — this candidate doesn't change it). - Determinism: 3 independent runs, compared by deterministic entry KEY (not raw id, which is randomly generated per store() and was a second bug this file's own first draft had to fix). - Latency/memory at N=5000 (matching this repo's own HNSW-benchmark convention): initially measured 9.8s for the candidate pass, which traced to an unset `ef` search parameter defaulting to efConstruction (200) — every dedup() search traversed a 200-candidate list for a duplicate-detection task that only needs very-close neighbors. Fixed by passing an explicit NEAR_DUP_SEARCH_EF=32 in consolidator.ts, re-measured at 3.7s (~2.7x), precision/recall unchanged (still exact). Real numbers logged in the test output either way, not asserted away. Full package suite: 478/479 passing (472 pre-existing + 7 new; same 1 pre-existing unrelated environmental failure as every prior commit on this branch). tsc --noEmit clean. Co-Authored-By: RuFlo <[email protected]> Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG
|
Pushed Pinned-model corpus — blocked, documented, not silently skipped. I tried both real ONNX backends this monorepo ships, directly, before writing anything else:
Neither is fixable in this PR's scope without touching unrelated native/module-resolution infrastructure. So: the semantic-dedup claim against a real pinned model remains open, exactly as the review says it should until that infra is fixed — I'm not claiming otherwise. What I could deliver instead, and did:
Full suite: 478/479 (472 pre-existing + 7 new this commit; the 1 failure is the same pre-existing unrelated environmental one on every commit this branch). CI is running on Generated by Claude Code Generated by Claude Code |
CI on a2e99cf failed with "1 unexpected failing file(s): + v3/@claude-flow/memory/src/consolidator-embedding-benchmark.test.ts". Root-caused via the vitest JSON report artifact (test-results-ubuntu-latest, downloaded and parsed directly, not guessed): a suite-level collection failure, "Cannot find package '@claude-flow/security' imported from '.../v3/@claude-flow/memory/src/agentdb-retrieval-guard.ts'" — the same unbuilt-sibling-package gap already accepted in scripts/ci-test-baseline.txt for 9 other @claude-flow/memory test files, including consolidator.test.ts itself (line 95). Every file in this package fails identically in this CI checkout state; my new file just wasn't in the baseline yet because it's new. Not a regression this PR introduced — confirmed by the failure being a pre-import-time module-resolution error, not a test assertion in the new file's own logic (which passes locally in a checkout with @claude-flow/ memory's dependencies actually built: see the 6/6 local run in the prior commit's message). Fix: add the new file to the baseline, alongside its 9 memory-package siblings already there, matching this repo's own established convention for this exact gap rather than working around it some other way. Co-Authored-By: RuFlo <[email protected]> Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG
|
CI red on Root-caused via the actual Pushed Generated by Claude Code Generated by Claude Code |
|
CI red on
This is unrelated to this PR's diff — I've touched only Per this PR's re-run budget (at most one flake-confirmation re-run in total, already spent on the earlier Generated by Claude Code Generated by Claude Code |
Records tonight's finding, verified GitHub fates for the trailing 7 nights (including a newly-discovered merge conflict on #3232), and the evaluation verdict. Co-Authored-By: RuFlo <[email protected]> Claude-Session: https://claude.ai/code/session_013LQSLa9h1uLdYsQBP5r36z
1. Hypothesis
Frozen before evaluation; condition (4) was added mid-session after an independent adversarial critic reproduced a violation of it (see §3/§8) — a legitimate strengthening the critique surfaced, not a post-hoc relaxation to match a weaker result.
2. Candidate
v3/@claude-flow/memory/src/consolidator.ts(2 commits, ~130 net lines):MemoryConsolidator.dedup()previously only bucketed entries bysha256(content). Every entry already carries.embedding?: Float32Array(types.ts:66), anddedup()already holds a handle to the adapter's incrementally-syncedHNSWIndex— used only forremovePoint()bookkeeping, never queried for similarity.Added: a second pass that, for hash-pass survivors with an embedding, calls the already-in-scope
index.search(entry.embedding, 8)and merges any cosine-similarity match ≥opts.similarityThreshold(new option, default 0.95) using the existing keeper-selection strategies — now factored into sharedselectKeeper/mergeGrouphelpers reused by both passes. Guarded to run only whenindex.getConfig().metric === 'cosine'and threshold< 1; otherwise byte-identical to pre-existing behavior.An orphaned second implementation (
MemoryDomainService.consolidate(),domain/services/memory-domain-service.ts, ~700 LOC, zero tests) already does real embedding-cosine grouping, but was confirmed not directly reusable: it's bound to an injectedIMemoryRepositoryrequiring a genuine per-entrysearchByVector()round-trip for a real backend (its own backing repo does a brute-force O(n) scan, not even HNSW-backed), andindex.test.ts:43explicitly asserts its backing class must not be part of the public export surface — a deliberate quarantine, not an oversight. The in-scopeHNSWIndexhandle is strictly cheaper and was chosen instead.3. Evaluation Receipt
evaluated: accepted. Real evaluator: Vitest 4.1.8, deterministic, $0, zero LLM calls.
NEAR_DUP_SEARCH_K-wide (8) neighborhood query per entry with no transitive step, so a duplicate cluster larger than 8 split into multiple leftover survivors within a singlededup()call (reproduced: 15 pairwise-identical-embedding entries →merged: 13, 2 survivors, instead ofmerged: 14, 1 survivor). Bounded severity — it self-heals across repeatedrunAll()calls (background timer /nightlyLearner) sinceconsumedresets every call — but a single manualdedup()call under-converged relative to its documented "collapse duplicates" contract.merged: 13) and passes with the fix-forward applied (merged: 14, single survivor).@claude-flow/memorysuite: 472/473 passing (466 pre-existing + 7 new; the 1 failure isauto-memory-bridge.test.ts's pre-existing chmod-based test, confirmed unrelated and identical with/without this diff — the sandbox runs as root, bypassing the permission check it relies on).tsc --noEmit: zero errors.4. Baseline Comparison
HNSWIndex, zero extra backend round-tripsdedup()calltsc --noEmit5. Darwin Lineage
Real interface confirmed tonight (
ruvector harness darwin --help:darwin <config> --execute). Skipped, with a reason distinct from prior binary-fix nights:similarityThresholdis a genuine continuous tunable with a real quality/false-positive tradeoff (Darwin-eligible in principle, unlike several recent invariant-only fixes), but running it honestly needs a labeled near-duplicate benchmark corpus (true-duplicate vs. true-distinct pairs with ground truth) that doesn't exist in this repo — out of scope tonight per the research-protection budget policy. The shipped default (0.95) matches the orphaned domain-service's own already-chosen constant and CrewAI's closely-adjacent 0.85 gate, not an arbitrary pick. Flagged as a concrete follow-up.6. Flywheel Evidence
Real interface confirmed (
ruvector harness flywheel --help:verify <bundle>/gate <evidence>). No signed bundle — the schema targets LLM-task-corpus-evaluated retrieval-policy candidates; this is a deterministic code-correctness fix, consistent with every accepted night since 2026-08-18. Evidence retained as: 7 new tests (2 commits) + the linked issue + the gist + two stash-isolated baseline/candidate comparisons + the independent adversarial critique.7. Reward Hack Check
No standalone reward-hack CLI reachable this session (
weight-eftis a LoRA-distillation tool, not a diff/benchmark scanner). Manual checklist, independently re-verified by the adversarial critic with no authoring context: no test weakened (diff on the test file is purely additive — confirmed by direct diff of the pre-existing 9 tests, byte-identical); no gold-label leakage; no cherry-picking (full suite run and reported honestly both ways, pre-existing failure included, not omitted); no seed manipulation (deterministic PRNG-seeded embeddings throughout); zero cost. One disclosed caveat below (§9).8. Security Review
Not the primary focus, but relevant to data-retention correctness: this pass deletes entries judged near-duplicate, so a coincidental (or, if the embedding path were ever exposed to untrusted input, adversarially-engineered) high-cosine collision could cause a legitimate memory to be silently dropped. Mitigated by: a conservative 0.95 default requiring near-exact vector alignment, not merely "related" content; the pass runs only during consolidation (background timer /
nightlyLearner), not the synchronous write path; no new network/filesystem/credential surface (confirmed line-by-line by the adversarial critic — pureMap/HNSWIndexoperations). No fix required tonight.9. Regression Analysis
Non-hash-exact-dedup behavior is unaffected when embeddings are absent or the index metric isn't cosine (explicit fallback tests). Disclosed, not hidden:
MemoryService.getConsolidator()does not overridesimilarityThreshold, so the near-dup pass is on by default at 0.95 for every existingMemoryServiceconsumer — a real behavior change for currentrunAll()/background-auto-run-timer users, not merely opt-in for new callers. Judged acceptable given the conservative default and the 2026-09-03 MMR-fix precedent (also default-on), but flagged explicitly here rather than left implicit for reviewers to discover.10. ADR
None created — this activates an already-computed-but-discarded signal at an existing call site (same class as 2026-09-03 MMR/#3169, 2026-08-25 PQ-dispatch/#3094, 2026-08-28 HybridBackend-weights/#3119, none of which created an ADR), not a new architectural decision.
11. Research Gist
docs/dream-cycle/dream-gist-2026-09-08.md(this branch) — full SOTA report (5-role parallel research fan-out), competitor comparison (CrewAI/Mem0/Zep/LangMem/Qdrant/SemDedup), plugins and automation scan findings, and the adversarial critique transcript summary.12. Issue
Closes #3231 (full 15-section report: ledger check with GitHub-verified fates for the last 14 rows, research, evaluation receipt, Darwin/Flywheel interface confirmation, reward-hack check, security review, plugins/automation scan findings, competitor comparison, witness).
13. Witness
a295c68703158377a6ce827738bf8f13b94bd69524117f596be1304943be8705b08d1d157bf5904919df355774a027b523ff6909f5eaa46114b6b67e8dcf01e38d73c61aabb9400f8fe6b662cc191b1dc9787827Verifier procedure: fetch
docs/dream-cycle/dream-gist-2026-09-08.mdfrom this branch, strip the witness table's filled values back toPENDING, SHA-256 it, concatenate with the session commit above, SHA-256 again — result must equal the witness stamp.14. Merge Policy
Human review required. Do not self-merge. Do not autonomously promote Flywheel state.
🤖 Generated with RuFlo
https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG
Generated by Claude Code