feat(retrieval): fact extraction + consolidation (Issue 3)#286
Conversation
7373465 to
3347356
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit b7d23c8. Configure here.
e9f84af to
952a0b1
Compare
|
Review follow-up pushed (952a0b1): |
- add eval/longmemeval/facts.ts: pure reconcile (ADD / UPDATE-supersede / DELETE-tombstone / NOOP by subject + event date) and applyOps curation - consolidateRecordFacts extracts atomic facts per session, reconciles them across sessions, and emits session-tagged fact chunks - add FactExtractor seam with mock (offline) and OpenAI implementations - index curated facts alongside raw chunks behind the facts lever and record one extraction LLM call per session in the cost report
reconcile compared each incoming fact only to the initial existing snapshot, so same-batch duplicate subjects, supersession chains, or a tombstone after an add produced wrong ops. Fold each op into the working map as the batch is walked so later facts see earlier add/update/delete results.
A fact restated identically across sessions kept only the first session's session_id (reconcile NOOP), so its fact chunk could carry a session_id absent from answer_session_ids and miss recall. Track every session asserting a fact's current statement and emit one fact chunk per source session.
Wrap the JSON.parse in parseFacts so a malformed LLM reply (or prose with stray brackets) degrades to no facts for that session instead of throwing and aborting the whole eval run. Export parseFacts and cover it with unit tests.
- reconcile ignores a stale tombstone dated before the curated fact (mirrors the supersede staleness check); a dateless tombstone still deletes - consolidateRecordFacts tags each per-session fact chunk with that session's own date instead of the first assertion's, so restated chunks order correctly
isNewer treated a missing candidate date as the empty string, so an undated later update always lost to any dated prior and the stale fact won. Sessions are reconciled chronologically, so a dateless candidate is the later assertion — give it the same winning default a dateless tombstone already has.
952a0b1 to
235ec1c
Compare
|
Closing per review — superseded by #299 (fact consolidation now implemented at the library level in @betterdb/agent-memory). Branch kept. |

Part of the LongMemEval recall→QA epic (item 205255004). Stack 3/4 — base
feature/longmemeval-context-assembly.Pure
reconcile(ADD / UPDATE-supersede / DELETE-tombstone / NOOP by subject+date) +applyOps;consolidateRecordFactsextracts per session and indexes curated session-tagged fact chunks behind thefactslever.FactExtractorseam (mock + OpenAI). One extraction LLM call/session in the cost report. 7 tests.QA lift target (+5pt knowledge-update/preference) pending a real Tier-2 eval.
Note
Low Risk
Changes are confined to the LongMemEval eval harness and optional
factslever; no production retrieval API or auth paths are modified.Overview
Introduces a facts ablation lever for LongMemEval that LLM-extracts atomic user facts per haystack session, merges them chronologically by subject (add / update / tombstone delete / noop), and appends curated fact chunks alongside normal session/turn chunks for indexing.
New
facts.tsdefinesreconcile/applyOps,consolidateRecordFacts(tracks every session that restates the same fact so recall can hit onsession_idwith the correct per-sessiondate), mock and OpenAIFactExtractorseams, and tolerantparseFactsfor model JSON.run.tswires the extractor whenLONGMEMEVAL_FACTSis on;runner.tsruns consolidation per record and records one LLM call per session in the cost report. Vitest covers reconciliation edge cases, parsing, consolidation tagging, and a harness check that recall does not regress vs baseline.Reviewed by Cursor Bugbot for commit 235ec1c. Bugbot is set up for automated code reviews on this repo. Configure here.