Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat(retrieval): fact extraction + consolidation (Issue 3)#286

Closed
jamby77 wants to merge 6 commits into
feature/longmemeval-context-assemblyfrom
feature/longmemeval-fact-consolidation
Closed

feat(retrieval): fact extraction + consolidation (Issue 3)#286
jamby77 wants to merge 6 commits into
feature/longmemeval-context-assemblyfrom
feature/longmemeval-fact-consolidation

Conversation

@jamby77

@jamby77 jamby77 commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

Part of the LongMemEval recall→QA epic (item 205255004). Stack 3/4 — base feature/longmemeval-context-assembly.

Pure reconcile (ADD / UPDATE-supersede / DELETE-tombstone / NOOP by subject+date) + applyOps; consolidateRecordFacts extracts per session and indexes curated session-tagged fact chunks behind the facts lever. FactExtractor seam (mock + OpenAI). One extraction LLM call/session in the cost report. 7 tests.

QA lift target (+5pt knowledge-update/preference) pending a real Tier-2 eval.


Note

Low Risk
Changes are confined to the LongMemEval eval harness and optional facts lever; no production retrieval API or auth paths are modified.

Overview
Introduces a facts ablation lever for LongMemEval that LLM-extracts atomic user facts per haystack session, merges them chronologically by subject (add / update / tombstone delete / noop), and appends curated fact chunks alongside normal session/turn chunks for indexing.

New facts.ts defines reconcile / applyOps, consolidateRecordFacts (tracks every session that restates the same fact so recall can hit on session_id with the correct per-session date), mock and OpenAI FactExtractor seams, and tolerant parseFacts for model JSON. run.ts wires the extractor when LONGMEMEVAL_FACTS is on; runner.ts runs consolidation per record and records one LLM call per session in the cost report. Vitest covers reconciliation edge cases, parsing, consolidation tagging, and a harness check that recall does not regress vs baseline.

Reviewed by Cursor Bugbot for commit 235ec1c. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread packages/retrieval/eval/longmemeval/facts.ts
@jamby77 jamby77 force-pushed the feature/longmemeval-fact-consolidation branch from 7373465 to 3347356 Compare June 30, 2026 08:12
Comment thread packages/retrieval/eval/longmemeval/facts.ts

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit b7d23c8. Configure here.

Comment thread packages/retrieval/eval/longmemeval/facts.ts
Comment thread packages/retrieval/eval/longmemeval/facts.ts
@jamby77 jamby77 force-pushed the feature/longmemeval-fact-consolidation branch 2 times, most recently from e9f84af to 952a0b1 Compare July 3, 2026 06:37
@jamby77

jamby77 commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator Author

Review follow-up pushed (952a0b1): isNewer treated a missing candidate date as the empty string, so an undated later update always lost to any dated prior (stale fact won). A dateless later candidate now wins, mirroring the dateless-tombstone default — sessions are reconciled chronologically. Latent with the real dataset (dates always present) but the reconcile semantics are consistent now.

jamby77 added 6 commits July 3, 2026 10:06
- add eval/longmemeval/facts.ts: pure reconcile (ADD / UPDATE-supersede /
  DELETE-tombstone / NOOP by subject + event date) and applyOps curation
- consolidateRecordFacts extracts atomic facts per session, reconciles them
  across sessions, and emits session-tagged fact chunks
- add FactExtractor seam with mock (offline) and OpenAI implementations
- index curated facts alongside raw chunks behind the facts lever and record
  one extraction LLM call per session in the cost report
reconcile compared each incoming fact only to the initial existing snapshot,
so same-batch duplicate subjects, supersession chains, or a tombstone after
an add produced wrong ops. Fold each op into the working map as the batch is
walked so later facts see earlier add/update/delete results.
A fact restated identically across sessions kept only the first session's
session_id (reconcile NOOP), so its fact chunk could carry a session_id absent
from answer_session_ids and miss recall. Track every session asserting a fact's
current statement and emit one fact chunk per source session.
Wrap the JSON.parse in parseFacts so a malformed LLM reply (or prose with
stray brackets) degrades to no facts for that session instead of throwing and
aborting the whole eval run. Export parseFacts and cover it with unit tests.
- reconcile ignores a stale tombstone dated before the curated fact (mirrors
  the supersede staleness check); a dateless tombstone still deletes
- consolidateRecordFacts tags each per-session fact chunk with that session's
  own date instead of the first assertion's, so restated chunks order correctly
isNewer treated a missing candidate date as the empty string, so an
undated later update always lost to any dated prior and the stale fact
won. Sessions are reconciled chronologically, so a dateless candidate is
the later assertion — give it the same winning default a dateless
tombstone already has.
@jamby77 jamby77 force-pushed the feature/longmemeval-fact-consolidation branch from 952a0b1 to 235ec1c Compare July 3, 2026 07:10

@KIvanow KIvanow left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented as part of #299 on the library level. Should be closed

@jamby77

jamby77 commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

Closing per review — superseded by #299 (fact consolidation now implemented at the library level in @betterdb/agent-memory). Branch kept.

@jamby77 jamby77 closed this Jul 6, 2026
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 6, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants