feat(retrieval): entity graph lever for LongMemEval#295
Closed
jamby77 wants to merge 1 commit into
Closed
Conversation
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 14c9780. Configure here.
14c9780 to
c5d90ea
Compare
457eae2 to
20ea924
Compare
- Add eval/longmemeval/graph.ts: buildEntityGraph + traverseGraph (BFS over entity co-occurrence, as-of temporal validity, deduped, capped) - EntityLinker seam with mock (capitalized words) and OpenAI (batch JSON, guarded parseEntityLists) implementations - consolidateRecordFacts also returns the curated facts (additive) - Wire LONGMEMEVAL_GRAPH reader-side only (requires LONGMEMEVAL_FACTS): question entities seed a 1-2 hop traversal whose facts are appended to reader contexts, deduped against evidence already present; recall stays measured on the unmodified rerank top-k - Knobs: LONGMEMEVAL_GRAPH_HOPS (2), LONGMEMEVAL_GRAPH_MAX_FACTS (5), LONGMEMEVAL_GRAPH_MODEL; cost report gets 1 link call per record plus 1 seed call per question
20ea924 to
fca29ff
Compare
c5d90ea to
6f3135a
Compare
Member
|
Closing it for now, as we prioritise other things and this doesn't move the needle sufficiently |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the LongMemEval recall→QA gap epic (Issue 6 — entity/relationship graph). Stacked on #291.
What
Adds the
graphlever: a lightweight in-memory entity graph built per record from the curated facts (Phase 4), traversed 1–2 hops at question time to pull connected facts the vector query misses.eval/longmemeval/graph.ts— purebuildEntityGraph(entity → fact adjacency, case-normalized) andtraverseGraph(BFS from seed entities; as-of temporal validity mirroring the temporal lever — facts dated after the question are dropped, dateless facts kept; discovery-order dedupe; capped).EntityLinkerseam (texts[] → entities[][], one batched call): mock (capitalized words) for offline runs, OpenAI (JSON array-of-arrays, guardedparseEntityListsthat degrades to empty lists on malformed replies).consolidateRecordFactsnow also returns the curated facts (additive return field).Wiring
LONGMEMEVAL_GRAPH=1, inert withoutLONGMEMEVAL_FACTS=1(the graph is built over curated facts). Reader-side only: question entities seed the traversal and the traversed facts are appended to the reader contexts, skipping any fact whose statement is already present in a context. Recall stays measured on the unmodified rerank top-k, so recall cannot regress by construction (same principle as the reworked assemble/temporal levers).Knobs:
LONGMEMEVAL_GRAPH_HOPS(default 2),LONGMEMEVAL_GRAPH_MAX_FACTS(default 5),LONGMEMEVAL_GRAPH_MODEL. Cost report: 1 batch entity-link LLM call per record + 1 seed-link call per question; nothing is stored on Valkey.Tests
12 new tests (traverse hop-1/hop-2/as-of/dedupe+limit/unknown-seeds, mock linker, parse guards, curated-facts return, 2 runEval integrations incl. no-duplicate-evidence). Full retrieval suite 184/184.
Offline Tier-0 smoke (
LONGMEMEVAL_FACTS=1 LONGMEMEVAL_GRAPH=1 LONGMEMEVAL_QA=1): baseline metrics unchanged, cost linesfacts llm=15,graph llm=6(3 records × 2 calls).The +5pt multi-session QA target awaits a real Tier-2 run on
_m.Note
Low Risk
Changes are confined to the LongMemEval eval harness and reader-side context assembly; retrieval scoring and production retrieval paths are untouched.
Overview
Adds a
graphLongMemEval lever that builds an in-memory entity graph from curated facts and expands reader contexts at question time without changing recall@k.graph.tsintroducesbuildEntityGraph, BFStraverseGraph(hop depth,asOftemporal filter, dedupe, limit), and anEntityLinkerseam (mock capitalized-word linker + OpenAI batch linker with safeparseEntityLists).consolidateRecordFactsnow also returnsfacts: Fact[]so the graph can be built over reconciled facts.runner/run: whengraphis on (requiresfacts), each record gets one batched entity-link call over fact statements; per question, question entities seed traversal and up toLONGMEMEVAL_GRAPH_MAX_FACTS(default 5) connected facts are appended to contexts, skipping statements already present. Env:LONGMEMEVAL_GRAPH,LONGMEMEVAL_GRAPH_HOPS,LONGMEMEVAL_GRAPH_MODEL. Graph costs are tracked in the cost report.12 new tests cover traversal, parsing, curated-facts return, and
runEvalintegration (including no duplicate evidence).Reviewed by Cursor Bugbot for commit 6f3135a. Bugbot is set up for automated code reviews on this repo. Configure here.