feat(retrieval): structured context assembly (Issue 1)#285
Conversation
|
Review follow-up pushed (228b3fb): enabling @KIvanow heads-up for interpreting your runs: the "assemble neutral" ablation measured the render-only pass. A real assemble ablation needs one of the option flags set — though given recall is already 92.8%, I wouldn't expect it to move the needle either. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 228b3fb. Configure here.
|
Fixed in 0075c23: a zero dedup threshold is rejected (treated as unset) — containment is never negative, so dedupe at 0 marked every later hit a duplicate and collapsed the reader to one context. MMR lambda 0 (pure diversity) stays valid. Unit-tested both. |
- add eval/longmemeval/assemble.ts: assembleContexts dedups near-duplicate chunks, MMR-selects k diverse hits, groups by session, and orders chronologically before they reach the reader - wire behind the assemble lever in runEval; operate on the rerank pool while recall stays measured on the unmodified top-k (no regression) - export tokenize from rerank.ts for reuse
dedupe and mmrSelect ranked candidates by QueryHit.score (raw vector distance), discarding the hybrid rerank order the runner passes in. Iterate in pool order and derive MMR relevance from pool position so the rerank-preferred near-duplicate and diverse contexts are kept.
On real LongMemEval-M, assemble (dedup + MMR + chronological re-sort by default) regressed QA hard by dropping and reordering retrieved evidence before the reader. Default assembleContexts to a non-destructive, relevance-first pass (rerank-ordered top-k + date prefix) and move dedup, MMR, and session grouping behind opt-in flags for isolated A/B.
Enabling LONGMEMEVAL_ASSEMBLE alone was a render-only no-op: dedup, MMR, and chronological grouping existed but nothing outside unit tests could enable them, so the lever's ablation point measured nothing. - Add resolveAssembleOptions: LONGMEMEVAL_DEDUP_THRESHOLD, LONGMEMEVAL_MMR_LAMBDA (unit-range floats), LONGMEMEVAL_GROUP - Thread assembleOptions through RunConfig into assembleContexts - Banner prints the active assemble features, or a render-only warning - Fix the stale runner comment describing structure that could not occur
containment is never negative, so dedupe at threshold 0 marks every later hit a duplicate and collapses the reader to a single context. Treat 0 as unset for the dedup threshold; MMR lambda 0 (pure diversity) stays valid.
0075c23 to
eea4759
Compare
Part of the LongMemEval recall→QA epic (item 205255004). Stack 2/4 — base
feature/longmemeval-ablation-foundation.assembleContexts: dedup near-duplicates → MMR-select k → group by session → chronological order, behind theassemblelever. Operates on the rerank pool; recall stays on the unmodified top-k (no regression). 6 tests.QA lift target (+5pt multi-session/temporal) pending a real Tier-2 eval.
Note
Low Risk
Changes are confined to LongMemEval eval harness and QA context formatting; production retrieval paths are unaffected and recall measurement is explicitly preserved.
Overview
Adds
assembleContextsfor the LongMemEvalassemblelever: by default it only date-prefixes the rerank-ordered top-k(same relevance order as before); dedup, MMR, and session chronological grouping are separate opt-ins viaAssembleOptions/ env (LONGMEMEVAL_DEDUP_THRESHOLD,LONGMEMEVAL_MMR_LAMBDA,LONGMEMEVAL_GROUP).When the lever is on, QA feeds the reader from the full rerank pool through
assembleContexts, while recall@k still uses the unmodified rerank top-k. The harness logs assemble mode (features vs render-only) and records assemble latency in the cost report.tokenizeis exported fromrerank.tsfor shared lexical overlap in dedup/MMR.New unit and integration tests cover defaults, each structure flag, env parsing (including rejecting a zero dedup threshold), and runner wiring.
Reviewed by Cursor Bugbot for commit eea4759. Bugbot is set up for automated code reviews on this repo. Configure here.