feat(retrieval): query-time temporal resolver (Issue 2, slim)#287
Conversation
7373465 to
3347356
Compare
407df8b to
d456cd4
Compare
|
LongMemEval Full 500, k=10, pool=30, same harness as the baseline. Levers on (assemble+temporal, no facts): recall@10 92.8% (unchanged), QA 28.2% vs the 63.8% baseline (-35.6 pts). Per-type QA collapses everywhere, worst on multi-session (13.5%) and temporal-reasoning (21.1%). Recall is identical because it is measured on the untouched rerank top-k, which proves the gold evidence is retrieved into the pool. The assemble/temporal stage then loses it before the reader. Root causes:
Suggestions: run each lever in isolation on Separately, the |
54d7bd3 to
e9f84af
Compare
d456cd4 to
12121bf
Compare
|
Reworked to address the regression (pushed):
Net: assemble+temporal on should now sit at ≈ baseline rather than dropping/reordering evidence. Could you re-run Separately, the |
|
Follow-up on the |
KIvanow
left a comment
There was a problem hiding this comment.
Reviewed the full PR and re-ran LongMemEval-M (full 500, k=10, pool=30, same reader/judge/embedder + live valkey-search).
Code is clean and safe to merge. The as-of filter is correct: it drops chunks dated after question_date, keeps undated chunks, falls back to the original pool if the filter would empty it, and preserves rerank order. Recall stays measured on the unmodified top-k (pool.slice(0, k)), so recall is invariant to the lever. With the lever off, readerPool.slice(0, k) is identical to the previous hits path, so there is no baseline regression. Tests cover the edge cases plus zero-cost and no-recall-regression, which is the right contract.
It also resolves the earlier regression. The prior version re-sorted survivors newest-first, which fed a recency-ordered pool into the assembler's position-based relevance and collapsed QA (we saw 28.2% on the combined levers). Scoping this down to an as-of filter only, with ordering left to rerank/assemble, brings QA back to baseline: 62.4% vs 63.8% (within noise), recall unchanged at 92.8%.
One honest caveat: it does not improve QA on this benchmark. LongMemEval questions are asked at the end of each timeline, so almost nothing is dated after question_date and the filter has little to remove (measured cost was +12ms with effectively no pool change). The +8pt temporal-reasoning target in the description is unmet in practice. That is a property of the benchmark structure, not a code defect. The real QA bottleneck is downstream reasoning (multi-session and temporal-reasoning lag despite 90%+ recall), which a pool filter cannot address.
Net: this is defensible, well-tested, flag-gated scaffolding that fixes the regression and adds no risk, so merging is fine. Just flagging that the temporal QA lift will need to come from the evidence/reasoning layer, not the as-of filter.
12121bf to
41e7e8e
Compare
e9f84af to
952a0b1
Compare
41e7e8e to
50b5a67
Compare
952a0b1 to
235ec1c
Compare
- add eval/longmemeval/temporal.ts: resolveTemporal applies an as-of
("before X") filter with empty-result safety, then orders survivors
newest-date-first for recency-sensitive questions
- wire behind the temporal lever in the reader-context path; recall stays
measured on the unmodified rerank top-k (no regression)
Drop the newest-first recency re-sort from resolveTemporal. Re-sorting the pool by date fed a recency-ordered list into the assembler's position-based relevance and regressed QA on LongMemEval-M. The lever now only filters out chunks dated after the question, preserving rerank order.
50b5a67 to
7b7fe6e
Compare
Part of the LongMemEval recall→QA epic (item 205255004). Stack 4/4 — base
feature/longmemeval-fact-consolidation.resolveTemporal: as-of/'before X' filter (with empty-result safety) + newest-first recency ordering, behind thetemporallever in the reader-context path; recall unchanged. Scoped down because Issue 3'sreconcilealready supersedes and the assembler already orders chronologically. 4 tests.QA lift target (+8pt temporal-reasoning) pending a real Tier-2 eval.
Note
Low Risk
Changes are confined to the LongMemEval eval harness behind an optional lever; production retrieval is unaffected unless separately wired.
Overview
Adds an opt-in
temporallever on the LongMemEval eval runner that filters the rerank pool before reader contexts are built, without changing recall (still scored on the unmodified top-k).resolveTemporaldrops chunks whosedatetag is afterrecord.question_date, keeps undated chunks, preserves incoming rerank order (no recency re-sort), and falls back to the full pool if the filter would remove everything. When the lever is off, behavior is unchanged.The QA path now builds contexts from a
readerPool(temporal-filtered when enabled) for both assemble and non-assemble modes; the non-assemble branch takes the first k from that pool instead of the recallhitsslice. Temporal work is recorded in the cost report as latency-only (no embed/LLM calls). Unit and integration tests cover filter semantics and lever wiring.Reviewed by Cursor Bugbot for commit 7b7fe6e. Bugbot is set up for automated code reviews on this repo. Configure here.