feat(retrieval): LongMemEval question-type filter + stratified slice#300
Conversation
Add two eval-harness knobs so a subset of the LongMemEval set can be run in isolation instead of the whole dataset: - LONGMEMEVAL_TYPE: comma-separated question_type allow-list (e.g. "temporal-reasoning,multi-session"). Unset = all types. The flat LONGMEMEVAL_LIMIT now counts only kept records, so N filtered questions run cleanly. - LONGMEMEVAL_PER_TYPE: stratified slice keeping N records of EACH question_type (balanced A/B), since _m/_s are grouped by type on disk and a flat limit would only sample the first type. Harness-only change under eval/; no library code touched.
Two edge cases in the stratified run-limit math: - A malformed LONGMEMEVAL_TYPE (e.g. only commas/whitespace) produced a zero type count, so runLimit=perType*0=0 stopped runEval immediately even though loadRecords treated the empty allow-list as "all types" and kept emitting a stratified sample - the harness ran no records. - typeCount counted every raw comma entry, so a repeated type inflated runLimit and the banner while loadRecords (Set-keyed allow-list) yields at most one stratum per distinct type. typeCount now derives from a deduped Set of trimmed types (mirroring loadRecords) and falls back to the full 6 when the selection is empty, so it is never zero and never double-counts.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 3210a0f. Configure here.
Stratified mode (LONGMEMEVAL_PER_TYPE) only early-stopped when an explicit LONGMEMEVAL_TYPE allow-list was set. With no filter the loader had no notion of "all types seen" so it scanned the whole file to EOF even after every per-type cap was full, draining a multi-GB dataset (longmemeval_m ~2.7GB). Pass the LongMemEval type count (6) as an expectedTypeCount hint so the loader can early-stop once counts.size >= expectedTypes and every count >= perType, even with no explicit filter. Two regression tests use a poison-tailed JSON stream to prove the loader stops before the malformed tail with the hint and reaches it (throws) without.
jamby77
left a comment
There was a problem hiding this comment.
Approving — clean, well-tested, harness-only (no src/ changes), and all three prior Bugbot comments are addressed in the current head (Set dedup + LME_TYPE_COUNT fallback + the expectedTypeCount early-stop, with a test proving it stops before the malformed tail).
One cheap follow-up left inline: the type-list parse is duplicated between run.ts and dataset.ts and the early-stop math relies on them staying identical — a shared parseTypeList() would remove that coupling. Also minor: the >6-types case is capped by the hardcoded LME_TYPE_COUNT=6 (fine for real LongMemEval), and a narrowed cap = perType ?? 0 local drops the two as number casts.
Addresses review feedback: the comma-separated question_type parse was copy-pasted between run.ts (sizing the per-record limit cap) and dataset.ts (filtering + stratified early-stop). The early-stop math relies on the two derivations being identical, so a drift in one copy would silently desync runLimit and the loader's expectedTypes. - Extract `parseTypeList(value?): Set<string>` in dataset.ts as the single source of truth; import it in both run.ts and loadRecords. - Narrow the per-type cap to a `const cap = perType ?? 0` local, dropping the two `perType as number` casts in loadRecords. - Lock parseTypeList semantics (trim, drop blanks, dedupe, empty=all) with a focused test.
…lter # Conflicts: # packages/retrieval/eval/longmemeval/run.ts

Summary
Two eval-harness knobs so a subset of the LongMemEval set can be run in isolation instead of the whole dataset. Both are harness-only (under
packages/retrieval/eval/longmemeval/); no library code is touched.Changes
LONGMEMEVAL_TYPE(run.ts + dataset.ts): comma-separatedquestion_typeallow-list, e.g.LONGMEMEVAL_TYPE=temporal-reasoning,multi-session. Unset = all types. The flatLONGMEMEVAL_LIMITnow counts only KEPT (filtered) records, so N of a given type run cleanly rather than N raw records that may all be filtered out.LONGMEMEVAL_PER_TYPE(run.ts + dataset.ts): stratified slice keeping N records of EACHquestion_type(a balanced, type-stratified sample for a paired A/B). Needed because_m/_sare grouped by type on disk, so a flat limit only samples the first type. In this mode the run total isperType x typeCount, and the runner's per-record cap is raised to match.dataset : ... (limit 1/type x6=6, type=...)).Usage:
Checklist
eval/), nosrc/changespnpm --filter @betterdb/retrieval typecheckcleanNote
Low Risk
Eval harness and test-only changes under
eval/longmemeval; no production retrieval library behavior is modified.Overview
Adds LongMemEval harness controls to run subsets and balanced samples without scanning multi‑GB files to EOF.
LONGMEMEVAL_TYPEfilters records by comma-separatedquestion_type; flatLONGMEMEVAL_LIMITnow counts only kept records.LONGMEMEVAL_PER_TYPEkeeps up to N records per type (for datasets grouped by type on disk). SharedparseTypeListkeeps the runner’srunLimit(perType × typeCount) aligned withloadRecordsfiltering and stratified early-stop viaexpectedTypeCount.The run banner shows stratified limits (e.g.
1/type x6=6) and active type filters. Tests cover stratified early-stop on streamed JSON andparseTypeListsemantics.Reviewed by Cursor Bugbot for commit 9469a69. Bugbot is set up for automated code reviews on this repo. Configure here.