Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat(retrieval): LongMemEval question-type filter + stratified slice#300

Merged
KIvanow merged 5 commits into
masterfrom
lme-harness-type-filter
Jul 8, 2026
Merged

feat(retrieval): LongMemEval question-type filter + stratified slice#300
KIvanow merged 5 commits into
masterfrom
lme-harness-type-filter

Conversation

@KIvanow

@KIvanow KIvanow commented Jul 3, 2026

Copy link
Copy Markdown
Member

Summary

Two eval-harness knobs so a subset of the LongMemEval set can be run in isolation instead of the whole dataset. Both are harness-only (under packages/retrieval/eval/longmemeval/); no library code is touched.

Changes

  • LONGMEMEVAL_TYPE (run.ts + dataset.ts): comma-separated question_type allow-list, e.g. LONGMEMEVAL_TYPE=temporal-reasoning,multi-session. Unset = all types. The flat LONGMEMEVAL_LIMIT now counts only KEPT (filtered) records, so N of a given type run cleanly rather than N raw records that may all be filtered out.
  • LONGMEMEVAL_PER_TYPE (run.ts + dataset.ts): stratified slice keeping N records of EACH question_type (a balanced, type-stratified sample for a paired A/B). Needed because _m/_s are grouped by type on disk, so a flat limit only samples the first type. In this mode the run total is perType x typeCount, and the runner's per-record cap is raised to match.
  • Run banner now reports the active type filter and stratified limit (e.g. dataset : ... (limit 1/type x6=6, type=...)).

Usage:

LONGMEMEVAL_TYPE=single-session-user LONGMEMEVAL_LIMIT=50 pnpm -F @betterdb/retrieval eval:longmemeval
LONGMEMEVAL_PER_TYPE=20 pnpm -F @betterdb/retrieval eval:longmemeval

Checklist

  • Harness-only (eval/), no src/ changes
  • pnpm --filter @betterdb/retrieval typecheck clean
  • Prettier applied
  • Smoke-run verified: type filter and stratified slice both honored (banner + per-type recall)

Note

Low Risk
Eval harness and test-only changes under eval/longmemeval; no production retrieval library behavior is modified.

Overview
Adds LongMemEval harness controls to run subsets and balanced samples without scanning multi‑GB files to EOF.

LONGMEMEVAL_TYPE filters records by comma-separated question_type; flat LONGMEMEVAL_LIMIT now counts only kept records. LONGMEMEVAL_PER_TYPE keeps up to N records per type (for datasets grouped by type on disk). Shared parseTypeList keeps the runner’s runLimit (perType × typeCount) aligned with loadRecords filtering and stratified early-stop via expectedTypeCount.

The run banner shows stratified limits (e.g. 1/type x6=6) and active type filters. Tests cover stratified early-stop on streamed JSON and parseTypeList semantics.

Reviewed by Cursor Bugbot for commit 9469a69. Bugbot is set up for automated code reviews on this repo. Configure here.

Add two eval-harness knobs so a subset of the LongMemEval set can be run
in isolation instead of the whole dataset:

- LONGMEMEVAL_TYPE: comma-separated question_type allow-list (e.g.
  "temporal-reasoning,multi-session"). Unset = all types. The flat
  LONGMEMEVAL_LIMIT now counts only kept records, so N filtered
  questions run cleanly.
- LONGMEMEVAL_PER_TYPE: stratified slice keeping N records of EACH
  question_type (balanced A/B), since _m/_s are grouped by type on disk
  and a flat limit would only sample the first type.

Harness-only change under eval/; no library code touched.
Comment thread packages/retrieval/eval/longmemeval/run.ts
Comment thread packages/retrieval/eval/longmemeval/run.ts
Two edge cases in the stratified run-limit math:

- A malformed LONGMEMEVAL_TYPE (e.g. only commas/whitespace) produced a
  zero type count, so runLimit=perType*0=0 stopped runEval immediately
  even though loadRecords treated the empty allow-list as "all types" and
  kept emitting a stratified sample - the harness ran no records.
- typeCount counted every raw comma entry, so a repeated type inflated
  runLimit and the banner while loadRecords (Set-keyed allow-list) yields
  at most one stratum per distinct type.

typeCount now derives from a deduped Set of trimmed types (mirroring
loadRecords) and falls back to the full 6 when the selection is empty, so
it is never zero and never double-counts.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 3210a0f. Configure here.

Comment thread packages/retrieval/eval/longmemeval/dataset.ts Outdated
Stratified mode (LONGMEMEVAL_PER_TYPE) only early-stopped when an explicit
LONGMEMEVAL_TYPE allow-list was set. With no filter the loader had no notion
of "all types seen" so it scanned the whole file to EOF even after every
per-type cap was full, draining a multi-GB dataset (longmemeval_m ~2.7GB).

Pass the LongMemEval type count (6) as an expectedTypeCount hint so the loader
can early-stop once counts.size >= expectedTypes and every count >= perType,
even with no explicit filter. Two regression tests use a poison-tailed JSON
stream to prove the loader stops before the malformed tail with the hint and
reaches it (throws) without.
@KIvanow KIvanow requested a review from jamby77 July 3, 2026 09:59
Comment thread packages/retrieval/eval/longmemeval/run.ts Outdated

@jamby77 jamby77 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — clean, well-tested, harness-only (no src/ changes), and all three prior Bugbot comments are addressed in the current head (Set dedup + LME_TYPE_COUNT fallback + the expectedTypeCount early-stop, with a test proving it stops before the malformed tail).

One cheap follow-up left inline: the type-list parse is duplicated between run.ts and dataset.ts and the early-stop math relies on them staying identical — a shared parseTypeList() would remove that coupling. Also minor: the >6-types case is capped by the hardcoded LME_TYPE_COUNT=6 (fine for real LongMemEval), and a narrowed cap = perType ?? 0 local drops the two as number casts.

KIvanow added 2 commits July 8, 2026 09:52
Addresses review feedback: the comma-separated question_type parse was
copy-pasted between run.ts (sizing the per-record limit cap) and dataset.ts
(filtering + stratified early-stop). The early-stop math relies on the two
derivations being identical, so a drift in one copy would silently desync
runLimit and the loader's expectedTypes.

- Extract `parseTypeList(value?): Set<string>` in dataset.ts as the single
  source of truth; import it in both run.ts and loadRecords.
- Narrow the per-type cap to a `const cap = perType ?? 0` local, dropping the
  two `perType as number` casts in loadRecords.
- Lock parseTypeList semantics (trim, drop blanks, dedupe, empty=all) with a
  focused test.
…lter

# Conflicts:
#	packages/retrieval/eval/longmemeval/run.ts
@KIvanow KIvanow merged commit 7f95164 into master Jul 8, 2026
3 checks passed
@KIvanow KIvanow deleted the lme-harness-type-filter branch July 8, 2026 07:39
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 8, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants