Streaming Evaluation of Agent Memory for Future-Oriented Assistance
Paper • Quickstart • Data • Evaluation • Memory Systems • Citation
- 2026-06-12 Paper released on arXiv. StreamMemBench: Streaming Evaluation of Agent Memory for Future-Oriented Assistance
- Code and benchmark data released.
StreamMemBench is an open benchmark for evaluating streaming persona memory systems. It turns time-ordered multimodal life streams into evidence-anchored tasks, then evaluates whether a memory system can use initial evidence, incorporate feedback, and reuse corrected memory in follow-up interactions.
- Two-step task sequence centered around evidence anchors drawn from EgoLife egocentric streams. An initial task tests evidence use, and a follow-up task tests whether feedback and interaction experience are reused.
- Four diagnostic metrics covering fidelity, initial evidence use, feedback incorporation, and followup reuse. All are normalized to
[0, 1]. - 8 memory systems evaluated across two backbone models, including RAG baselines (
rag_raw,rag_extracted) and upstream memory systems (mem0,memos,evermemos,memskill,memoryos,a_mem). - 6 participants × 7 days of egocentric stream data, yielding 2,854 evidence-task items and 16,214 tasks.
- Chinese and English benchmark versions with shared English public schema fields.
Install the package and run the deterministic smoke baseline.
python -m pip install -e .
streammembench eval --system rag_raw --config configs/evaluation/smoke.yaml --max-items 1 --output results/quickstart/smoke.jsonl
streammembench aggregate --results results/quickstartFor API-backed construction and evaluation, copy .env.example to .env and set LLM_API_KEY. The default endpoint is DeepSeek. Set LLM_BASE_URL and LLM_MODEL to use another OpenAI-compatible provider.
python -m pip install -e .streammembench build --config configs/datasets/streammembench_v1.yaml
streammembench prepare --config configs/construction/egolife_v1.yaml --participant A1_JAKE --day DAY1
streammembench construct --config configs/construction/egolife_v1.yaml --participant A1_JAKE --day DAY1 --max-segments 1
streammembench eval --system rag_raw --config configs/evaluation/smoke.yaml --output results/local/rag_raw.jsonl
streammembench eval --system rag_raw --config configs/evaluation/paper_sample.yaml --output results/local/paper_sample.jsonl
streammembench systems
streammembench aggregate --results results/local
streammembench reproduce --system rag_raw --config configs/evaluation/release_sample.yaml --output-dir results/reproductionThe registered systems include two RAG baselines (rag_raw, rag_extracted) and memory systems (mem0, memos, evermemos, memskill, memoryos, a_mem). rag_raw runs without external API keys; rag_extracted uses an extraction model when configured. Third-party memory systems require their upstream packages, local checkouts, credentials, or service endpoints as documented in docs/memory_systems.md.
For eval and reproduce, the CLI automatically merges the main benchmark config with the registered system config under configs/memory_systems/. Use --system-config to point at a different system config for a local run.
See configs/README.md for how public configs map to the original research pipeline.
data/streammembench_v1 contains the converted full benchmark release.
- 6 participants
- 7 days
- 42 segment files
- 3,347 stream segments with source observations in
segments/ - 2,854 evidence-task items, covering 2,854 unique stream segments with accepted evidence anchors
- 493 stream segments without a public evidence-task item after extraction/review
- 8,107 evidence anchors
- 16,214 tasks
stream_segments denotes the complete set of released stream windows provided
as input to memory systems. evidence_task_items denotes the evaluated subset
for which at least one evidence anchor and its associated task sequence were
accepted during benchmark construction.
StreamMemBench is released in Chinese and English under a shared public schema
with English field names. The two language versions are provided in
data/streammembench_v1 and data/streammembench_v1_en, respectively, and
share the same participant coverage, temporal segmentation, and task-set size.
For dataset scope, release boundaries, and known limitations, see docs/dataset_card.md.
StreamMemBench v1 is derived from the EgoLife project. EgoLife should be cited alongside StreamMemBench when using this data.
The converted benchmark uses six EgoLife participants across seven days. Public stream files are split by participant/day and then by the processed five-minute stream windows. See docs/data_sources.md.
Each benchmark item contains one stream segment and one or more evidence anchors.
{
"participant": "A1_JAKE",
"day": "DAY1",
"segment_id": 1,
"time_range": "11:10:00 - 11:15:00",
"stream_segment": {
"text": "...",
"observations": []
},
"evidence_anchors": [
{
"evidence_id": "jake_workstation_setup",
"evidence_statement": "...",
"evidence_type": "stated_or_inferred",
"subject": "Jake",
"supporting_observations": [],
"source_span": {
"time_window": "11:10:00 - 11:15:00",
"clip": "A1_JAKE/DAY1",
"raw_indices": []
},
"tasks": {
"initial_task": {
"user_request": "...",
"expected_behavior": "..."
},
"followup_task": {
"user_request": "...",
"expected_behavior": "..."
}
}
}
]
}The public schema includes the following fields.
evidence,evidence_anchor,evidence_statementsupporting_observationsinitial_task,followup_taskfeedbackfidelity,initial_evidence_use,feedback_incorporation,followup_reuse
Construction supports the Chinese and English benchmark workflows with shared English public schema fields. Prompt templates are under prompts/, and their stage mapping is documented in prompts/README.md.
The released data preserves outputs from the two-round evidence/task construction and review pipeline. See docs/construction_pipeline.md for the source-to-release flow and multi-stage artifact layout.
The stable release flow is shown below.
EgoLife raw/processed events
-> five-minute stream segments
-> evidence anchor construction
-> two-round construction trace
-> public evidence-task files
-> memory-system evaluation
-> aggregate metrics
Stable public artifacts are saved in the following locations.
data/streammembench_v1/segments/{participant}/{day}.jsondata/streammembench_v1/evidence_tasks/{participant}/{day}.clean.jsonresults/construction/construction_logs/for newly generated construction tracesdata/source/egolife_text/{dense_caption,transcript}.jsonfor local stream reconstructionresults/{run_id}/...for generated evaluation outputs
Memory systems implement MemorySystem and are evaluated as agent-like systems. A system can wrap an in-process object, a local package, or a remote API as long as it can add stream records to memory, generate responses, revise with feedback, store the evaluated interaction, and optionally retrieve memory records for analysis. See docs/memory_systems.md.
Registered system names are listed below.
rag_raw,rag_extractedmem0,memos,evermemos,memskill,memoryos,a_mem
rag_raw and rag_extracted are complete local baselines. mem0, memos, memoryos, a_mem, evermemos, and memskill have direct entry points under src/streammembench/memory/systems/; they require the corresponding upstream package, local checkout, or service configuration before their results should be reported.
Evaluation results use four canonical metric keys.
fidelityinitial_evidence_usefeedback_incorporationfollowup_reuse
All metric values are normalized to [0, 1]. See docs/evaluation.md.
Each evaluation result contains run_id, system_name, participant, day, evidence_id, the four metric fields, initial_answer, feedback, revised_answer, followup_answer, and metadata. revised_answer is empty when the feedback simulator judges that the first answer needs no correction.
For the prediction/result JSONL contract and label-leakage rules, see docs/submission_format.md.
configs/ Dataset, evaluation, and memory-system configs
data/ Released Chinese/English benchmark data and smoke sample
prompts/ Construction and evaluation prompt templates
src/streammembench/ Python package
docs/ Dataset and reproduction documentation
results/ Local result output directory
If you use StreamMemBench in your research, please cite the paper.
@misc{streammembench2026,
title = {StreamMemBench: Streaming Evaluation of Agent Memory for Future-Oriented Assistance},
author = {Guanming Liu and Yuqi Ren and Hansu Gu and Peng Zhang and Weihang Wang and Jiahao Liu and Ning Gu and Tun Lu},
year = {2026},
eprint = {2606.14571},
archivePrefix = {arXiv},
primaryClass = {cs.AI}
}StreamMemBench v1 is derived from EgoLife. Please also cite EgoLife when using the benchmark data.
The StreamMemBench code is released under the MIT License. Benchmark data is derived from EgoLife and remains subject to the upstream EgoLife license and Hugging Face access terms.
StreamMemBench builds on the EgoLife project. We thank the EgoLife authors for making their egocentric stream data publicly available.
The benchmark integrates or evaluates upstream memory-system projects including A-Mem, Mem0, MemOS, MemoryOS, MemSkill, and EverMemOS. We thank their authors for releasing code that supports reproducible comparison.
For questions and feedback, open an issue on GitHub or contact the maintainers.