Verbative Benchmarks

Reproduce Verbative's memory-benchmark results — LoCoMo and AMA-Bench (all six domains) — against your own installed Verbative memory engine. No engine source lives here: the benchmarks drive the same closed-source engine that ships in the product, so the numbers you get are the numbers the product actually produces.

What this is: the test harness (dataset loaders, context assembly, reader/ judge prompts, scoring, orchestration) — everything except the engine. What it needs: the verbative-memory package (the engine) plus a Verbative account with the memory entitlement (Advanced, or an active trial), and one API key for the reader/judge model.

---

Requirements

Need	Why
A Verbative account with memory (Advanced or trial)	The engine gates on your entitlement — that's what makes the run your installed version.
pip install verbative-memory	The closed-source engine (extraction, embeddings, retrieval, ranking, reranker). ~4 GB of on-device models download on first run.
An OpenRouter and/or OpenAI key	The reader answers and the judge grades. AMA defaults to Qwen3-32B via OpenRouter; LoCoMo defaults to gpt-4o-mini via OpenAI. Both are fully configurable.
Python ≥ 3.10	Harness runtime.

Everything is local except the reader/judge API calls — the memory engine (capture, embeddings, retrieval, reranking) runs entirely on your machine.

---

Published results

published-results/ holds the evidence behind the numbers we quote: the final validated run's per-question answers and judge verdicts (join them to the dataset by episode_id + question index — the dataset itself is not redistributed here), plus the run's aggregate scores and configuration. It is generated by tools/export_published_results.py, which allow-lists fields and hard-fails on any absolute path or username, so what ships is exactly the model's own outputs — nothing from your machine, nothing from the dataset.

---

Quickstart

# 1. Install the harness + engine
pip install -r requirements.txt          # pulls verbative-memory (the engine)
verbative-memory setup                    # one-time: download the on-device models (~4 GB)

# 2. Configure
cp config.example.toml config.toml        # defaults reproduce the published numbers
cp .env.example .env && $EDITOR .env       # add your OPENROUTER_API_KEY / OPENAI_API_KEY

# 3. Get the datasets (downloaded, never redistributed)
python scripts/fetch_datasets.py           # LoCoMo auto; AMA prints how to fetch it

# 4. Preflight — engine installed, entitled, models loaded?
python scripts/check_engine.py

# 5. Run
python -m locomo.run_locomo               # LoCoMo retrieval + judged J-score
python -m ama.run_ama                      # AMA-Bench (AMA_SPLIT=SOFTWARE default; TEXT2SQL/WEB/Game/EMBODIED_AI/OPENWORLD_QA)

Both runners checkpoint — every question/episode is written as it finishes, so a rerun skips completed work and survives Ctrl-C, reboots, and crashes.

---

The benchmarks

LoCoMo — long-term conversational memory

Multi-session dialogues; the engine ingests each conversation, then answers questions that require recalling and reasoning across sessions. Scored with the micro-averaged J-score using the mem0 memory-benchmarks reader/judge protocol (gpt-4o-mini), so it's directly comparable to published cloud-memory numbers. 10 conversations, 1,540 questions (multi-hop, single-hop, open-domain, temporal).

Published (verbative.de):

Top-k contexts	J-score	Mean context tokens
10	65.4	205
20	68.6	407
30	69.5	598
50	69.9	958
200	69.6	3,398

The headline: top-20 already edges the cloud reference (68.6) at ~4× less context.

AMA-Bench — agent-trajectory memory (six domains)

Question-answering over recorded software-agent trajectories (an AI fixing a bug step by step). Reader and judge are Qwen3-32B (thinking on) via OpenRouter, following the AMA-Bench leaderboard protocol. 36 episodes, 432 questions across four categories: Recall (A), Causal (B), State-Updating (C), Abstraction (D).

Published (verbative.de): 48.1% macro / 47.9% micro on SOFTWARE under the validated Nebius judge — vs the official AMA-Agent baseline's 45.8% on the authors' own instrument.

Provider purity matters — and the judge host is chosen by measurement (full study: ama/README.md). The authors publish two SOFTWARE numbers for their own baseline: 45.8% (Google Drive evaluation_result.json, per-question verdicts, vLLM unquantized judge) and 63.55 (leaderboard dashboard, self-reported capability scores). We judged both official targets on every OpenRouter host serving qwen/qwen3-32b, allow_fallbacks = false. The config pins provider_pin = "Nebius" because Nebius reproduces the authors' published verdicts at 99.0% per-verdict agreement (95/96; SiliconFlow 93.8%, Groq 88.5%, DeepInfra 84.4%). Judging the authors' complete published run (432 answers), no host lands within 11 points of the leaderboard's 63.55 (Nebius 36.8% … DeepInfra 52.8%). The leaderboard's own code aggregates submitters' self-reported judge booleans as-is and promises an organizer re-judge with an undisclosed judge config ("weekly verification") — whichever of the two produced 63.55, it is not reproducible by any available qwen/qwen3-32b host, nor by the authors' own published vLLM instrument. Do not use Alibaba as the judge: in the earlier measured trial it over-scored by +26 pt with only 75% agreement (72% vs the true 45.8%); during the fresh study its endpoint only returned 502 errors.

---

Configuration

Everything lives in config.toml (copy from config.example.toml). It's fully configurable — pick any OpenAI-compatible provider and model per role:

[ama.reader]
provider     = "openrouter"
model        = "qwen/qwen3-32b"
provider_pin = "Nebius"       # VERIFIED against the official judge (see ama/README.md); NOT Alibaba — lenient
thinking     = true

Any value can be overridden by an environment variable of the same UPPER_SNAKE path — e.g. AMA_READER_MODEL=meta-llama/llama-3.1-70b or LOCOMO_JUDGE_PROVIDER=openrouter — handy for sweeps without editing the file. API keys are read from the env var named by each provider's api_key_env; they are never written to the config.

See config.example.toml for every knob and its default.

---

How it works (and why there's no engine source here)

   dataset ──►  [ verbative-memory engine ]  ──►  reader (LLM)  ──►  judge (LLM)  ──►  score
   (download)      ↑ closed-source, installed         ↑ your API key
                   │ via pip; runs on-device
              this repo = everything around it

The harness ingests each dataset into an isolated benchmark memory store (data_dir in the config — it never touches your real project memory), retrieves through the engine exactly as the product does, then reads and judges. The engine is imported from the installed verbative-memory package (verbative_bench/engine.py) — its code is never copied into this repository. That's the point: you're measuring the engine you actually run.

---

Licensing & attribution

• Harness (this repo): MIT — see LICENSE.
• Engine (verbative-memory): closed-source, proprietary; not covered by the MIT license.
• Datasets: downloaded from upstream (LoCoMo — Snap Research; AMA-Bench — MIT), each under its own license; not redistributed here.
• Prompts & judge: the LoCoMo reader/judge prompts are adapted from mem0's memory-benchmarks (Apache-2.0, see locomo/prompts.py); the AMA-Bench judge prompt and evaluation metrics are reproduced from AMA-Bench (MIT — https://github.com/AMA-Bench/AMA-Bench, see ama/judge.py), with attribution retained.

---

Troubleshooting

• The Verbative memory engine is not installed — pip install verbative-memory, or set VERBATIVE_ENGINE_PATH to a local engine hooks dir.
• Embedder unavailable — run verbative-memory setup to download the on-device models.
• PURITY VIOLATION: pinned 'Alibaba' but served by … — OpenRouter routed to a different host; retry, or change provider_pin. This guard is deliberate.
• AMA dataset skipped — AMA-Bench has no pinned public URL yet; follow the printed instructions to point AMA_LOCAL / AMA_URL at the release file.