A reproducible, keyless benchmark that runs the same golden questions through the major enterprise RAG retrieval strategies, each with a small, runnable reference implementation.
Retrieval augmented generation has moved through several strategies, from a naive dense baseline to hybrid, agentic, multi agent, and multimodal systems. This repository compares them on identical ground, so the trade offs are measured, not just claimed. Each strategy has a public, runnable reference implementation, and they all expose the same POST /v1/chat API, so one harness scores them all.
| Strategy | Reference implementation | What it does |
|---|---|---|
| Naive | RagFlow | Single dense retrieval over Chroma |
| Advanced | RagFlowPlus | Hybrid retrieval fused with RRF, plus a cross encoder reranker |
| Modular | RagFlowPro | pgvector on Postgres, RRF computed in one SQL query, streaming |
| Agentic | RagFlowProPlus | A bounded self correcting loop with confidence grading |
| Multi agent | RagFlowProMax | A supervisor routes specialist agents and a verifier checks grounding |
Each reference implementation is public, has green CI, a security policy, and a live keyless demo.
Each strategy adds one idea and keeps the ones before it. Read top to bottom for how RAG retrieval has evolved.
- Naive. One embedding, one dense similarity search, top matches stuffed into the prompt. The honest baseline: it works on easy questions and has nothing to fall back on when they get hard.
- Advanced. Retrieval becomes hybrid, dense plus keyword, fused with reciprocal rank fusion, then a cross encoder reranker reorders the finalists. This rescues keyword heavy and synonym mismatched queries that dense search alone misses.
- Modular. The data plane moves to pgvector on Postgres, and the whole hybrid retrieval and fusion runs in a single SQL query next to the data. Answers stream, and a measurable evaluation harness lands. This is the production shape.
- Agentic. A bounded loop grades its own retrieved context, rewrites the query and retries when confidence is low, and declines rather than guessing when the grounding is weak. It returns a reasoning trace and a confidence score the simpler strategies do not.
- Multi agent. A supervisor routes the question to specialist agents, a synthesizer merges their findings, and a verifier checks the answer is grounded before it is returned. The whole path is bounded and traced, so every step is observable.
Three more reference implementations sit alongside the main progression, each public, green, and verified keyless in its own repository. The benchmark below scores the five above; these are demonstrated in their own repos.
- Knowledge augmented (RagFlowKAG). Extracts a knowledge graph of subject, predicate, object triples from the documents at index time, then links the question's entities, pulls their one hop neighbourhood, and grounds the answer in vector chunks plus graph facts.
- Cache augmented (RagFlowCache). Skips retrieval entirely: the whole small corpus is preloaded into context, and a semantic cache of past questions returns a stored answer with no model call when a new question is close enough.
- Multimodal (UltimateRAG). Retrieves across text and images in one vector space. A local vision model captions images at index time so they become searchable, and the images that grounded the answer are shown.
Same seven golden questions, same three documents, same local model (llama3.2:3b), scored on every strategy:
| Strategy | Facts answered | Declined, no hallucination | Score |
|---|---|---|---|
| Naive (RagFlow) | 5 / 5 | 2 / 2 | 7 / 7 |
| Advanced (RagFlowPlus) | 5 / 5 | 2 / 2 | 7 / 7 |
| Modular (RagFlowPro) | 5 / 5 | 2 / 2 | 7 / 7 |
| Agentic (RagFlowProPlus) | 5 / 5 | 2 / 2 | 7 / 7 |
| Multi agent (RagFlowProMax) | 5 / 5 | 2 / 2 | 7 / 7 |
All five score 100 percent, and that result is worth reading correctly rather than dressing up. On clean documents with direct questions, even the naive baseline retrieves the fact and refuses to invent one, so the strategy does not change the outcome here. Reporting a manufactured gap would be the opposite of the point.
The advanced strategies earn their keep where this small set does not reach: hybrid retrieval helps on keyword heavy or synonym mismatched queries that dense search alone misses, reranking decides between near duplicate chunks, the agentic loop adds self correction and hallucination pressure handling on harder, multi hop, or noisier corpora, and the multi agent strategy routes specialists and verifies grounding before answering. What this benchmark does prove is narrower and still real: every strategy is measured on identical ground, none of them hallucinate on the two out of document questions, the agentic strategy additionally returns a reasoning trace and a confidence score, and the multi agent strategy returns a full agent trace and an explicit grounding check, none of which the simpler strategies provide. A harder benchmark on a larger, noisier corpus is the natural next step to separate the strategies on accuracy, not just on transparency.
The harness in benchmark/ posts the same golden question set to each strategy's /v1/chat API, keyless on a local Ollama model (llama3.2:3b), and scores the answers. Answer items count as correct when the answer contains the known fact. Decline items count as correct only when the system refuses a question the documents do not cover, which is how hallucination resistance is measured. Every strategy runs on the same three sample documents that ship with the repos: an HR handbook, a product FAQ, and a real SEC 10-K excerpt.
This is an honest, small benchmark: a short labeled set, one small local model, a single run, and a keyword based answer check rather than an LLM judge. It shows direction, not a leaderboard, and the numbers move with the model and the questions. The value is that every strategy is scored on identical ground.
Bring up any reference implementation (see its README, most run keyless on Ollama and Docker), then:
cd benchmark
python run.py http://localhost:8000 llama3.2:3b change_meThe same command scores every strategy, since they all expose POST /v1/chat. The golden questions live in benchmark/golden.py.
Malav Patel. GitHub @mlvpatel.
Released under the MIT License. See LICENSE.