Visdoc

VisDoc is a document-image RAG project built with Qdrant for the ViDoRe computer science dataset. It retrieves relevant textbook page images, reranks them with a late-interaction retrieval pipeline, and sends the retrieved pages to a vision language model for final answer generation.

The repository includes the full workflow for loading the dataset, creating the Qdrant collection, indexing page images, searching documents, evaluating retrieval quality, and testing retrieval-augmented answer generation.

Flow

VisDoc RAG Flow

flowchart TD
    A[Page images] --> B[Page embeds]
    B --> C[Multivectors]
    B --> D[MUVERA vectors]
    A --> E[Page markdown]
    E --> F[miniCOIL vectors]
    C --> G[Qdrant collection]
    D --> G
    F --> G
    H[User query] --> I[Query embed]
    I --> J[MUVERA query]
    H --> K[miniCOIL query]
    J --> L[Visual prefetch]
    K --> M[Markdown prefetch]
    L --> N[RRF fusion]
    M --> N
    N --> O[Multivector rerank]
    O --> P[Retrieved pages]
    P --> Q[Answer model]

Loading

1. Load ViDoRe corpus pages, queries, and qrels with visdoc_dataset.py.
2. Convert each page image into ColModernVBert late-interaction vectors.
3. Convert the same multivectors into MUVERA fixed-size vectors for prefetch.
4. Convert page markdown into miniCOIL sparse vectors for lexical retrieval.
5. Upload full multivectors, MUVERA vectors, miniCOIL vectors, and payloads into Qdrant.
6. Embed the user query with ColModernVBert, MUVERA, and miniCOIL.
7. Search Qdrant with MUVERA and miniCOIL prefetch, fuse candidates with RRF, and rerank with full multivectors.
8. Send retrieved page images to Qwen, Gemma through vLLM, or OpenAI for final answer extraction.
9. Evaluate the retrieval flow with recall, MRR, NDCG, and latency metrics.

Concepts

• Page-level Retrieval treats each document page image as one searchable unit.
• Late-interaction Vectors keep multiple token/page vectors instead of one pooled document vector. This preserves fine-grained visual/text evidence on the page and gives better matching for specific details.
• MUVERA converts multi-vectors into fixed-size vectors so Qdrant can use HNSW for fast candidate prefetch before full multivector reranking.
• miniCOIL creates sparse vectors from page markdown, adding lexical retrieval over the text extracted from each page.
• RRF Fusion combines MUVERA visual candidates and miniCOIL markdown candidates before the final multivector rerank.
• Indexing embeds each page image, stores the full multivector, MUVERA vector, and miniCOIL markdown vector in Qdrant, and attaches payload metadata for retrieval and filtering.
• TurboQuant applies Google's 4-bit vector quantization in Qdrant, reducing memory use while keeping full multivector reranking available.
• Multivector Reranking reranks prefetched candidates with the original late-interaction vectors for better precision.
• Payloads store page metadata such as doc_id, markdown, and page_number_in_doc alongside vectors, and can be used later for filtering search results.
• Answer Extraction sends retrieved page images to Qwen, Gemma through vLLM, or OpenAI to produce the final RAG answer.

Setup

Install the Python dependencies:

uv sync

Run Qdrant locally with GPU indexing enabled:

docker run \
  --name qdrant \
  --gpus=all \
  -p 6333:6333 \
  -p 6334:6334 \
  -e QDRANT__GPU__INDEXING=1 \
  -v "$(pwd)/qdrant_storage:/qdrant/storage:z" \
  qdrant/qdrant:gpu-nvidia-latest

Use docker run only the first time. After the container exists, restart the same Qdrant instance with the same GPU, ports, environment, storage mount, and collections:

docker start qdrant

Useful follow-up commands:

docker stop qdrant
docker logs -f qdrant
curl http://localhost:6333/collections

• --gpus=all exposes the NVIDIA GPU to the container.
• QDRANT__GPU__INDEXING=1 enables GPU indexing in Qdrant.
• qdrant_storage/ keeps collections across container restarts, so indexed documents do not need to be rebuilt each time.

Optionally, for Gemma answer extraction, install vLLM and start a local OpenAI-compatible server for google/gemma-4-E2B-it-qat-w4a16-ct.

VLLM_WORKER_MULTIPROC_METHOD=spawn \
python -m vllm.entrypoints.openai.api_server \
    --model google/gemma-4-E2B-it-qat-w4a16-ct \
    --max-model-len 9500 \
    --max-num-seqs 1 \
    --trust-remote-code \
    --limit-mm-per-prompt '{"image": 8, "audio": 0}' \
    --enforce-eager \
    --gpu-memory-utilization 0.80 \
    --mm-processor-kwargs '{"max_soft_tokens": 1120}'

Usage

1. Inspect the ViDoRe computer science dataset:

uv run python visdoc_dataset.py

visdoc_dataset.py loads the configured ViDoRe dataset and prints basic dataset details plus a sample qrels entry.

2. Run the main Qdrant document search smoke test:

uv run python visdoc_search.py

visdoc_search.py contains the main QdrantDocumentSearch class for indexing, searching, answer extraction, and evaluation. Running it performs a basic local indexing, search, and evaluation test.

3. Evaluate retrieval on the ViDoRe dataset:

uv run python visdoc_eval.py

visdoc_eval.py indexes the configured collection when needed and reports retrieval metrics such as recall, MRR, NDCG, and latency.

4. Test retrieval plus answer extraction:

uv run python visdoc_test.py

visdoc_test.py searches an already-indexed collection, saves retrieved page images under retrieved_pages/, and asks the document answer model for the final RAG answer.

5. Run document answer models directly:

uv run python visdoc_answer.py

visdoc_answer.py contains the Qwen, Gemma, and OpenAI document answer extractors used after retrieval in the RAG pipeline.

6. Review shared configuration:

visdoc_config.py stores model, dataset, Qdrant, evaluation, and test settings.

Libraries

• Qdrant stores document vectors, payloads, indexes, and runs retrieval.
• ColPali Engine provides the ColModernVBert late-interaction document model and processor.
• PyTorch runs the local embedding model on CPU or CUDA.
• FastEmbed provides the MUVERA postprocessor and miniCOIL sparse text inference.
• Polars loads and transforms the ViDoRe corpus, queries, and qrels from Hugging Face datasets.
• Transformers runs the local Qwen document answer model.
• vLLM serves the optional Gemma document answer model through an OpenAI-compatible API.
• OpenAI provides an optional document answer extractor after retrieval.
• Ranx computes retrieval metrics such as recall, MRR, and NDCG.

Models

• ModernVBERT/colmodernvbert creates late-interaction document and query embeddings for retrieval.
• Qwen/Qwen3.5-4B answers questions from retrieved document page images locally.
• google/gemma-4-E2B-it-qat-w4a16-ct answers questions from retrieved document page images through a local vLLM server.
• gpt-5.4-mini provides the optional OpenAI answer extractor for retrieved document pages.
• MUVERA converts variable-length multi-vector embeddings into fixed-size vectors for fast prefetch.
• Qdrant/minicoil-v1 creates sparse vectors from page markdown for lexical prefetch.

Dataset

This project uses vidore/vidore_v3_computer_science from Hugging Face. It is a corpus of textbooks from the OpenStax website for long-document understanding tasks, and is one of the 10 corpora in the ViDoRe v3 Benchmark.

• corpus provides document pages. We use corpus_id as the document ID and image.bytes as the page image to embed.
• The Qdrant payload stores the remaining corpus fields: corpus_id, doc_id, markdown, and page_number_in_doc.
• queries provides evaluation questions. We use English rows from language, then keep query_id and query.
• qrels provides relevance labels. We join on query_id and use corpus_id plus score for evaluation.
• documents_metadata is loaded for dataset inspection and summary output.

Dataset sizes: corpus has 1,360 pages, queries has 1,290 rows, qrels has 6,294 relevance rows, and documents_metadata has 2 source documents.

Evaluation

The evaluation script checks whether queries retrieve their expected ViDoRe page IDs. It indexes the collection when needed and reports Ranx metrics plus average latency.

• Recall@K: whether a relevant page appears in the top K results.
• MRR: how early the first relevant page appears.
• NDCG@10: graded relevance quality in the top 10 results.
• Average latency: mean search time per query.

Defaults: recall@1, recall@3, recall@5, recall@10, recall@20, recall@50, mrr, and ndcg@10.

Current baseline:

Model	Recall@1	Recall@3	Recall@5	Recall@10	Recall@20	Recall@50	MRR	NDCG@10
muvera_minicoil_rrf_full_rerank	0.213	0.384	0.458	0.573	0.679	0.801	0.714	0.533

Sample Qwen3.5-4B answer after retrieval with the top 6 pages:

Query: What are the key differences between the Waterfall and V-model software development life cycle approaches in terms of how testing is integrated?

Answer: The Waterfall model is a sequential process where testing occurs at the very end, after all design and implementation phases are complete. In contrast, the V-model integrates testing throughout the development lifecycle. For each design phase in the V-model, there is a corresponding testing phase: unit testing corresponds to module design, integration testing to architecture design, system testing to system analysis, and acceptance testing to requirement analysis.

Sample runtime on an NVIDIA GeForce RTX 5070 Ti laptop GPU with 12 GB VRAM, 1024 px image resizing, 6 retrieved pages, and the Qwen answer model preloaded:

Qdrant search: 713.00 ms
Qwen3.5-4B answer extraction: 5,054.08 ms

Notes

• For the full GPU pipeline, use a machine with around 12 GB of GPU VRAM or more.
• The OpenAI answer extractor requires OPENAI_API_KEY in the environment.
• The images/ directory contains sample test images used by local smoke tests.
• After starting Docker, the Qdrant dashboard is available at http://localhost:6333/dashboard.
• Tune vLLM settings such as --max-model-len, --limit-mm-per-prompt, --gpu-memory-utilization, and --mm-processor-kwargs for the dataset, input size, number of retrieved pages, and available GPU memory.

References

• Multi-Vector Search: Qdrant multi-vector search course
• MUVERA: Weaviate MUVERA Podcast
• miniCOIL: Qdrant miniCOIL Talk
• ColPali: Qdrant ColPali Podcast
• ColModernVBERT: Hugging Face Model
• Qwen3.5-4B: Hugging Face Model
• Visual Document Retrieval: Towards AI Blog
• Polars Datasets: Huggingface Documentation