• This is the repo for the paper PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented Generation.
• 🎉 Accepted to AAAI 2026

中文文档 | Usage Guide | Contributing | Changelog

PRGB (Placeholder RAG Benchmark) is a benchmark tool focused on evaluating document faithfulness and external knowledge utilization efficiency in Retrieval-Augmented Generation (RAG) systems. It comprehensively evaluates model performance through progressive dimensions such as multi-level filtering and cross-entity reasoning, using placeholders with noise-injected datasets to help researchers and developers analyze the performance of mainstream RAG models in complex scenarios.

1. ✅ API-based Inference: Provide inference and verification functionalities based on APIs.
2. ✅ Data Quality Improvement: The dataset is currently undergoing manual verification and will soon have a more accurate version with improved data quality and consistency.
3. Retrieval Testing Version: A version specifically designed for testing retrieval systems is currently under development and will be released soon.

• 🎯 Multi-Model Support: Supports multiple large language models with local vllm inference
• 📊 Standardized Evaluation: Provides unified evaluation metrics and processes
• 🔧 Flexible Configuration: Supports noise configuration, placeholder configuration, and other parameter adjustments
• 🌍 Multi-Language Support: Supports Chinese and English dataset evaluation
• 📈 Detailed Reports: Generates comprehensive evaluation results and score reports

In our experiments, we uniformly set the following configurations: noise_config: '{"noise_doc_level1":4,"noise_doc_level2":4,"noise_doc_level3":1}', num_iterations: 3, and shuffle: True.

The table below shows the performance of various state-of-the-art models on Chinese datasets, sorted by Overall score from high to low. Bold values indicate the best experimental results, and italic bold values indicate the second-best experimental results.

The table below shows the performance of various state-of-the-art models on English datasets, sorted by Overall score from high to low. Bold values indicate the best experimental results, and italic bold values indicate the second-best experimental results.

• Python 3.7+
• CUDA (if using GPU inference)

1. Clone the repository

git clone https://github.com/Alipay-Med/PRGB.git
cd PRGB

2. Install dependencies

pip install -r requirements.txt

3. Verify installation

python test_imports.py

Before running evaluations, it's recommended to verify that imports work correctly:

python test_imports.py

If you only need to modify the model path, using Makefile is recommended

# View all available commands
make help

# Set environment variables and run evaluation
export EVAL_MODEL_PATH=/path/to/your/model
make eval

# Or set environment variables in one line
EVAL_MODEL_PATH=/path/to/your/model make eval

# Chinese evaluation (using data/zh.jsonl)
# Chinese evaluation with inference mode (using data/zh.jsonl)
EVAL_MODEL_PATH=/path/to/your/model make eval-ch-infer

# English evaluation (using data/en.jsonl)
# English evaluation with inference mode (using data/en.jsonl)
EVAL_MODEL_PATH=/path/to/your/model make eval-en-infer

# Test evaluation (no real model needed)
make eval-test

# Export error samples (requires evaluation result file path)
EVAL_RESULT_FILE=results/model_eval_result.jsonl make export-errors

If you need to modify other parameters, using the shell is recommended.

# Run with default parameters (requires model path)
./run_eval.sh /path/to/your/model

# Pass all parameters
./run_eval.sh /path/to/your/model data/zh.jsonl Qwen3_infer ./results

# Basic usage
python eval.py \
    --model-name "Qwen3" \
    --model-path "/path/to/your/model" \
    --data-path "tests/test.jsonl" \
    --output-path "./results"

# Complete parameter example
python eval.py \
    --model-name "Qwen3" \
    --model-path "/path/to/your/model" \
    --data-path "your_data.jsonl" \
    --output-path "./results" \
    --batch-size 16 \
    --temperature 0.7 \
    --noise-config '{"noise_doc_level1":4,"noise_doc_level2":4,"noise_doc_level3":1}' \
    --custom_config "config/default_prompt_config.json" \
    --shuffle True \
    --num-iterations 3 \
    --verbose

• --model-path: Model path or API key
  • Local model: /path/to/your/model
  • API key: sk-xxxxxxxxxxxxxxxxxxxxxxxx

• --model-name: Model name (default: Qwen3)
  • Supported model types: Qwen3, Qwen2.5, Gemma3, Claude, GPT4, Deepseek, etc.
  • Used to specify the specific type and version of the model

• --data-path: Data file path (default: tests/test.jsonl)
  • Supports JSONL format data files
  • Chinese data: data/zh.jsonl
  • English data: data/en.jsonl
• --num-iterations: Number of evaluation iterations (default: 3)
  • For each query, randomly select n different placeholders to run evaluation
  • Each placeholder represents a different version of the same query with different variable substitutions
  • Used for multiple evaluations to get average scores
  • Recommended values: 1-5 times
• --shuffle: Whether to shuffle data (default: True)
  • Controls whether to randomly shuffle evaluation data order
  • Helps reduce order bias

• --output-path: Output directory (default: ./results)
  • Path to save evaluation results and score files
  • Will automatically create directory if it doesn't exist

• --batch-size: Batch size (default: 16)
  • Controls GPU memory usage and inference speed
  • Adjust based on GPU memory size: 8-32
• --temperature: Generation temperature (default: 0.7)
  • Controls randomness of generated text
  • Range: 0.0-1.0, 0.0 for deterministic generation
• --noise-config: Noise configuration JSON string (default: '{"noise_doc_level1":4,"noise_doc_level2":4,"noise_doc_level3":1}')
  • Controls the number of noise documents at different levels
  • noise_doc_level1: Number of level 1 noise documents
  • noise_doc_level2: Number of level 2 noise documents
  • noise_doc_level3: Number of level 3 noise documents
• --custom_config: Custom prompt configuration file path (default: None)
  • Used to specify custom prompt configuration files
  • Supports JSON format configuration files
  • Default uses language-related configuration files (Chinese/English)

• --inference-mode: Whether to use inference mode (default: False)
  • When set to True, enables inference mode for the model
  • Used with inference-optimized models like Qwen3_infer
  • Typically used with the eval-ch-infer and eval-en-infer make commands
• --verbose: Enable detailed logging (default: False)
  • Outputs detailed evaluation process and debug information
  • Helps with problem troubleshooting and performance analysis

The parameter order for run_eval.sh script:

1. MODEL_NAME (default: "Qwen3_infer")
2. MODEL_PATH (required)
3. DATA_PATH (default: "data/zh.jsonl")
4. OUTPUT_PATH (default: "./results")

# Run basic example
python examples/basic_evaluation.py

# Run custom example
python examples/basic_evaluation.py --mode custom

# Export error samples
python examples/export_errors.py

PRGB/
├── README.md                 # Project documentation
├── README_zh.md             # Chinese documentation
├── USAGE.md                 # Usage guide
├── pyproject.toml           # Project configuration
├── requirements.txt         # Python dependencies
├── eval.py                 # Main evaluation script
├── example_usage.py        # Example usage script
├── run_eval.sh             # Run script
├── run_http_eval.sh        # HTTP evaluation script
├── Makefile                # Build and development commands
├── CONTRIBUTING.md         # Contribution guidelines
├── CHANGELOG.md            # Change log
├── LEGAL.md                # Legal notice
├── .gitignore              # Git ignore file
├── .flake8                 # Code style configuration
├── .pre-commit-config.yaml # Pre-commit hook configuration
├── .gitattributes          # Git attributes
│
├── core/                   # Core functionality modules
│   ├── __init__.py        # Module initialization
│   ├── eval.py            # Main evaluation logic
│   ├── data.py            # Data processing
│   ├── eval_types.py      # Evaluation type definitions
│   ├── logger.py          # Logging functionality
│   └── models/            # Model implementations
│       ├── __init__.py    # Models module initialization
│       ├── api_models.py  # API-based model implementations
│       └── vllm_models.py # VLLM-based model implementations
│
├── config/                 # Configuration files
│   ├── api_prompt_config_ch.json  # Chinese API prompt configuration
│   ├── api_prompt_config_en.json  # English API prompt configuration
│   └── default_prompt_config.json # Default prompt configuration
│
├── utils/                  # Utility functions
│   ├── __init__.py        # Module initialization
│   └── transfer_csv_to_jsonl.py  # CSV to JSONL conversion tool
│
├── examples/               # Example scripts
│   ├── basic_evaluation.py # Basic usage example
│   └── export_errors.py    # Error sample export example
│
├── tests/                  # Test files
│   ├── test_imports.py     # Import test script
│   ├── test_data_process.py    # Data processing tests
│   ├── test_checkanswer.py     # Answer checking tests
│   ├── test_eval.py            # Evaluation tests
│   └── test_import_models.py   # Model import tests
│
├── data/                   # Data files
│   ├── zh.jsonl           # Chinese evaluation data
│   └── en.jsonl           # English evaluation data
│
├── results/                # Evaluation results (generated)
│
└── pic/                    # Images and figures
    ├── benchmark.jpg       # Benchmark figure
    └── examples.jpg        # Examples figure

The project uses the following tools to ensure code quality:

• black: Code formatting
• flake8: Code style checking
• pre-commit: Pre-commit hooks

# Run all tests
pytest tests/

# Using Makefile
make test

# Format code
make format

# Check code style
make lint

Please see the LICENSE file for detailed license information.

For questions or suggestions, please contact us through:

• Submit an Issue
• Send email to: [email protected] or [email protected]

Note: This project is for research and evaluation purposes only. Please ensure compliance with relevant laws and regulations and model usage terms.