LLM-as-a-Judge for Extractive QA Datasets

This is the repository for the paper: LLM-as-a-Judge: Reassessing the Performance of LLMs in Extractive QA

Reproduction of Results

• download data.zip
• download judge.zip

Table 2: Pearson correlation coefficients

python3 get_correlation_score.py

Table 3: Automatic evaluation scores (EM and F1) and LLM-as-a-judge scores

python3 run_eval.py

Running process

Step 1: Run the QA Task

python3 run_qa.py

Postprocessing the generated response from the QA task

python3 postprocess_qa.py

Step 2: Run LLM-as-a-judge

python3 run_judge.py

Step 3: Evaluation

python3 run_eval.py

Calculate correlation scores:

python3 get_correlation_score.py

Data files include:

• data: input data
• data/human_result.json: human judgement dataset
• qa_inference: predicted answers from 8 QA models
• qa_postprocess: postprocess on the predicted answers
• judge/mistral-v0.3: judged by mistral-v0.3
• judge/llama-3.3-70b: judged by llama-3.3-70b
• judge/qwen-2.5-72b: judged by qwen-2.5-72b