This repo contains the code and data of: VoiceBench: Benchmarking LLM-Based Voice Assistants

• 2025.04.20 Released wildvoice, a crowd-sourced dataset comprising human-recorded speech with diverse accents.
• 2025.04.12 Released bbh, a crowd-sourced dataset comprising human-recorded speech, for evaluating the reasoning ability of voice assistants.
• 2024.12.11 Updated the VoiceBench Leaderboard to include mmsu.
• 2024.12.10 Added a curated list of awesome voice assistants.
• 2024.11.24 Expanded the test samples in VoiceBench to include mmsu, covering 12 diverse domains from mmlu-pro.
• 2024.11.12 Updated the VoiceBench Leaderboard to include: 1) Mini-Omni2, GPT-4o-Audio, and Whisper-v3+GPT-4o, and 2) multiple-choice QA from OpenBookQA.
• 2024.10.30 Expanded the test samples in VoiceBench to include: 1) the complete set of open-ended QA from alpacaeval, and 2) multiple-choice QA from openbookqa.

• Leaderboard
• Setup
• Dataset
• Evaluation
• Awesome Voice Assistants
• Citation

We encourage you to submit new voice assistant results directly through the issue tracker. The ranking list will be updated accordingly.

conda create -n voicebench python=3.10
conda activate voicebench
pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu121
pip install xformers==0.0.23 --no-deps
pip install -r requirements.txt

The data used in this project is available at VoiceBench Dataset hosted on Hugging Face.

You can access it directly via the link and integrate it into your project by using the Hugging Face datasets library.

To load the dataset in your Python environment:

from datasets import load_dataset

# Load the VoiceBench dataset
# Available subset: alpacaeval, commoneval, sd-qa, ifeval, advbench, ...
dataset = load_dataset("hlt-lab/voicebench", 'alpacaeval')

PS: alpacaeval contains helpful_base and vicuna data, while alpacaeval_full is constructed with the complete data. alpacaeval_full is used in the leaderboard.

To obtain the responses from the voice assistant model, run the following command:

python main.py --model naive --data alpacaeval --split test --modality audio

Supported Arguments:

• --model: Specifies the model to use for generating responses. Replace naive with the model you want to test (e.g., qwen2, diva).
• --data: Selects the subset of the dataset. Replace alpacaeval with other subsets like commoneval, sd-qa, etc., depending on your evaluation needs.
• --split: Chooses the data split to evaluate.
  • For most datasets (alpacaeval, commoneval, ifeval, advbench), use test as the value.
  • For the sd-qa subset, you should provide a region code instead of test, such as aus for Australia, usa for the United States, etc.
• --modality: Use audio for spoken instructions, text for text-based instructions.

This will generate the output and save it to a file named naive-alpacaeval-test-audio.jsonl.

For datasets alpacaeval, commoneval, wildvoice, and sd-qa, we use gpt-4o-mini to evaluate the responses. Run the following command to get the GPT score:

python api_judge.py --src_file naive-alpacaeval-test-audio.jsonl

The GPT evaluation scores will be saved to result-naive-alpacaeval-test-audio.jsonl.

Note: This step should be skipped for other datasets, as they are not evaluated using GPT-4.

To generate the final evaluation results, run:

python evaluate.py --src_file result-naive-alpacaeval-test-audio.jsonl --evaluator open

Supported Arguments:

• --evaluator: Specifies the evaluator type:
  • Use open for alpacaeval, commoneval, and wildvoice.
  • Use qa for sd-qa.
  • Use ifeval for ifeval.
  • Use harm for advbench.
  • Use mcq for openbookqa and mmsu.
  • Use bbh for bbh.

Title	Date	Code
OSUM-EChat: Enhancing End-to-End Empathetic Spoken Chatbot via Understanding-Driven Spoken Dialogue	2025-08-13	Github
DIFFA: Large Language Diffusion Models Can Listen and Understand	2025-07-24	Github
Voxtral	2025-07-17	--
Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models	2025-07-10	Github
DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment	2025-07-03	Github
Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model	2025-06-16	Github
Ming-Omni: A Unified Multimodal Model for Perception and Generation	2025-06-11	Github
Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model	2025-06-10	--
VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model	2025-05-06	Github
LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis	2025-05-05	Github
Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Play	2025-05-05	Github
Kimi-Audio Technical Report	2025-04-25	Github
Qwen2.5-Omni Technical Report	2025-03-26	Github
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs	2025-03-03	HF
Nexus-O: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision	2025-02-26	--
M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance	2025-02-26	--
Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction	2025-02-24	Github
LLM-Enhanced Dialogue Management for Full-Duplex Spoken Dialogue Systems	2025-02-19	--
FlexDuo: A Pluggable System for Enabling Full-Duplex Capabilities in Speech Dialogue Systems	2025-02-19	--
Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction	2025-02-17	Github
DuplexMamba: Enhancing Real-time Speech Conversations with Duplex and Streaming Capabilities	2025-02-16	Github
Ola: Pushing the Frontiers of Omni-Modal Language Model with Progressive Modality Alignment	2025-02-06	Github
SpeechGPT 2.0-preview	2025-01-26	Github
Baichuan-Omni-1.5 Technical Report	2025-01-26	Github
MiniCPM-o 2.6: A GPT-4o Level MLLM for Vision, Speech, and Multimodal Live Streaming on Your Phone	2025-01-24	Github
MinMo: A Multimodal Large Language Model for Seamless Voice Interaction	2025-01-10	--
OpenOmni: Large Language Models Pivot Zero-shot Omnimodal Alignment across Language with Real-time Self-Aware Emotional Speech Synthesis	2025-01-08	Github
VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction	2025-01-03	Github
OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarios	2025-01-02	--
SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training	2024-12-20	Github
MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Models	2024-12-13	HF
Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition	2024-12-12	Github
Continuous Speech Tokens Makes LLMs Robust Multi-Modality Learners	2024-12-06	--
GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot	2024-12-03	Github
Advancing Speech Language Models by Scaling Supervised Fine-Tuning with Over 60,000 Hours of Synthetic Speech Dialogue Data	2024-12-02	--
SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation	2024-11-27	--
Ultravox: An Open-Weight Alternative to GPT-4o Realtime	2024-11-12	Github
Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM	2024-11-01	Github
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation	2024-10-23	--
Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistant	2024-10-20	Github
Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities	2024-10-15	Github
Baichuan-Omni Technical Report	2024-10-11	--
IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities	2024-10-09	--
Distilling an End-to-End Voice Assistant Without Instruction Training Data	2024-10-03	HF
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions	2024-09-26	--
Moshi: a Speech-Text Foundation Model for Real-Time Dialogue	2024-09-17	Github
LLaMA-Omni: Seamless Speech Interaction with Large Language Models	2024-09-10	Github
Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming	2024-08-29	Github
VITA: Towards Open-Source Interactive Omni Multimodal LLM	2024-08-09	Github
Qwen2-Audio Technical Report	2024-07-15	Github
PSLM: Parallel Generation of Text and Speech with LLMs for Low-Latency Spoken Dialogue Systems	2024-06-18	--
LLaSM: Large Language and Speech Model	2023-08-30	Github

If you use the VoiceBench in your research, please cite the following paper:

@article{chen2024voicebench,
  title={VoiceBench: Benchmarking LLM-Based Voice Assistants},
  author={Chen, Yiming and Yue, Xianghu and Zhang, Chen and Gao, Xiaoxue and Tan, Robby T. and Li, Haizhou},
  journal={arXiv preprint arXiv:2410.17196},
  year={2024}
}