• 2025-04-02 First Release Open-Omni-Nexus. a fully open-source implementation of a GPT-4o-like speech-to-speech video understanding model.
• 2025-04-01 Meet M4. M4 enables multiplexed modeling capabilities for a video language model at minimal cost.
• 2025-04-01 First Release OmniMMI. We're excited to share the first version of OmniMMI.

Table of Contents

• OmniMMI
  • Introduction
  • Statistics
  • Data
• Evaluation
  • Installation
  • Usage
• Leaderboard

we introduce OmniMMI, a comprehensive multi-modal interaction benchmark tailored for OmniLLMs in streaming video contexts. OmniMMI encompasses over 1,121 interactive videos and 2,290 questions, addressing two critical yet underexplored challenges in existing video benchmarks: streaming video understanding and proactive reasoning, across six distinct subtasks.

• Streaming Temporal State Awareness. Streaming video understanding must build an understanding w.r.t. the current and historical temporal state incrementally, without accessing the future context. This contrasts with traditional MLLM that can leverage the entire multi-modal contexts, posing challenges in our distinguished tasks of action prediction (AP), state grounding (SG) and multi-turn dependencies (MD).
• Proactive Reasoning and Turn-Taking. Generating responses proactively and appropriately anticipating the turn-taking time spot w.r.t. user's intentions and dynamic contexts is a crucial feature for general interactive agents. This typically requires models to identify speakers (SI), distinguish between noise or legitimate query (PT), and proactively initiate a response (PA).

You can download the OmniMMI from huggingface, containing both json and raw videos.

videohallucer_datasets                
    ├── clips/ # video clips for assessing offline models
    ├── videos/
    ├── action_prediction.json
    ├── dynamic_state_grounding.json
    ├── multiturn_dependency_reasoning.json
    ├── proactive_alerting.json
    ├── proactive_turntaking.json
    └── speaker_identification.json
  

We offer a selection of data samples (multiturn dependency reasoning) from our dataset for further elucidation:

[
    {
        "video": "jGM8EImhpXg.mp4",
        "qa": [
            {
                "question": "What is the task of the instruction video?",
                "answer": "Cook Omelet",
                "timestamp": "0.0 -- 36.0"
            },
            {
                "question": "What's the first step of ##ANSWER##?",
                "answer": "Pouring the egg into the pot",
                "timestamp": "36.0 -- 40.0"
            },
            {
                "question": "What do you do after the step from ##ANSWER##?",
                "answer": "fry eggs",
                "timestamp": "41.0 -- 70.0"
            }
        ]
    },
...
]

If you want to upload results from your models, feel free to submit a PR following one of these baselines, or send an email to me ([email protected]) to update your results on our page.

Available Baselines

Large Video Language Model

• VideoChatGPT-7B
• VideoChat2-7B
• Video-LLaVA-7B
• LLaMA-VID-7B/13B
• MiniGPT4-Video-7B
• PLLaVA-7B/13B/34B
• LLaVA-NeXT-Video-DPO-DPO-7B/34B
• ShareGPT4Video-8B
• LongVA-7B
• LongVILA-8B
• LongLLaVA-9B
• VideoLLM-online-8B
• VideoLLaMB-7B
• InternLM-XComposer-2.5-OmniLive-1.5B
• Gemini-1.5-pro
• GPT4O (Azure)

Large Omni Language Model

• VideoLLaMA2-7B
• VITA-7B
• MiniOmni2-0.5B
• M4-7B

For detailed instructions on installation and checkpoints of these baselines, please consult the INSTALLATION guide.

Note: To evaluate audio-input models, we use a TTS model to automatically convert text queries into audio. Please install ChatTTS beforehand.

debug inference pipeline

cd baselines
python ../model_testing_zoo.py --model_name Gemini-1.5-pro # ["VideoChatGPT", "VideoChat2", "VideoLLaVA", "LLaMA-VID", "PLLaVA", "PLLaVA-13B", "PLLaVA-34B", "LLaVA-NeXT-Video", "LLaVA-NeXT-Video-34B", "LongVA", "LongVILA", "LongLLaVA", "VideoLLaMB", "VideoOnline", "VideoLLaMBOneline", "Gemini-1.5-pro", "GPT4O"])

cd baselines
bash run_all.sh
bash eval_all.sh

for each evaluation script for specific subtask, e.g. run_ap.sh

2.1 set the model and conda environment

model_names=("LongVA" "PLLaVA-13B")
environments=("llongva" "pllava")

There are two groups of models

(i) common models

"MiniGPT4-Video" "VideoChatGPT" "VideoLLaVA" "VideoChat2" "LLaMA-VID" "PLLaVA" "LLaVA-NeXT-Video" "ShareGPT4Video" "PLLaVA-13B" "PLLaVA-34B" "LLaVA-NeXT-Video-34B" "LongVA" "LongVILA" "LongLLaVA" "VideoLLaMB" "InternLMXCO" "InterSuit" "InterSuitAV"

(ii) online models

"VideoOnline" "VideoLLaMBOnline" "InterSuitOnline" "InterSuitOnlineAV"

2.2 set the conda environment path

source ~/scratch/anaconda3/bin/activate

2.3 run the evaluation script for each subtask

cd baselines
bash run_ap.sh # action prediction
bash eval_ap.sh

Please check the detailed leaderboard in our page

If you find our work helpful, please consider citing it.

@article{omnimmi,
    title={OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts},
    author={Wang, Yuxuan and Wang, Yueqian and Chen, Bo and Wu, Tong and Zhao, Dongyan and Zheng, Zilong},
    journal={arxiv},
    year={2025}
}