Keda Tao, Kele Shao, Bohan Yu, Weiqiang Wang, Jian liu, Huan Wang, "OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models"

[Paper]

• 2025-11-19: The paper is released.
• 2025-11-18: This repo is released.

Abstract: Omnimodal large language models (OmniLLMs) have attracted increasing research attention of late towards unified audio-video understanding, wherein processing audio-video token sequences creates a significant computational bottleneck, however. Existing token compression methods have yet to accommodate this emerging need of jointly compressing multimodal tokens. To bridge this gap, we present OmniZip, a training-free, audio-guided audio-visual token-compression framework that optimizes multimodal token representation and accelerates inference. Specifically, OmniZip first identifies salient audio tokens, then computes an audio retention score for each time group to capture information density, thereby dynamically guiding video token pruning and preserving cues from audio anchors enhanced by cross-modal similarity. For each time window, OmniZip compresses the video tokens using an interleaved spatio-temporal scheme. Extensive empirical results demonstrate the merits of OmniZip - it achieves 3.42 $\times$ inference speedup and 1.4 $\times$ memory reduction over other top-performing counterparts, while maintaining performance with no training.

• Release code
• Release paper
• Release evaluation script for all benchmarks
• Support more models

git clone https://github.com/KD-TAO/OmniZip.git
cd OmniZip

conda create -n omnizip python=3.10 -y
conda activate omnizip
pip install --upgrade pip
bash setup.sh

cd lmms-eval
pip install -e .

# Recommend
# pip install torch==2.6.0 torchvision==0.21.0
pip install flash-attn --no-build-isolation

You can adjust the number of input frames and the maximum pixel size according to your own computing resources.

VIDEO_MIN_PIXELS = 128 * 28 * 28
VIDEO_MAX_PIXELS = 768 * 28 * 28 # <- 1 (We use 128*28*28 in the paper.)
FRAME_FACTOR = 2
FPS = 2.0
FPS_MIN_FRAMES = 4
FPS_MAX_FRAMES = 768 # <- 2

To have a quick demo of OmniZip with Qwen2.5-Omni, please run

python demo.py --omnizip

You can set the relevant parameters by

python demo.py --omnizip --rho_audio 0.4 --rho_video 0.7

• We use the lmms-eval toolkit to evaluate our models. It's worth noting that you can specify OmniZip Settings via parameters in eval.sh, such as:

export WRAPPER=OmniZip
OMNIZIP_RHO_AUDIO=0.3
OMNIZIP_RHO_VIDEO=0.6
OMNIZIP_G=3
OMNIZIP_CONTEXTUAL_RATIO=0.05
...

• Then you can run for evaluation

bash eval.sh

• We will sort out the code as soon as possible.

This project is based on Qwen2.5-Omni. Thanks for their awesome work.

If you have any questions, please feel free to contact me at [email protected]

If you find this work useful for your research, please consider citing our paper:

@article{omnizip,
  title={OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models}, 
  author={Keda Tao and Kele Shao and Bohan Yu and Weiqiang Wang and Jian liu and Huan Wang},
  journal={arXiv preprint arXiv:2511.14582},
  year={2025}
}