Xiaohu Huang^*, Hao Zhou^*, Qiangpeng Yang, Shilei Wen, Kai Han^†
The University of Hong Kong · ByteDance

^*Equal contribution  ·  ^†Corresponding author

JoVA is a unified framework for joint video-audio generation, with a particular focus on human speech synchronized with lip movements.
Unlike prior unified video-audio methods that rely on explicit fusion or modality-specific alignment modules (e.g., additional cross-attention stacks), JoVA enables direct cross-modal interaction by applying joint self-attention over concatenated video, audio, and text tokens within each transformer layer.

To further improve fine-grained lip–speech synchronization, JoVA introduces a simple yet effective mouth-aware supervision strategy based on facial keypoint detection, increasing training emphasis on the mouth region in the VAE latent space—without complicating the architecture.

Please refer to the project page for qualitative results, demos, and additional visualizations:

• Project page: https://visual-ai.github.io/jova

If you find this work useful, please cite:

@article{huang2025jova,
  title   = {JoVA: Unified Multimodal Learning for Joint Video-Audio Generation},
  author  = {Huang, Xiaohu and Zhou, Hao and Yang, Qiangpeng and Wen, Shilei and Han, Kai},
  journal = {arXiv preprint},
  year    = {2025},
}