📄 Paper  |  🌐 Project Page  |  🗂 Dataset  |  🤗 Models  |  🚀 Live Demo

🧢 CapSpeech comprises over 10 million machine-annotated audio-caption pairs and nearly 0.36 million human-annotated audio-caption pairs. CapSpeech provides a new benchmark including these tasks:

1. CapTTS: style-captioned TTS

2. CapTTS-SE: text-to-speech synthesis with sound effects

3. AccCapTTS: accent-captioned TTS

4. EmoCapTTS: emotion-captioned TTS

5. AgentTTS: text-to-speech synthesis for chat agent

Explore CapSpeech directly in your browser — no installation needed.

• 🚀 Live Demo: 🤗 Spaces

Install and Run CapSpeech locally.

• 💿 Installation & Usage: 📄 Instrucitons

Please refer to the following documents to prepare the data, train the model, and evaluate its performance.

• Data Preparation
• Training
• Evaluation

• Helin Wang at Johns Hopkins University
• Jiarui Hai at Johns Hopkins University

If you find this work useful, please consider contributing to this repo and cite this work:

@misc{wang2025capspeechenablingdownstreamapplications,
      title={CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech}, 
      author={Helin Wang and Jiarui Hai and Dading Chong and Karan Thakkar and Tiantian Feng and Dongchao Yang and Junhyeok Lee and Laureano Moro Velazquez and Jesus Villalba and Zengyi Qin and Shrikanth Narayanan and Mounya Elhiali and Najim Dehak},
      year={2025},
      eprint={2506.02863},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2506.02863}, 
}

All datasets, listening samples, source code, pretrained checkpoints, and the evaluation toolkit are licensed under the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).
See the LICENSE file for details.

This implementation is based on Parler-TTS, F5-TTS, SSR-Speech, Data-Speech, EzAudio, and Vox-Profile. We appreciate their awesome work.

If you find this repo helpful or interesting, consider dropping a ⭐ — it really helps and means a lot!