ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model

Mingyuan Zhang¹  Xinying Guo¹  Liang Pan¹  Zhongang Cai^1,2  Fangzhou Hong¹  Huirong Li¹ 
Lei Yang²  Ziwei Liu¹⁺

¹S-Lab, Nanyang Technological University  ²SenseTime Research 

⁺corresponding author

---

[Project Page] • [arXiv] • [Video]

Abstract: 3D human motion generation is crucial for creative industry. Recent advances rely on generative models with domain knowledge for text-driven motion generation, leading to substantial progress in capturing common motions. However, the performance on more diverse motions remains unsatisfactory. In this work, we propose ReMoDiffuse, a diffusion-model-based motion generation framework that integrates a retrieval mechanism to refine the denoising process.

Pipeline Overview: ReMoDiffuse is a retrieval-augmented 3D human motion diffusion model. Benefiting from the extra knowledge from the retrieved samples, ReMoDiffuse is able to achieve high-fidelity on the given prompts. It contains three core components: a) Hybrid Retrieval database stores multi-modality features of each motion sequence. b) Semantics-modulated transformer incorporates several identical decoder layers, including a Semantics-Modulated Attention (SMA) layer and an FFN layer. The SMA layer will adaptively absorb knowledge from both retrived samples and the given prompts. c) Condition Mxture technique is proposed to better mix model's outputs under different combinations of conditions.

The code will be available soon.