This is the Official Repository of "Reinforcing Diffusion Models by Direct Group Preference Optimization", by Yihong Luo, Tianyang Hu, Jing Tang.

Our proposed DGPO shows a near 30 times faster training compared to Flow-GRPO on improving GenEval score (Left Figure). The notable improvement is achieved while maintaining strong performance on other out-of-domain metrics (Right Figure).

The key insight of our work is that the success of methods like GRPO stems from leveraging fine-grained, relative preference information within a group of samples, not from the policy-gradient formulation itself. Existing methods for diffusion models force the use of inefficient stochastic (SDE) samplers to fit the policy-gradient framework, leading to slow training and suboptimal sample quality.

DGPO circumvents this problem by optimizing group-level preferences directly, extending the Direct Preference Optimization (DPO) framework to handle pairwise groups instead of pairwise samples. This allows us to:

• Use Efficient Samplers: Employ fast and high-fidelity deterministic ODE samplers for generating training data, leading to better-quality rollouts.
• Learn Directly from Preferences: Optimize the model by maximizing the likelihood of group-wise preferences, eliminating the need for a stochastic policy and inefficient random exploration.
• Train Efficiently: Avoid training on the entire sampling trajectory, significantly reducing the computational cost of each iteration.

For a group of generated samples, we partition them into a positive group $\mathcal{G}^+$ and a negative group $\mathcal{G}^-$ based on their reward. The model with parameter $\theta$ is trained by maximum likelihood learning of a group-level reward:

$$ \max_{\theta} E_{(\mathcal{G}^+, \mathcal{G}^-, c) \sim \mathcal{D}} \log p_\theta(\mathcal{G}^+ \succ \mathcal{G}^-|c) = E_{(\mathcal{G}^+, \mathcal{G}^-, c) \sim \mathcal{D}} \log\sigma(R_\theta(\mathcal{G}^+|c) - R_\theta(\mathcal{G}^-|c)). $$

We refer to our paper for more details.

DGPO consistently outperforms Flow-GRPO on target metrics for Compositional Generation, Text Rendering, and Human Preference Alignment, while also showing strong or superior performance on out-of-domain quality and preference scores.

Please contact Yihong Luo ([email protected]) if you have any questions about this work.

@misc{luo2025dgpo,
      title={Reinforcing Diffusion Models by Direct Group Preference Optimization}, 
      author={Yihong Luo and Tianyang Hu and Jing Tang},
      year={2025},
      eprint={2510.08425},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2510.08425}, 
}