🤗 HF Models and Datasets Collection | 📑 Arxiv Preprint

For any questions, feel free to open an issue or directly contact to Qingyang Zhang, happy to help and discuss!

If you find this repo helpful, please consider to star🌟 this repo for support our work.

• [2025-09-20] EMPO has been accepted by NeurIPS as a Spotlight! See you in San Diego!
• [2025-04-30] We release the training and evaluation code for both mathematical reasoning and free-form natural reasoning tasks.
• [2025-04-08] We introduce EMPO, which makes the first attempt on fully unsupervised LLM reasoning incentivization. Check out our arxiv preprint (first released at 2025.04.08): https://arxiv.org/abs/2504.05812

• EMPO: Fully Unsupervised LLM Reasoning Incentivization
  • Table of Contents
  • Overview
  • Repository Structure
  • TRL Quick Start
  • Verl Quick Start
  • Acknowledgement
  • Related Works
  • Citation

EMPO (Entropy Minimized Policy Optimization) does not require any supervised information for incentivizing reasoning capabilities (i.e., neither verifiable reasoning traces, problems with golden answers, nor additional pre-trained reward models). By continuously minimizing the predictive entropy of LLMs on unlabeled user queries, EMPO enables self-supervised RL for reasoning capabilities.

This repository contains two self-contained implementations of EMPO:

• trl: Based on Hugging Face’s trl, a cutting-edge library designed for post-training foundation models.

   ↳ Built on commit v0.14-release

• verl-empo: Based on VERL, a high-performance RL training library designed for LLMs.

   ↳ Built on commit v0.4x

Both are licensed under Apache 2.0 and include their respective LICENSE and NOTICE files.

Developed upon trl 0.14.0. See trl for details.

cd trl
pip install -r requirements.txt
sh empo-1.5B-NM-COT-20K.sh

As trl 0.14.0 is already a relatively outdated training framework. We highly recommend verl for further development for efficiency and compatibility.

Developed upon verl==0.4.x. See verl for details.

The recommended docker image with pre-built dependency can be founded in here.

Place the train and val data from 'math_data' in your local path.

cd verl-empo
sh recipe/empo/scripts/run_empo_qwen2.5_math_7b.sh

Remember to modify the code here if you want to track the training dynamics with wandb.

Load verl checkpoints by modifying trainer.resume_from_path.

Calculate pass@1 accuracy with greedy decoding by setting actor_rollout_ref.rollout.val_kwargs.do_sample=False.

We report pass@1 with greedy decoding for all datasets.

Noted that due to different 1) evaluation proxy 2) RL framework and 3) GPU hardware, the above results are different from those reported in our early preprint.

This repo is built upon Semantic Entropy, Open-R1, Online-DPO-R1, and TTRL. We thank all these researchers for generously sharing their insights, model weights, data, and codes.

There are many awesome works related to this paper that you may also interested with:

• LLM Uncertainty Quantification: Semantic Entropy (ICLR'23, Nature'24)
• Test-time Adaption in Computer Vision: COME (ICLR'25)
• Pseudo Feedback for LLM Reasoning: PFPO (ICLR'25 spotlight)
• Test-time Reinforcement Learning: TTRL (cocurrent work)

More papers are listed in Awesome Reinforcement Learning with Internal Reward Paper list.

If you find this work helpful, please consider to star🌟 this repo. Thanks for your support!

@article{zhang2025right,
  title={Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization},
  author={Zhang, Qingyang and Wu, Haitao and Zhang, Changqing and Zhao, Peilin and Bian, Yatao},
  journal={Advances in neural information processing systems},
  year={2025}
}