Roadmap

Principle

The development principle of OpenRLHF is to optimize performance as much as possible while maintaining ease of use and ease of understanding (CleanRLHF).

Easy of Use

• Remove / decouple the ppo_trainer.py without ray
• Single Controller @xiaoxigua999
• Refactor packing samples and ring attention
• Upgrade PyTorch container to 25.0x

Performance Optimization

• AutoEP
• Partial rollout
• FP8 rollouts
• Migrate to FSDP2 + DTensor TP Plan(Add FSDP backend and --dist_backend flag across CLIs; introduce FSDPStrategy #1115)
• New Agentic RL Arch & Streaming sampling(feat: restructure and clean up streaming async sampling (#1130) #1152)
• Support FlashAttention 3/4
• Larger generation batch size (vLLM)
• HybridEngine (Inference Engine Offload) @xiaoxigua999 c10f02b
  • CUDA IPC weights sync @xiaoxigua999
  • Deepspeed engine offloading
• vLLM weights sync acceleration (NCCL)
• Sequence parallelism (RingAttention)
  • DPO
  • RM
  • SFT @UbeCc Support ring-attention during sft phase #576
  • PPO @Wraythh Wraythh/ppo ring #685
• Packing samples
  • SFT
  • DPO
  • PPO
  • RM
• DeepSpeed config optimization
  • Verify the RLHF convergence of {"overlap_comm": true} @xiaoxigua999 5fd5101
• Liger-Kernel Add support for Liger Kernels for better training throughput #695
• DeepSpeed AutoTP training in https://github.com/deepspeedai/DeepSpeed/blob/master/blogs/huggingface-tp/README.md
• Dynamic batch size

New Algorithms

• Multi-modal RL
• Multi-turn SFT dataset @UbeCc Add support for multiturn sft training with packing_samples #586
• PRM
• REINFORCE++
  • REINFORCE++-baseline
• RLOO
• GRPO
  • Dr. GRPO
• LoRA
• LoRA weights merge script @UbeCc
• Reinforcement Finetuning (RFT) custom reward func for reinforced finetuning #707
  • Rule-based evaluation datasets
• Entropy loss
• Better logs:
  • entropy log
  • clip ratio
  • reward std
  • grad norm
  • ppo_kl
• Dynamic Sampling (Data filtering)
• Clip-Higher
• Length Penalty
• Multi-turn / Agent RLHF
• Async / Pipe RLHF
• Token-in-token-out agentic RL
• Agent loop
• Truncated Importance Sampling & Icepop
• Muon Optimizer
• On-policy distillation