Code for the paper Balancing the Budget: Understanding Trade-offs Between Supervised and Preference-Based Finetuning

Link - https://arxiv.org/pdf/2502.11284

conda create --name <env> --file requirements.txt

Also add the IFEval repository in the root folder.

Process all the datasets

bash tuning/data_processing.sh

1. Edit the train_sizes list in tuning/run.sh to add different #train examples to train the models.
2. Run bash tuning/run.sh and select the task, sft-pft ratio and base model.

If you find our work or code useful, please cite the paper:

@misc{raghavendra2025balancingbudgetunderstandingtradeoffs,
      title={Balancing the Budget: Understanding Trade-offs Between Supervised and Preference-Based Finetuning}, 
      author={Mohit Raghavendra and Junmo Kang and Alan Ritter},
      year={2025},
      eprint={2502.11284},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2502.11284}, 
}