Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs

1 Introduction

1. What makes PPO costly and hard to use for RLHF?
   It keeps up to four models in memory at once (the policy, a reference model for the KL penalty, a critic or value network, and the reward model), interleaves training of policy and critic, and its online optimization is unstable and sensitive to hyperparameters, needing RL expertise.
   • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 12248
2. How does “Back to Basics” differ from RL-free methods like DPO?
   DPO, IPO and iterative fine-tuning drop reinforcement learning altogether. Ahmadian et al. stay within RL but ask whether much simpler policy-gradient methods than PPO suffice for RLHF.
   • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, pp. 12248–12249
3. Why might PPO’s design be unnecessary for fine-tuning a pretrained LLM?
   PPO was built for deep-RL settings that start from random policies, where gradient estimates have high variance and large off-policy updates destabilize training. A pretrained, supervised-fine-tuned model concentrates probability on few tokens, so variance is much less of a problem.
   • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 12249
4. What were the main findings of “Back to Basics”?
   Vanilla policy gradient beat PPO by 3.2–20.3 win-rate points; REINFORCE Leave-One-Out (RLOO) beat PPO, DPO and RAFT across datasets and models; modelling partial completions turned out unnecessary; and RLOO was relatively robust to reward noise and the KL penalty.
   • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 12249