Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs

5 Results and Discussion

1. How did modelling the whole generation compare with per-token modelling?
   REINFORCE with a moving-average baseline, RLOO and RAFT (whole-sequence methods) achieved higher rewards than PPO and vanilla policy gradient (per-token), while needing no value network or learned baseline.
   • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §5, pp. 12254–12255
2. How did RLOO compare with RAFT given the same samples?
   RLOO matched or beat RAFT even with half the samples (\(k = 2\) vs \(k = 4\)); averaged over settings its win rates were 61.3 and 61.9 for \(k = 2\) and \(k = 4\), vs 56.1 and 59.5 for RAFT, because it learns from every sample.
   • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §5, p. 12255
3. Why is RAFT more fragile than RLOO?
   Learning only from the top-ranked sample makes it sensitive to anything that corrupts the ranking: with a stronger KL penalty RAFT optimized reward worse and drifted further from the reference policy, and it suffered more from noise added to rewards.
   • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §5, p. 12256