learn.ifs.name

Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs

Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers))

Subscriber deck — free during early access

12 cards

1 Introduction

  1. What makes PPO costly and hard to use for RLHF?
    It keeps up to four models in memory at once (the policy, a reference model for the KL penalty, a critic or value network, and the reward model), interleaves training of policy and critic, and its online optimization is unstable and sensitive to hyperparameters, needing RL expertise.
    • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 12248
  2. How does “Back to Basics” differ from RL-free methods like DPO?
    DPO, IPO and iterative fine-tuning drop reinforcement learning altogether. Ahmadian et al. stay within RL but ask whether much simpler policy-gradient methods than PPO suffice for RLHF.
    • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, pp. 12248–12249
  3. Why might PPO’s design be unnecessary for fine-tuning a pretrained LLM?
    PPO was built for deep-RL settings that start from random policies, where gradient estimates have high variance and large off-policy updates destabilize training. A pretrained, supervised-fine-tuned model concentrates probability on few tokens, so variance is much less of a problem.
    • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 12249
  4. What were the main findings of “Back to Basics”?
    Vanilla policy gradient beat PPO by 3.2–20.3 win-rate points; REINFORCE Leave-One-Out (RLOO) beat PPO, DPO and RAFT across datasets and models; modelling partial completions turned out unnecessary; and RLOO was relatively robust to reward noise and the KL penalty.
    • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 12249

2.3 REINFORCE Leave-One-Out (RLOO)

  1. What is the RLOO estimator?
    Sample \(k\) completions per prompt; for each, use the mean reward of the other \(k-1\) as its baseline: \(\frac{1}{k}\sum_{i}\big[R(y_i, x) - \frac{1}{k-1}\sum_{j \ne i} R(y_j, x)\big]\nabla \log \pi(y_i \mid x)\). This is an unbiased, on-the-fly, parameter-free substitute for a learned value function.
    samplerewardbaselineadvantagey₁0.90.37+0.53y₂0.30.57−0.27y₃0.60.47+0.13y₄0.20.60−0.40baseline = mean reward of the other three samples
    Every sample is pushed up or down relative to its siblings; RAFT would train on y₁ only (invented rewards, k = 4)
    • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §2.3, p. 12251
  2. How does RLOO differ from RAFT (reward-ranked fine-tuning)?
    RAFT samples \(k\) completions and fine-tunes only on the top-ranked one, discarding the rest; RLOO uses all \(k\) samples, both as baselines for each other and in a multi-sample gradient estimate.
    • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §2.3, p. 12251

3.1 Revisiting the need for low-variance estimators

  1. What did varying GAE’s \(\lambda\) in PPO show?
    Lower \(\lambda\) reduces variance but adds bias via the learned value function; reward fell steadily as \(\lambda\) decreased, and \(\lambda = 1\) (unbiased, highest variance, equivalent to vanilla policy gradient) did best. With a strong initial policy, trading bias for lower variance doesn’t pay.
    • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §3.1, p. 12252

3.2 Clipping is rarely necessary

  1. Is PPO’s clipping needed in RLHF?
    Hardly: the loss was clipped in under 5% of cases per batch, showing training is nearly on-policy, and removing clipping (and the probability ratio) didn’t hurt and even helped slightly.
    • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §3.2, p. 12253

3.3 Modeling partial completions is not necessary

  1. Why can RLHF generation be treated as a bandit problem?
    The reward comes only at the end of a completion, and the “environment” is deterministic (each token just extends the context), so the whole generation can be one action taken from the prompt state; per-token states and values add complexity without real intermediate rewards.
    • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §3.3, p. 12253

5 Results and Discussion

  1. How did modelling the whole generation compare with per-token modelling?
    REINFORCE with a moving-average baseline, RLOO and RAFT (whole-sequence methods) achieved higher rewards than PPO and vanilla policy gradient (per-token), while needing no value network or learned baseline.
    • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §5, pp. 12254–12255
  2. How did RLOO compare with RAFT given the same samples?
    RLOO matched or beat RAFT even with half the samples (\(k = 2\) vs \(k = 4\)); averaged over settings its win rates were 61.3 and 61.9 for \(k = 2\) and \(k = 4\), vs 56.1 and 59.5 for RAFT, because it learns from every sample.
    • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §5, p. 12255
  3. Why is RAFT more fragile than RLOO?
    Learning only from the top-ranked sample makes it sensitive to anything that corrupts the ranking: with a stronger KL penalty RAFT optimized reward worse and drifted further from the reference policy, and it suffered more from noise added to rewards.
    • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §5, p. 12256

Source

  • Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün and Sara Hooker. 2024. Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12248–12267. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2024.acl-long.662 https://aclanthology.org/2024.acl-long.662/