Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs

2.3 REINFORCE Leave-One-Out (RLOO)

1. What is the RLOO estimator?
   Sample \(k\) completions per prompt; for each, use the mean reward of the other \(k-1\) as its baseline: \(\frac{1}{k}\sum_{i}\big[R(y_i, x) - \frac{1}{k-1}\sum_{j \ne i} R(y_j, x)\big]\nabla \log \pi(y_i \mid x)\). This is an unbiased, on-the-fly, parameter-free substitute for a learned value function.

   • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §2.3, p. 12251
2. How does RLOO differ from RAFT (reward-ranked fine-tuning)?
   RAFT samples \(k\) completions and fine-tunes only on the top-ranked one, discarding the rest; RLOO uses all \(k\) samples, both as baselines for each other and in a multi-sample gradient estimate.
   • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §2.3, p. 12251