Training language models to follow instructions with human feedback

3.4 Models

1. How is the InstructGPT reward model trained?
   A 6B GPT-3 model maps a prompt and response to a scalar. Labelers rank \(K = 4\) to 9 responses, giving \(\binom{K}{2}\) pairs, all used as one batch element: \(\text{loss}(\theta) = -\frac{1}{\binom{K}{2}} \mathbb{E}_{(x, y_w, y_l)} [\log \sigma(r_\theta(x, y_w) - r_\theta(x, y_l))]\), where \(y_w\) is preferred over \(y_l\).
   • Ouyang et al., 2022, Training language models to follow instructions with human feedback (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §3.4
2. How many pairwise comparisons does one ranking of \(K\) responses give, and why train them together?
   \(\binom{K}{2}\): 6 for \(K = 4\), 36 for \(K = 9\). Training on all of them as one batch element per prompt is computationally efficient: each of the \(K\) completions takes part in \(K - 1\) comparisons but needs only one forward pass.
   • Ouyang et al., 2022, Training language models to follow instructions with human feedback (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §3.4
3. How is the InstructGPT policy optimized with PPO?
   As a bandit: present a random prompt, take the response, get the reward model’s score, end the episode. A per-token KL penalty from the SFT model limits over-optimization of the reward model; the value function starts from the RM.
   • Ouyang et al., 2022, Training language models to follow instructions with human feedback (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §3.4
4. What is PPO-ptx, and why was it introduced?
   PPO with pretraining gradients mixed in (increasing the log-likelihood of the pretraining distribution), which mostly removed RLHF’s performance regressions on public NLP data sets without hurting labeler preferences; it worked better than raising the KL coefficient. “InstructGPT” usually means PPO-ptx.
   • Ouyang et al., 2022, Training language models to follow instructions with human feedback (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §3.4