Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs
Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers))
Subscriber deck — free during early access
1 cards in “3.1 Revisiting the need for low-variance estimators” · whole deck
3.1 Revisiting the need for low-variance estimators
- What did varying GAE’s \(\lambda\) in PPO show?Lower \(\lambda\) reduces variance but adds bias via the learned value function; reward fell steadily as \(\lambda\) decreased, and \(\lambda = 1\) (unbiased, highest variance, equivalent to vanilla policy gradient) did best. With a strong initial policy, trading bias for lower variance doesn’t pay.
- Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §3.1, p. 12252
Source
- Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün and Sara Hooker. 2024. Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12248–12267. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2024.acl-long.662 https://aclanthology.org/2024.acl-long.662/