Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs
Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers))
Subscriber deck — free during early access
1 cards in “3.3 Modeling partial completions is not necessary” · whole deck
3.3 Modeling partial completions is not necessary
- Why can RLHF generation be treated as a bandit problem?The reward comes only at the end of a completion, and the “environment” is deterministic (each token just extends the context), so the whole generation can be one action taken from the prompt state; per-token states and values add complexity without real intermediate rewards.
- Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §3.3, p. 12253
Source
- Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün and Sara Hooker. 2024. Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12248–12267. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2024.acl-long.662 https://aclanthology.org/2024.acl-long.662/