learn.ifs.name

Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs

Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers))

Subscriber deck — free during early access

1 cards in “3.2 Clipping is rarely necessary” · whole deck

3.2 Clipping is rarely necessary

  1. Is PPO’s clipping needed in RLHF?
    Hardly: the loss was clipped in under 5% of cases per batch, showing training is nearly on-policy, and removing clipping (and the probability ratio) didn’t hurt and even helped slightly.
    • Ahmadian et al., 2024, Back to Basics (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §3.2, p. 12253

Source

  • Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün and Sara Hooker. 2024. Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12248–12267. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2024.acl-long.662 https://aclanthology.org/2024.acl-long.662/