Tülu 3: Pushing Frontiers in Open Language Model Post-Training
Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124)
Subscriber deck — free during early access
1 cards in “6 Reinforcement Learning with Verifiable Rewards” · whole deck
6 Reinforcement Learning with Verifiable Rewards
- What is RLVR?RL in which the reward comes from a deterministic check instead of a learned reward model: the policy earns \(\alpha\) (10) only if its answer is verifiably correct, otherwise 0, minus the usual KL penalty to the reference policy, optimized with PPO. Tülu 3 used GSM8K and MATH answers and IFEval-style constraint checkers.
RLVR: reward only for verifiably correct answers - Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124), §6
Source
- Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi and Hannaneh Hajishirzi. 2024. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. In arXiv preprint arXiv:2411.15124. https://arxiv.org/abs/2411.15124