Tülu 3: Pushing Frontiers in Open Language Model Post-Training
Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124)
Subscriber deck — free during early access
1 cards in “6.2.1 Key Findings” · whole deck
6.2.1 Key Findings
- What did Tülu 3 find when training with RLVR?It improved the targeted evaluations (GSM8K, MATH, IFEval); adding reward-model scores to the verifiable reward hurt; initializing PPO’s value function from a general reward model worked best; starting from a weaker SFT model reached the same training reward but at higher KL; and too little KL penalty led to over-optimization.
- Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124), §6.2.1
Source
- Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi and Hannaneh Hajishirzi. 2024. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. In arXiv preprint arXiv:2411.15124. https://arxiv.org/abs/2411.15124