Tülu 3: Pushing Frontiers in Open Language Model Post-Training
Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124)
Subscriber deck — free during early access
1 cards in “8.2 Insights from the Unfruitful” · whole deck
8.2 Insights from the Unfruitful
- Which techniques did Tülu 3 try but leave out of the final recipe?Online DPO (sampling pairs from the current policy and labeling them with a reward model), which gave little or no gain on GSM8K and hurt MATH; and rejection sampling, whose gains were too small for its compute. They noted it needs a strong judge, and it worked better when the original response stayed among the candidates.
- Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124), §8.2
Source
- Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi and Hannaneh Hajishirzi. 2024. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. In arXiv preprint arXiv:2411.15124. https://arxiv.org/abs/2411.15124