learn.ifs.name

Tülu 3: Pushing Frontiers in Open Language Model Post-Training

Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124)

Subscriber deck — free during early access

1 cards in “5.1.2 Policy Optimization” · whole deck

5.1.2 Policy Optimization

  1. Which DPO variant did Tülu 3 adopt, and why?
    Length-normalized DPO: the usual DPO loss with each response’s log-probability ratio divided by its length, which counters the length bias in human and model preferences. It beat SimPO and standard DPO in their tests and was much cheaper to iterate with than PPO.
    • Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124), §5.1.2

Source

  • Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi and Hannaneh Hajishirzi. 2024. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. In arXiv preprint arXiv:2411.15124. https://arxiv.org/abs/2411.15124