learn.ifs.name

Tülu 3: Pushing Frontiers in Open Language Model Post-Training

Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124)

Subscriber deck — free during early access

1 cards in “5.2.1 From Prompts to Preference Data” · whole deck

5.2.1 From Prompts to Preference Data

  1. How did Tülu 3 build its preference data?
    For each prompt, four responses came from models sampled from a pool, often with one from the Tülu 3 SFT model itself (on-policy). GPT-4o rated each from 1 to 5 on helpfulness, instruction following, honesty and truthfulness; the best mean rating became “chosen” and a random lower-rated one “rejected”.
    • Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124), §5.2.1

Source

  • Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi and Hannaneh Hajishirzi. 2024. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. In arXiv preprint arXiv:2411.15124. https://arxiv.org/abs/2411.15124