learn.ifs.name

Tülu 3: Pushing Frontiers in Open Language Model Post-Training

Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124)

Subscriber deck — free during early access

1 cards in “3.2 Prompt Decontamination” · whole deck

3.2 Prompt Decontamination

  1. How did Tülu 3 decontaminate its training prompts?
    It compared prompts only (completions are often regenerated) by 8-gram overlap: a test item counts as matching if over 50% of its tokens fall in 8-grams shared with one training item. Datasets overlapping more than 2% of an unseen evaluation were dropped; for development evaluations, matching items or whole datasets were removed.
    • Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124), §3.2

Source

  • Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi and Hannaneh Hajishirzi. 2024. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. In arXiv preprint arXiv:2411.15124. https://arxiv.org/abs/2411.15124