Tülu 3: Pushing Frontiers in Open Language Model Post-Training

Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124)

Subscriber deck — free during early access

12 cards

1 Introduction

What gap does Tülu 3 address?
Post-training (instruction tuning, preference tuning, RL) had become central to frontier models, but its data and recipes were rarely published, and fully open recipes lagged on core skills such as MATH, GSM8K and IFEval. Tülu 3 releases models (on Llama 3.1 bases, 8B to 405B) together with all data, code, recipes and evaluation tools.
- Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124), §1

Why does Tülu 3 Eval have separate development and unseen splits?
The team chose data and methods using only the development evaluations and never looked at the unseen ones during development, so the unseen split measures how much those decisions overfit. They did overfit somewhat, e.g. to IFEval’s specific constraint types.
- Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124), §2.2

What are the four stages of the Tülu 3 recipe?
(1) Curating prompts aimed at core skills (knowledge, reasoning, math, coding, instruction following, chat, safety); (2) supervised fine-tuning; (3) preference tuning with DPO on partly on-policy data; (4) reinforcement learning with verifiable rewards (RLVR). All stages are guided by one evaluation suite.
- Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124), §2.3

How did Tülu 3 keep its synthetic prompts diverse?
By persona-driven generation: GPT-4o was given one of about 250K personas (e.g. “a machine learning researcher focused on neural networks”) along with a request such as “create a coding problem”, which avoids the repetitive mode collapse of plain prompting. This produced math, coding and verifiable instruction-following data.
- Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124), §3.1.2

How did Tülu 3 decontaminate its training prompts?
It compared prompts only (completions are often regenerated) by 8-gram overlap: a test item counts as matching if over 50% of its tokens fall in 8-grams shared with one training item. Datasets overlapping more than 2% of an unseen evaluation were dropped; for development evaluations, matching items or whole datasets were removed.
- Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124), §3.2

What loss-aggregation pitfall did Tülu 3 run into during SFT, and how did they fix it?
Averaging the loss over tokens within each micro-batch, then over micro-batches in gradient accumulation or across devices, weights examples differently from averaging over all tokens at once, so results depended on the hardware setup. They switched to a summed loss (with a lower learning rate, 5×10⁻⁶), weighting every token equally.
- Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124), §4.3.2

Which DPO variant did Tülu 3 adopt, and why?
Length-normalized DPO: the usual DPO loss with each response’s log-probability ratio divided by its length, which counters the length bias in human and model preferences. It beat SimPO and standard DPO in their tests and was much cheaper to iterate with than PPO.
- Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124), §5.1.2

How did Tülu 3 build its preference data?
For each prompt, four responses came from models sampled from a pool, often with one from the Tülu 3 SFT model itself (on-policy). GPT-4o rated each from 1 to 5 on helpfulness, instruction following, honesty and truthfulness; the best mean rating became “chosen” and a random lower-rated one “rejected”.
- Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124), §5.2.1

What did Tülu 3’s ablations find about preference data?
More unique prompts helped, while duplicating prompts with extra responses did not; prompts unused in SFT beat reused ones (a mix was best); and including on-policy responses from the SFT model improved results over purely off-policy data.
- Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124), §5.3

What is RLVR?
RL in which the reward comes from a deterministic check instead of a learned reward model: the policy earns \(\alpha\) (10) only if its answer is verifiably correct, otherwise 0, minus the usual KL penalty to the reference policy, optimized with PPO. Tülu 3 used GSM8K and MATH answers and IFEval-style constraint checkers.
RLVR: reward only for verifiably correct answers
- Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124), §6

What did Tülu 3 find when training with RLVR?
It improved the targeted evaluations (GSM8K, MATH, IFEval); adding reward-model scores to the verifiable reward hurt; initializing PPO’s value function from a general reward model worked best; starting from a weaker SFT model reached the same training reward but at higher KL; and too little KL penalty led to over-optimization.
- Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124), §6.2.1

Which techniques did Tülu 3 try but leave out of the final recipe?
Online DPO (sampling pairs from the current policy and labeling them with a reward model), which gave little or no gain on GSM8K and hurt MATH; and rejection sampling, whose gains were too small for its compute. They noted it needs a strong judge, and it worked better when the original response stayed among the candidates.
- Lambert et al., 2024, Tülu 3 (arXiv preprint arXiv:2411.15124), §8.2

Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi and Hannaneh Hajishirzi. 2024. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. In arXiv preprint arXiv:2411.15124. https://arxiv.org/abs/2411.15124