learn.ifs.name

The Llama 3 Herd of Models

Meta, 2024, The Llama 3 Herd of Models (arXiv preprint arXiv:2407.21783)

Subscriber deck — free during early access

2 cards in “4.1 Modeling” · whole deck

4.1 Modeling

  1. Describe one round of Llama 3 post-training.
    Train a reward model on human preference data; use it for rejection sampling (pick the best of several generated responses) on human prompts; supervised fine-tuning on that plus other and synthetic data; then DPO on fresh preference data from the latest models; finally average models from different runs. Six such rounds were done.
    reward modelrejection samplingSFTDPOmodel averaging× 6
    Each round’s best model generates data for the next round
  2. What two tweaks stabilized DPO for Llama 3?
    Masking special formatting tokens (headers, terminators) from the DPO loss, since tokens shared by chosen and rejected responses get conflicting gradients and caused tail repetition or abrupt termination; and adding a negative log-likelihood term (weight 0.2) on chosen responses to preserve formatting.

Source