learn.ifs.name

Introduction

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1

22 cards in “1.6 How language models are trained” · whole deck

1.6 How language models are trained

  1. What is the difference between training and inference?
    Training sets the weights of the network. Inference is running (prompting) the trained model to generate text.
  2. What are the four stages of training a modern LLM?
    (1) Pretraining on huge corpora to predict the next token → base model; (2) instruction tuning (SFT); (3) preference alignment; (4) RL for reasoning (RL with verifiable rewards).
    1. Pretrainingweb text2. Instruction tuninginstructions + responses3. Preferenceschosen vs rejected4. RL (verifiable)checkable answers
  3. What is a base model?
    The model that results from pretraining: it can predict tokens and generate text, but has not yet been tuned to follow instructions.
  4. What kind of training data is used in preference alignment?
    An instruction plus two responses, one chosen and one rejected (by humans or another LLM). The model is trained to produce the chosen rather than the rejected continuation.
  5. Why is pretraining like playing a Shannon game with the model?
    The model predicts each token of running text from the prior context; when it is wrong it is told the correct token and its weights are updated to make that token more likely. Through these corrections it also learns useful embeddings.
  6. What is the main source of pretraining data for open language models?
    The Common Crawl: snapshots of the web, each with billions of pages, plus more curated data like Wikipedia and books.
  7. What quality filters are applied to pretraining data?
    Deduplication (removing duplicate documents), boilerplate removal, and removing adult content. Safety filtering additionally removes toxic material using toxicity classifiers.
  8. What problems do toxicity filters for pretraining data have?
    They often mistakenly flag non-toxic text, e.g. in minority dialects like African American English. And models trained on toxicity-filtered data, while somewhat less toxic, are worse at detecting toxicity themselves.
  9. What three ethical and legal problems arise from training on web-scraped data?
    Copyright (much text is copyrighted; fair use is unsettled and creators are not compensated), privacy (phone numbers, emails slip through filters), and skew (authors from the US and developed countries are overrepresented).
  10. What is data-centric AI?
    A movement holding that training data, its properties and its provenance are essential: builders should ask where data comes from, who created it, whether they are compensated, and whether it is okay to use.
  11. Why is pretraining alone not sufficient? Give an example of the failure.
    A base model just generates text that is related to or predictable from the prompt, rather than following it: asked to “Translate to French: The small dog”, it may continue “The small dog crossed the road.”
  12. What is instruction tuning?
    Further training a pretrained base LLM on a corpus of tens or hundreds of thousands of instructions with responses (translation, QA, classification, meal planning…), so it learns to follow instructions. Also called supervised fine-tuning (SFT) or instruct tuning.
  13. In instruction tuning, which tokens is the model trained to predict?
    Only the response tokens: the instruction is the conditioning context, and the model predicts the output tokens one by one conditioned on it.
  14. Why does the exact set of tasks in instruction tuning matter less than one might think?
    Instruction-tuned models seem to learn to follow instructions in general rather than just the training tasks, a form of meta-learning. What helps is diversity of instruction types.
  15. How does instruction-tuning data address safety?
    It includes examples where harmful requests (e.g. obtaining someone’s social security number) are answered with refusals.
  16. What are preference judgments, and how are they most simply obtained?
    Rankings of potential responses to a prompt. Simplest: have a model generate two candidate responses, then have humans (or an LLM) judge which is better.
  17. What kind of feedback does preference alignment provide that instruction tuning does not?
    It tells the model what not to say, by contrasting a chosen response with a rejected one.
  18. Do the “chosen” answers in preference datasets have to be good?
    No. Many are not very good (even false); training still helps as long as the chosen answer is better than the rejected one.
  19. How is a model trained on preference data?
    With reinforcement-learning-related algorithms such as PPO and DPO, using a reward function to make chosen responses more probable and rejected ones less probable.
  20. What is RL for reasoning (RL with verifiable rewards)?
    Training in domains where responses can be automatically judged right or wrong without human judgments (math is canonical). The model gets a reward for verifiably correct responses, which is especially useful for training chain-of-thought (reasoning) models.
  21. How can “instruction-following” constraints make almost any task verifiable?
    By specifying checkable constraints on the output: required or forbidden keywords, output format (JSON, markdown), length, output language, or answers from a fixed set. What is verified is the form of the output, not its correctness.
  22. What is RLVR?
    Reinforcement Learning with Verifiable Rewards: the model is rewarded for responses that are verifiably correct, nudging its parameters to prefer correct responses over incorrect ones.

Source