learn.ifs.name

Post-training

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 8

12 cards in “8.4 Alignment via preference-based learning” · whole deck

8.4 Alignment via preference-based learning

  1. How do the elements of reinforcement learning map onto LLM alignment?
    RLLLM alignment
    actionchoice of the next token
    statecontext: tokens generated so far
    policy \(\pi\)the language model
    rewardscore from a reward model learned from preferences
  2. What objective does preference-based RL alignment optimize, in its simplest form?
    Find the policy that maximizes expected reward of its own outputs: \(\pi^* = \arg\max_{\pi_\theta}\mathbb{E}_{x \sim D,\, o \sim \pi_\theta(o \mid x)}[r(x, o)]\).
  3. What are two key differences between traditional RL and RL for LLM alignment?
    (1) The reward comes not from the environment (win or lose) but from a learned reward model, a noisy surrogate for the true reward. (2) Learning starts not from a random policy but from a strong pretrained, instruction-tuned model that should only be nudged toward preferred behavior.
  4. Why is a KL-divergence penalty added to the alignment objective?
    Optimizing only for reward on a small amount of preference data would make the model forget what it learned in pretraining. The penalty \(\beta D_{KL}[\pi_\theta \,\|\, \pi_{ref}]\) keeps the trained policy close to the reference (starting) model.
  5. Write the KL-constrained alignment objective.
    \(\pi^* = \arg\max_{\pi_\theta}\mathbb{E}\big[r(x, o) - \beta\log\frac{\pi_\theta(o \mid x)}{\pi_{ref}(o \mid x)}\big]\), where \(\beta\) controls the strength of the penalty.
  6. What does KL divergence measure?
    A distance between two probability distributions; for LLM policies, the log of the ratio of the trained policy’s probability to the reference policy’s.
  7. What are the two approaches to aligning LLMs with this objective?
    Train an explicit reward model and optimize the policy with RL (e.g. PPO), or rearrange the closed-form solution to fine-tune directly from preference data (DPO).
  8. What is Direct Preference Optimization (DPO)?
    A method (Rafailov et al., 2023) that optimizes an LLM directly on preference data with gradient descent, without learning an explicit reward model or sampling from the model being trained.
  9. What is the key insight of DPO?
    Rewrite the closed-form optimum of the KL-constrained objective so the reward is expressed through policies: \(r(x, o) = \beta\log\frac{\pi(o \mid x)}{\pi_{ref}(o \mid x)} + \beta\log Z(x)\). In the Bradley-Terry difference of two rewards, the intractable partition function \(Z(x)\) cancels.
  10. Write the DPO loss for one preference pair.
    \(L_{DPO} = -\log\sigma\Big(\beta\log\frac{\pi_\theta(o_w \mid x)}{\pi_{ref}(o_w \mid x)} - \beta\log\frac{\pi_\theta(o_l \mid x)}{\pi_{ref}(o_l \mid x)}\Big)\)
  11. What does the DPO update do, intuitively?
    It increases the likelihood of preferred outputs and decreases that of dispreferred ones, relative to the reference model, while the KL term (via \(\beta\), typically 0.01–0.1) keeps it from straying too far.
  12. What are the advantages of DPO over PPO?
    No explicit reward model to train; it learns directly from the preference dataset without expensive online sampling from the policy; and it keeps only 2 LLMs in memory during training instead of 4.

Source