learn.ifs.name

Post-training

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 8

14 cards in “8.3 Learning from preferences” · whole deck

8.3 Learning from preferences

  1. What problems can remain after instruction tuning that preference learning addresses?
    Hallucinations, unsafe or harmful outputs, and responses that are technically correct but not as helpful as they could be; it targets qualities like honesty, helpfulness and harmlessness.
  2. Why are preference judgments easier to obtain than instructions?
    They don’t require knowing how to do something, only an opinion about the result. People can express preferences about outputs even with no expertise in producing them.
  3. What forms do preference judgments take?
    Picking the better of a pair, selecting one from a set, ranking a set of possibilities, or accepting/rejecting an option in isolation.
  4. What does LLM preference data consist of, and how is a preference written?
    A prompt \(x\) with several outputs \(o\) sampled from an LLM, and judgments such as \((o_i \succ o_j \mid x)\): \(o_i\) is preferred to \(o_j\) given \(x\).
  5. How many preference pairs does a ranking of \(N\) outputs give, and how many does choosing one best of \(N\) give?
    A full ranking of \(N\) outputs gives \(\binom{N}{2}\) pairs (e.g. 6 for 4 outputs); selecting the single best gives \(N - 1\) pairs.
  6. What are the three sources of preference data for LLM alignment?
    Human annotator judgments, implicit preferences extracted from online resources, and fully synthetic preferences from LLMs acting as annotators.
  7. How can social media provide implicit preference data?
    On sites like Reddit or StackExchange, a post serves as the prompt and the responses as sampled outputs; accumulated user votes rank the responses, which can be turned into preference pairs.
  8. How are synthetic preference datasets like UltraFeedback created?
    Outputs for each prompt are generated by a diverse set of LLMs, and a strong LLM (e.g. GPT-4) ranks them.
  9. What are aspect-based scalar judgments, and what is their advantage?
    Annotators (human or LLM) rate each output on a Likert scale along dimensions like helpfulness, honesty, correctness, complexity and verbosity. Pairs can be derived per dimension or from averages, avoiding costly pairwise comparisons.
  10. What is the Bradley-Terry model of preferences?
    Each item has a hidden scalar score; the probability that \(o_i\) is preferred to \(o_j\) is the sigmoid of the score difference: \(P(o_i \succ o_j \mid x) = \sigma(z_i - z_j)\).
  11. Why is the Bradley-Terry formulation convenient?
    Tiny score differences give probabilities near 0.5 (weak or no preference), larger differences quickly approach 0 or 1, and the sigmoid’s derivative allows learning with a binary cross-entropy loss. The score difference is interpreted as the log odds of the preference.
  12. How is a reward model trained from preference data?
    Learn \(r(x, o)\) that gives the scores in Bradley-Terry, minimizing \(-\log\sigma(r(x, o_w) - r(x, o_l))\) over pairs of preferred (winner) and dispreferred (loser) outputs, averaged over the preference dataset.
  13. How is a reward model typically built?
    Initialize it from a pretrained LLM, remove the language modeling head, and replace it with a single dense linear layer that outputs a scalar; train with the Bradley-Terry loss.
  14. Besides alignment, what are reward models used for?
    Best-of-N sampling (picking the preferred response among several sampled ones) and selecting data for instruction tuning.

Source