Post-training

Introduction

1. In what two ways are pretrained-only LLMs insufficiently helpful?
   They aren’t good at following instructions (asked how to steal from a store, a base model may just generate a related question), and they aren’t always good at complex problem solving that needs long chains of reasoning.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 8, Introduction, p. 209
2. Why are pretrained-only LLMs also potentially harmful?
   Pretraining doesn’t make them safe: they can attack users, suggest harmful actions, or give dangerously false answers (e.g. medical).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 8, Introduction, p. 209
3. What is the underlying reason LLMs are too harmful and insufficiently helpful after pretraining?
   The pretraining objective (predicting words in text) is misaligned with the human need for models to be helpful and non-harmful.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 8, Introduction, p. 209
4. What are the three stages of post-training?
   (1) Instruction tuning (SFT): fine-tuning on instructions and responses. (2) Preference alignment (e.g. RLHF, DPO): using a model of human preferences to fine-tune the model. (3) RL with verifiable rewards (RLVR): training on tasks with clear right answers, like math or programming, to solve multi-step problems.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 8, Introduction, pp. 209–210
5. What is a base model?
   A model that has been pretrained but hasn’t yet gone through any post-training steps.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 8, Introduction, p. 210