Post-training

They aren’t good at following instructions (asked how to steal from a store, a base model may just generate a related question), and they aren’t always good at complex problem solving that needs long chains of reasoning.

Pretraining doesn’t make them safe: they can attack users, suggest harmful actions, or give dangerously false answers (e.g. medical).

The pretraining objective (predicting words in text) is misaligned with the human need for models to be helpful and non-harmful.

(1) Instruction tuning (SFT): fine-tuning on instructions and responses. (2) Preference alignment (e.g. RLHF, DPO): using a model of human preferences to fine-tune the model. (3) RL with verifiable rewards (RLVR): training on tasks with clear right answers, like math or programming, to solve multi-step problems.

A model that has been pretrained but hasn’t yet gone through any post-training steps.

Fine-tuning a base LLM on a corpus of instructions and responses for many tasks (translation, meal planning…). Beyond those tasks it learns to follow instructions in general, a form of meta-learning.

The same language-modeling cross-entropy objective as pretraining, but trained only on the response: the instruction is the conditioning context, and the model predicts the response tokens one by one.

Unlike pretraining text, each instruction comes with a supervised target: a correct answer or response.

Taking a fully pretrained model and running additional training passes with cross-entropy loss on new data, adapting all or only some of the parameters.

Much less: several epochs over instruction datasets of at most millions of examples rather than trillions of tokens, so it costs a small fraction of pretraining.

Anything: code to write, questions to answer, translations, writing tasks, shopping lists; with parts like length restrictions, personas to assume, and demonstrations.

Aya (multilingual, many tasks, partly written by volunteer fluent speakers of many languages), Super-NaturalInstructions, Flan 2022, OPT-IML, Alpaca.

Writing high-quality instructions by hand is slow and costly. Fields of existing datasets (e.g. SQuAD QA, NLI premise/hypothesis/label, sentiment text/label) are extracted as key/value pairs and inserted into instruction templates, often paraphrased by LLMs for diversity.

“Suppose {{premise}} Can we infer that {{hypothesis}}? Yes, no, or maybe?”, with the dataset’s label as the target response.

Having language models write it, based on various datasets.

Take harmful questions, have an LLM paraphrase them and write safe refusals, review them manually, and add them to the instruction data. Even a few hundred safety instructions mixed into a large dataset substantially reduce harmfulness.

As prompts to an LLM to generate instruction-tuning examples for that task.

Since the goal is following instructions in general, they are tested on held-out tasks (leave-one-out). Many datasets are near-duplicates of a task, so datasets are grouped into clusters by task similarity and a whole cluster (e.g. all sentiment datasets) is withheld.

To adapt it to a new task, domain (legal, medical) or language/variety whose data wasn’t sufficiently represented in pretraining.

Every batch must backpropagate through enormous numbers of parameters in many huge layers, which is very expensive in compute, memory and time; many labs can’t do it.

Fine-tuning that freezes most parameters and updates only a small chosen subset.

Low-Rank Adaptation: freeze the dense weight matrices (e.g. \(W^Q, W^K, W^V, W^O\)) and instead learn a low-rank update with far fewer parameters.

It learns \(A\) \([k \times r]\) and \(B\) \([r \times d]\) with \(r \ll \min(d, k)\), replacing \(W + \Delta W\) by \(W + AB\). The forward pass becomes \(h = xW + xAB\).

Much lower hardware requirements (no gradients for most parameters); no extra inference time, since \(AB\) has the shape of \(W\) and can be added into it; and modules for different domains can be swapped in and out by adding or subtracting them.

Only the attention matrices: \(W^Q, W^K, W^V, W^O\). Many variants exist.

Hallucinations, unsafe or harmful outputs, and responses that are technically correct but not as helpful as they could be; it targets qualities like honesty, helpfulness and harmlessness.

They don’t require knowing how to do something, only an opinion about the result. People can express preferences about outputs even with no expertise in producing them.

Picking the better of a pair, selecting one from a set, ranking a set of possibilities, or accepting/rejecting an option in isolation.

A prompt \(x\) with several outputs \(o\) sampled from an LLM, and judgments such as \((o_i \succ o_j \mid x)\): \(o_i\) is preferred to \(o_j\) given \(x\).

A full ranking of \(N\) outputs gives \(\binom{N}{2}\) pairs (e.g. 6 for 4 outputs); selecting the single best gives \(N - 1\) pairs.

Human annotator judgments, implicit preferences extracted from online resources, and fully synthetic preferences from LLMs acting as annotators.

On sites like Reddit or StackExchange, a post serves as the prompt and the responses as sampled outputs; accumulated user votes rank the responses, which can be turned into preference pairs.

Outputs for each prompt are generated by a diverse set of LLMs, and a strong LLM (e.g. GPT-4) ranks them.

Annotators (human or LLM) rate each output on a Likert scale along dimensions like helpfulness, honesty, correctness, complexity and verbosity. Pairs can be derived per dimension or from averages, avoiding costly pairwise comparisons.

Each item has a hidden scalar score; the probability that \(o_i\) is preferred to \(o_j\) is the sigmoid of the score difference: \(P(o_i \succ o_j \mid x) = \sigma(z_i - z_j)\).

Tiny score differences give probabilities near 0.5 (weak or no preference), larger differences quickly approach 0 or 1, and the sigmoid’s derivative allows learning with a binary cross-entropy loss. The score difference is interpreted as the log odds of the preference.

Learn \(r(x, o)\) that gives the scores in Bradley-Terry, minimizing \(-\log\sigma(r(x, o_w) - r(x, o_l))\) over pairs of preferred (winner) and dispreferred (loser) outputs, averaged over the preference dataset.

Initialize it from a pretrained LLM, remove the language modeling head, and replace it with a single dense linear layer that outputs a scalar; train with the Bradley-Terry loss.

Best-of-N sampling (picking the preferred response among several sampled ones) and selecting data for instruction tuning.

Find the policy that maximizes expected reward of its own outputs: \(\pi^* = \arg\max_{\pi_\theta}\mathbb{E}_{x \sim D,\, o \sim \pi_\theta(o \mid x)}[r(x, o)]\).

(1) The reward comes not from the environment (win or lose) but from a learned reward model, a noisy surrogate for the true reward. (2) Learning starts not from a random policy but from a strong pretrained, instruction-tuned model that should only be nudged toward preferred behavior.

Optimizing only for reward on a small amount of preference data would make the model forget what it learned in pretraining. The penalty \(\beta D_{KL}[\pi_\theta \,\|\, \pi_{ref}]\) keeps the trained policy close to the reference (starting) model.

\(\pi^* = \arg\max_{\pi_\theta}\mathbb{E}\big[r(x, o) - \beta\log\frac{\pi_\theta(o \mid x)}{\pi_{ref}(o \mid x)}\big]\), where \(\beta\) controls the strength of the penalty.

A distance between two probability distributions; for LLM policies, the log of the ratio of the trained policy’s probability to the reference policy’s.

Train an explicit reward model and optimize the policy with RL (e.g. PPO), or rearrange the closed-form solution to fine-tune directly from preference data (DPO).

A method (Rafailov et al., 2023) that optimizes an LLM directly on preference data with gradient descent, without learning an explicit reward model or sampling from the model being trained.

Rewrite the closed-form optimum of the KL-constrained objective so the reward is expressed through policies: \(r(x, o) = \beta\log\frac{\pi(o \mid x)}{\pi_{ref}(o \mid x)} + \beta\log Z(x)\). In the Bradley-Terry difference of two rewards, the intractable partition function \(Z(x)\) cancels.

\(L_{DPO} = -\log\sigma\Big(\beta\log\frac{\pi_\theta(o_w \mid x)}{\pi_{ref}(o_w \mid x)} - \beta\log\frac{\pi_\theta(o_l \mid x)}{\pi_{ref}(o_l \mid x)}\Big)\)

It increases the likelihood of preferred outputs and decreases that of dispreferred ones, relative to the reference model, while the KL term (via \(\beta\), typically 0.01–0.1) keeps it from straying too far.

No explicit reward model to train; it learns directly from the preference dataset without expensive online sampling from the policy; and it keeps only 2 LLMs in memory during training instead of 4.