learn.ifs.name

Post-training

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 8

13 cards in “8.1 Instruction tuning” · whole deck

8.1 Instruction tuning

  1. What is instruction tuning, and what does the model learn beyond the training tasks?
    Fine-tuning a base LLM on a corpus of instructions and responses for many tasks (translation, meal planning…). Beyond those tasks it learns to follow instructions in general, a form of meta-learning.
  2. What objective and which tokens are used in instruction tuning?
    The same language-modeling cross-entropy objective as pretraining, but trained only on the response: the instruction is the conditioning context, and the model predicts the response tokens one by one.
  3. Why is instruction tuning called supervised fine-tuning, though it uses next-token prediction?
    Unlike pretraining text, each instruction comes with a supervised target: a correct answer or response.
  4. What is fine-tuning in general?
    Taking a fully pretrained model and running additional training passes with cross-entropy loss on new data, adapting all or only some of the parameters.
  5. How much data does instruction tuning need compared to pretraining?
    Much less: several epochs over instruction datasets of at most millions of examples rather than trillions of tokens, so it costs a small fraction of pretraining.
  6. What kinds of content can instructions contain?
    Anything: code to write, questions to answer, translations, writing tasks, shopping lists; with parts like length restrictions, personas to assume, and demonstrations.
  7. What instruction-tuning datasets exist?
    Aya (multilingual, many tasks, partly written by volunteer fluent speakers of many languages), Super-NaturalInstructions, Flan 2022, OPT-IML, Alpaca.
  8. Why use existing NLP datasets to create instruction data, and how?
    Writing high-quality instructions by hand is slow and costly. Fields of existing datasets (e.g. SQuAD QA, NLI premise/hypothesis/label, sentiment text/label) are extracted as key/value pairs and inserted into instruction templates, often paraphrased by LLMs for diversity.
  9. Give an example of an instruction template built from an NLI dataset.
    “Suppose {{premise}} Can we infer that {{hypothesis}}? Yes, no, or maybe?”, with the dataset’s label as the target response.
  10. What is the most common way to generate instruction-tuning data today?
    Having language models write it, based on various datasets.
  11. How can instruction data make models safer, and how much is needed?
    Take harmful questions, have an LLM paraphrase them and write safe refusals, review them manually, and add them to the instruction data. Even a few hundred safety instructions mixed into a large dataset substantially reduce harmfulness.
  12. How can annotation guidelines written for crowdworkers be reused?
    As prompts to an LLM to generate instruction-tuning examples for that task.
  13. How are instruction-tuned models evaluated, and why with task clusters?
    Since the goal is following instructions in general, they are tested on held-out tasks (leave-one-out). Many datasets are near-duplicates of a task, so datasets are grouped into clusters by task similarity and a whole cluster (e.g. all sentiment datasets) is withheld.

Source