learn.ifs.name

Self-Instruct: Aligning Language Models with Self-Generated Instructions

Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers))

Subscriber deck — free during early access

16 cards

1 Introduction

  1. What bottleneck of instruction tuning does Self-Instruct address?
    Instruction-tuned models depend on human-written instruction data, which is costly and limited in quantity, diversity and creativity (mostly popular NLP tasks), limiting how well the tuned model generalizes.
    • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 13484
  2. What is Self-Instruct in one sentence?
    An almost annotation-free, iterative bootstrapping method in which a vanilla pretrained language model generates new task instructions and input–output instances from a small seed set, filters them, and is then fine-tuned on its own generated data.
    • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 13485
  3. What were Self-Instruct’s headline results with GPT-3?
    From 175 seed tasks, vanilla GPT-3 generated about 52K instructions with 82K instances; fine-tuning GPT-3 on them improved SuperNaturalInstructions by 33 points absolute, nearly matching InstructGPT-001 (trained on private user data and human labels), and left only a 5% gap to it in human evaluation on novel tasks.
    • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 13485

2.1 Defining Instruction Data

  1. How does Self-Instruct define a task, and why can the input be empty?
    A task \(t\) is an instruction \(I_t\) plus one or more instances \((X_{t,i}, Y_{t,i})\) with \(M(I_t, X_{t,i}) = Y_{t,i}\). The boundary between instruction and input is fluid (“write an essay about school safety” vs “write an essay about the following topic” + “school safety”), so instructions without input are allowed for format diversity.
    • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §2.1, p. 13486

2.2 Automatic Instruction Data Generation

  1. List the four steps of Self-Instruct’s generation pipeline.
    (1) Generate new task instructions; (2) decide whether each instruction is a classification task; (3) generate instances, input-first or output-first; (4) filter low-quality or near-duplicate tasks and add the rest to the task pool.
    task pool175 seeds → 52Knew instructionsclassification?yes: output-firstno: input-firstfilter (ROUGE-L < 0.7)
    Self-Instruct bootstraps a large instruction dataset from 175 seed tasks; filtered tasks go back into the pool
    • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §2.2, p. 13486
  2. How does Self-Instruct prompt for new instructions?
    Starting from a pool of 175 hand-written seed tasks, each step samples 8 instructions as in-context examples, 6 human-written and 2 model-generated, to promote diversity, and lets the model continue the list with new instructions.
    • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §2.2, p. 13486
  3. Why does Self-Instruct generate classification instances output-first?
    Input-first generation (invent an input, then answer it) tends to produce inputs biased toward one label, e.g. mostly grammatical sentences for grammar error detection. Generating the class label first and then an input for each label gives balanced data.
    • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §2.2, p. 13486
  4. How does Self-Instruct filter generated tasks?
    A new instruction enters the pool only if its ROUGE-L similarity to every existing instruction is below 0.7; instructions with words like “image” or “graph” that a text model can’t handle are dropped; duplicate instances, same-input-different-output instances, and heuristically invalid generations (too long, too short, repeating the input) are removed.
    • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §2.2, p. 13487

2.3 Finetuning the LM to Follow Instructions

  1. How was the model fine-tuned on Self-Instruct data, and why use several templates?
    Instruction and input are concatenated as the prompt and the model is trained to generate the output. Multiple templates (with or without “Task:”, “Input:”, “Output:” prefixes, varying line breaks) make it robust to different formats.
    • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §2.3, p. 13487

3 Self-Instruct Data from GPT3

  1. How diverse was the generated instruction data?
    Very: only about half the instructions had a simple verb–object structure, and the 20 most common root verbs with their top objects covered just 14% of the data; most generated instructions had little ROUGE-L overlap with the seed tasks.
    • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §3, p. 13487
  2. What did a manual quality check of Self-Instruct data find?
    Of 200 sampled instructions with one instance each: 92% described a valid task, 79% had an appropriate input, 58% a correct output, and 54% were fully valid. Even flawed instances were mostly in the right format or partly correct, still useful for learning to follow instructions.
    • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §3, p. 13487

4.3 Experiment 1: Zero-Shot Generalization on SuperNI benchmark

  1. How did GPT-3 fine-tuned on Self-Instruct data compare with other models on SuperNI?
    It outperformed vanilla GPT-3 (which mostly produced irrelevant, repetitive text) by 33 points and beat T0 and GPT-3 fine-tuned on T0’s human-written data, nearly matching InstructGPT-001. Models trained on SuperNI’s own training set did better, but adding Self-Instruct data to it still helped.
    • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §4.3, p. 13489

4.4 Experiment 2: Generalization to User-oriented Instructions on Novel Tasks

  1. Why did Wang et al. write 252 new user-oriented instructions, and how were they evaluated?
    SuperNI tasks come from research datasets skewed toward classification; the new set covers practical uses (email, social media, programming, entertainment) in varied formats. Since many outputs (programs, logic) need expertise, the instructions’ authors rated outputs on a four-level scale from valid to irrelevant.
    • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §4.4, pp. 13489–13490

4.5 Effect of Data Size and Quality

  1. How did the amount and quality of Self-Instruct data affect results?
    Human-rated quality improved with more data but nearly plateaued after about 16K instructions (on SuperNI already after hundreds). Regenerating the outputs with InstructGPT-003 improved the fine-tuned model by 10%, showing much room for better-quality data.
    • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §4.5, p. 13491

5 Related Work

  1. How does Self-Instruct differ from Unnatural Instructions and from knowledge distillation?
    Unnatural Instructions seeded from SuperNI tasks and generated with already instruction-tuned InstructGPT-002, distilling from it; Self-Instruct uses only a vanilla model. Unlike usual distillation, source and target are the same model, and what is distilled is whole tasks (instructions with instances).
    • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §5, pp. 13491–13492

8 Limitations

  1. What limitations of Self-Instruct do Wang et al. acknowledge?
    It inherits language-model weaknesses on tail phenomena, so gains may favour frequent kinds of tasks; it may work best with large models, raising access barriers; and the iterative loop could amplify social biases, as seen in its difficulty producing balanced labels.
    • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §8, p. 13493

Source

  • Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi and Hannaneh Hajishirzi. 2023. Self-Instruct: Aligning Language Models with Self-Generated Instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484–13508. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2023.acl-long.754 https://aclanthology.org/2023.acl-long.754/