Self-Instruct: Aligning Language Models with Self-Generated Instructions

2.2 Automatic Instruction Data Generation

1. List the four steps of Self-Instruct’s generation pipeline.
   (1) Generate new task instructions; (2) decide whether each instruction is a classification task; (3) generate instances, input-first or output-first; (4) filter low-quality or near-duplicate tasks and add the rest to the task pool.

   • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §2.2, p. 13486
2. How does Self-Instruct prompt for new instructions?
   Starting from a pool of 175 hand-written seed tasks, each step samples 8 instructions as in-context examples, 6 human-written and 2 model-generated, to promote diversity, and lets the model continue the list with new instructions.
   • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §2.2, p. 13486
3. Why does Self-Instruct generate classification instances output-first?
   Input-first generation (invent an input, then answer it) tends to produce inputs biased toward one label, e.g. mostly grammatical sentences for grammar error detection. Generating the class label first and then an input for each label gives balanced data.
   • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §2.2, p. 13486
4. How does Self-Instruct filter generated tasks?
   A new instruction enters the pool only if its ROUGE-L similarity to every existing instruction is below 0.7; instructions with words like “image” or “graph” that a text model can’t handle are dropped; duplicate instances, same-input-different-output instances, and heuristically invalid generations (too long, too short, repeating the input) are removed.
   • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §2.2, p. 13487