Self-Instruct: Aligning Language Models with Self-Generated Instructions

3 Self-Instruct Data from GPT3

1. How diverse was the generated instruction data?
   Very: only about half the instructions had a simple verb–object structure, and the 20 most common root verbs with their top objects covered just 14% of the data; most generated instructions had little ROUGE-L overlap with the seed tasks.
   • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §3, p. 13487
2. What did a manual quality check of Self-Instruct data find?
   Of 200 sampled instructions with one instance each: 92% described a valid task, 79% had an appropriate input, 58% a correct output, and 54% were fully valid. Even flawed instances were mostly in the right format or partly correct, still useful for learning to follow instructions.
   • Wang et al., 2023, Self-Instruct (Proceedings of the 61st Annual Meeting of ACL (Volume 1: Long Papers)), §3, p. 13487