Language Models are Few-Shot Learners
Brown et al., 2020, Language Models are Few-Shot Learners (Advances in Neural Information Processing Systems 33 (NeurIPS 2020))
Subscriber deck — free during early access
1 cards in “3.1 Language Modeling, Cloze, and Completion Tasks” · whole deck
3.1 Language Modeling, Cloze, and Completion Tasks
- How did GPT-3 do on LAMBADA, and what trick helped few-shot?Zero-shot accuracy 76.2%, 8 points above the previous state of the art; few-shot with a fill-in-the-blank format (“Alice was friends with Bob. Alice went to visit her friend, ___. → Bob”) reached 86.4%. The format didn’t help one-shot, perhaps because several examples are needed to recognize the pattern.
- Brown et al., 2020, Language Models are Few-Shot Learners (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §3.1
Source
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020). https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html