Language Models are Few-Shot Learners
Brown et al., 2020, Language Models are Few-Shot Learners (Advances in Neural Information Processing Systems 33 (NeurIPS 2020))
Subscriber deck — free during early access
2 cards in “3.4 SuperGLUE” · whole deck
3.4 SuperGLUE
- How did few-shot GPT-3 do on SuperGLUE?With 32 examples per task: 71.8 average vs 69.0 for fine-tuned BERT-large and 89.0 state of the art. Near state of the art on COPA and ReCoRD; around BERT-large on BoolQ, WSC, MultiRC and RTE; chance level on WiC.
- Brown et al., 2020, Language Models are Few-Shot Learners (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §3.4
- What weakness did WiC, RTE and CB reveal in GPT-3?GPT-3 was weak in few- and one-shot settings at tasks that compare two sentences or snippets (whether a word means the same in two sentences, or whether one sentence entails another); no prompt phrasing made WiC work.
- Brown et al., 2020, Language Models are Few-Shot Learners (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §3.4
Source
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020). https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html