Language Models are Few-Shot Learners
Brown et al., 2020, Language Models are Few-Shot Learners (Advances in Neural Information Processing Systems 33 (NeurIPS 2020))
Subscriber deck — free during early access
2 cards in “3.2 Question Answering” · whole deck
3.2 Question Answering
- How did GPT-3 do on closed-book TriviaQA?Without any retrieval or fine-tuning: 64.3% zero-shot (beating fine-tuned closed-book T5-11B), 68.0% one-shot (matching an open-domain system with learned retrieval over 21M documents), and 71.2% few-shot.
- Brown et al., 2020, Language Models are Few-Shot Learners (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §3.2
- Where did GPT-3 underperform on question answering and reading comprehension?On Natural Questions (fine-grained Wikipedia knowledge), it trailed fine-tuned T5-11B; on DROP (discrete reasoning and numeracy), few-shot 36.5 F1 vs 89.1 for the state of the art. On CoQA it came within 3 points of humans (85.0 F1 few-shot).
- Brown et al., 2020, Language Models are Few-Shot Learners (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §3.2
Source
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020). https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html