Language Models are Few-Shot Learners

Brown et al., 2020, Language Models are Few-Shot Learners (Advances in Neural Information Processing Systems 33 (NeurIPS 2020))

Subscriber deck — free during early access

1 cards in “4 Measuring and Preventing Memorization Of Benchmarks” · whole deck

4 Measuring and Preventing Memorization Of Benchmarks

How did the GPT-3 authors check for benchmark contamination?
For each benchmark they built a “clean” subset removing examples with any 13-gram overlap with the pre-training data and compared scores. Performance mostly changed negligibly, with no correlation between contamination level and score difference, so either contamination was overestimated or had little effect.
- Brown et al., 2020, Language Models are Few-Shot Learners (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §4

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020). https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html