Language Models are Few-Shot Learners

1 Introduction

1. What is GPT-3, and what did its paper set out to test?
   A 175-billion-parameter autoregressive language model, 10× larger than any previous dense LM, used to test whether scaling improves task-agnostic few-shot performance: tasks and demonstrations are given purely as text, with no gradient updates or fine-tuning.
   • Brown et al., 2020, Language Models are Few-Shot Learners (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §1
2. What remaining limitation of the pretrain-then-fine-tune paradigm motivated GPT-3?
   Even with task-agnostic architectures, a task-specific fine-tuning step on a large labelled data set remained; GPT-2 had suggested it might be unnecessary, but its zero-shot results were mostly far below supervised baselines.
   • Brown et al., 2020, Language Models are Few-Shot Learners (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §1
3. What is in-context learning, and how did the GPT-3 paper frame it?
   Learning a task from instructions and demonstrations placed in the model’s context, without weight updates. Language models can be seen as meta-learners: slow outer-loop gradient descent during pre-training plus fast inner-loop “in-context” learning within the activations.
   • Brown et al., 2020, Language Models are Few-Shot Learners (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §1
4. How does the gap between zero-, one- and few-shot performance change with model size?
   It generally grows with capacity: zero-shot performance improves steadily with size, but few-shot improves faster, suggesting larger models are better in-context learners.
   • Brown et al., 2020, Language Models are Few-Shot Learners (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §1