Language Models are Few-Shot Learners

2 Approach

1. Distinguish fine-tuning, few-shot, one-shot and zero-shot in the GPT-3 paper.
   Fine-tuning updates weights on thousands of labelled examples. Few-shot gives \(K\) demonstrations (typically 10–100, as many as fit in the 2048-token context) plus the query, with no updates. One-shot uses \(K = 1\). Zero-shot gives only a natural language task description.

   setting	prompt (invented example)
   zero-shot	Translate English to French: bread =>
   one-shot	Translate English to French: cat => chat bread =>
   few-shot	Translate English to French: cat => chat house => maison … bread =>

   • Brown et al., 2020, Language Models are Few-Shot Learners (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §2
2. What are the advantages and drawbacks of few-shot learning versus fine-tuning, according to Brown et al.?
   Few-shot needs far less task-specific data and avoids overfitting to narrow training distributions or exploiting spurious features; but its results were generally well below fine-tuned state of the art, and it still needs a few examples. One- and zero-shot are the fairest comparisons with humans.
   • Brown et al., 2020, Language Models are Few-Shot Learners (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §2