Language Models are Unsupervised Multitask Learners

1. Introduction

1. What is the central claim of the GPT-2 paper?
   Language models trained on a large, diverse web corpus begin to perform tasks like question answering, translation, reading comprehension and summarization zero-shot, without explicit supervision or any parameter or architecture changes, and this ability grows with model capacity.
   • Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 1. Introduction
2. Why did Radford et al. think single-task supervised learning limits generalization?
   Systems trained on one task and domain are narrow experts, brittle under small changes in data distribution or task specification. From a meta-learning view, each (dataset, objective) pair is one training example, and multitask NLP efforts had only 10–17 such pairs, far too few to generalize.
   • Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 1. Introduction
3. How does GPT-2 continue the trend in transfer learning?
   From word vectors fed to task architectures, to transferred recurrent representations, to fine-tuning whole Transformers (GPT, BERT) without task-specific architectures, and now to performing tasks with no supervised training or modification at all.
   • Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 1. Introduction