Language Models are Unsupervised Multitask Learners
Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report)
Subscriber deck — free during early access
1 cards in “6. Discussion” · whole deck
6. Discussion
- What limitations of GPT-2’s zero-shot abilities did its authors acknowledge?Zero-shot performance was competitive on reading comprehension but rudimentary on summarization and far from usable in practice; on many tasks it was probably no better than random, and language models only beat trivial baselines with enough capacity. The ceiling with fine-tuning, and whether scale overcomes unidirectionality, were open questions.
- Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 6. Discussion
Source
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. In OpenAI technical report. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf