Language Models are Unsupervised Multitask Learners
Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report)
Subscriber deck — free during early access
1 cards in “3.3. LAMBADA” · whole deck
3.3. LAMBADA
- What did GPT-2 achieve on LAMBADA, and what error pattern did it show?Perplexity fell from 99.8 to 8.6 and accuracy rose from 19% to 52.66%. Most errors were valid sentence continuations but not valid final words; a stop-word filter approximating that constraint raised accuracy to 63.24%.
- Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 3.3. LAMBADA
Source
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. In OpenAI technical report. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf