Language Models are Unsupervised Multitask Learners
Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report)
Subscriber deck — free during early access
1 cards in “3.5. Reading Comprehension” · whole deck
3.5. Reading Comprehension
- How was GPT-2 prompted for CoQA, and how did it do?Conditioned on the document, the conversation history and a final “A:”, greedy decoding reached 55 F1, matching or beating 3 of 4 supervised baselines without their 127,000+ training examples (supervised state of the art was near the 89 F1 human level). It often used simple heuristics, e.g. answering a “who” question with a name from the document.
- Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 3.5. Reading Comprehension
Source
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. In OpenAI technical report. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf