Language Models are Unsupervised Multitask Learners
Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report)
Subscriber deck — free during early access
1 cards in “3.6. Summarization” · whole deck
3.6. Summarization
- How did GPT-2 summarize zero-shot, and how well?Append “TL;DR:” to the article, sample 100 tokens with top-\(k\) sampling (\(k = 2\)), and keep the first 3 sentences. ROUGE barely beat 3 random sentences and stayed below classic neural baselines; removing the TL;DR: hint cost 6.4 points, showing the hint invokes task behaviour.
- Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 3.6. Summarization
Source
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. In OpenAI technical report. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf