Language Models are Unsupervised Multitask Learners

2.1. Training Dataset

1. How was the WebText corpus built, and why that way?
   Common Crawl has serious quality problems, so WebText takes only pages linked from Reddit posts with at least 3 karma, a cheap human filter for quality: about 45M links, giving over 8M documents and 40 GB of text after de-duplication and cleaning. Wikipedia was removed to avoid overlap with test sets.
   • Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 2.1. Training Dataset
2. What naturally occurring task demonstrations did the GPT-2 authors find in WebText?
   For example, English–French translations appearing inside ordinary text (a quoted French phrase followed by its English meaning), showing how a language model could pick up tasks from web text without explicit supervision.
   • Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 2.1. Training Dataset