Language Models are Unsupervised Multitask Learners

4. Generalization vs Memorization

1. How did the GPT-2 authors check whether test data leaked into WebText?
   With Bloom filters of WebText 8-grams: common LM test sets had 1–6% 8-gram overlap with WebText (average 3.2%), less than with their own training splits (average 5.9%). Overlap gave a small, consistent benefit (e.g. LAMBADA 8.6 → 8.7 perplexity when overlapping examples were removed).
   • Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 4. Generalization vs Memorization
2. Why did the GPT-2 authors conclude the model was underfitting rather than memorizing WebText?
   Training and held-out WebText performance were similar and improved together as model size grew; they recommended n-gram overlap de-duplication as a sanity check when creating new datasets.
   • Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 4. Generalization vs Memorization