Language Models are Unsupervised Multitask Learners

3.1. Language Modeling

1. How did GPT-2 do on language modelling benchmarks zero-shot?
   State of the art on 7 of 8 datasets without any training on them, with large gains on small datasets (Penn Treebank, WikiText-2) and long-dependency tests (LAMBADA, CBT). It was much worse on the One Billion Word Benchmark, whose sentence shuffling removes long-range structure.
   • Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 3.1. Language Modeling
2. How could a byte-level model be evaluated fairly on tokenized LM benchmarks?
   Invertible de-tokenizers removed preprocessing artifacts (disconnected punctuation, split contractions) before scoring, a simple form of domain adaptation that still allows computing the dataset’s log-probability per canonical unit; they gained 2.5–5 perplexity.
   • Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 3.1. Language Modeling