Language Models are Unsupervised Multitask Learners

Language models trained on a large, diverse web corpus begin to perform tasks like question answering, translation, reading comprehension and summarization zero-shot, without explicit supervision or any parameter or architecture changes, and this ability grows with model capacity.

Systems trained on one task and domain are narrow experts, brittle under small changes in data distribution or task specification. From a meta-learning view, each (dataset, objective) pair is one training example, and multitask NLP efforts had only 10–17 such pairs, far too few to generalize.

From word vectors fed to task architectures, to transferred recurrent representations, to fine-tuning whole Transformers (GPT, BERT) without task-specific architectures, and now to performing tasks with no supervised training or modification at all.

A general system should model \(p(\text{output} \mid \text{input}, \text{task})\), and language can specify the task, input and output all as a sequence of symbols, e.g. (translate to french, english text, french text). A language model may learn to infer and perform tasks demonstrated in natural text in order to predict it better.

A supervised objective is the same as the language-modelling objective evaluated only on a subset of the sequence (the output), so the global minimum of the unsupervised objective is also a minimum of the supervised one; the question is whether it can be optimized well enough in practice.

Common Crawl has serious quality problems, so WebText takes only pages linked from Reddit posts with at least 3 karma, a cheap human filter for quality: about 45M links, giving over 8M documents and 40 GB of text after de-duplication and cleaning. Wikipedia was removed to avoid overlap with test sets.

For example, English–French translations appearing inside ordinary text (a quoted French phrase followed by its English meaning), showing how a language model could pick up tasks from web text without explicit supervision.

Byte-level BPE: a base vocabulary of 256 bytes plus merges, 50,257 tokens in total. It can assign a probability to any Unicode string, so the model can be evaluated on any dataset regardless of preprocessing, while keeping the performance of word-level models.

Plain byte-level BPE wastes vocabulary on many variants of common words with attached punctuation (dog., dog!, dog?). Blocking merges across categories (except for spaces) avoids that while barely fragmenting words.

Layer normalization is moved to the input of each sub-block (pre-norm) with an extra one after the final block; residual-layer weights are initialized scaled by \(1/\sqrt{N}\) for \(N\) residual layers; vocabulary 50,257; context 1024 tokens instead of 512; batch size 512.

Four, roughly log-uniformly spaced: 117M parameters (12 layers, \(d_{\text{model}} = 768\), the size of GPT), 345M (24, 1024, the size of BERT-large), 762M (36, 1280) and 1542M (48, 1600), the last being GPT-2. All still underfit WebText.

State of the art on 7 of 8 datasets without any training on them, with large gains on small datasets (Penn Treebank, WikiText-2) and long-dependency tests (LAMBADA, CBT). It was much worse on the One Billion Word Benchmark, whose sentence shuffling removes long-range structure.

Invertible de-tokenizers removed preprocessing artifacts (disconnected punctuation, split contractions) before scoring, a simple form of domain adaptation that still allows computing the dataset’s log-probability per canonical unit; they gained 2.5–5 perplexity.

Perplexity fell from 99.8 to 8.6 and accuracy rose from 19% to 52.66%. Most errors were valid sentence continuations but not valid final words; a stop-word filter approximating that constraint raised accuracy to 63.24%.

Conditioned on the document, the conversation history and a final “A:”, greedy decoding reached 55 F1, matching or beating 3 of 4 supervised baselines without their 127,000+ training examples (supervised state of the art was near the 89 F1 human level). It often used simple heuristics, e.g. answering a “who” question with a name from the document.

Append “TL;DR:” to the article, sample 100 tokens with top-\(k\) sampling (\(k = 2\)), and keep the first 3 sentences. ROUGE barely beat 3 random sentences and stayed below classic neural baselines; removing the TL;DR: hint cost 6.4 points, showing the hint invokes task behaviour.

Prompted with example pairs “english sentence = french sentence”: 5 BLEU English→French and 11.5 French→English, beating some unsupervised MT baselines but far below the best (33.5). Non-English pages had been filtered out, leaving only about 10 MB of French in WebText.

It answered 4.1% exactly (5.3 times the smallest model, which didn’t beat a 1% most-common-answer baseline), suggesting capacity matters for stored knowledge. Its probabilities were well calibrated: 63.1% accuracy on the 1% of questions it was most confident about. Retrieval-based systems reached 30–50%.

By writing the task into the context and letting the model continue:

task	how the task is invoked
summarization	article … TL;DR:
translation	example pairs english = french, then english =
reading comprehension	document, dialogue history, A:
question answering	example question–answer pairs, then the question

With Bloom filters of WebText 8-grams: common LM test sets had 1–6% 8-gram overlap with WebText (average 3.2%), less than with their own training splits (average 5.9%). Overlap gave a small, consistent benefit (e.g. LAMBADA 8.6 → 8.7 perplexity when overlapping examples were removed).

Training and held-out WebText performance were similar and improved together as model size grew; they recommended n-gram overlap de-duplication as a sanity check when creating new datasets.

Zero-shot performance was competitive on reading comprehension but rudimentary on summarization and far from usable in practice; on many tasks it was probably no better than random, and language models only beat trivial baselines with enough capacity. The ceiling with fine-tuning, and whether scale overcomes unidirectionality, were open questions.