Language Models are Unsupervised Multitask Learners
Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report)
Subscriber deck — free during early access
2 cards in “2.2. Input Representation” · whole deck
2.2. Input Representation
- What input representation does GPT-2 use, and why?Byte-level BPE: a base vocabulary of 256 bytes plus merges, 50,257 tokens in total. It can assign a probability to any Unicode string, so the model can be evaluated on any dataset regardless of preprocessing, while keeping the performance of word-level models.
- Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 2.2. Input Representation
- Why did GPT-2 forbid BPE merges across character categories?Plain byte-level BPE wastes vocabulary on many variants of common words with attached punctuation (dog., dog!, dog?). Blocking merges across categories (except for spaces) avoids that while barely fragmenting words.
- Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 2.2. Input Representation
Source
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. In OpenAI technical report. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf