Language Models are Unsupervised Multitask Learners

2.3. Model

1. How does the GPT-2 architecture differ from the original GPT?
   Layer normalization is moved to the input of each sub-block (pre-norm) with an extra one after the final block; residual-layer weights are initialized scaled by \(1/\sqrt{N}\) for \(N\) residual layers; vocabulary 50,257; context 1024 tokens instead of 512; batch size 512.
   • Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 2.3. Model
2. What model sizes did the GPT-2 paper train?
   Four, roughly log-uniformly spaced: 117M parameters (12 layers, \(d_{\text{model}} = 768\), the size of GPT), 345M (24, 1024, the size of BERT-large), 762M (36, 1280) and 1542M (48, 1600), the last being GPT-2. All still underfit WebText.
   • Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 2.3. Model