Attention Is All You Need

5.3 Optimizer

1. What learning rate schedule did the Transformer use?
   Adam (\(\beta_1 = 0.9\), \(\beta_2 = 0.98\), \(\epsilon = 10^{-9}\)) with \(\text{lrate} = d_{\text{model}}^{-0.5} \cdot \min(\text{step}^{-0.5}, \text{step} \cdot \text{warmup}^{-1.5})\): linear warm-up for 4000 steps, then decay proportional to the inverse square root of the step number.
   • Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §5.3