Attention Is All You Need

5.4 Regularization

1. Which regularizers did the Transformer use?
   Residual dropout (on each sub-layer output before the residual addition, and on embeddings plus positional encodings; \(P_{\text{drop}} = 0.1\) for the base model) and label smoothing of 0.1, which hurts perplexity but improves accuracy and BLEU.
   • Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §5.4