Attention Is All You Need

3.1 Encoder and Decoder Stacks

1. Describe a Transformer encoder layer.
   Two sub-layers: multi-head self-attention, then a position-wise feed-forward network. Each sub-layer is wrapped as \(\text{LayerNorm}(x + \text{Sublayer}(x))\), a residual connection followed by layer normalization; all outputs have dimension \(d_{\text{model}} = 512\). The encoder stacks \(N = 6\) such layers.
   • Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §3.1
2. How does a Transformer decoder layer differ from an encoder layer?
   It has a third sub-layer, multi-head attention over the encoder output, and its self-attention is masked so position \(i\) can’t attend to later positions; with outputs shifted right by one, predictions for position \(i\) depend only on outputs before \(i\).
   • Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §3.1