Attention Is All You Need

3.2.2 Multi-Head Attention

1. What is multi-head attention?
   Project queries, keys and values \(h\) times with different learned matrices, run attention in parallel on each projection, concatenate and project again: \(\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h) W^O\), \(\text{head}_i = \text{Attention}(Q W_i^Q, K W_i^K, V W_i^V)\).
   • Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §3.2.2
2. Why use several attention heads, and what does it cost?
   Heads can jointly attend to information from different representation subspaces at different positions; a single head averages them. With \(h = 8\) and \(d_k = d_v = d_{\text{model}}/h = 64\), the total cost is similar to single-head attention with full dimensionality.
   • Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §3.2.2