Attention Is All You Need

3.2.1 Scaled Dot-Product Attention

1. Give the formula for scaled dot-product attention.
   \(\text{Attention}(Q, K, V) = \text{softmax}\big(\frac{QK^\top}{\sqrt{d_k}}\big) V\), with queries and keys of dimension \(d_k\) and values of dimension \(d_v\), packed as rows of matrices \(Q\), \(K\), \(V\).

   • Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §3.2.1
2. Why does the Transformer scale dot products by \(1/\sqrt{d_k}\)?
   If query and key components are independent with mean 0 and variance 1, \(q \cdot k = \sum_{i=1}^{d_k} q_i k_i\) has variance \(d_k\). For large \(d_k\) dot products get large, pushing the softmax into regions with extremely small gradients; scaling keeps the variance at 1.
   • Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §3.2.1
3. Why use dot-product rather than additive attention?
   Similar theoretical complexity, but dot-product attention is much faster and more space-efficient in practice with optimized matrix multiplication. Without scaling, additive attention outperforms it for large \(d_k\); the scaling fixes that.
   • Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §3.2.1