Transformers and Pretraining

7.4 The input: token and position embeddings

1. How is the input matrix \(X\) computed?
   For each of the \(N\) tokens, look up its token embedding (row of \(E\), shape \([|V| \times d]\), selected by token id or by a one-hot vector) and add a positional embedding for its position.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.4, pp. 191–192
2. Why do transformers need positional embeddings?
   Token embeddings don’t depend on position, and attention by itself is order-insensitive, so word order must be represented explicitly.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.4, p. 192
3. What are absolute positional embeddings?
   Randomly initialized embeddings for each position up to a maximum length, learned along with the other parameters (a matrix \(E_{pos}\)), added to the token embedding.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.4, pp. 192–193
4. What problem do learned absolute position embeddings have, and what alternatives exist?
   Later positions have fewer training examples, so their embeddings may be poorly trained and generalize badly. Alternatives: sinusoidal embeddings (static sine/cosine functions of position, as in the original transformer), or relative position methods applied inside attention at each layer, most popularly RoPE (Rotary Position Embedding).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.4, p. 193
5. What advantage might sinusoidal position embeddings have beyond handling arbitrary lengths?
   They may capture relationships among positions, e.g. that position 4 is closer to position 5 than to position 17.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.4, p. 193