learn.ifs.name

Transformers and Pretraining

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7

8 cards in “7.7 Pretraining transformer LLMs” · whole deck

7.7 Pretraining transformer LLMs

  1. Why is LLM pretraining called self-supervised?
    No gold labels need to be added: the natural sequence of words supervises itself, as the model is trained to predict the true next token at every position.
  2. What loss is minimized in pretraining, and what does it reduce to?
    Cross-entropy between the predicted distribution and the one-hot true next token, \(-\sum_{w}y_t[w]\log\hat{y}_t[w]\), which reduces to \(-\log\hat{y}_t[w_{t+1}]\): the negative log probability of the correct next token.
  3. What is teacher forcing?
    Always feeding the model the correct history \(w_{1:t}\) to predict the next token, rather than its own previous predictions.
  4. What is the loss for a training sequence, and what gets updated?
    The average over positions, \(\frac{1}{T}\sum_t -\log\hat{y}_t[w_{t+1}]\), minimized by gradient descent with backpropagation. All weights are updated, including the embedding matrix \(E\).
  5. Why can transformer pretraining be done in parallel over a sequence?
    The desired outputs are known in advance and the causal mask prevents each position from attending to its own target, so all \(N\) positions are scored against their true next tokens in one pass: \(N\) training examples at once.
  6. How is the context window filled during pretraining?
    Completely, with text; shorter documents are packed together, separated by a special end-of-text token. Batch sizes are very large (millions of tokens).
  7. How is perplexity defined for an LLM, and how does it relate to the training loss?
    \(\text{Perplexity}(w_{1:T}) = P(w_{1:T})^{-1/T}\), the inverse probability normalized by length. It is the exponent of the mean cross-entropy loss (in nats, since log is natural log).
  8. Why is perplexity hard to compare between models with different tokenizers?
    It depends on the number of tokens in a text, so it is very sensitive to the tokenization algorithm. It is best used to compare models with the same tokenizer.

Source