learn.ifs.name

Masked Language Models

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 9

17 cards in “9.3 Training bidirectional encoders” · whole deck

9.3 Training bidirectional encoders

  1. Why can’t a bidirectional encoder be trained with next-word prediction?
    Without the causal mask, the next word is directly visible in the context, so the task is trivial.
  2. What is the cloze task?
    A fill-in-the-blank task: given a sequence with one or more elements missing, predict the missing elements (“The ___ of Walden Pond is so beautifully…”).
  3. What is denoising as a training method?
    Corrupting the input (masking, substituting, reordering, deleting, inserting words) and training the model to recover the original.
  4. Describe the BERT masking procedure.
    A sample of tokens (15% in BERT) is chosen. Of these, 80% are replaced with [MASK], 10% with a random token sampled by unigram probability, and 10% are left unchanged. The model must predict the original token at each chosen position.

    “So long and thanks for all the fish”

    chosen tokenmodel input
    long[MASK]
    thanks[MASK]
    theapricot (random)
    The loss is computed only at the chosen positions.
  5. Why doesn’t BERT always replace chosen tokens with [MASK]?
    [MASK] never appears at fine-tuning or inference time. If the model only predicted at [MASK] positions it would learn to predict only there; random replacements and unchanged tokens make it try to predict every input token.
  6. How does a masked LM predict the token at a masked position?
    The final layer’s output \(h^L_i\) for that position goes through the LM head: logits \(u_i = h^L_i E^\top\), probabilities \(y_i = \text{softmax}(u_i)\).
  7. What is the MLM loss?
    The average negative log probability of the correct tokens over the set \(M\) of chosen positions: \(L_{MLM} = -\frac{1}{|M|}\sum_{i \in M}\log P(x_i \mid h^L_i)\).
  8. In what sense is BERT training inefficient?
    Only the chosen tokens (about 15%) contribute to the loss; the others play no role in learning. (ELECTRA uses all positions.)
  9. What is Next Sentence Prediction (NSP), and what is it for?
    A second BERT objective: given a pair of sentences, predict whether they are actually adjacent in the corpus or the second was randomly chosen (50/50). It targets sentence-pair relationships: paraphrase, entailment, discourse coherence.
  10. What special tokens and embeddings does BERT add for sentence pairs?
    [CLS] prepended to the input, [SEP] between the sentences and at the end, and segment embeddings (first/second) added to the token and position embeddings.
  11. How is NSP predicted?
    From the final-layer vector of the [CLS] token, with a learned classification head: \(y = \text{softmax}(h^L_{CLS}W_{NSP})\), trained with cross-entropy, combined with the MLM loss.
  12. Why does the [CLS] representation tend to capture the whole sequence?
    Because it is the direct input to the NSP classifier, training pushes it to contain information about the sequence as a whole.
  13. How does RoBERTa’s training regime differ from BERT’s?
    It drops next sentence prediction: inputs are simply contiguous sentences (starting with [CLS]), packed across document boundaries with separators up to the context length, with large batches.
  14. How is the training data sampled to build a multilingual tokenizer vocabulary?
    Not uniformly (which would favor English), but by reweighting each language’s share: \(q_i = \frac{p_i^\alpha}{\sum_j p_j^\alpha}\) with \(p_i\) its proportion of sentences and \(\alpha\) about 0.3, which upweights less-represented languages.
  15. What are the advantages of a multilingual encoder over many monolingual ones?
    One model instead of a hundred, and better performance on low-resource languages by leveraging similar higher-resource languages in the training data.
  16. What is the curse of multilinguality?
    When the number of languages grows very large, performance on each language degrades compared with a model trained on fewer languages.
  17. What does it mean that multilingual models “have an accent”?
    Grammatical structures of higher-resource languages (often English) bleed into the representations of lower-resource languages, making them slightly more English-like.

Source