learn.ifs.name

Masked Language Models

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 9

7 cards in “9.2 Bidirectional transformer encoders” · whole deck

9.2 Bidirectional transformer encoders

  1. For what kinds of tasks is the left-to-right restriction of decoders a limitation?
    Tasks where processing a token benefits from future tokens, especially sequence labeling: named entity tagging, part-of-speech tagging, parsing.
  2. What is the focus of a bidirectional encoder, compared with a causal model?
    Computing contextualized representations of the input tokens, using the entire input sequence, rather than generating the next token.
  3. Why are masked LMs called encoder-only?
    They produce an encoding for each input token but generally aren’t used to produce running text by decoding/sampling.
  4. In what two ways do bidirectional encoders differ from causal transformers?
    Attention isn’t causal (a token can attend to following tokens), and training predicts tokens in the middle of the text rather than at the end.
  5. How is bidirectional attention implemented?
    By simply removing the causal mask: \(\text{head} = \text{softmax}\big(\frac{QK^\top}{\sqrt{d_k}}\big)V\). Everything else in the transformer block stays the same.
  6. What tokenization do BERT-style encoders use, and what complication does it cause?
    Subword tokens (BPE, WordPiece or SentencePiece Unigram LM). For tasks that need words, like parsing, subwords must sometimes be mapped back to words.
  7. How large are masked language models compared to causal LLMs?
    Much smaller: typically orders of magnitude fewer parameters than large causal LLMs.

Source