Masked Language Models

9.1 Three architectures for language models

1. What are the three common LM architectures?
   The decoder, the encoder, and the encoder-decoder.

   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 9, §9.1, p. 227
2. What does a decoder model do, and which LLMs are decoders?
   It takes input tokens and generates output tokens one at a time, with information flowing left to right: a generative model. GPT, Claude, Llama and Mistral are decoders.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 9, §9.1, p. 227
3. What does an encoder model do, and how is it typically used?
   It takes a token sequence and outputs a vector representation for each token. Encoders are usually masked LMs (BERT, RoBERTa); they aren’t used to generate text but are fine-tuned as classifiers (sentiment, topic…).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 9, §9.1, p. 227
4. What distinguishes the encoder-decoder architecture, and where is it used?
   It maps input tokens to output tokens with only a loose relationship between them: outputs can come from a very different token set and be much longer or shorter. Used for machine translation and speech recognition.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 9, §9.1, p. 228