learn.ifs.name

Masked Language Models

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 9

7 cards in “9.4 Fine-tuning for classification” · whole deck

9.4 Fine-tuning for classification

  1. What are the two ways to use pretrained LMs for downstream tasks?
    Prompting the model in natural language so it generates what we want, or fine-tuning: adding application-specific circuitry (a head) on top and training it on labeled data, freezing or only minimally adjusting the pretrained parameters.
  2. How is an encoder fine-tuned for sequence classification, e.g. sentiment?
    Pass the text through the encoder, take the final-layer [CLS] vector (or the mean of the token outputs), and feed it to a classifier head: \(y = \text{softmax}(h^L_{CLS}W_C)\), with \(W_C\) of shape \([d \times \text{classes}]\), trained with cross-entropy.
  3. During classification fine-tuning, are the pretrained weights updated?
    They can be, but reasonable performance usually needs only minimal changes, often limited to the last few transformer layers.
  4. How is sequence-pair classification done with an encoder?
    As in NSP training: input [CLS] sentence A [SEP] sentence B [SEP], and classify the final [CLS] vector with a learned head.
  5. What is natural language inference (NLI)?
    Also called recognizing textual entailment: given a premise and a hypothesis, classify their relationship as entails, contradicts or neutral, in the loose sense of how a typical human would read them.
  6. Classify: premise “I’m confused.”, hypothesis “Not all of it is very clear to me.”
    Entails.
  7. Name a dataset for natural language inference.
    MultiNLI (Multi-Genre Natural Language Inference).

Source