Masked Language Models

Introduction

1. What is masked language modeling?
   A training paradigm for bidirectional encoders (e.g. BERT): instead of predicting the next word, a word in the middle is masked and the model guesses it from the words on both sides, so it sees both left and right context.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 9, Introduction, p. 227
2. How are masked language models usually fine-tuned for a downstream task?
   Add a neural classifier on top of the pretrained network’s top layer and train it on labeled data for the task (e.g. named entity tagging, natural language inference).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 9, Introduction, p. 227
3. What is transfer learning?
   Acquiring knowledge from one task or domain and applying (transferring) it to a new task, as in the pretrain–fine-tune paradigm.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 9, Introduction, p. 227