Masked Language Models

A training paradigm for bidirectional encoders (e.g. BERT): instead of predicting the next word, a word in the middle is masked and the model guesses it from the words on both sides, so it sees both left and right context.

Add a neural classifier on top of the pretrained network’s top layer and train it on labeled data for the task (e.g. named entity tagging, natural language inference).

Acquiring knowledge from one task or domain and applying (transferring) it to a new task, as in the pretrain–fine-tune paradigm.

The decoder, the encoder, and the encoder-decoder.

It takes input tokens and generates output tokens one at a time, with information flowing left to right: a generative model. GPT, Claude, Llama and Mistral are decoders.

It takes a token sequence and outputs a vector representation for each token. Encoders are usually masked LMs (BERT, RoBERTa); they aren’t used to generate text but are fine-tuned as classifiers (sentiment, topic…).

It maps input tokens to output tokens with only a loose relationship between them: outputs can come from a very different token set and be much longer or shorter. Used for machine translation and speech recognition.

Tasks where processing a token benefits from future tokens, especially sequence labeling: named entity tagging, part-of-speech tagging, parsing.

Computing contextualized representations of the input tokens, using the entire input sequence, rather than generating the next token.

They produce an encoding for each input token but generally aren’t used to produce running text by decoding/sampling.

Attention isn’t causal (a token can attend to following tokens), and training predicts tokens in the middle of the text rather than at the end.

By simply removing the causal mask: \(\text{head} = \text{softmax}\big(\frac{QK^\top}{\sqrt{d_k}}\big)V\). Everything else in the transformer block stays the same.

Subword tokens (BPE, WordPiece or SentencePiece Unigram LM). For tasks that need words, like parsing, subwords must sometimes be mapped back to words.

Much smaller: typically orders of magnitude fewer parameters than large causal LLMs.

Without the causal mask, the next word is directly visible in the context, so the task is trivial.

A fill-in-the-blank task: given a sequence with one or more elements missing, predict the missing elements (“The ___ of Walden Pond is so beautifully…”).

Corrupting the input (masking, substituting, reordering, deleting, inserting words) and training the model to recover the original.

A sample of tokens (15% in BERT) is chosen. Of these, 80% are replaced with [MASK], 10% with a random token sampled by unigram probability, and 10% are left unchanged. The model must predict the original token at each chosen position.

“So long and thanks for all the fish”

chosen token	model input
long	[MASK]
thanks	[MASK]
the	apricot (random)

[MASK] never appears at fine-tuning or inference time. If the model only predicted at [MASK] positions it would learn to predict only there; random replacements and unchanged tokens make it try to predict every input token.

The final layer’s output \(h^L_i\) for that position goes through the LM head: logits \(u_i = h^L_i E^\top\), probabilities \(y_i = \text{softmax}(u_i)\).

The average negative log probability of the correct tokens over the set \(M\) of chosen positions: \(L_{MLM} = -\frac{1}{|M|}\sum_{i \in M}\log P(x_i \mid h^L_i)\).

Only the chosen tokens (about 15%) contribute to the loss; the others play no role in learning. (ELECTRA uses all positions.)

A second BERT objective: given a pair of sentences, predict whether they are actually adjacent in the corpus or the second was randomly chosen (50/50). It targets sentence-pair relationships: paraphrase, entailment, discourse coherence.

[CLS] prepended to the input, [SEP] between the sentences and at the end, and segment embeddings (first/second) added to the token and position embeddings.

From the final-layer vector of the [CLS] token, with a learned classification head: \(y = \text{softmax}(h^L_{CLS}W_{NSP})\), trained with cross-entropy, combined with the MLM loss.

Because it is the direct input to the NSP classifier, training pushes it to contain information about the sequence as a whole.

It drops next sentence prediction: inputs are simply contiguous sentences (starting with [CLS]), packed across document boundaries with separators up to the context length, with large batches.

Not uniformly (which would favor English), but by reweighting each language’s share: \(q_i = \frac{p_i^\alpha}{\sum_j p_j^\alpha}\) with \(p_i\) its proportion of sentences and \(\alpha\) about 0.3, which upweights less-represented languages.

One model instead of a hundred, and better performance on low-resource languages by leveraging similar higher-resource languages in the training data.

When the number of languages grows very large, performance on each language degrades compared with a model trained on fewer languages.

Grammatical structures of higher-resource languages (often English) bleed into the representations of lower-resource languages, making them slightly more English-like.

Prompting the model in natural language so it generates what we want, or fine-tuning: adding application-specific circuitry (a head) on top and training it on labeled data, freezing or only minimally adjusting the pretrained parameters.

Pass the text through the encoder, take the final-layer [CLS] vector (or the mean of the token outputs), and feed it to a classifier head: \(y = \text{softmax}(h^L_{CLS}W_C)\), with \(W_C\) of shape \([d \times \text{classes}]\), trained with cross-entropy.

They can be, but reasonable performance usually needs only minimal changes, often limited to the last few transformer layers.

As in NSP training: input [CLS] sentence A [SEP] sentence B [SEP], and classify the final [CLS] vector with a learned head.

Also called recognizing textual entailment: given a premise and a hypothesis, classify their relationship as entails, contradicts or neutral, in the loose sense of how a typical human would read them.

Entails.

MultiNLI (Multi-Genre Natural Language Inference).

Assigning each token a label from a small fixed set, e.g. named entity recognition or part-of-speech tagging.

Roughly anything referred to by a proper name: a person, location, organization. Named entity recognition finds the spans of text that are proper names and tags their type. The term is often extended to dates, times and prices.

PER (person), LOC (location), ORG (organization), GPE (geo-political entity).

Linking text to knowledge sources like Wikipedia, measuring sentiment toward particular entities, and anonymizing text for privacy.

Segmentation ambiguity (which tokens form an entity span, when most words aren’t entities) and type ambiguity: Washington can be a person, a sports team (ORG), a city (LOC) or a government (GPE).

A way to turn span recognition into per-token labeling: B marks the first token of a span, I a token inside a span, O a token outside any span, with B and I tags per entity type (\(2n + 1\) tags for \(n\) types).

word	BIO	BIOES
Jane	B-PER	B-PER
Villanueva	I-PER	E-PER
of	O	O
United	B-ORG	B-ORG
Airlines	I-ORG	E-ORG
Chicago	B-LOC	S-LOC

IO drops the B tag, losing information about where adjacent spans begin. BIOES adds E for the end of a span and S for a single-token span.

Pass each token’s final output vector through a classifier \(y_i = \text{softmax}(h^L_iW_K)\) over the \(k\) tags and take the argmax per token, or pass the distributions to a CRF layer that models tag transitions.

Training: give every subword token the tag of the word it came from. Decoding: use the tag predicted for the first subword of each word (or combine the subwords’ tag distributions).

Precision, recall and F1 (with paired bootstrap tests for significance), counting entities, not words, as units: Jane Villanueva is one response.

Labeling only Jane instead of Jane Villanueva as PER causes two errors (a false positive and a false negative), and training is per word while evaluation is per entity.

A word sense is a discrete representation of one aspect of a word’s meaning; contextual embeddings give a richer continuous, high-dimensional model of meaning in context.

By the cosine between their contextual embeddings, although some transformations of the embeddings are needed first.