Embeddings

Words that occur in similar contexts tend to have similar meanings; the difference in meaning between two words corresponds roughly to the difference in their environments. Formulated in the 1950s by linguists Joos, Harris and Firth.

Vector representations of word meaning learned directly from word distributions in texts.

The field that studies embeddings and their meanings: representing word meaning as points in a semantic space derived from word distributions.

The useful representations of the input are learned automatically (self-supervised) from text, instead of being designed by hand through feature engineering.

Static embeddings (this chapter) give each word one fixed vector. Contextualized (dynamic) embeddings like BERT give a different vector for each occurrence, depending on its context.

It says nothing about meaning: it can’t tell us that cat is similar to dog, cold is the opposite of hot, happy is positive, or that buy, sell and pay describe one event from different perspectives.

The linguistic study of word meaning.

The lemma (citation form) is the dictionary form: mouse for mice, sing for sang, dormir for Spanish duermes. Wordforms are the specific inflected forms that occur.

A sense is one aspect of a word’s meaning (mouse the rodent vs the cursor device). A polysemous lemma has multiple senses; choosing the right one in context is word sense disambiguation.

Two senses are synonyms if their meanings are (nearly) identical (couch/sofa, car/automobile). Formally, two words are synonymous if they are substitutable in any sentence without changing its truth conditions.

Any difference in linguistic form comes with some difference in meaning (e.g. H₂O is scientific, water fits a hiking guide). So true synonymy hardly exists; “synonym” really means approximate synonymy.

Few words have synonyms, but most have many similar words (cat and dog). Similarity is a graded relation between words rather than between senses, which avoids committing to a representation of senses.

Ask people to rate how similar word pairs are. SimLex-999 rates pairs on a 0–10 scale, from near-synonyms (vanish/disappear) to unrelated pairs (hole/agreement).

Words can be related without being similar: coffee and cup share no features but co-participate in an everyday event; scalpel and surgeon are related eventively.

A set of words covering a semantic domain and bearing structured relations to each other, e.g. hospitals (surgeon, scalpel, nurse, anesthetic) or restaurants (waiter, menu, plate, chef).

Unsupervised methods like Latent Dirichlet Allocation (LDA) that induce sets of associated words (akin to semantic fields) from large text collections, useful for discovering topical structure in documents.

The aspects of a word’s meaning related to emotions, sentiment, opinions or evaluations: fake/knockoff/forgery vs copy/replica/reproduction, innocent vs naive.

Valence (pleasantness), arousal (intensity of emotion provoked) and dominance (degree of control exerted). Happy is high valence, excited high arousal, controlling high dominance.

Representing each word by three numbers made a word a point in a three-dimensional space: the first expression of the idea that meaning can be a vector.

Osgood’s representation of connotation as a point in space, and the distributional idea (Joos, Harris, Firth) of defining a word by its distribution: its neighboring words or grammatical environments.

It occurs with words like garlic, rice, delicious, salty, leaves, as do spinach, chard and collard greens. Computationally: count the words in its context.

A classifier on word strings needs the same words in training and test. With embeddings it can assign sentiment as long as it has seen words with similar meanings, which lie nearby in space.

Sparse vectors (count-based) are long (dimension \(|V|\)) and mostly zeros. Dense vectors (e.g. word2vec) are short (50–1000 dimensions) and contain real values, possibly negative.

A \(|V| \times |V|\) matrix with a row per target word and a column per context word; each cell counts how often the context word occurs near the target (e.g. within ±4 words) in a training corpus.

	a	computer	pie
cherry	1	0	1
strawberry	0	0	2
digital	0	1	0
information	1	1	0

Each row is that word’s vector.

A collection of vectors characterized by its dimension: the number of elements each vector has. In a word-word matrix each word is a point in \(|V|\)-dimensional space.

Cherry and strawberry both have high counts for pie and sugar; digital and information both for computer and data. Each pair is closer to each other than to the other pair.

Their dimension is the vocabulary size, typically 10,000–50,000 most frequent words. Most entries are zero because most words never occur near each other.

tf-idf (introduced in Chapter 11); historically many other weightings too (e.g. PPMI).

It is high when two vectors have large values in the same dimensions; orthogonal vectors (nonzero in different dimensions) have a dot product of 0.

It favors long vectors. Frequent words co-occur with more words and with higher counts, so they have longer vectors and higher dot products regardless of meaning.

\(|v| = \sqrt{\sum_{i=1}^{N} v_i^2}\)

The dot product normalized by the vector lengths, equal to the cosine of the angle between the vectors: \(\cos(v, w) = \frac{v \cdot w}{|v||w|} = \frac{\sum_i v_i w_i}{\sqrt{\sum_i v_i^2}\sqrt{\sum_i w_i^2}}\).

A vector divided by its length, so it has length 1. For unit vectors the dot product equals the cosine.

From 1 (same direction) through 0 (orthogonal) to −1 (opposite). Raw counts are non-negative, so cosines between count vectors lie in [0, 1].

No: \(\cos(cv, w) = \cos(v, w)\) for any \(c > 0\). That is what we want, since it removes the effect of word frequency on vector length.

Compute the cosine between the target and every other word in the vocabulary, sort, and take the top 10. Uses include finding paraphrases, tracking meaning change, and discovering word meanings in different corpora.

Not fully understood, but: classifiers need far fewer weights with 300 dimensions than 50,000 (helping generalization and avoiding overfitting), and dense vectors better capture synonymy, since in sparse vectors car and automobile are unrelated dimensions.

Skip-gram with negative sampling, one of the two algorithms in the word2vec package (the other is CBOW), often loosely called word2vec. It is fast, efficient and widely available with pretrained embeddings.

Instead of counting co-occurrences, train a binary classifier on “Is word \(c\) likely to show up near word \(w\)?”, then throw away the task and keep the learned classifier weights as the embeddings.

Running text supplies its own supervision: a word that actually occurs near the target is a gold positive answer, so no hand labeling is needed. The idea comes from neural language models (Bengio et al., 2003; Collobert et al., 2011).

It simplifies the task (binary classification instead of word prediction) and the architecture (logistic regression instead of a multi-layer network).

(1) Treat the target word and a neighboring context word as positive examples. (2) Randomly sample other words as negative examples. (3) Train logistic regression to distinguish the two. (4) Use the learned weights as embeddings.

From the similarity of their embeddings, measured by the dot product and turned into a probability by the sigmoid: \(P(+ \mid w, c) = \sigma(c \cdot w) = \frac{1}{1 + \exp(-c \cdot w)}\), and \(P(- \mid w, c) = \sigma(-c \cdot w)\).

It assumes they are independent and multiplies their probabilities: \(P(+ \mid w, c_{1:L}) = \prod_{i=1}^{L}\sigma(c_i \cdot w)\), or in logs \(\sum_i \log\sigma(c_i \cdot w)\).

A target (input) embedding in matrix \(W\) and a context (output) embedding in matrix \(C\). The parameters \(\theta\) are these \(2|V|\) vectors of dimension \(d\).

Starting from random vectors, it iteratively shifts each word’s embedding to be more like the embeddings of words that occur nearby and less like those of words that don’t.

(apricot, tablespoon), (apricot, of), (apricot, jam), (apricot, a): one per context word in the ±2 window.

For each positive pair, sample \(k\) noise words (not equal to the target) and pair them with the target, e.g. (apricot, aardvark), (apricot, seven).

From the weighted unigram distribution \(P_\alpha(w) = \frac{\text{count}(w)^\alpha}{\sum_{w'}\text{count}(w')^\alpha}\) with \(\alpha = 0.75\). This gives rare words a bit more probability than their raw frequency (e.g. .01 becomes about .03).

Maximize the similarity (dot product) of target–context pairs from positive examples, and minimize the similarity of target–noise pairs from negative examples.

\(L = -\Big[\log\sigma(c_{pos} \cdot w) + \sum_{i=1}^{k}\log\sigma(-c_{neg_i} \cdot w)\Big]\)

It moves the target embedding of apricot and the context embedding of jam closer (higher dot product), and moves apricot away from the noise words’ context embeddings (lower dot product).

\(\frac{\partial L}{\partial c_{pos}} = [\sigma(c_{pos} \cdot w) - 1]w\), \(\frac{\partial L}{\partial c_{neg_i}} = \sigma(c_{neg_i} \cdot w)\,w\), and \(\frac{\partial L}{\partial w} = [\sigma(c_{pos} \cdot w) - 1]c_{pos} + \sum_i \sigma(c_{neg_i} \cdot w)\,c_{neg_i}\).

Commonly by the sum of its target and context embeddings, \(w_i + c_i\), or by the target embedding \(w_i\) alone (discarding \(C\)).

Subword models: each word is represented as itself plus a bag of character n-grams with boundary symbols (for where, n = 3: <wh, whe, her, ere, re>), with an embedding learned per n-gram. Unknown or rare words (e.g. in morphologically rich languages) get the sum of their n-gram embeddings.

Global Vectors: a widely used static embedding model based on ratios of probabilities from the global word-word co-occurrence matrix.

Word2vec can be seen as implicitly optimizing a function of a count matrix with a particular (PPMI) weighting (Levy & Goldberg, 2014).

List a word’s nearest neighbors by cosine (e.g. frog → frogs, toad, rana, lizard…); hierarchical clustering of embedding vectors; and projecting the high-dimensional vectors down to 2 dimensions with a method like t-SNE.

Short windows (±1–3) give more syntactic representations: the nearest words are semantically similar words of the same part of speech. Long windows give topically related words that aren’t necessarily similar.

±2: other fictional schools (Sunnydale, Evernight). ±5: words from the Harry Potter topic (Dumbledore, Malfoy, half-blood).

First-order (syntagmatic) association: words typically near each other (wrote and book). Second-order (paradigmatic) association: words with similar neighbors (wrote and said, remarked).

For “a is to b as a* is to ?” (apple : tree :: grape : ?), add the offset \(b - a\) to \(a^*\) and return the nearest word (vine): \(\hat{b}^* = \arg\min_x \text{distance}(x, b - a + a^*)\).

king − man + woman ≈ queen; Paris − France + Italy ≈ Rome. Offsets also seem to capture relations like comparative/superlative morphology.

The nearest vector is usually one of the input words or a morphological variant (cherry : red :: potato : potato), so those must be excluded; it works for frequent words, small distances and certain relations (capitals, inflections) but poorly for others, and is probably too simple a model of human analogy.

Train separate embedding spaces on texts from each period (e.g. per decade) and compare a word’s neighbors over time: gay moved from “cheerful” to “homosexual”, broadcast from sowing seeds to transmitting signals, awful from “full of awe” to “terrible” (pejoration).

In word2vec trained on news, computer programmer − man + woman is closest to homemaker, and father : doctor :: mother : nurse.

When a system allocates resources (jobs, credit) unfairly to different groups, e.g. a hiring search using biased embeddings that downweights documents with women’s names.

Embeddings don’t just reflect input statistics but exaggerate them: gendered terms become more gendered in embedding space than in the text, and biases exceed actual employment statistics.

The IAT measures people’s associations between concepts and attributes by response latencies. Caliskan et al. replicated its findings using GloVe cosines instead of latencies, e.g. African American names closer to unpleasant words, European American names to pleasant ones.

Methods transform the space to remove stereotypes while keeping definitional gender, or change the training procedure. They reduce but do not eliminate bias; it remains an open problem.

Cosines between occupation words and gendered or ethnic names in each decade correlate with historical occupation percentages, replicate 1930s stereotype surveys, and show competence adjectives being closer to male words, a bias slowly decreasing since 1960.

Extrinsic: whether the vectors improve performance on an NLP task. Intrinsic: the correlation between the model’s word similarity scores (e.g. cosine) and human similarity ratings.

Out of context: WordSim-353, SimLex-999 (similarity rather than relatedness), TOEFL synonym questions. In context: SCWS (Stanford Contextual Word Similarity) and WiC (Word-in-Context). Sentence level: semantic textual similarity (STS). Plus analogy datasets (morphological, lexicographic, encyclopedic relations, SemEval-2012 Task 2).

Embedding algorithms are inherently variable (random initialization, random negative sampling), and individual documents can strongly affect results. Best practice: train multiple embeddings with bootstrap sampling over documents and average.

Linguistics (distributionalists Harris, Joos, Firth: meaning from distribution), psychology (Osgood: meaning as a point in space, similarity as distance), and mechanical indexing, i.e. information retrieval (the vector space model, idf, mutual information).

“The meaning of a word is its use in the language”: words should be defined by how people use them rather than by formal definitions or truth values.

Symbols for primitive meanings used to define words (e.g. hen = +female, +chicken, +adult; chick = +chicken, −adult), from Hjelmslev and early generative grammar.

Latent semantic analysis: apply singular value decomposition (SVD) to a weighted term-document matrix and keep the top ~300 dimensions as dense embeddings. It was used as a cognitive model and for many tasks.

In LSA it meant the mapping from sparse count space to the dense latent space; it shifted metonymically to mean the resulting dense vector itself.

Mikolov et al. (2013a) proposed skip-gram and CBOW (simplifying neural LM hidden layers); negative sampling was proposed in Mikolov et al. (2013b).