learn.ifs.name

Embeddings

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5

8 cards in “5.4 Cosine for measuring similarity” · whole deck

5.4 Cosine for measuring similarity

  1. Why does the dot product act as a similarity metric?
    It is high when two vectors have large values in the same dimensions; orthogonal vectors (nonzero in different dimensions) have a dot product of 0.
  2. What is the problem with the raw dot product as a similarity metric?
    It favors long vectors. Frequent words co-occur with more words and with higher counts, so they have longer vectors and higher dot products regardless of meaning.
  3. Define the vector length \(|v|\).
    \(|v| = \sqrt{\sum_{i=1}^{N} v_i^2}\)
  4. Define cosine similarity.
    The dot product normalized by the vector lengths, equal to the cosine of the angle between the vectors: \(\cos(v, w) = \frac{v \cdot w}{|v||w|} = \frac{\sum_i v_i w_i}{\sqrt{\sum_i v_i^2}\sqrt{\sum_i w_i^2}}\).
    “computer” →“pie”cherrydigitalinformationlarge angle
    Small angle (high cosine) between digital and information; large angle to cherry.
  5. What is a unit vector, and why is it useful?
    A vector divided by its length, so it has length 1. For unit vectors the dot product equals the cosine.
  6. What is the range of the cosine, and why is it 0–1 for count vectors?
    From 1 (same direction) through 0 (orthogonal) to −1 (opposite). Raw counts are non-negative, so cosines between count vectors lie in [0, 1].
  7. Is cosine similarity affected by scaling a vector, and why does that matter?
    No: \(\cos(cv, w) = \cos(v, w)\) for any \(c > 0\). That is what we want, since it removes the effect of word frequency on vector length.
  8. How can cosine similarity find the words most similar to a target word?
    Compute the cosine between the target and every other word in the vocabulary, sort, and take the top 10. Uses include finding paraphrases, tracking meaning change, and discovering word meanings in different corpora.

Source