learn.ifs.name

Embeddings

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5

21 cards in “5.5 Word2vec” · whole deck

5.5 Word2vec

  1. Why do dense embeddings work better than sparse vectors?
    Not fully understood, but: classifiers need far fewer weights with 300 dimensions than 50,000 (helping generalization and avoiding overfitting), and dense vectors better capture synonymy, since in sparse vectors car and automobile are unrelated dimensions.
  2. What is SGNS, and how does it relate to word2vec?
    Skip-gram with negative sampling, one of the two algorithms in the word2vec package (the other is CBOW), often loosely called word2vec. It is fast, efficient and widely available with pretrained embeddings.
  3. What is the core intuition of word2vec?
    Instead of counting co-occurrences, train a binary classifier on “Is word \(c\) likely to show up near word \(w\)?”, then throw away the task and keep the learned classifier weights as the embeddings.
  4. What is self-supervision in word2vec?
    Running text supplies its own supervision: a word that actually occurs near the target is a gold positive answer, so no hand labeling is needed. The idea comes from neural language models (Bengio et al., 2003; Collobert et al., 2011).
  5. In what two ways is word2vec simpler than a neural language model?
    It simplifies the task (binary classification instead of word prediction) and the architecture (logistic regression instead of a multi-layer network).
  6. Summarize the four steps of skip-gram.
    (1) Treat the target word and a neighboring context word as positive examples. (2) Randomly sample other words as negative examples. (3) Train logistic regression to distinguish the two. (4) Use the learned weights as embeddings.
  7. How does skip-gram compute the probability that \(c\) is a real context word of \(w\)?
    From the similarity of their embeddings, measured by the dot product and turned into a probability by the sigmoid: \(P(+ \mid w, c) = \sigma(c \cdot w) = \frac{1}{1 + \exp(-c \cdot w)}\), and \(P(- \mid w, c) = \sigma(-c \cdot w)\).
  8. How does skip-gram handle the several context words in a window?
    It assumes they are independent and multiplies their probabilities: \(P(+ \mid w, c_{1:L}) = \prod_{i=1}^{L}\sigma(c_i \cdot w)\), or in logs \(\sum_i \log\sigma(c_i \cdot w)\).
  9. What two embeddings does skip-gram learn for each word?
    A target (input) embedding in matrix \(W\) and a context (output) embedding in matrix \(C\). The parameters \(\theta\) are these \(2|V|\) vectors of dimension \(d\).
  10. What does the skip-gram learning algorithm do to embeddings, intuitively?
    Starting from random vectors, it iteratively shifts each word’s embedding to be more like the embeddings of words that occur nearby and less like those of words that don’t.
  11. For the window “lemon, a [tablespoon of apricot jam, a] pinch”, what are the positive training examples?
    (apricot, tablespoon), (apricot, of), (apricot, jam), (apricot, a): one per context word in the ±2 window.
  12. How are negative examples created in SGNS?
    For each positive pair, sample \(k\) noise words (not equal to the target) and pair them with the target, e.g. (apricot, aardvark), (apricot, seven).
  13. How are noise words chosen, and why use the exponent 0.75?
    From the weighted unigram distribution \(P_\alpha(w) = \frac{\text{count}(w)^\alpha}{\sum_{w'}\text{count}(w')^\alpha}\) with \(\alpha = 0.75\). This gives rare words a bit more probability than their raw frequency (e.g. .01 becomes about .03).
  14. What are the two goals of the skip-gram learning algorithm?
    Maximize the similarity (dot product) of target–context pairs from positive examples, and minimize the similarity of target–noise pairs from negative examples.
  15. Write the SGNS loss for one positive pair and \(k\) noise words.
    \(L = -\Big[\log\sigma(c_{pos} \cdot w) + \sum_{i=1}^{k}\log\sigma(-c_{neg_i} \cdot w)\Big]\)
  16. What does one gradient step of skip-gram do for “…apricot jam…”?
    It moves the target embedding of apricot and the context embedding of jam closer (higher dot product), and moves apricot away from the noise words’ context embeddings (lower dot product).
    apricotjammatrixTolstoycontext: closernoise: apart
  17. What are the SGNS gradients?
    \(\frac{\partial L}{\partial c_{pos}} = [\sigma(c_{pos} \cdot w) - 1]w\), \(\frac{\partial L}{\partial c_{neg_i}} = \sigma(c_{neg_i} \cdot w)\,w\), and \(\frac{\partial L}{\partial w} = [\sigma(c_{pos} \cdot w) - 1]c_{pos} + \sum_i \sigma(c_{neg_i} \cdot w)\,c_{neg_i}\).
  18. After training skip-gram, how is each word represented?
    Commonly by the sum of its target and context embeddings, \(w_i + c_i\), or by the target embedding \(w_i\) alone (discarding \(C\)).
  19. What does fasttext add to word2vec, and why?
    Subword models: each word is represented as itself plus a bag of character n-grams with boundary symbols (for where, n = 3: <wh, whe, her, ere, re>), with an embedding learned per n-gram. Unknown or rare words (e.g. in morphologically rich languages) get the sum of their n-gram embeddings.
  20. What is GloVe?
    Global Vectors: a widely used static embedding model based on ratios of probabilities from the global word-word co-occurrence matrix.
  21. How are dense embeddings like word2vec related to count-based embeddings?
    Word2vec can be seen as implicitly optimizing a function of a count matrix with a particular (PPMI) weighting (Levy & Goldberg, 2014).

Source