learn.ifs.name

Distributed Representations of Words and Phrases and their Compositionality

Mikolov et al., 2013, Distributed Representations of Words and Phrases and their Compositionality (Advances in Neural Information Processing Systems 26 (NIPS 2013))

Subscriber deck — free during early access

4 cards in “1 Introduction” · whole deck

1 Introduction

  1. What does Mikolov et al.’s 2013 paper add to the Skip-gram model?
    Subsampling of frequent words (2–10× faster training and better vectors for rare words), negative sampling as a simple alternative to hierarchical softmax, and a data-driven method to find phrases and learn vectors for millions of them, plus the observation that vectors compose by addition.
  2. Why is Skip-gram training so efficient?
    Unlike earlier neural architectures for word vectors, it involves no dense matrix multiplications (no hidden non-linear layer); an optimized single-machine implementation could train on more than 100 billion words a day.
  3. What linear regularities do Skip-gram vectors show?
    Many relations are approximately constant vector offsets: vec(“Madrid”) − vec(“Spain”) + vec(“France”) is closer to vec(“Paris”) than to any other word vector.
    SpainMadridFrancePariscapital-of
    A schematic 2-D picture: the country→capital offset is roughly the same vector
  4. Why are word vectors not enough for idiomatic phrases?
    Some phrases aren’t compositions of their words’ meanings: “Boston Globe” is a newspaper, not a combination of Boston and Globe, and “Air Canada” isn’t Air + Canada. Representing such phrases by their own vectors makes the model more expressive.

Source