learn.ifs.name

Embeddings

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5

6 cards in “5.3 Simple count-based embeddings” · whole deck

5.3 Simple count-based embeddings

  1. What is a word-context (co-occurrence) matrix?
    A \(|V| \times |V|\) matrix with a row per target word and a column per context word; each cell counts how often the context word occurs near the target (e.g. within ±4 words) in a training corpus.
  2. Build co-occurrence vectors over the contexts a, computer, pie from: “…cherry pie, a traditional dessert”, “…strawberry rhubarb pie. Apple pie”, “…personal digital assistants…”, “a computer. This includes information available…”. (Treat the windows as given.)
    acomputerpie
    cherry101
    strawberry002
    digital010
    information110
    Each row is that word’s vector.
  3. What is a vector space, and what is its dimension?
    A collection of vectors characterized by its dimension: the number of elements each vector has. In a word-word matrix each word is a point in \(|V|\)-dimensional space.
  4. What do real co-occurrence counts from a large corpus show for cherry, strawberry, digital, information?
    Cherry and strawberry both have high counts for pie and sugar; digital and information both for computer and data. Each pair is closer to each other than to the other pair.
  5. How large are count-based vectors, and why are they sparse?
    Their dimension is the vocabulary size, typically 10,000–50,000 most frequent words. Most entries are zero because most words never occur near each other.
  6. What weighting is commonly applied to co-occurrence counts?
    tf-idf (introduced in Chapter 11); historically many other weightings too (e.g. PPMI).

Source