Neural Networks

6.5 Embeddings as input to neural classifiers

1. What is the difference between static and contextual embeddings as network inputs?
   A static embedding is a fixed vector per token, trained once and looked up in a dictionary. A contextual embedding differs for each occurrence and is learned by the network as part of the task (e.g. language modeling).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.5, p. 163
2. What is the embedding matrix \(E\)?
   The dictionary of embeddings: one row per vocabulary token, shape \([|V| \times d]\). Inputs are converted to token indices, and the corresponding rows are selected.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.5, p. 164
3. How is selecting an embedding equivalent to a matrix multiplication?
   Represent the token as a one-hot vector of shape \([1 \times |V|]\); multiplying it by \(E\) selects that token’s row. A whole sequence is an \([N \times |V|]\) one-hot matrix times \(E\), giving \([N \times d]\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.5, p. 164
4. What are the two common ways of passing \(N\) token embeddings to a classifier?
   Concatenation into one long \([1 \times Nd]\) vector (keeps order and all information, but needs a big network), or pooling into a single \(d\)-dimensional vector (smaller and efficient, loses order).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.5, pp. 164–165
5. Why is pooling appropriate for sentiment classification?
   For sentiment, the identity of words (e.g. great) matters more than their exact position.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.5, p. 165
6. What are mean-pooling and max-pooling?
   Mean-pooling averages the embeddings: \(x_{mean} = \frac{1}{N}\sum_i e(w_i)\). Max-pooling takes, for each dimension, the element-wise maximum over all the inputs.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.5, p. 165
7. What are the advantages and disadvantages of neural language models compared with n-gram models?
   Advantages: much longer histories, better generalization over contexts of similar words, far more accurate prediction. Disadvantages: slower, more complex, vast energy to train, less interpretable; n-grams may still suit small tasks.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.5, p. 165
8. What does a feedforward neural language model approximate?
   Like an n-gram model, \(P(w_t \mid w_{1:t-1}) \approx P(w_t \mid w_{t-N+1:t-1})\), but representing the previous words by embeddings rather than by their identities.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.5, p. 166
9. Why can a neural LM predict fed after “…that the dog gets” having only seen “…that the cat gets fed”?
   Cat and dog have similar embeddings, so the network generalizes from the cat context. An n-gram model, which only knows word identities, cannot.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.5, pp. 166–167
10. Why does a feedforward LM concatenate rather than pool its input embeddings?
    To predict the next token it helps to know each previous token and their order, which pooling would lose.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.5, p. 167
11. Write the equations of a feedforward neural LM with a 3-word window.
    \(e = [Ex_{t-3}; Ex_{t-2}; Ex_{t-1}]\) (concatenation), \(h = \sigma(We + b)\), \(z = Uh\), \(\hat{y} = \text{softmax}(z)\): a distribution over all \(|V|\) possible next tokens.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.5, pp. 167–168