RNNs and LSTMs

14.2 RNNs as language models

1. What advantage do RNN language models have over n-gram and feedforward LMs?
   No limited or fixed context window: the hidden state can in principle represent information about all preceding words back to the start of the sequence.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.2, p. 306
2. Write the forward computation of an RNN language model.
   \(e_t = Ex_t\) (embedding of the one-hot input), \(h_t = g(Uh_{t-1} + We_t)\), \(\hat{y}_t = \text{softmax}(Vh_t)\); \(P(w_{t+1} = k \mid w_{1:t}) = \hat{y}_t[k]\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.2, pp. 306–307
3. How is the probability of a whole sequence computed with an RNN LM?
   The product of the probabilities of each true word: \(P(w_{1:n}) = \prod_i \hat{y}_i[w_i]\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.2, p. 307
4. Why is RNN language-model training called self-supervised?
   The text supplies its own labels: at each step the model is trained (with cross-entropy) to predict the true next word, with no added gold annotation.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.2, p. 307
5. What is teacher forcing in training an RNN LM?
   At each step the model is given the correct previous word (with the hidden state) rather than its own prediction from the previous step.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.2, p. 308
6. What is weight tying in an RNN language model, and what does it gain?
   Using the embedding matrix \(E\) both at the input and, transposed, in place of the output matrix \(V\): \(\hat{y}_t = \text{softmax}(E^\top h_t)\). Both encode word embeddings; tying improves perplexity and greatly reduces parameters.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.2, p. 308