RNNs and LSTMs

14.5 The LSTM

1. Why is “The flights the airline was canceling were full” hard for an RNN LM?
   Predicting were requires remembering the distant plural flights, while the closer singular airline intervenes; RNN hidden states tend to encode mostly local information.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.5, pp. 314–315
2. What two problems make simple RNNs bad at using distant information?
   The hidden layer must both provide information for the current decision and carry forward information for future ones; and backpropagation through time multiplies gradients repeatedly, driving them toward zero (vanishing gradients).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.5, p. 315
3. How do LSTMs manage context?
   Long short-term memory networks (Hochreiter & Schmidhuber, 1997) add an explicit context (cell) layer and learn, via gates, to remove information no longer needed and add information likely to be needed later.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.5, p. 315
4. What is the common design of an LSTM gate, and why the sigmoid?
   A feedforward layer, then a sigmoid, then element-wise multiplication with the layer being gated. The sigmoid pushes values toward 0 or 1, so the product acts like a binary mask: values passed nearly unchanged or erased.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.5, p. 316
5. What does the forget gate do? Write its equations.
   It deletes information from the context that is no longer needed: \(f_t = \sigma(U_fh_{t-1} + W_fx_t)\), \(k_t = c_{t-1} \odot f_t\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.5, p. 316
6. What is the Hadamard product?
   Element-wise multiplication of two vectors of the same dimension, written \(\odot\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.5, p. 316
7. How does an LSTM add new information to its context?
   Compute candidate information \(g_t = \tanh(U_gh_{t-1} + W_gx_t)\), mask it with the add (input) gate \(i_t = \sigma(U_ih_{t-1} + W_ix_t)\), \(j_t = g_t \odot i_t\), and add it to the filtered context: \(c_t = j_t + k_t\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.5, p. 316
8. What does the output gate do?
   Decides what information the current hidden state needs (as opposed to what to preserve for the future): \(o_t = \sigma(U_oh_{t-1} + W_ox_t)\), \(h_t = o_t \odot \tanh(c_t)\).

   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.5, p. 316
9. What are the inputs and outputs of an LSTM unit?
   Inputs: current input \(x_t\), previous hidden state \(h_{t-1}\), previous context \(c_{t-1}\). Outputs: new hidden state \(h_t\) (the unit’s output) and updated context \(c_t\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.5, pp. 316–317
10. Why is the LSTM’s complexity not a problem for building architectures?
    It is encapsulated within the unit: externally an LSTM differs from a simple recurrent unit only by the extra context vector, so LSTM (or GRU) units can be swapped into any RNN architecture (stacked, bidirectional) and trained by unrolling.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.5, p. 317
11. Which recurrent unit became standard in practice?
    LSTMs (rather than simple RNNs) for any modern system using recurrent networks.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.5, p. 318