learn.ifs.name

RNNs and LSTMs

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14

9 cards in “14.1 Recurrent neural networks” · whole deck

14.1 Recurrent neural networks

  1. What is a recurrent neural network (RNN), and what is an Elman network?
    Any network with a cycle in its connections, so some unit’s value depends (directly or indirectly) on its own earlier outputs. Elman networks (simple recurrent networks) are a constrained, effective class in which the hidden layer receives its own previous value as extra input.
  2. What does the recurrent link in an Elman network provide?
    The previous time step’s hidden layer is added to the hidden layer’s input, acting as memory or context of earlier processing, with no fixed length limit: it can carry information back to the sequence start.
  3. What new set of weights does an RNN add compared with a feedforward network?
    \(U\), connecting the previous hidden layer \(h_{t-1}\) to the current hidden layer; like the other weights, it is trained by backpropagation.
  4. Write the equations of a simple RNN.
    \(h_t = g(Uh_{t-1} + Wx_t)\) and \(y_t = f(Vh_t)\), usually \(y_t = \text{softmax}(Vh_t)\), with \(W\) \([d_h \times d_{in}]\), \(U\) \([d_h \times d_h]\), \(V\) \([d_{out} \times d_h]\).
    x₁x₂x₃y₁y₂y₃h₁h₂h₃UU…WV
    Unrolled in time: the same U, V, W are shared at every step.
  5. Why must RNN inference proceed incrementally from start to end?
    Computing the hidden layer at time \(t\) requires the hidden layer from time \(t-1\).
  6. What does it mean to unroll an RNN in time?
    Copy the network’s layers for each time step to show their different values over time, while the weight matrices \(U\), \(V\), \(W\) are shared across all steps.
  7. What two complications does training an RNN add over a feedforward network?
    The loss at time \(t\) depends on the hidden layer at \(t-1\), and the hidden layer at \(t\) affects both the output at \(t\) and all later hidden layers and losses; its error must account for both.
  8. What is backpropagation through time (BPTT)?
    A two-pass algorithm: a forward pass computing and saving hidden states and accumulating the loss, then a backward pass over the sequence in reverse computing gradients and error terms for each step.
  9. How are RNNs trained in practice with modern frameworks, and what is done for very long inputs?
    Unroll the network for the given input into a feedforward computation graph and train with ordinary backpropagation. For very long inputs (speech, character streams), unroll fixed-length segments treated as separate training items.

Source