Long Short-Term Memory

4.5 Learning

1. How was LSTM trained in the original paper?
   With a variant of RTRL using truncated gradients: errors arriving at a memory cell’s net inputs (cell, input gate, output gate) are not propagated further back in time, but they still change incoming weights; only inside the cell does error flow back through previous internal states.
   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §4.5
2. Trace an error signal through an LSTM memory cell under truncated backprop.
   It enters at the cell output, is scaled by the output gate activation and \(h'\), then flows back through the CEC indefinitely without being scaled; leaving through the input gate and \(g\), it is scaled once more by the input gate activation and \(g'\), updates the incoming weights, and is truncated.
   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §4.5