Long Short-Term Memory

6.1 Limitations of LSTM

1. What limitations of LSTM did its authors acknowledge?
   Truncated LSTM struggles with nondecomposable tasks like strongly delayed XOR (storing one input alone doesn’t reduce error); each block needs two extra gate units; it behaves like a feedforward net seeing the whole input, so e.g. 500-step parity is hard; and like all gradient methods it can’t count time steps precisely (99 vs 100 steps).
   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §6.1
2. Why didn’t the authors recommend computing the full (untruncated) gradient for LSTM?
   It increases computational complexity, constant error flow through the CECs can be shown only for truncated LSTM, and in experiments the full gradient made no significant difference because error flow outside the CECs vanishes quickly anyway.
   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §6.1