Long Short-Term Memory

1 Introduction

1. What problem does LSTM address?
   Recurrent networks could in principle store recent inputs in their activations (short-term memory), but gradient-based training (BPTT, RTRL) takes too long or fails when the time lags between inputs and the corresponding teacher signals are long, because error signals flowing back in time vanish or blow up.
   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §1
2. Summarize the key claims of the LSTM paper.
   A gradient-based architecture that enforces constant error flow through “constant error carousels” in special units, with multiplicative gates controlling access; it bridges time lags over 1000 steps, is local in space and time with O(1) cost per weight and time step, and solves long-time-lag tasks no previous recurrent algorithm had solved.
   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §1