Long Short-Term Memory

5 Experiments

1. What makes a good test of a long-time-lag learning algorithm, according to Hochreiter and Schmidhuber?
   Minimal time lags must be long for all training sequences (no short-lag exemplars to generalize from), and tasks must be complex enough not to be solved quickly by random weight guessing, which turned out to solve many earlier benchmarks faster than the proposed algorithms.
   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §5
2. Which tasks did the LSTM paper use?
   The embedded Reber grammar; noise-free and noisy sequences with long lags (up to 1000 steps); noise and signal on the same channel; the adding and multiplication problems (continuous values, distributed); and temporal order tasks with widely separated relevant symbols. LSTM succeeded more often and learned much faster than RTRL, BPTT, Elman nets, recurrent cascade correlation and neural sequence chunking.
   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §5
3. What is the adding problem?
   Each sequence element is a pair: a random real value and a marker. Exactly two elements are marked, and the target at the end is the sum of their values, so the network must store two values across long lags while ignoring noise.
   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §5