Long Short-Term Memory

3.1 Exponentially decaying error

1. How does an error signal scale when propagated back \(q\) steps from unit \(u\) to unit \(v\) in a standard RNN?
   By a sum over paths of products of terms \(f'_{l_m}(\text{net}_{l_m}(t - m))\, w_{l_m l_{m-1}}\), one per step: a product of \(q\) factors along each path.
   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §3.1
2. When does backpropagated error blow up, and when does it vanish?
   If \(|f'(\text{net})\, w| > 1\) at every step, the largest product grows exponentially with \(q\): errors blow up, causing oscillating weights and unstable learning. If \(|f'(\text{net})\, w| < 1\) at every step, it decreases exponentially: errors vanish and nothing is learned in acceptable time.
   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §3.1
3. Why does error tend to vanish with logistic sigmoid units?
   The sigmoid’s derivative is at most 0.25, so \(|f' w| < 1\) whenever \(|w| < 4\), as in early training. Larger initial weights don’t help (the derivative shrinks faster than the weight grows), nor does a larger learning rate, which doesn’t change the ratio of long- to short-range error flow.
   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §3.1