Long Short-Term Memory

Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation)

Subscriber deck — free during early access

1 cards in “4.8 Internal state drift and remedies” · whole deck

4.8 Internal state drift and remedies

What is internal state drift, and how is it countered?
If a cell’s inputs are mostly of one sign, its state drifts, making \(h'(s)\) tiny so the gradient vanishes. A simple fix is to bias the input gate toward zero initially; its downside is negligible compared with the drift.
- Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §4.8

Source

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. In Neural Computation, pp. 1735–1780. doi:10.1162/neco.1997.9.8.1735