Long Short-Term Memory
Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation)
Subscriber deck — free during early access
1 cards in “4.7 Abuse problem and solutions” · whole deck
4.7 Abuse problem and solutions
- What is the “abuse problem” in early LSTM training, and its remedies?Early on, errors can be reduced without storing anything over time, so the network may abuse memory cells, e.g. as constant bias cells, and take long to release them. Remedies: add cells and gates sequentially when error stops decreasing, or give output gates negative initial biases so cells get allocated later.
- Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §4.7
Source
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. In Neural Computation, pp. 1735–1780. doi:10.1162/neco.1997.9.8.1735