Long Short-Term Memory

3.2 Constant error flow: naive approach

1. How can error flow through a single self-connected unit \(j\) be kept constant?
   Its local backflow is \(\vartheta_j(t) = f'_j(\text{net}_j(t))\, \vartheta_j(t+1)\, w_{jj}\), so we need \(f'_j(\text{net}_j)\, w_{jj} = 1\): a linear activation (the identity) with self-weight \(w_{jj} = 1.0\). Then \(y_j(t+1) = y_j(t)\). This is the constant error carousel (CEC).
   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §3.2
2. What is the input weight conflict?
   The same incoming weight to the CEC unit must both store relevant inputs (switch the unit on) and protect the stored value by ignoring irrelevant later inputs, so it gets conflicting update signals. Write operations need a context-sensitive control.
   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §3.2
3. What is the output weight conflict?
   The same outgoing weight must both retrieve the stored content at certain times and keep it from disturbing other units at others, e.g. a unit later starts causing errors in short-time-lag outputs that were already learned. Read operations need a context-sensitive control.
   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §3.2