Long Short-Term Memory

Recurrent networks could in principle store recent inputs in their activations (short-term memory), but gradient-based training (BPTT, RTRL) takes too long or fails when the time lags between inputs and the corresponding teacher signals are long, because error signals flowing back in time vanish or blow up.

A gradient-based architecture that enforces constant error flow through “constant error carousels” in special units, with multiplicative gates controlling access; it bridges time lags over 1000 steps, is local in space and time with O(1) cost per weight and time step, and solves long-time-lag tasks no previous recurrent algorithm had solved.

By a sum over paths of products of terms \(f'_{l_m}(\text{net}_{l_m}(t - m))\, w_{l_m l_{m-1}}\), one per step: a product of \(q\) factors along each path.

If \(|f'(\text{net})\, w| > 1\) at every step, the largest product grows exponentially with \(q\): errors blow up, causing oscillating weights and unstable learning. If \(|f'(\text{net})\, w| < 1\) at every step, it decreases exponentially: errors vanish and nothing is learned in acceptable time.

The sigmoid’s derivative is at most 0.25, so \(|f' w| < 1\) whenever \(|w| < 4\), as in early training. Larger initial weights don’t help (the derivative shrinks faster than the weight grows), nor does a larger learning rate, which doesn’t change the ratio of long- to short-range error flow.

Its local backflow is \(\vartheta_j(t) = f'_j(\text{net}_j(t))\, \vartheta_j(t+1)\, w_{jj}\), so we need \(f'_j(\text{net}_j)\, w_{jj} = 1\): a linear activation (the identity) with self-weight \(w_{jj} = 1.0\). Then \(y_j(t+1) = y_j(t)\). This is the constant error carousel (CEC).

The same incoming weight to the CEC unit must both store relevant inputs (switch the unit on) and protect the stored value by ignoring irrelevant later inputs, so it gets conflicting update signals. Write operations need a context-sensitive control.

The same outgoing weight must both retrieve the stored content at certain times and keep it from disturbing other units at others, e.g. a unit later starts causing errors in short-time-lag outputs that were already learned. Read operations need a context-sensitive control.

A linear unit with a fixed self-connection of weight 1.0 (the CEC), plus a multiplicative input gate that protects the stored content from irrelevant inputs, and a multiplicative output gate that protects other units from currently irrelevant memory contents.

Gates: \(y^{\text{in}_j}(t) = f_{\text{in}_j}(\text{net}_{\text{in}_j}(t))\), \(y^{\text{out}_j}(t) = f_{\text{out}_j}(\text{net}_{\text{out}_j}(t))\). State: \(s_{c_j}(t) = s_{c_j}(t-1) + y^{\text{in}_j}(t)\, g(\text{net}_{c_j}(t))\), with \(s_{c_j}(0) = 0\). Output: \(y^{c_j}(t) = y^{\text{out}_j}(t)\, h(s_{c_j}(t))\).

It has only input and output gates: the state accumulates \(s(t) = s(t-1) + \ldots\) with a fixed self-weight of 1. The forget gate, which multiplies \(s(t-1)\) by a learned value, was introduced later by Gers, Schmidhuber and Cummins (2000).

The input gate decides when to keep or override information in the cell (scaling errors that leave the CEC); the output gate decides when to access the cell and when to keep it from perturbing other units (deciding which errors get trapped in the CEC). Multiplicative gates open and close access to constant error flow.

No: with local output encoding output gates aren’t strictly required (weights can be zeroed), and some tasks work with input gates only. Output gates still help by keeping hard-to-learn long-lag memories from perturbing easy short-lag outputs; distributed output representations typically need them.

\(S\) memory cells sharing the same input gate and output gate. Blocks make it easier to store distributed information and are slightly more efficient, since the whole block needs only two gate units.

With a variant of RTRL using truncated gradients: errors arriving at a memory cell’s net inputs (cell, input gate, output gate) are not propagated further back in time, but they still change incoming weights; only inside the cell does error flow back through previous internal states.

It enters at the cell output, is scaled by the output gate activation and \(h'\), then flows back through the CEC indefinitely without being scaled; leaving through the input gate and \(g\), it is scaled once more by the input gate activation and \(g'\), updates the incoming weights, and is truncated.

\(O(W)\) per time step (\(O(1)\) per weight), like BPTT and far better than RTRL. Local in space: update cost per weight doesn’t depend on network size. Local in time: storage doesn’t depend on sequence length, unlike full BPTT, which must store activations for the whole sequence.

Early on, errors can be reduced without storing anything over time, so the network may abuse memory cells, e.g. as constant bias cells, and take long to release them. Remedies: add cells and gates sequentially when error stops decreasing, or give output gates negative initial biases so cells get allocated later.

If a cell’s inputs are mostly of one sign, its state drifts, making \(h'(s)\) tiny so the gradient vanishes. A simple fix is to bias the input gate toward zero initially; its downside is negligible compared with the drift.

Minimal time lags must be long for all training sequences (no short-lag exemplars to generalize from), and tasks must be complex enough not to be solved quickly by random weight guessing, which turned out to solve many earlier benchmarks faster than the proposed algorithms.

The embedded Reber grammar; noise-free and noisy sequences with long lags (up to 1000 steps); noise and signal on the same channel; the adding and multiplication problems (continuous values, distributed); and temporal order tasks with widely separated relevant symbols. LSTM succeeded more often and learned much faster than RTRL, BPTT, Elman nets, recurrent cascade correlation and neural sequence chunking.

Each sequence element is a pair: a random real value and a marker. Exactly two elements are marked, and the target at the end is the sum of their values, so the network must store two values across long lags while ignoring noise.

Truncated LSTM struggles with nondecomposable tasks like strongly delayed XOR (storing one input alone doesn’t reduce error); each block needs two extra gate units; it behaves like a feedforward net seeing the whole input, so e.g. 500-step parity is hard; and like all gradient methods it can’t count time steps precisely (99 vs 100 steps).

It increases computational complexity, constant error flow through the CECs can be shown only for truncated LSTM, and in experiments the full gradient made no significant difference because error flow outside the CECs vanishes quickly anyway.

Bridging very long time lags; handling noise, distributed representations and continuous values; no a priori finite number of states (unlike finite automata or HMMs); good generalization; no need for parameter fine-tuning; and O(1) update complexity per weight and time step while being local in space and time.

Each memory cell’s internal architecture guarantees constant error flow within its CEC (given truncated backprop), and multiplicative input and output gates learn to open and close access to that error flow, protecting the stored content and the rest of the network.