Long Short-Term Memory

4.1 Memory cells and gate units

1. What is an LSTM memory cell (in the 1997 paper)?
   A linear unit with a fixed self-connection of weight 1.0 (the CEC), plus a multiplicative input gate that protects the stored content from irrelevant inputs, and a multiplicative output gate that protects other units from currently irrelevant memory contents.

   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §4.1
2. Give the equations of the original LSTM memory cell.
   Gates: \(y^{\text{in}_j}(t) = f_{\text{in}_j}(\text{net}_{\text{in}_j}(t))\), \(y^{\text{out}_j}(t) = f_{\text{out}_j}(\text{net}_{\text{out}_j}(t))\). State: \(s_{c_j}(t) = s_{c_j}(t-1) + y^{\text{in}_j}(t)\, g(\text{net}_{c_j}(t))\), with \(s_{c_j}(0) = 0\). Output: \(y^{c_j}(t) = y^{\text{out}_j}(t)\, h(s_{c_j}(t))\).
   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §4.1
3. How does the 1997 memory cell differ from the LSTM usually taught today?
   It has only input and output gates: the state accumulates \(s(t) = s(t-1) + \ldots\) with a fixed self-weight of 1. The forget gate, which multiplies \(s(t-1)\) by a learned value, was introduced later by Gers, Schmidhuber and Cummins (2000).
   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §4.1