Long Short-Term Memory

4.2 Why gate units?

1. What do the input and output gates learn to do?
   The input gate decides when to keep or override information in the cell (scaling errors that leave the CEC); the output gate decides when to access the cell and when to keep it from perturbing other units (deciding which errors get trapped in the CEC). Multiplicative gates open and close access to constant error flow.
   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §4.2
2. Are both gates always necessary?
   No: with local output encoding output gates aren’t strictly required (weights can be zeroed), and some tasks work with input gates only. Output gates still help by keeping hard-to-learn long-lag memories from perturbing easy short-lag outputs; distributed output representations typically need them.
   • Hochreiter & Schmidhuber, 1997, Long Short-Term Memory (Neural Computation), §4.2