RNNs and LSTMs

Historical notes

1. Where were RNNs investigated in the 1980s, and with what aim?
   In the Parallel Distributed Processing (PDP) group at UC San Diego, mostly for human cognitive modeling; Elman (1990) introduced recurrence at the hidden layer, Jordan (1986) from the output layer.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, Historical notes, p. 326
2. What changed the practical prospects of RNNs?
   LSTMs (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), which brought big gains in phoneme recognition, handwriting recognition and, most significantly, speech recognition.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, Historical notes, p. 326
3. What did Collobert and Weston’s work (2008, 2011) demonstrate?
   Near state-of-the-art performance on POS tagging, chunking, NER and semantic role labeling using learned word embeddings, convolutional networks and end-to-end training, without hand-engineered features.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, Historical notes, p. 326
4. Who coined “encoder-decoder”, and where did attention come from?
   Cho et al. (2014) coined the name (with Sutskever et al. 2014 using RNNs for both parts). Attention, a soft weighting over inputs for a generative decoder, was first developed by Graves (2013) for handwriting; Bahdanau et al. (2015) named it attention and applied it to MT.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, Historical notes, p. 327