Neural Networks

Historical notes

1. Trace the early history of neural networks up to their decline.
   McCulloch–Pitts neuron (1943); perceptron research in the late 1950s–60s (Rosenblatt at Cornell, Widrow at Stanford, who turned the threshold into a bias); then decline after Minsky & Papert (1969) showed a perceptron can’t model XOR.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, Historical notes, p. 176
2. What revived neural networks in the 1980s, and what ideas came from that period?
   Practical tools for deeper networks, especially backpropagation (Rumelhart et al., 1986). The connectionist / parallel distributed processing movement contributed distributed representations, recurrent networks and tensors for compositionality.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, Historical notes, p. 176
3. When and why did the term “deep learning” arise?
   In the early 2000s, when better hardware and optimization techniques made it possible to train much larger and deeper networks (Hinton et al., 2006; Bengio et al., 2007).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, Historical notes, p. 176