learn.ifs.name

Introduction

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1

9 cards in “1.4 History of speech and language processing” · whole deck

1.4 History of speech and language processing

  1. In what period do the central ideas of modern LMs (word prediction, neural networks, embeddings) originate?
    The late 1940s, 1950s and early 1960s: Shannon’s word prediction, small neural networks at Cornell and Stanford, Osgood’s spatial meaning, and Harris’s and Joos’s ideas that meaning relates to context.
  2. What are the four stages in the history of NLP shown in the book’s timeline?
    (1) Statistical and neural foundations (1940s–60s); (2) the symbolic turn (≈1965–early 1990s); (3) the long revival of empiricism (≈1975/1988–2017); (4) the LLM era, beginning with prompting (2019–).
  3. What caused the “symbolic turn” of the 1960s?
    Influential arguments by Chomsky against statistical models of language and by Minsky & Papert (1969) against neural models. AI, NLP and cognitive science moved toward symbolic reasoning and explicit symbolic structures.
  4. Where did the “long revival of empiricism” begin, and what did that group contribute?
    In speech recognition at IBM’s Thomas J. Watson Research Center (Fred Jelinek and colleagues, 1975–1985). They coined the term “language model” and developed n-gram LMs and algorithms for training and inference of statistical models.
  5. What is connectionism (parallel distributed processing)?
    The cognitive-science paradigm of neural networks that spread in the late 1980s, after improved training algorithms based on error backpropagation (Rumelhart et al., 1986).
  6. What hardware development made very large, deep networks trainable in the 2000s?
    GPUs, originally developed for graphics.
  7. What architecture culminated the neural line of work around 2017, and what was it invented for?
    The transformer (Vaswani et al., 2017), invented for machine translation, which quickly became a general-purpose architecture.
  8. What was the third major inflection point in NLP history, and why was it a revolution?
    Prompting (Radford et al., 2019). Instead of labeling training data and training a classifier for each task, you could simply type a question to an LLM and have it predict the answer.
  9. What is generative AI?
    The area concerned with using computational models to generate text, code, speech and images. Once prompted LMs became central, NLP shifted from parsing and interpreting text toward generation.

Source