Words and Tokens

Introduction

1. How did ELIZA produce its responses?
   By simple pattern matching on words: it recognized phrases like “I need X” and transformed them into outputs like “What would it mean to you if you got X?”, imitating a Rogerian psychotherapist.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, Introduction, p. 33
2. What is tokenization?
   Separating out (tokenizing) words and word parts from running text: the first step in modern NLP.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, Introduction, p. 33
3. What is edit distance, and where does NLP use it?
   A measure of how similar two strings are, based on the number of insertions, deletions and substitutions needed to turn one into the other. It is used whenever strings must be compared, e.g. in the word error rate metric for speech recognition.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, Introduction, p. 33