Words and Tokens

Historical notes

1. Where does BPE come from, and how did it enter NLP?
   It was a text compression method (Gage, 1994), applied to subword tokenization for neural MT by Sennrich et al. (2016), then adopted as the default by OpenAI’s GPT-2 and included in SentencePiece.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, Historical notes, p. 65
2. Who first defined regular expressions, and how did grep get its name?
   Kleene (1951, 1956), along with finite automata, based on the McCulloch–Pitts neuron. Ken Thompson built regexes into editors; his ed command g/regular expression/p (Global Regular Expression Print) became Unix grep.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, Historical notes, p. 65
3. Why did Bellman call his method “dynamic programming”?
   To hide that it was mathematical research from a Secretary of Defense who hated the word “research”: “dynamic” conveys multi-stage and is impossible to use pejoratively.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, Historical notes, p. 65