Words and Tokens

2.7 Simple Unix tools for word tokenization

1. What does tr -sc 'A-Za-z' '\n' < sh.txt do?
   Tokenizes naively: -c complements the set (all non-letters), which are changed to newlines, and -s squeezes runs into a single newline, giving one word per line.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.7, pp. 56–57
2. Write a Unix pipeline that lists the most frequent words in a text file, case-insensitively.
   tr -sc 'A-Za-z' '\n' < f.txt | tr A-Z a-z | sort | uniq -c | sort -n -r
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.7, p. 57
3. What are the most frequent words in any corpus, Shakespeare included?
   Short function words: articles, pronouns, prepositions (the, and, i, to, of, a…).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.7, p. 58