Words and Tokens

2.8 Rule-based tokenization

1. When might we want rule-based word tokenization instead of BPE subwords?
   When tokens must be words: e.g. parsers needing grammatical words as input, linguistic studies with a predefined token definition, or social-science work on orthographic words.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.8, p. 58
2. What does a good English rule-based tokenizer need to handle?
   Split off punctuation, but keep word-internal punctuation (m.p.h., Ph.D., AT&T), prices and dates ($45.55, 01/02/06), URLs, hashtags, emails, and numbers with commas (555,500.50). Number formats differ by language (555 500,50).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.8, p. 58
3. How can a rule-based tokenizer treat clitic contractions and multiword expressions?
   Expand contractions (what’re → what are, we’re → we are), and optionally keep multiword expressions like New York or rock ’n’ roll as one token using a dictionary, which ties tokenization to named entity recognition.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.8, p. 58
4. Describe the Penn Treebank tokenization standard.
   Used for LDC treebanks: it separates clitics (doesn’t → does + n’t), keeps hyphenated words together, and separates all punctuation (and $ from amounts).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.8, p. 59
5. How are rule-based tokenizers made fast?
   Deterministic regular expressions compiled into efficient finite-state automata (e.g. NLTK’s regexp_tokenize).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.8, p. 59
6. What ambiguity of the apostrophe must a careful tokenizer resolve?
   Genitive marker (the book’s cover), quotation mark (‘The other class’, she said), or clitic (they’re).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.8, p. 59
7. What is sentence segmentation, and which cues does it use in English?
   Splitting text into sentences. Question and exclamation marks are fairly unambiguous boundaries; the period is ambiguous.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.8, p. 59
8. Why is the period hard for sentence segmentation, and how is it handled?
   It marks both sentence ends and abbreviations (Dr., Inc.), sometimes both at once. Segmenters decide whether a period belongs to a word, using rules or ML plus an abbreviation dictionary, often jointly with word tokenization.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.8, p. 60
9. How does Stanford CoreNLP decide that a sentence ends?
   Deterministically from tokenization: a sentence ends when ., ! or ? is not already part of another token (abbreviation, number), optionally followed by closing quotes or brackets.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.8, p. 60