Words and Tokens

2.1 Words

1. Should punctuation count as words?
   It depends on the task. Punctuation marks boundaries (commas, periods, colons) and some meaning (question marks, quotation marks). LLMs generally treat punctuation as separate words.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.1, p. 34
2. What is an utterance?
   The spoken correlate of a sentence.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.1, p. 34
3. What are the two kinds of disfluency in “I do uh main- mainly business data processing”?
   A fragment (the broken-off word main-) and a filled pause or filler (uh).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.1, p. 34
4. Why might a speech recognizer keep fillers like uh and um as words?
   They help predict the upcoming word, since they may signal that the speaker is restarting a clause or idea. They can also be a cue to speaker identity.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.1, p. 34
5. What is the difference between word types and word instances?
   Types are the distinct words in a corpus (their number is the vocabulary size \(|V|\)); instances are the running words, \(N\). “The cat saw the dog” has 5 instances but 4 types.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.1, p. 34
6. Why does the book avoid calling word instances “word tokens”?
   It reserves “token” for the output of subword tokenization algorithms.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.1, p. 34
7. Should They and they be the same word type?
   It depends on the task: they may be merged where formatting doesn’t matter, but capitalization is a useful feature for other tasks. Sometimes both cased and uncased versions of a model are kept.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.1, p. 34
8. Why is the orthographic word I’m really two grammatical words?
   It functions as the subject pronoun I plus the verb ’m (short for am).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.1, p. 35
9. Which writing systems have no orthographic words, and why is that a problem?
   Chinese, Japanese and Thai don’t use spaces to mark word boundaries, so deciding what counts as a word requires a (debatable) standard.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.1, p. 35
10. What is a Chinese character called, and what does it usually correspond to?
    A hanzi. Each character generally represents a single morpheme and is pronounced as a single syllable.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.1, p. 35
11. How can the Chinese sentence “Yao Ming reaches the finals” be segmented into 3, 5 or 7 units?
    3 words under the Chinese Treebank standard (names are one word: YaoMing / reaches / finals); 5 under the Peking University standard (Yao / Ming / reaches / overall / finals); or 7 characters if words are ignored entirely.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.1, p. 35
12. Why can characters serve as basic units for Chinese but not for Japanese or Thai?
    Chinese characters are at a reasonable semantic level (roughly morphemes) for most applications; in Japanese and Thai the individual character is too small a unit.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.1, p. 35
13. State Herdan’s Law (Heaps’ Law).
    The number of word types grows with the number of instances as \(|V| = kN^{\beta}\), with constants \(k > 0\) and \(0 < \beta < 1\). Roughly, vocabulary grows a bit faster than the square root of text length.

    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.1, p. 36
14. How do function words and content words behave differently as a corpus grows?
    Function words (a, of) form a roughly fixed set that is exhausted early. Content words, especially nouns such as names and technical terms, keep growing indefinitely. So vocabulary growth has two regimes, with different \(\beta\) values.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.1, p. 36
15. Why does Herdan’s Law cause a problem for computational models?
    No vocabulary, however big, contains every word that might occur, so models constantly meet unknown words they have never seen before.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.1, p. 36
16. What two problems make words a poor unit of processing for NLP models?
    Many languages lack orthographic words, and defining them post hoc is hard; and the number of words grows without bound. So models use smaller subword units that can be recombined into unseen words.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.1, p. 37