Words and Tokens

2.5 Corpora

1. Why should NLP algorithms be tested on more than English?
   Algorithms should work across languages with different properties; developing and testing only on English (or a few big national languages) excludes most of the world’s roughly 7,000 languages.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.5, p. 47
2. What are AAE and MAE, and why do they matter for tokenization?
   African American English and Mainstream American English. AAE features such as iont (MAE I don’t) or talmbout (talking about) affect word segmentation, so tools must handle such varieties.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.5, p. 47
3. What is code switching?
   Using multiple languages within a single utterance, e.g. Spanish–English or Hindi–English mixing in social media posts. It is very common worldwide.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.5, p. 47
4. What are vernacular and vehicular languages?
   Speakers of mainly oral languages are often diglossic: an oral vernacular for family and local communication, and a vehicular language, usually written, for the wider economy, society or state.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.5, p. 47
5. Besides language, what dimensions of variation shape a corpus?
   Genre (news, fiction, Wikipedia, conversation, legal or medical text…), the demographics of writers/speakers (age, gender, race, class), and time (language changes).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.5, pp. 47–48
6. What is a datasheet (or data statement) for a corpus?
   Documentation by the corpus creator of its motivation, situation, language variety, speaker demographics, collection process (size, sampling, consent, preprocessing), annotation process, and distribution (copyright/IP restrictions).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.5, p. 48