Words and Tokens

2.3 Unicode

1. What is Unicode?
   A standard for representing text written in any character of any script of the world’s languages (including dead languages like Sumerian cuneiform and invented ones like Klingon), plus symbols, emoji, etc.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.3, p. 39
2. What is ASCII, and how does it relate to Unicode?
   The 1960s American Standard Code for Information Interchange: one byte per character, using only 128 values (high-order bit 0), for the Latin characters used in English. It survives as Unicode’s first block (“Basic Latin”).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.3, p. 39
3. Why is ASCII insufficient?
   Even Latin-script languages need characters outside it (Spanish ñ, ó), and many languages use other scripts entirely (Devanagari, Chinese characters…).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.3, p. 39
4. What is a Unicode code point?
   A unique abstract id for a character, a number from 0 to 0x10FFFF, written with the prefix U+ (e.g. U+0061 for a).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.3, p. 40
5. Why do the first 128 Unicode code points match ASCII?
   Unicode was designed to be backwards compatible with ASCII.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.3, p. 40
6. What is the difference between a code point and a glyph?
   A code point is the abstract character; a glyph is its visual representation, stored in a typeface. a in Times, Courier, bold or italic is the same code point U+0061.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.3, p. 40
7. What is a character encoding?
   The way a character (code point) is actually written as bytes in a text string or file. UTF-8 is by far the most frequent; almost the whole web uses it.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.3, p. 40
8. Why don’t we store text as raw 4-byte code points (UTF-32)?
   It makes files 4 times longer than ASCII and full of zero bytes, and zero bytes break ASCII-based systems that historically used 0 as an end-of-string marker.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.3, p. 41
9. What does it mean that UTF-8 is a variable-length encoding?
   Characters take different numbers of bytes: ASCII characters take 1 byte, other code points 2, 3 or 4 bytes. Each multi-byte byte is between 128 and 255, and the first byte’s leading bits say how many bytes follow.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.3, p. 41
10. Show how UTF-8 packs code points into bytes.

    code points	bytes
    U+0000–007F	0xxxxxxx
    U+0080–07FF	110yyyyy 10xxxxxx
    U+0800–FFFF	1110zzzz 10yyyyyy 10xxxxxx
    U+10000–10FFFF	11110uuu 10uuzzzz 10yyyyyy 10xxxxxx

    The code point’s bits fill the x, y, z, u slots.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.3, p. 41
11. How is ñ (U+00F1) encoded in UTF-8?
    It is in the U+0080–07FF range, so it takes 2 bytes: its bits 00011 110001 fill 110yyyyy 10xxxxxx, giving 11000011 10110001 = 0xC3 0xB1.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.3, p. 41
12. Roughly how many UTF-8 bytes do characters of different scripts need?
    ASCII: 1 byte; most other European, Middle Eastern and African scripts: 2 bytes; most Chinese, Japanese and Korean characters: 3 bytes; rarer CJKV characters, emoji and some symbols: 4 bytes.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.3, pp. 41–42
13. What are the advantages of UTF-8?
    It is efficient for common characters, uses no zero bytes (except for NULL itself), is backwards compatible with ASCII (ASCII files are valid UTF-8), and is self-synchronizing: after corruption you can find the next or previous character start by moving at most 3 bytes.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.3, p. 42
14. In Python 3, what does len() of a string count?
    Code points (characters), not bytes: Python 3 strings are sequences of Unicode code points. Bytes appear only when encoding/decoding to a file.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.3, p. 42
15. Is there such a thing as a plain text file without an encoding?
    No. Every file is in some encoding (UTF-8 or older ones like ASCII or Latin-1), which must be specified when reading or writing.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 2, §2.3, p. 42