learn.ifs.name

A Mathematical Theory of Communication

Shannon, 1948, A Mathematical Theory of Communication (Bell System Technical Journal)

Subscriber deck — free during early access

3 cards in “3 The series of approximations to English” · whole deck

3 The series of approximations to English

  1. What is Shannon’s series of approximations to English?
    Text generated by increasingly complex stochastic processes over 26 letters and space:
    ordermodel
    zeroletters independent and equiprobable
    firstletters independent, with English frequencies
    seconddigram structure
    thirdtrigram structure
    word, first orderwords independent, with their frequencies
    word, second orderword transition probabilities
    The output looks noticeably more like English at each step: an early demonstration of n-gram language models.
  2. What did Shannon observe about how far the structure of approximated text extends?
    Samples have reasonably good structure out to about twice the range the model takes into account: digram text gives plausible four-letter sequences, and word-digram text gives runs of four or more words that fit into sentences.
  3. How did Shannon generate his higher-order approximations to English without frequency tables?
    By sampling from a book: pick a random letter, open another page and read until that letter appears, record the letter after it, and repeat. This samples from the true digram (or trigram, word) statistics.

Source