learn.ifs.name

A Mathematical Theory of Communication

Shannon, 1948, A Mathematical Theory of Communication (Bell System Technical Journal)

Subscriber deck — free during early access

3 cards in “2 The discrete source of information” · whole deck

2 The discrete source of information

  1. Why can statistical knowledge of a source reduce the channel capacity needed?
    Messages aren’t random: in English E is more frequent than Q, TH more than XP. Proper encoding exploits this, as Morse code already does by giving the most common letter, E, the shortest symbol (a dot).
  2. How does Shannon model a discrete information source?
    As a stochastic process choosing successive symbols with probabilities that may depend on preceding choices; conversely any stochastic process producing discrete symbols from a finite set is a discrete source (natural languages, quantized speech, abstract processes).
  3. How are letter, transition and digram probabilities related in a source where each letter depends only on the previous one?
    \(p(i, j) = p(i)\,p_i(j)\) and \(p(i) = \sum_j p(i, j) = \sum_j p(j)\,p_j(i)\), with \(\sum_j p_i(j) = 1\). The general \(n\)-gram case needs \(n\)-gram probabilities or transition probabilities \(p_{i_1 \ldots i_{n-1}}(i_n)\).

Source