A Mathematical Theory of Communication

4 Graphical representation of a Markoff process

1. How does a discrete Markov process become an information source?
   There are finitely many states and transition probabilities \(p_i(j)\); a letter is produced at each transition. The states encode the “residue of influence” from preceding letters, so a trigram source needs up to \(n^2\) states (pairs of preceding letters).
   • Shannon, 1948, A Mathematical Theory of Communication (Bell System Technical Journal), §4