A Mathematical Theory of Communication

10 Discussion and examples

1. Why compare encoding to an electrical transformer?
   For maximum power transfer a transformer matches a generator to its load; likewise the encoder should match the source to the channel statistically, so the channel sees the input statistics that maximize its entropy.
   • Shannon, 1948, A Mathematical Theory of Communication (Bell System Technical Journal), §10
2. Encode a source emitting A, B, C, D with probabilities ½, ¼, ⅛, ⅛ (independently) optimally in binary.
   \(H = \tfrac12 \cdot 1 + \tfrac14 \cdot 2 + 2 \cdot \tfrac18 \cdot 3 = \tfrac74\) bits per symbol, achieved exactly by

   A	0
   B	10
   C	110
   D	111

   The coded 0s and 1s are equally likely, one bit each, and the average length is \(\tfrac74\) digits per letter.
   • Shannon, 1948, A Mathematical Theory of Communication (Bell System Technical Journal), §10
3. How much channel capacity does a fully deterministic source need, and what if we ignore some of what we know?
   A source that can produce only one message (like a machine computing the digits of π) has zero entropy and needs no channel. If we choose to use only some statistics (e.g. letter frequencies of English), the needed capacity is the entropy of the maximum-entropy source satisfying those statistics.
   • Shannon, 1948, A Mathematical Theory of Communication (Bell System Technical Journal), §10