learn.ifs.name

A Mathematical Theory of Communication

Shannon, 1948, A Mathematical Theory of Communication (Bell System Technical Journal)

Subscriber deck — free during early access

7 cards in “Introduction” · whole deck

Introduction

  1. What did Shannon call “the fundamental problem of communication”?
    Reproducing at one point, exactly or approximately, a message selected at another point.
  2. Why does Shannon set the meaning of messages aside?
    Semantic aspects are irrelevant to the engineering problem. What matters is that the actual message is one selected from a set of possible messages, and the system must work for every possible selection, not just the one actually sent.
  3. Give Shannon’s three reasons for measuring information with a logarithm.
    It is practically more useful (time, bandwidth or number of relays vary linearly with the log of the number of possibilities); it matches intuition (two punched cards should store twice as much as one); it is mathematically more convenient for limiting operations.
  4. What is a bit, and where does the word come from?
    The unit of information when logarithms are taken to base 2 (“binary digit”), a word suggested by J. W. Tukey. A device with two stable states stores one bit; \(N\) such devices store \(\log_2 2^N = N\) bits.
  5. How do information units change with the base of the logarithm?
    Base 2 gives bits, base 10 decimal digits (about 3⅓ bits each), base \(e\) natural units. Changing from base \(a\) to base \(b\) multiplies by \(\log_b a\).
  6. What are the five parts of Shannon’s general communication system?
    Information source → transmitter → channel (where a noise source acts) → receiver → destination.
    sourcetrans-mitterre-ceiverdesti-nationnoisechannel
    Message → signal → received signal → message
  7. How does Shannon classify communication systems?
    Discrete (message and signal are sequences of discrete symbols, e.g. telegraphy), continuous (both are continuous functions, e.g. radio, television) and mixed (e.g. PCM speech). The discrete case comes first and founds the others.

Source