A Mathematical Theory of Communication

28 The rate for a source relative to a fidelity evaluation

1. Why can’t a continuous source be given an entropy rate the way a discrete one is?
   Exact specification of a continuous quantity needs infinitely many bits, so exact transmission needs infinite capacity. The useful question is transmission within a tolerance, measured by a fidelity criterion.
   • Shannon, 1948, A Mathematical Theory of Communication (Bell System Technical Journal), §28
2. How did Shannon define the rate of a continuous source for a given fidelity?
   With a fidelity criterion \(v = \iint \rho(x, y) P(x, y)\,dx\,dy\) (an average distortion), the rate \(R_1\) is the minimum of \(\iint P(x, y) \log \frac{P(x, y)}{P(x)P(y)}\,dx\,dy\) over systems achieving fidelity \(v_1\). Theorem 21: transmission at that fidelity is possible over a channel iff \(R_1 \le C\). This is the origin of rate–distortion theory.
   • Shannon, 1948, A Mathematical Theory of Communication (Bell System Technical Journal), §28