A Mathematical Theory of Communication

20 Entropy of a continuous distribution

1. How is entropy defined for a continuous distribution, and how does it differ from the discrete case?
   \(H = -\int p(x) \log p(x)\,dx\). Unlike discrete entropy it is relative to the coordinate system (it changes by the expected log Jacobian under a change of variables) and can be negative; rates and capacities, being differences of entropies, are unaffected.
   • Shannon, 1948, A Mathematical Theory of Communication (Bell System Technical Journal), §20
2. Which distribution maximizes entropy for a fixed standard deviation \(\sigma\), and what is its entropy?
   The Gaussian, with \(H = \log \sqrt{2\pi e}\,\sigma\). (On a half line with fixed mean \(a\), the exponential, with \(H = \log ea\).)
   • Shannon, 1948, A Mathematical Theory of Communication (Bell System Technical Journal), §20