Entropy, information and channel capacity

(i) Entropy

The average information per source symbol: H = −Σ pᵢ log₂ pᵢ bits/symbol. It measures the uncertainty of the source, with 0 ≤ H ≤ log₂ M. Example: a fair coin has H = 1 bit.

(ii) Information

The information in a message of probability p is I = log₂(1/p) = −log₂ p bits. The less likely the event, the more information it carries; a certain event carries 0 bits. Example: an event of probability 1/8 carries 3 bits.

(iii) Channel capacity

The channel capacity C is the maximum rate (bits per second) at which information can be sent over a channel with an arbitrarily small error probability. By the Shannon-Hartley theorem, for a band-limited channel with Gaussian noise:

C = B log₂(1 + S/N) bits/s

where B is bandwidth in Hz and S/N is the signal-to-noise power ratio. Example: B = 3000 Hz and S/N = 1000 (30 dB) gives C = 3000 × log₂(1001) ≈ 29.9 kbps. Shannon's theorem says that if the information rate R ≤ C, error-free transmission is possible with suitable coding. For a binary symmetric channel with error probability p, C = 1 − H(p) bits per use.