Information and entropy; capacity as bandwidth → ∞

Information and entropy

• Information: the amount of information in a message $m_k$ of probability $p_k$ is $I_k = \log_2\frac{1}{p_k}$ bits. Less likely messages carry more information; a certain message carries none.
• Entropy: the average information per symbol of a source:

$$H = \sum_{k=1}^{M} p_k\log_2\frac{1}{p_k}\ \text{bits/symbol}, \qquad 0 \le H \le \log_2 M$$

Upper limit of channel capacity as $B \to \infty$

Shannon–Hartley: for an AWGN channel with signal power $S$ and noise PSD $N_0$ (so $N = N_0 B$),

$$C = B\log_2\left(1 + \frac{S}{N_0B}\right)$$

Let $x = \dfrac{S}{N_0 B}$, so $B = \dfrac{S}{N_0}\cdot\dfrac1x$:

$$C = \frac{S}{N_0}\cdot\frac{1}{x}\log_2(1+x) = \frac{S}{N_0}\log_2(1+x)^{1/x}$$

When $B \to \infty$, $x \to 0$, and the standard limit $\lim_{x \to 0}(1+x)^{1/x} = e$ gives

$$C_\infty = \frac{S}{N_0}\log_2 e = 1.44\,\frac{S}{N_0}\ \text{bits/s}$$

Capacity stays finite because the noise power grows in proportion to the bandwidth. At this limit the energy per bit satisfies $E_b/N_0 = \ln 2 = 0.693$ (−1.6 dB), the Shannon limit.

Answer: $C_\infty = 1.44\,S/N_0$ bits/s.