Entropy and information rate; Shannon limit as bandwidth → ∞

Relation between entropy and information rate

Entropy $H$ is the average information per symbol, in bits/symbol. Information rate $R$ is the average information produced per second, in bits/s.

Derivation: let a discrete memoryless source emit $r$ symbols per second from an alphabet $\{s_1, \dots, s_M\}$ with probabilities $p_1, \dots, p_M$.

1. In a long time $T$ seconds the source emits $N = rT$ symbols.
2. Symbol $s_k$ occurs about $N p_k = rTp_k$ times, and each occurrence carries $\log_2(1/p_k)$ bits.
3. Total information in $T$ seconds:

$$I_T = \sum_{k=1}^{M} rT\,p_k\log_2\frac{1}{p_k} = rT\,H$$

4. Information per second:

$$R = \frac{I_T}{T} = r\,H\ \text{bits/s}$$

where $r$ is the symbol rate (symbols/s) and $H$ the entropy (bits/symbol). Example: $r = 1000$ symbols/s and $H = 1.75$ bits/symbol give $R = 1750$ bits/s. For equally likely symbols, $R_{max} = r\log_2 M$.

Limit of Shannon capacity when $B \to \infty$

Shannon–Hartley theorem for an AWGN channel (noise PSD $N_0$, noise power $N = N_0B$):

$$C = B\log_2\left(1+\frac{S}{N_0B}\right)\ \text{bits/s}$$

Rewrite with $x = \dfrac{S}{N_0B}$, so $B = \dfrac{S}{N_0 x}$:

$$\begin{aligned} C &= \frac{S}{N_0}\cdot\frac{1}{x}\log_2(1+x) = \frac{S}{N_0}\log_2(1+x)^{1/x} \end{aligned}$$

As $B \to \infty$, $x \to 0$, and $\lim_{x \to 0}(1+x)^{1/x} = e$:

$$C_\infty = \frac{S}{N_0}\log_2 e = \frac{S}{N_0}\cdot\frac{1}{\ln 2} \approx 1.44\,\frac{S}{N_0}\ \text{bits/s}$$

Meaning:

• Capacity does not become infinite when bandwidth becomes infinite, because noise power $N_0 B$ also increases. It approaches a finite limit set by $S/N_0$.
• At this limit, with energy per bit $E_b = S/C$:

$$\frac{E_b}{N_0} = \frac{S}{N_0 C_\infty} = \ln 2 = 0.693 \;\Rightarrow\; -1.6\ \text{dB}$$

This is the Shannon limit: no coding scheme can give reliable transmission with $E_b/N_0$ below −1.6 dB.

 C
 |           ......................  1.44 S/N0
 |       ..'
 |    .'
 |  .'
 | /
 +-------------------------------->  B