Entropy and information rate of 6-level PCM signal

Given: band limit $f_m = 5$ kHz; 6 quantization levels with probabilities $\frac12, \frac14, \frac18, \frac1{16}, \frac1{32}, \frac1{32}$ (sum = 1).

Entropy

$$H = \sum_{i=1}^{6} p_i\log_2\frac{1}{p_i}$$ $$\begin{aligned} H &= \tfrac12(1) + \tfrac14(2) + \tfrac18(3) + \tfrac1{16}(4) + \tfrac1{32}(5) + \tfrac1{32}(5) \\ &= 0.5 + 0.5 + 0.375 + 0.25 + 0.15625 + 0.15625 \\ &= 1.9375\ \text{bits/sample} \end{aligned}$$

For comparison, if the six levels were equally likely, $H_{max} = \log_2 6 = 2.585$ bits/sample. The unequal probabilities reduce the average information.

Information rate

The signal is sampled at the Nyquist rate:

$$r = f_s = 2f_m = 2 \times 5\ \text{kHz} = 10\ 000\ \text{samples/s}$$

Each sample is one message (level), so

$$R = rH = 10\ 000 \times 1.9375 = 19\ 375\ \text{bits/s}$$

Remarks

• A fixed-length PCM code would need $\lceil\log_2 6\rceil = 3$ bits/sample, i.e. $30\ 000$ bits/s, so its efficiency would be $1.9375/3 = 64.58\%$.
• Because all probabilities are powers of $\frac12$, a Huffman code (lengths 1, 2, 3, 4, 5, 5) gives exactly $\bar L = 1.9375$ bits/sample and 100 % efficiency, i.e. the line rate can be brought down to the information rate of 19.375 kbps.

Answer: entropy $H = 1.9375$ bits/sample; information rate $R = 19\ 375$ bits/s ≈ 19.375 kbps.