---
title: "Entropy and information rate of 6-level PCM signal"
subject: "Communication Systems"
chapter: 6
key: "semester_6/communication_systems:entropy-info-rate-5khz-six-levels"
source: https://www.bctengineer.com/q/semester_6/communication_systems/entropy-info-rate-5khz-six-levels
site: BCT Engineering
---

# Entropy and information rate of 6-level PCM signal

Communication Systems · Chapter 6: Baseband Digital Data Transmission

- Asked in: 2074 Asoj (CS II), 5+5 marks
- Video: [Calculate Source Entropy and Information Rate | Communication system.](https://www.youtube.com/watch?v=pi1XA8j5dcY) — ELECTRICAL HUB
- Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14
- Page: https://www.bctengineer.com/q/semester_6/communication_systems/entropy-info-rate-5khz-six-levels
- Whole chapter: https://www.bctengineer.com/md/semester_6/communication_systems/chapter-6

## Question

A continuous signal is band limited to 5 kHz. The signal is quantized in 6 levels of a PCM system with the probabilities 1/2, 1/4, 1/8, 1/16, 1/32 and 1/32. Calculate the entropy and information rate.

## Short answer

Entropy is 1.9375 bits/sample. Information rate equals Nyquist sampling rate (10 kHz) times entropy, giving 19,375 bits/s (19.375 kbps).

## Answer

**Given:** band limit $f_m = 5$ kHz; 6 quantization levels with probabilities $\frac12, \frac14, \frac18, \frac1{16}, \frac1{32}, \frac1{32}$ (sum = 1).

#### Entropy

$$
H = \sum_{i=1}^{6} p_i\log_2\frac{1}{p_i}
$$

| Symbol | $p_i$ | $\log_2(1/p_i)$ | $p_i\log_2(1/p_i)$ |
|---|---|---|---|
| Q1 | 0.5 | 1.0000 | 0.5000 |
| Q2 | 0.25 | 2.0000 | 0.5000 |
| Q3 | 0.125 | 3.0000 | 0.3750 |
| Q4 | 0.0625 | 4.0000 | 0.2500 |
| Q5 | 0.03125 | 5.0000 | 0.15625 |
| Q6 | 0.03125 | 5.0000 | 0.15625 |
| **Total** | 1 | | **1.9375** |

$$
\begin{aligned}
H &= \tfrac12(1) + \tfrac14(2) + \tfrac18(3) + \tfrac1{16}(4) + \tfrac1{32}(5) + \tfrac1{32}(5) \\
&= 0.5 + 0.5 + 0.375 + 0.25 + 0.15625 + 0.15625 \\
&= 1.9375\ \text{bits/sample}
\end{aligned}
$$

For comparison, if the six levels were equally likely, $H_{max} = \log_2 6 = 2.585$ bits/sample. The unequal probabilities reduce the average information.

#### Information rate

The signal is sampled at the Nyquist rate:

$$
r = f_s = 2f_m = 2 \times 5\ \text{kHz} = 10\ 000\ \text{samples/s}
$$

Each sample is one message (level), so

$$
R = rH = 10\ 000 \times 1.9375 = 19\ 375\ \text{bits/s}
$$

#### Remarks

- A fixed-length PCM code would need $\lceil\log_2 6\rceil = 3$ bits/sample, i.e. $30\ 000$ bits/s, so its efficiency would be $1.9375/3 = 64.58\%$.
- Because all probabilities are powers of $\frac12$, a Huffman code (lengths 1, 2, 3, 4, 5, 5) gives exactly $\bar L = 1.9375$ bits/sample and 100 % efficiency, i.e. the line rate can be brought down to the information rate of 19.375 kbps.

**Answer:** entropy $H = 1.9375$ bits/sample; information rate $R = 19\ 375$ bits/s ≈ **19.375 kbps**.

## Check yourself

The answer's key points as multiple-choice questions, with the correct option and why.

1. Which formula gives the entropy?
   A. H = sum p_i log2(1/p_i)
   B. H = sum p_i^2
   C. H = log2 6 always
   D. H = sum p_i
   Answer: A. Entropy is the average information per symbol, the sum of p_i log2(1/p_i). (Key point: Entropy formula)

2. The entropy of this 6-level source is:
   A. 2.585 bits/sample
   B. 1.9375 bits/sample
   C. 1 bit/sample
   D. 3 bits/sample
   Answer: B. 0.5 + 0.5 + 0.375 + 0.25 + 0.15625 + 0.15625 = 1.9375. (Key point: Entropy value)

3. A 5 kHz band-limited signal at the Nyquist rate gives:
   A. 5000 samples/s
   B. 10000 samples/s
   C. 2500 samples/s
   D. 20000 samples/s
   Answer: B. fs = 2 fm = 10 kHz, so r = 10000 symbols/s. (Key point: Sampling rate)

4. The information rate is:
   A. 30000 bits/s
   B. 19375 bits/s
   C. 10000 bits/s
   D. 25850 bits/s
   Answer: B. R = rH = 10000 x 1.9375 = 19375 bits/s; fixed 3-bit PCM would use 30000 bits/s. (Key point: Information rate)

## Related in Communication Systems

- [Information rate of a quantized 100 Hz signal](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Ainformation-rate-100hz-four-symbols) — Communication Systems, chapter 6
- [Bits per sample and information rate (4.5 kHz, 8 levels)](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Abits-info-rate-4-5khz-8-levels) — Communication Systems, chapter 6
- [PCM bit rate for 4.5 MHz TV signal with 1024 levels](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Apcm-tv-4-5mhz-1024) — Communication Systems, chapter 4
- [Uniform vs non-uniform quantization; 16-level PCM numerical](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Auniform-nonuniform-16-level-pcm) — Communication Systems, chapter 4
- [Symbol rate, entropy and information rate (6 symbols per 10 μs)](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Asymbol-rate-six-symbols-10us) — Communication Systems, chapter 6
- [PAM, PWM, PPM; PCM of 4.8 MHz TV signal with 512 levels](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Apam-pwm-ppm-tv-4-8mhz) — Communication Systems, chapter 4

## Related in other subjects

- [Band-limited signals; aliasing; Nyquist rate](https://www.bctengineer.com/question.md?key=semester_4%2Fsignals_and_systems%3Abandlimited-aliasing-nyquist) — Signals and Systems, chapter 4
- [Aliasing in a band-limited signal](https://www.bctengineer.com/question.md?key=semester_4%2Fsignals_and_systems%3Aaliasing-bandlimited-w) — Signals and Systems, chapter 4
- [Entropy and information rate, 5 outcomes](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Aentropy-info-rate-5-outcomes) — Data Communication, chapter 8
- [File size: 20 s, 44.1 KHz, 16-bit stereo](https://www.bctengineer.com/question.md?key=semester_8%2Fmultimedia_system%3Aaudio-size-20s-44khz-16bit) — Multimedia System, chapter 2, asked 3×