---
title: "Information and average information (entropy)"
subject: "Communication Systems"
chapter: 6
key: "semester_6/communication_systems:information-average-entropy"
source: https://www.bctengineer.com/q/semester_6/communication_systems/information-average-entropy
site: BCT Engineering
---

# Information and average information (entropy)

Communication Systems · Chapter 6: Baseband Digital Data Transmission

- Asked in: 2068 Jestha (old course), 4+6 marks
- Video: [Entropy (Basics, Definition, Calculation & Properties) Explained in Digital Communication](https://www.youtube.com/watch?v=BBdilQkH3gE) — Engineering Funda
- Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14
- Page: https://www.bctengineer.com/q/semester_6/communication_systems/information-average-entropy
- Whole chapter: https://www.bctengineer.com/md/semester_6/communication_systems/chapter-6

## Question

Define information. Derive the expression for evaluating the average amount of information contained in a statistically independent long sequence of symbols.

## Short answer

Information of event x_k is I(x_k) = -log2(p_k) bits. For a long independent sequence of N symbols, total information is N Σ p_k log2(1/p_k), so average entropy H = -Σ p_k log2(p_k) bits/symbol.

## Answer

#### Information
The **amount of information** in an event measures the uncertainty it removes: the less likely the event, the more information its occurrence gives. For a symbol $x_k$ with probability $p_k$, the self-information is

$$
I(x_k) = \log_2 \frac{1}{p_k} = -\log_2 p_k\ \text{bits}
$$

Properties:
- $I = 0$ when $p_k = 1$ (a certain event gives no information).
- $I \ge 0$, and $I(x_j) > I(x_k)$ if $p_j < p_k$.
- For independent events, information adds: $I(x_j x_k) = I(x_j) + I(x_k)$.

Units: bits (base 2), nats (base $e$) or decits (base 10). Example: a fair coin toss gives $\log_2 2 = 1$ bit.

#### Average information (entropy) of a long sequence
Consider a discrete memoryless source with alphabet $\{x_1, x_2, \dots, x_M\}$ and probabilities $\{p_1, p_2, \dots, p_M\}$, where $\sum p_k = 1$. The symbols are statistically independent.

**Step 1.** Take a long message of $N$ symbols ($N \to \infty$). By the law of large numbers, symbol $x_k$ occurs about

$$
n_k = p_k N \text{ times}
$$

**Step 2.** Each occurrence of $x_k$ carries $\log_2 (1/p_k)$ bits. So the information from all occurrences of $x_k$ is

$$
I_k = n_k \log_2 \frac{1}{p_k} = N p_k \log_2 \frac{1}{p_k}
$$

**Step 3.** The symbols are independent, so the information adds. The total information in the message is

$$
I_{total} = \sum_{k=1}^{M} I_k = N \sum_{k=1}^{M} p_k \log_2 \frac{1}{p_k}
$$

**Step 4.** The average information per symbol, called the **entropy**, is

$$
H = \frac{I_{total}}{N} = \sum_{k=1}^{M} p_k \log_2 \frac{1}{p_k} = -\sum_{k=1}^{M} p_k \log_2 p_k\ \text{bits/symbol}
$$

**Information rate:** if the source emits $r$ symbols/s, then $R = rH$ bits/s.

**Properties of entropy**
- $0 \le H \le \log_2 M$.
- $H = 0$ when one symbol has $p = 1$ (no uncertainty).
- $H = \log_2 M$ (the maximum) when all symbols are equally likely.
- For a binary source with $P(1) = p$: $H(p) = -p\log_2 p - (1-p)\log_2(1-p)$, which peaks at 1 bit when $p = 0.5$.

**Example:** a source with probabilities $\tfrac12, \tfrac14, \tfrac18, \tfrac18$ has

$$
H = \tfrac12(1) + \tfrac14(2) + \tfrac18(3) + \tfrac18(3) = 1.75\ \text{bits/symbol}
$$

## Check yourself

The answer's key points as multiple-choice questions, with the correct option and why.

1. What is the information in a symbol of probability p?
   A. 1/p
   B. log2(1/p)
   C. p^2
   D. p log2 p
   Answer: B. I = log2(1/p) = -log2 p bits, so rarer symbols carry more information. (Key point: Self-information)

2. In the derivation, how many times does symbol x_k appear in a long message of N symbols?
   A. p_k N
   B. N/p_k
   C. N p_k^2
   D. N + p_k
   Answer: A. By the law of large numbers n_k = p_k N, and each occurrence carries log2(1/p_k) bits. (Key point: Derivation step)

3. Dividing total information by N gives:
   A. H = N sum p_k log2 p_k
   B. H = sum log2 p_k
   C. H = sum p_k^2
   D. H = sum p_k log2(1/p_k)
   Answer: D. H = I_total/N = sum p_k log2(1/p_k) bits/symbol. (Key point: Entropy result)

4. When is entropy of an M-symbol source maximum?
   A. One symbol has p=1
   B. Probabilities are very unequal
   C. All symbols equally likely, H = log2 M
   D. H is never bounded
   Answer: C. 0 <= H <= log2 M, the maximum occurring when all symbols are equally likely. (Key point: Maximum entropy)

## Related in Communication Systems

- [Symbol rate, entropy and information rate (6 symbols per 10 μs)](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Asymbol-rate-six-symbols-10us) — Communication Systems, chapter 6
- [Entropy and information rate; Shannon limit as bandwidth → ∞](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Aentropy-info-rate-shannon-limit) — Communication Systems, chapter 6
- [Message vs information; 5 symbols per 10 μs](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Amessage-info-five-symbols-10us) — Communication Systems, chapter 6
- [Message vs information; entropy](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Amessage-vs-information-entropy) — Communication Systems, chapter 6
- [Entropy and information rate of 6-level PCM signal](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Aentropy-info-rate-5khz-six-levels) — Communication Systems, chapter 6
- [Information and entropy; capacity as bandwidth → ∞](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Ainformation-entropy-capacity-infinite) — Communication Systems, chapter 6

## Related in other subjects

- [Define information and entropy](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Adefine-information-entropy) — Data Communication, chapter 8, asked 3×
- [Entropy, information and channel capacity](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Ainformation-entropy-capacity) — Data Communication, chapter 8
- [Entropy and information rate, 5 outcomes](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Aentropy-info-rate-5-outcomes) — Data Communication, chapter 8
- [Entropy, information gain and ID3 tree](https://www.bctengineer.com/question.md?key=semester_7%2Fdata_mining%3Aid3-entropy-information-gain) — Data Mining, chapter 3