---
title: "Define information and entropy"
subject: "Data Communication"
chapter: 8
key: "semester_5/data_communication:define-information-entropy"
source: https://www.bctengineer.com/q/semester_5/data_communication/define-information-entropy
site: BCT Engineering
---

# Define information and entropy

Data Communication · Chapter 8: Information Theory and Coding

- Asked in: 2073 Shrawan, 2 marks; 2071 Shrawan, 2+2 marks; 2068 Chaitra, 4 marks (asked 3 times)
- Video: [Information entropy | Journey into information theory | Computer Science | Khan Academy](https://www.youtube.com/watch?v=2s3aJfRr9gE) — Khan Academy Labs
- Answer checked by Claude Sonnet 5.5 on 2026-10-14
- Page: https://www.bctengineer.com/q/semester_5/data_communication/define-information-entropy
- Whole chapter: https://www.bctengineer.com/md/semester_5/data_communication/chapter-8

## Question

Define information and entropy with examples.

## Short answer

Information of an event x with probability p(x) is I(x) = -log2 p(x) bits; a fair coin toss gives 1 bit. Entropy H(X) is the average information per symbol, H = -Σ p_i log2 p_i bits/symbol. For symbols with probabilities 1/2,1/4,1/8,1/8, H = 1.75 bits/symbol.

## Answer

#### Information
The **information** carried by a message is the amount of surprise or uncertainty removed when it is received. For a message x with probability *p(x)*:

I(x) = log₂ (1 / p(x)) = −log₂ p(x) bits

Rarer messages carry more information; a certain event (p = 1) carries 0 bits. Example: a fair coin toss gives I = log₂ 2 = 1 bit. Drawing one of 8 equally likely cards: I = log₂ 8 = 3 bits.

#### Entropy
The **entropy** H(X) of a source is the **average information per symbol**:

H(X) = −Σ pᵢ log₂ pᵢ  bits/symbol

It measures the uncertainty of the source. Properties: 0 ≤ H ≤ log₂ M for M symbols, with the maximum when all symbols are equally likely, and H = 0 when one symbol is certain.

**Example:** a source with symbols A, B, C, D having p = 1/2, 1/4, 1/8, 1/8.
H = ½·1 + ¼·2 + ⅛·3 + ⅛·3 = 0.5 + 0.5 + 0.375 + 0.375 = **1.75 bits/symbol**
(If all four were equally likely, H = log₂ 4 = 2 bits.)

If the source emits r symbols per second, the information rate R = r·H bits/s.

## Check yourself

The answer's key points as multiple-choice questions, with the correct option and why.

1. What is information I(x)?
   A. log2(1/p(x)) bits
   B. p(x)
   C. p(x) log p
   D. 1/log p
   Answer: A. Rarer messages carry more information. (Key point: Information)

2. A certain event (p = 1) carries
   A. 1 bit
   B. 0 bits
   C. Infinite
   D. 2 bits
   Answer: B. log2(1) = 0. (Key point: Certain event)

3. Entropy is
   A. Total information in a message
   B. Average information per symbol
   C. Maximum symbol probability
   D. Information rate in bits/s
   Answer: B. H = -sum p log2 p bits/symbol. (Key point: Entropy)

4. For p = 1/2, 1/4, 1/8, 1/8 entropy is
   A. 2 bits/symbol
   B. 1.75 bits/symbol
   C. 1.5 bits/symbol
   D. 1 bit/symbol
   Answer: B. 0.5+0.5+0.375+0.375 = 1.75. (Key point: Example)

## Related in Data Communication

- [Entropy, information and channel capacity](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Ainformation-entropy-capacity) — Data Communication, chapter 8
- [Entropy and information rate, 5 outcomes](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Aentropy-info-rate-5-outcomes) — Data Communication, chapter 8

## Related in other subjects

- [Message vs information; entropy](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Amessage-vs-information-entropy) — Communication Systems, chapter 6
- [Entropy and information rate; Shannon limit as bandwidth → ∞](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Aentropy-info-rate-shannon-limit) — Communication Systems, chapter 6
- [Entropy and information rate of 6-level PCM signal](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Aentropy-info-rate-5khz-six-levels) — Communication Systems, chapter 6
- [Information and entropy; capacity as bandwidth → ∞](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Ainformation-entropy-capacity-infinite) — Communication Systems, chapter 6