---
title: "Entropy, information and channel capacity"
subject: "Data Communication"
chapter: 8
key: "semester_5/data_communication:information-entropy-capacity"
source: https://www.bctengineer.com/q/semester_5/data_communication/information-entropy-capacity
site: BCT Engineering
---

# Entropy, information and channel capacity

Data Communication · Chapter 8: Information Theory and Coding

- Asked in: 2069 Asar, 5 marks
- Video: [Information Theory Basics](https://www.youtube.com/watch?v=bkLHszLlH34) — Intelligent Systems Lab
- Answer checked by Claude Sonnet 5.5 on 2026-10-14
- Page: https://www.bctengineer.com/q/semester_5/data_communication/information-entropy-capacity
- Whole chapter: https://www.bctengineer.com/md/semester_5/data_communication/chapter-8

## Question

Explain the following terms with reference to information theory (i) Entropy (ii) Information (iii) Channel capacity.

## Short answer

Entropy H = −Σ pᵢ log₂ pᵢ is the average bits per symbol, measuring source uncertainty. Information of an event is I = −log₂ p bits; less probable events carry more information. Channel capacity C = B log₂(1+S/N) gives the maximum error-free data rate for bandwidth B and SNR; Shannon’s theorem states error-free transmission is possible if R ≤ C.

## Answer

#### (i) Entropy
The average information per source symbol: H = −Σ pᵢ log₂ pᵢ bits/symbol. It measures the uncertainty of the source, with 0 ≤ H ≤ log₂ M. Example: a fair coin has H = 1 bit.

#### (ii) Information
The information in a message of probability p is I = log₂(1/p) = −log₂ p bits. The less likely the event, the more information it carries; a certain event carries 0 bits. Example: an event of probability 1/8 carries 3 bits.

#### (iii) Channel capacity
The **channel capacity** C is the maximum rate (bits per second) at which information can be sent over a channel with an arbitrarily small error probability. By the Shannon-Hartley theorem, for a band-limited channel with Gaussian noise:

C = B log₂(1 + S/N) bits/s

where B is bandwidth in Hz and S/N is the signal-to-noise power ratio. Example: B = 3000 Hz and S/N = 1000 (30 dB) gives C = 3000 × log₂(1001) ≈ 29.9 kbps. Shannon's theorem says that if the information rate R ≤ C, error-free transmission is possible with suitable coding. For a binary symmetric channel with error probability p, C = 1 − H(p) bits per use.

## Check yourself

The answer's key points as multiple-choice questions, with the correct option and why.

1. What is the information carried by an event of probability 1/8?
   A. 1 bit
   B. 3 bits
   C. 8 bits
   D. 0 bits
   Answer: B. I = log2(1/p) = log2 8 = 3 bits; rarer events carry more information. (Key point: Information formula)

2. How does entropy relate to information?
   A. Entropy is the average information per source symbol, H = -Σ p log2 p
   B. Entropy is the information of the least likely symbol
   C. Entropy is the channel bandwidth
   D. Entropy is always log2 M
   Answer: A. Entropy is the probability-weighted average of the information of the symbols. (Key point: Entropy)

3. Which formula gives capacity of a band-limited Gaussian channel?
   A. C = B log2(1 + S/N)
   B. C = B (S/N)
   C. C = B log2(S/N)
   D. C = 2B log2 M only
   Answer: A. Shannon-Hartley: C = B log2(1 + S/N) bits/s, the maximum rate for arbitrarily small error. (Key point: Channel capacity)

## Related in Data Communication

- [Define information and entropy](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Adefine-information-entropy) — Data Communication, chapter 8, asked 3×
- [Entropy and information rate, 5 outcomes](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Aentropy-info-rate-5-outcomes) — Data Communication, chapter 8
- [Shannon-Hartley channel capacity theorem](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Ashannon-hartley-theorem) — Data Communication, chapter 1, asked 3×
- [Channel capacity for 6 kHz, 6 dB](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Acapacity-6khz-6db) — Data Communication, chapter 1
- [State Nyquist and Shannon capacity formulae](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Anyquist-shannon-capacity-formula) — Data Communication, chapter 1, asked 2×
- [Capacity for 3.1 kHz, SNR 0 dB](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Acapacity-3-1khz-0db) — Data Communication, chapter 1

## Related in other subjects

- [Information and entropy; capacity as bandwidth → ∞](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Ainformation-entropy-capacity-infinite) — Communication Systems, chapter 6
- [Entropy and information rate; Shannon limit as bandwidth → ∞](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Aentropy-info-rate-shannon-limit) — Communication Systems, chapter 6
- [Duobinary error; entropy; capacity as B → ∞](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Aduobinary-entropy-capacity-infinite-b) — Communication Systems, chapter 6
- [Message vs information; entropy](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Amessage-vs-information-entropy) — Communication Systems, chapter 6