---
title: "Entropy and information rate; Shannon limit as bandwidth → ∞"
subject: "Communication Systems"
chapter: 6
key: "semester_6/communication_systems:entropy-info-rate-shannon-limit"
source: https://www.bctengineer.com/q/semester_6/communication_systems/entropy-info-rate-shannon-limit
site: BCT Engineering
---

# Entropy and information rate; Shannon limit as bandwidth → ∞

Communication Systems · Chapter 6: Baseband Digital Data Transmission

- Asked in: 2076 Chaitra (CS II), 4+6 marks
- Video: [Channel Capacity by Shannon-Hartley | Basics, Proof & Maximum Bandwidth Condition](https://www.youtube.com/watch?v=3ekWsXeZ8TM) — Engineering Funda
- Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14
- Page: https://www.bctengineer.com/q/semester_6/communication_systems/entropy-info-rate-shannon-limit
- Whole chapter: https://www.bctengineer.com/md/semester_6/communication_systems/chapter-6

## Question

State and derive the relation between entropy and information rate. Derive the expression which shows the limit of Shannon's channel capacity theorem when bandwidth tends to infinity.

## Short answer

Entropy H is the average information per symbol in bits/symbol. For a source emitting r symbols per second, information rate R = r H bits/s. As bandwidth B → ∞, Shannon capacity approaches C∞ = (S/N0) log2 e ≈ 1.44 S/N0 bits/s, with a required Eb/N0 of at least -1.6 dB.

## Answer

#### Relation between entropy and information rate

**Entropy** $H$ is the average information per symbol, in bits/symbol. **Information rate** $R$ is the average information produced per second, in bits/s.

**Derivation:** let a discrete memoryless source emit $r$ symbols per second from an alphabet $\{s_1, \dots, s_M\}$ with probabilities $p_1, \dots, p_M$.

1. In a long time $T$ seconds the source emits $N = rT$ symbols.
2. Symbol $s_k$ occurs about $N p_k = rTp_k$ times, and each occurrence carries $\log_2(1/p_k)$ bits.
3. Total information in $T$ seconds:

$$
I_T = \sum_{k=1}^{M} rT\,p_k\log_2\frac{1}{p_k} = rT\,H
$$

4. Information per second:

$$
R = \frac{I_T}{T} = r\,H\ \text{bits/s}
$$

where $r$ is the symbol rate (symbols/s) and $H$ the entropy (bits/symbol). Example: $r = 1000$ symbols/s and $H = 1.75$ bits/symbol give $R = 1750$ bits/s. For equally likely symbols, $R_{max} = r\log_2 M$.

#### Limit of Shannon capacity when $B \to \infty$

Shannon–Hartley theorem for an AWGN channel (noise PSD $N_0$, noise power $N = N_0B$):

$$
C = B\log_2\left(1+\frac{S}{N_0B}\right)\ \text{bits/s}
$$

Rewrite with $x = \dfrac{S}{N_0B}$, so $B = \dfrac{S}{N_0 x}$:

$$
\begin{aligned}
C &= \frac{S}{N_0}\cdot\frac{1}{x}\log_2(1+x) = \frac{S}{N_0}\log_2(1+x)^{1/x}
\end{aligned}
$$

As $B \to \infty$, $x \to 0$, and $\lim_{x \to 0}(1+x)^{1/x} = e$:

$$
C_\infty = \frac{S}{N_0}\log_2 e = \frac{S}{N_0}\cdot\frac{1}{\ln 2} \approx 1.44\,\frac{S}{N_0}\ \text{bits/s}
$$

**Meaning:**

- Capacity does not become infinite when bandwidth becomes infinite, because noise power $N_0 B$ also increases. It approaches a finite limit set by $S/N_0$.
- At this limit, with energy per bit $E_b = S/C$:

$$
\frac{E_b}{N_0} = \frac{S}{N_0 C_\infty} = \ln 2 = 0.693 \;\Rightarrow\; -1.6\ \text{dB}
$$

  This is the **Shannon limit**: no coding scheme can give reliable transmission with $E_b/N_0$ below −1.6 dB.

```text
 C
 |           ......................  1.44 S/N0
 |       ..'
 |    .'
 |  .'
 | /
 +-------------------------------->  B
```

## Check yourself

The answer's key points as multiple-choice questions, with the correct option and why.

1. The information rate of a source emitting r symbols/s with entropy H is:
   A. R = H/r
   B. R = r H
   C. R = r + H
   D. R = H^r
   Answer: B. In T seconds the source gives rT H bits, so R = rH bits/s. (Key point: Information rate relation)

2. In the derivation, how many times does symbol s_k occur in time T?
   A. About r T p_k times
   B. About p_k / (rT) times
   C. Exactly M times
   D. About T times
   Answer: A. N = rT symbols are emitted and s_k occurs a fraction p_k of them, each carrying log2(1/p_k) bits. (Key point: Derivation step)

3. What does C approach as B tends to infinity?
   A. Infinity
   B. Zero
   C. 1.44 S/N0 bits/s
   D. S/N0 squared
   Answer: C. Using lim (1+x)^(1/x) = e as x = S/(N0 B) goes to 0, C_inf = (S/N0) log2 e = 1.44 S/N0. (Key point: Infinite bandwidth limit)

4. Why does capacity stay finite even with infinite bandwidth?
   A. Signal power drops to zero
   B. Noise power N0 B grows with bandwidth
   C. Sampling rate is limited
   D. The channel is nonlinear
   Answer: B. More bandwidth admits more noise, so capacity is limited by S/N0. (Key point: Why capacity saturates)

5. What is the Shannon limit for Eb/N0?
   A. 0 dB
   B. ln 2 = 0.693, about -1.6 dB
   C. 3 dB
   D. 1.44 dB
   Answer: B. Eb/N0 = S/(N0 C_inf) = ln 2, i.e. -1.6 dB; below it reliable communication is impossible. (Key point: Shannon limit)

## Related in Communication Systems

- [Information and entropy; capacity as bandwidth → ∞](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Ainformation-entropy-capacity-infinite) — Communication Systems, chapter 6
- [Shannon-Hartley theorem; capacity limit 1.44S/N₀](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Ashannon-hartley-1-44) — Communication Systems, chapter 6, asked 2×
- [Shannon channel capacity theorem and its limitations](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Ashannon-capacity-limitations) — Communication Systems, chapter 6, asked 2×
- [Shannon channel capacity theorem and its implication](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Ashannon-capacity-implication) — Communication Systems, chapter 6
- [Duobinary error; entropy; capacity as B → ∞](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Aduobinary-entropy-capacity-infinite-b) — Communication Systems, chapter 6
- [Entropy and information rate of 6-level PCM signal](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Aentropy-info-rate-5khz-six-levels) — Communication Systems, chapter 6

## Related in other subjects

- [Entropy, information and channel capacity](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Ainformation-entropy-capacity) — Data Communication, chapter 8
- [Shannon-Hartley channel capacity theorem](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Ashannon-hartley-theorem) — Data Communication, chapter 1, asked 3×
- [Entropy and information rate, 5 outcomes](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Aentropy-info-rate-5-outcomes) — Data Communication, chapter 8
- [State Nyquist and Shannon capacity formulae](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Anyquist-shannon-capacity-formula) — Data Communication, chapter 1, asked 2×