---
title: "Message vs information; entropy"
subject: "Communication Systems"
chapter: 6
key: "semester_6/communication_systems:message-vs-information-entropy"
source: https://www.bctengineer.com/q/semester_6/communication_systems/message-vs-information-entropy
site: BCT Engineering
---

# Message vs information; entropy

Communication Systems · Chapter 6: Baseband Digital Data Transmission

- Asked in: 2080 Bhadra (CS II), 2+6 marks
- Video: [Entropy, Average Information and Conditions on Entropy](https://www.youtube.com/watch?v=Jvm2FKsvj4s) — RTU Wallah
- Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14
- Page: https://www.bctengineer.com/q/semester_6/communication_systems/message-vs-information-entropy
- Whole chapter: https://www.bctengineer.com/md/semester_6/communication_systems/chapter-6

## Question

Differentiate between message and information. Relate the message, the entropy and the information.

## Short answer

A message is a physical symbol, while information measures uncertainty removal: I = log2(1/p) bits. Entropy H = -∑ p log2 p is the average information per message. Thus, a source emitting messages with probabilities yields an expected information content given by its entropy.

## Answer

A **message** is the actual symbol or sequence of symbols sent by the source (a letter, a word, a sample value). **Information** is the measure of the uncertainty removed when that message is received; it depends only on how probable the message is, not on its meaning.

#### Message vs information

| Message | Information |
|---|---|
| Physical symbol(s) produced by the source | Quantity carried by the message |
| Has content and meaning | Independent of meaning |
| Counted in symbols, words or samples | Measured in bits: $I = \log_2(1/p)$ |
| A sure (expected) message is still a message | A sure message carries zero information |
| Example: "Sun rises in the east" | Almost 0 bits, since $p \approx 1$ |

#### Relation between message, information and entropy

1. **Information of one message.** If message $m_k$ occurs with probability $p_k$, its information content is

$$
I_k = \log_2\frac{1}{p_k}\ \text{bits}
$$

   Properties: $I_k \ge 0$; $I_k = 0$ for $p_k = 1$; $I_k > I_j$ if $p_k < p_j$; for independent messages $I(m_k m_j) = I_k + I_j$.

2. **Entropy = average information per message.** A source emits $M$ different messages with different information. In a long sequence of $N$ messages, $m_k$ appears about $N p_k$ times, so the total information is

$$
I_{total} = \sum_{k=1}^{M} N p_k \log_2\frac{1}{p_k}
$$

   Dividing by $N$:

$$
H = \frac{I_{total}}{N} = \sum_{k=1}^{M} p_k \log_2\frac{1}{p_k}\ \text{bits/message}
$$

   So entropy is the expected (mean) value of the information, $H = E[I_k]$. It lies in $0 \le H \le \log_2 M$, with the maximum when all messages are equally likely.

3. **Information rate.** If the source sends $r$ messages per second, the information rate is $R = rH$ bits/s.

#### Example

A source sends four messages with probabilities $\tfrac12, \tfrac14, \tfrac18, \tfrac18$.

| Message | $p_k$ | $I_k$ (bits) |
|---|---|---|
| $m_1$ | 1/2 | 1 |
| $m_2$ | 1/4 | 2 |
| $m_3$ | 1/8 | 3 |
| $m_4$ | 1/8 | 3 |

$$
H = \tfrac12(1)+\tfrac14(2)+\tfrac18(3)+\tfrac18(3) = 1.75\ \text{bits/message}
$$

A message sequence of 1000 symbols therefore carries about $1000 \times 1.75 = 1750$ bits of information, although each individual message carries 1, 2 or 3 bits. The message is the carrier; information is what it carries; entropy is the average amount carried per message.

## Check yourself

The answer's key points as multiple-choice questions, with the correct option and why.

1. Which statement is correct?
   A. Message and information are identical
   B. Information depends only on the message's probability
   C. Certain messages carry maximum information
   D. Information depends on meaning
   Answer: B. Information is I = log2(1/p), independent of meaning; a certain message carries zero. (Key point: Message vs information)

2. How is entropy related to information?
   A. Entropy is the average (expected) information per message
   B. Entropy is the least information
   C. Entropy is the total information
   D. Entropy is the maximum information
   Answer: A. H = E[I_k] = sum p_k log2(1/p_k). (Key point: Entropy relation)

3. For probabilities 1/2,1/4,1/8,1/8, the entropy is:
   A. 1.75 bits/message
   B. 2 bits/message
   C. 2.5 bits
   D. 1.5 bits
   Answer: A. 0.5(1)+0.25(2)+0.125(3)+0.125(3) = 1.75. (Key point: Example)

4. A 1000-message sequence from that source carries about:
   A. 1750 bits
   B. 2000 bits
   C. 1000 bits
   D. 3000 bits
   Answer: A. N x H = 1000 x 1.75 = 1750 bits. (Key point: Total info)

## Related in Communication Systems

- [Message vs information; 5 symbols per 10 μs](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Amessage-info-five-symbols-10us) — Communication Systems, chapter 6
- [Entropy and information rate; Shannon limit as bandwidth → ∞](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Aentropy-info-rate-shannon-limit) — Communication Systems, chapter 6
- [Entropy and information rate of 6-level PCM signal](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Aentropy-info-rate-5khz-six-levels) — Communication Systems, chapter 6
- [Information and entropy; capacity as bandwidth → ∞](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Ainformation-entropy-capacity-infinite) — Communication Systems, chapter 6
- [Symbol rate, entropy and information rate (6 symbols per 10 μs)](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Asymbol-rate-six-symbols-10us) — Communication Systems, chapter 6
- [Information and average information (entropy)](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Ainformation-average-entropy) — Communication Systems, chapter 6

## Related in other subjects

- [Define information and entropy](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Adefine-information-entropy) — Data Communication, chapter 8, asked 3×
- [Entropy, information and channel capacity](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Ainformation-entropy-capacity) — Data Communication, chapter 8
- [Entropy and information rate, 5 outcomes](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Aentropy-info-rate-5-outcomes) — Data Communication, chapter 8
- [Entropy, information gain and ID3 tree](https://www.bctengineer.com/question.md?key=semester_7%2Fdata_mining%3Aid3-entropy-information-gain) — Data Mining, chapter 3