Message vs information; entropy

A message is the actual symbol or sequence of symbols sent by the source (a letter, a word, a sample value). Information is the measure of the uncertainty removed when that message is received; it depends only on how probable the message is, not on its meaning.

Message vs information

Relation between message, information and entropy

1. Information of one message. If message $m_k$ occurs with probability $p_k$, its information content is

$$I_k = \log_2\frac{1}{p_k}\ \text{bits}$$

Properties: $I_k \ge 0$; $I_k = 0$ for $p_k = 1$; $I_k > I_j$ if $p_k < p_j$; for independent messages $I(m_k m_j) = I_k + I_j$.

2. Entropy = average information per message. A source emits $M$ different messages with different information. In a long sequence of $N$ messages, $m_k$ appears about $N p_k$ times, so the total information is

$$I_{total} = \sum_{k=1}^{M} N p_k \log_2\frac{1}{p_k}$$

Dividing by $N$:

$$H = \frac{I_{total}}{N} = \sum_{k=1}^{M} p_k \log_2\frac{1}{p_k}\ \text{bits/message}$$

So entropy is the expected (mean) value of the information, $H = E[I_k]$. It lies in $0 \le H \le \log_2 M$, with the maximum when all messages are equally likely.

3. Information rate. If the source sends $r$ messages per second, the information rate is $R = rH$ bits/s.

Example

A source sends four messages with probabilities $\tfrac12, \tfrac14, \tfrac18, \tfrac18$.

$$H = \tfrac12(1)+\tfrac14(2)+\tfrac18(3)+\tfrac18(3) = 1.75\ \text{bits/message}$$

A message sequence of 1000 symbols therefore carries about $1000 \times 1.75 = 1750$ bits of information, although each individual message carries 1, 2 or 3 bits. The message is the carrier; information is what it carries; entropy is the average amount carried per message.