Define information and entropy

Information

The information carried by a message is the amount of surprise or uncertainty removed when it is received. For a message x with probability p(x):

I(x) = log₂ (1 / p(x)) = −log₂ p(x) bits

Rarer messages carry more information; a certain event (p = 1) carries 0 bits. Example: a fair coin toss gives I = log₂ 2 = 1 bit. Drawing one of 8 equally likely cards: I = log₂ 8 = 3 bits.

Entropy

The entropy H(X) of a source is the average information per symbol:

H(X) = −Σ pᵢ log₂ pᵢ bits/symbol

It measures the uncertainty of the source. Properties: 0 ≤ H ≤ log₂ M for M symbols, with the maximum when all symbols are equally likely, and H = 0 when one symbol is certain.

Example: a source with symbols A, B, C, D having p = 1/2, 1/4, 1/8, 1/8. H = ½·1 + ¼·2 + ⅛·3 + ⅛·3 = 0.5 + 0.5 + 0.375 + 0.375 = 1.75 bits/symbol (If all four were equally likely, H = log₂ 4 = 2 bits.)

If the source emits r symbols per second, the information rate R = r·H bits/s.