---
title: "Variable length coding; entropy and fixed code efficiency"
subject: "Communication Systems"
chapter: 6
key: "semester_6/communication_systems:variable-length-coding-entropy-six"
source: https://www.bctengineer.com/q/semester_6/communication_systems/variable-length-coding-entropy-six
site: BCT Engineering
---

# Variable length coding; entropy and fixed code efficiency

Communication Systems · Chapter 6: Baseband Digital Data Transmission

- Asked in: 2072 Kartik (CS II), 2+3 marks
- Video: [Source Coding Terms | Codeword | Code Efficiency | Codeword Length | Code Redundancy](https://www.youtube.com/watch?v=0u3SX-XoaPc) — EngineersTutor
- Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14
- Page: https://www.bctengineer.com/q/semester_6/communication_systems/variable-length-coding-entropy-six
- Whole chapter: https://www.bctengineer.com/md/semester_6/communication_systems/chapter-6

## Question

Write down the significance of variable length coding with relevant example. A fixed length source encoder generates six symbols with probabilities {0.2, 0.15, 0.25, 0.05, 0.3, 0.05} and encoded by unique code word. Find entropy and maximum code efficiency.

## Short answer

Variable-length coding assigns shorter codewords to more frequent symbols, reducing average length and approaching entropy. For example, four symbols 0.5,0.25,0.125,0.125 achieve 100% efficiency vs 87.5% fixed. For given probabilities, entropy is 2.3282 bits/symbol; fixed-length code has maximum efficiency 77.61%.

## Answer

#### Significance of variable length coding

In **variable-length coding**, frequent symbols get short code words and rare symbols get long code words. Its significance:

- The average length $\bar L = \sum p_i l_i$ becomes smaller than a fixed-length code and can approach the entropy $H$, so bit rate, bandwidth and storage are reduced.
- It uses the source statistics; when probabilities are unequal, fixed-length codes waste bits.
- Prefix (instantaneous) variable-length codes such as Huffman and Shannon–Fano remain uniquely decodable.

**Example:** four symbols with $p = 0.5, 0.25, 0.125, 0.125$. Fixed code: 2 bits each, $\bar L = 2$. Variable code 0, 10, 110, 111: $\bar L = 0.5(1) + 0.25(2) + 0.125(3) + 0.125(3) = 1.75$ bits = $H$, so efficiency rises from 87.5 % to 100 %. (Morse code uses the same idea: "E" is a single dot.)

#### Numerical

Probabilities $\{0.2, 0.15, 0.25, 0.05, 0.3, 0.05\}$ (sum = 1).

**Entropy:**

| Symbol | $p_i$ | $\log_2(1/p_i)$ | $p_i\log_2(1/p_i)$ |
|---|---|---|---|
| x1 | 0.2 | 2.3219 | 0.4644 |
| x2 | 0.15 | 2.7370 | 0.4105 |
| x3 | 0.25 | 2.0000 | 0.5000 |
| x4 | 0.05 | 4.3219 | 0.2161 |
| x5 | 0.3 | 1.7370 | 0.5211 |
| x6 | 0.05 | 4.3219 | 0.2161 |
| **Total** | 1 | | **2.3282** |

$$
H = 2.3282\ \text{bits/symbol}
$$

**Fixed-length code:** 6 symbols need $\lceil\log_2 6\rceil = 3$ bits each, so $\bar L = 3$ bits/symbol.

**Maximum code efficiency** of this fixed-length encoder:

$$
\eta_{max} = \frac{H}{\bar L} = \frac{2.3282}{3} = 0.7761 = 77.61\%
$$

(Redundancy = 22.39 %.) A variable-length code, e.g. Huffman with $\bar L = 2.35$ bits, would raise the efficiency to $2.3282/2.35 = 99.07\%$.

(If "maximum efficiency" is read instead as the source efficiency relative to the maximum entropy $H_{max} = \log_2 6 = 2.585$ bits/symbol, it is $2.3282/2.585 = 90.07\%$.)

**Answer:** $H = 2.3282$ bits/symbol; maximum efficiency of the fixed-length code = **77.61 %**.

## Check yourself

The answer's key points as multiple-choice questions, with the correct option and why.

1. What is the main idea of variable-length coding?
   A. All symbols get equal length codes
   B. Frequent symbols get short codes and rare symbols get long codes
   C. Rare symbols get short codes
   D. Codes are chosen at random to avoid prefixes
   Answer: B. This lowers the average length towards the entropy, saving bits when probabilities are unequal. (Key point: Variable-length idea)

2. How many bits per symbol does a unique fixed-length code need for six symbols?
   A. 2
   B. 3
   C. 4
   D. 6
   Answer: B. ceil(log2 6) = 3 bits. (Key point: Fixed-length bits for six symbols)

3. With H = 2.3282 bits/symbol and the 3-bit fixed code, what is the code efficiency?
   A. 66.7%
   B. 77.61%
   C. 88.2%
   D. 99.07%
   Answer: B. eta = H/L = 2.3282/3 = 0.7761 = 77.61%. (Key point: Maximum efficiency)

## Related in Communication Systems

- [Importance of source coding; Shannon-Fano vs fixed length (seven symbols)](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Asource-coding-sf-fixed-seven) — Communication Systems, chapter 6
- [Source coding; Huffman for five symbols (0.3, 0.25, 0.2, 0.15, 0.1)](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Asource-coding-huffman-five-s0) — Communication Systems, chapter 6
- [Importance of source coding; fixed vs Shannon-Fano (five symbols)](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Asource-coding-fixed-sf-five) — Communication Systems, chapter 6
- [Importance of source coding; Huffman for eight symbols at 1000 symbols/s](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Asource-coding-huffman-eight-1000) — Communication Systems, chapter 6
- [Importance of source coding; Shannon-Fano vs BCD (four symbols)](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Asource-coding-sf-bcd-four) — Communication Systems, chapter 6
- [Source vs channel coding; Huffman for five symbols](https://www.bctengineer.com/question.md?key=semester_6%2Fcommunication_systems%3Asource-channel-huffman-five) — Communication Systems, chapter 6

## Related in other subjects

- [Source coding; fixed vs variable length codes](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Asource-coding-fixed-variable) — Data Communication, chapter 8
- [Huffman code for symbols A to E (0.4, 0.3, ...)](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Ahuffman-table-abcde) — Data Communication, chapter 8
- [Shannon-Fano code for six symbols](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Ashannon-fano-six-symbols) — Data Communication, chapter 8
- [Shannon-Fano code for eight symbols](https://www.bctengineer.com/question.md?key=semester_5%2Fdata_communication%3Ashannon-fano-eight-symbols) — Data Communication, chapter 8