---
title: "Overfitting and pruning"
subject: "Data Mining"
chapter: 3
key: "semester_7/data_mining:overfitting-pruning"
source: https://www.bctengineer.com/q/semester_7/data_mining/overfitting-pruning
site: BCT Engineering
---

# Overfitting and pruning

Data Mining · Chapter 3: Classification

- Asked in: year not given
- Video: [How to Avoid Overfitting in Decision Tree Learning | Machine Learning | Data Mining by Mahesh Huddar](https://www.youtube.com/watch?v=__syf8H9-Pg) — Mahesh Huddar
- Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14
- Page: https://www.bctengineer.com/q/semester_7/data_mining/overfitting-pruning
- Whole chapter: https://www.bctengineer.com/md/semester_7/data_mining/chapter-3

## Question

What do you mean by overfitting? What are approaches to avoid it? Differentiate Pre-pruning and Post-pruning.

## Short answer

Overfitting occurs when a model fits training noise, giving low training error but high test error. Avoid it by pruning, using more and cleaner data, limiting model depth, or cross-validation. Pre-pruning stops tree growth early if a split is not good enough, while post-pruning grows the full tree and then replaces subtrees with leaves when error does not increase; pre-pruning is faster but may underfit, and post-pruning is usually more accurate.

## Answer

**Overfitting** occurs when a model fits the training data too closely, including noise and chance patterns, so it has very low training error but a high error on unseen (test) data. In a decision tree it appears as a very large tree with many branches that reflect noise or outliers. Causes: noisy data, too few training records, and a too complex model.

#### Approaches to avoid overfitting
- **Pruning** (pre- and post-).
- Use a larger and cleaner training set; remove noise and outliers.
- Limit the model: maximum depth, minimum records per node, minimum gain.
- Use a **validation set** or cross-validation to select the model size.
- Use the **MDL principle** (prefer the model with the shortest total description length).
- Use ensembles (bagging, random forest).

#### Pre-pruning vs post-pruning
| Basis | Pre-pruning (early stopping) | Post-pruning |
| --- | --- | --- |
| When | During tree construction | After the full tree is grown |
| Method | Stop splitting if gain/records/purity fall below a threshold | Replace subtrees by leaves if error does not increase |
| Tree size | Never grows the full tree | Grows the full tree first |
| Cost | Cheaper, faster | More computation |
| Risk | Underfitting; stopping too early may miss a good later split (horizon effect) | Wasted effort on growing branches later cut |
| Accuracy | Usually less accurate | Usually more accurate and preferred |
| Examples | Minimum support, χ² test, max depth | Reduced-error pruning, cost-complexity pruning (CART), pessimistic pruning (C4.5) |

## Check yourself

The answer's key points as multiple-choice questions, with the correct option and why.

1. Overfitting means the model has:
   A. High error on both
   B. Low test error but high training error
   C. Very low training error but high error on unseen data
   D. Too few parameters
   Answer: C. It fits noise. (Key point: Overfitting)

2. Pre-pruning:
   A. Cuts subtrees after full growth
   B. Removes outliers only
   C. Grows the full tree first
   D. Stops splitting during construction
   Answer: D. Early stopping. (Key point: Pre-pruning)

3. What is the risk of pre-pruning?
   A. Underfitting from stopping too early (horizon effect)
   B. Wasted effort growing branches later cut
   C. Higher cost
   D. Overfitting always
   Answer: A. Post-pruning wastes effort instead. (Key point: Risk)

4. Which is usually more accurate?
   A. Pre-pruning
   B. Post-pruning
   C. Neither
   D. They are always equal
   Answer: B. Post-pruning is preferred though costlier. (Key point: Accuracy)

## Related in Data Mining

- [ID3 steps, overfitting, TP/FP/recall](https://www.bctengineer.com/question.md?key=semester_7%2Fdata_mining%3Aid3-steps-overfitting-recall) — Data Mining, chapter 3
- [Major clustering approaches](https://www.bctengineer.com/question.md?key=semester_7%2Fdata_mining%3Aclustering-approaches) — Data Mining, chapter 5
- [Overfitting in ID3 and decision tree rules](https://www.bctengineer.com/question.md?key=semester_7%2Fdata_mining%3Aid3-overfitting-rules) — Data Mining, chapter 3

## Related in other subjects

- [Approaches to avoid distributed deadlock](https://www.bctengineer.com/question.md?key=semester_7%2Fdistributed_system%3Adeadlock-avoidance-approaches) — Distributed System, chapter 8
- [Pre-qualification steps vs post-qualification](https://www.bctengineer.com/question.md?key=semester_8%2Fconstruction_management%3Aprequalification-steps-post) — Construction Management, chapter 5
- [FDNR to avoid inductors; 3rd order LPF at 4000 rad/s](https://www.bctengineer.com/question.md?key=semester_5%2Ffilter_design%3Afdnr-third-order-4000) — Filter Design, chapter 8, asked 2×
- [Semi-theoretical, Kennedy, Lacey approaches](https://www.bctengineer.com/question.md?key=semester_6%2Firrigation_and_drainage_engineering%3Asemi-theoretical-kennedy-lacey-compare) — Irrigation and Drainage Engineering, chapter 4, asked 2×