- Year not given
Overfitting and pruning
Short answer
Overfitting occurs when a model fits training noise, giving low training error but high test error. Avoid it by pruning, using more and cleaner data, limiting model depth, or cross-validation. Pre-pruning stops tree growth early if a split is not good enough, while post-pruning grows the full tree and then replaces subtrees with leaves when error does not increase; pre-pruning is faster but may underfit, and post-pruning is usually more accurate.
What do you mean by overfitting? What are approaches to avoid it? Differentiate Pre-pruning and Post-pruning.
Overfitting occurs when a model fits the training data too closely, including noise and chance patterns, so it has very low training error but a high error on unseen (test) data. In a decision tree it appears as a very large tree with many branches that reflect noise or outliers. Causes: noisy data, too few training records, and a too complex model.
Approaches to avoid overfitting
- Pruning (pre- and post-).
- Use a larger and cleaner training set; remove noise and outliers.
- Limit the model: maximum depth, minimum records per node, minimum gain.
- Use a validation set or cross-validation to select the model size.
- Use the MDL principle (prefer the model with the shortest total description length).
- Use ensembles (bagging, random forest).
Pre-pruning vs post-pruning
| Basis | Pre-pruning (early stopping) | Post-pruning |
|---|---|---|
| When | During tree construction | After the full tree is grown |
| Method | Stop splitting if gain/records/purity fall below a threshold | Replace subtrees by leaves if error does not increase |
| Tree size | Never grows the full tree | Grows the full tree first |
| Cost | Cheaper, faster | More computation |
| Risk | Underfitting; stopping too early may miss a good later split (horizon effect) | Wasted effort on growing branches later cut |
| Accuracy | Usually less accurate | Usually more accurate and preferred |
| Examples | Minimum support, χ² test, max depth | Reduced-error pruning, cost-complexity pruning (CART), pessimistic pruning (C4.5) |
✓ Answer checked by Claude Opus 5.5 (high effort)
Related questions
- ID3 steps, overfitting, TP/FP/recall · Data Mining, chapter 3
- Major clustering approaches · Data Mining, chapter 5
- Overfitting in ID3 and decision tree rules · Data Mining, chapter 3
- Approaches to avoid distributed deadlock · Distributed System, chapter 8
- Pre-qualification steps vs post-qualification · Construction Management, chapter 5
- FDNR to avoid inductors; 3rd order LPF at 4000 rad/s · Filter Design, chapter 8 · asked 2×
- Semi-theoretical, Kennedy, Lacey approaches · Irrigation and Drainage Engineering, chapter 4 · asked 2×