- Year not given · 2+3+3 marks
Preprocessing steps, binning, normalization
Short answer
Data preprocessing steps: cleaning, integration, transformation, reduction, discretization. Binning smooths noisy data by sorting, partitioning into equal-frequency or equal-width bins, and replacing values with bin mean, median, or boundary. Normalization scales attributes: min-max to [0,1], z-score using (v-mean)/σ, decimal scaling by dividing by 10^j.
List the steps of data preprocessing. Explain binning method to handle noisy data and normalization techniques for data transformation.
Steps of data preprocessing
- Data cleaning
- Data integration
- Data transformation
- Data reduction
- Data discretisation (and concept hierarchy generation)
Binning method for noisy data
Binning smooths sorted data by using its neighbourhood ("local smoothing"). Steps: sort the data, partition into equal-frequency (equal depth) or equal-width bins, then replace each value by the bin mean, median or nearest boundary.
Data: 4, 8, 15, 21, 21, 24, 25, 28, 34 (bins of 3 values)
| Bin | Original | By means | By boundaries |
|---|---|---|---|
| 1 | 4, 8, 15 | 9, 9, 9 | 4, 4, 15 |
| 2 | 21, 21, 24 | 22, 22, 22 | 21, 21, 24 |
| 3 | 25, 28, 34 | 29, 29, 29 | 25, 25, 34 |
Normalisation techniques
Normalisation scales attribute values to a small range so that attributes with large ranges do not dominate. Example data: 200, 300, 400, 600, 1000 (min = 200, max = 1000, mean = 500, standard deviation σ = 282.84 using the population formula).
- Min-max normalisation to [0, 1]: v' = (v − min) / (max − min)
- 200 → 0; 300 → 100/800 = 0.125; 400 → 0.25; 600 → 0.5; 1000 → 1.
- Z-score normalisation: v' = (v − mean) / σ
- 200 → −1.06; 300 → −0.71; 400 → −0.35; 600 → 0.35; 1000 → 1.77.
- Decimal scaling: v' = v / 10ʲ, with j the smallest integer so max(|v'|) < 1. Here j = 4.
- 200 → 0.02; 300 → 0.03; 400 → 0.04; 600 → 0.06; 1000 → 0.1.
Answer: min-max gives 0, 0.125, 0.25, 0.5, 1; z-score gives −1.06, −0.71, −0.35, 0.35, 1.77; decimal scaling gives 0.02, 0.03, 0.04, 0.06, 0.1.
✓ Answer checked by Claude Opus 5.5 (high effort)
Related questions
- Data preprocessing and handling noisy data · Data Mining, chapter 2
- Data cleaning: missing, noisy, outliers · Data Mining, chapter 2
- Short note: Data preprocessing · Data Mining, chapter 2
- Processes in data preprocessing · Data Mining, chapter 2
- How transformation reduces design steps; band stop design · Filter Design, chapter 3