Data preprocessing and handling noisy data

Data preprocessing

Real-world data is dirty (incomplete, noisy, inconsistent), and poor data gives poor mining results. Preprocessing makes the data fit for mining. The major tasks are:

1. Data cleaning: fill missing values, smooth noisy data, identify outliers, resolve inconsistencies.
2. Data integration: merge data from several sources, remove redundancy.
3. Data transformation: normalisation, aggregation, generalisation, attribute construction.
4. Data reduction: reduce volume but keep the analytical result: dimensionality reduction, numerosity reduction, compression.
5. Data discretisation: replace numeric values by intervals or concept labels.

Noisy data and the techniques to handle it

Noise is random error or variance in a measured variable (e.g. age = 250).

1. Binning: sort the data and divide it into bins (equal-frequency or equal-width); then smooth each value by its bin neighbours (local smoothing).
2. Regression: fit the data to a function (linear/multiple regression); values are smoothed to the fitted line.
3. Clustering: group similar values; values outside the clusters are outliers and are removed.
4. Combined computer and human inspection: the computer flags suspicious values, a human checks them.

Binning example

Price data (sorted): 4, 8, 15, 21, 21, 24, 25, 28, 34. Use equal-frequency bins of depth 3.

Means: (4+8+15)/3 = 9; (21+21+24)/3 = 22; (25+28+34)/3 = 29. In smoothing by boundaries, each value moves to the closer of the bin's minimum and maximum (8 is 4 from 4 and 7 from 15, so 8 → 4).

Larger bin width gives stronger smoothing.