Processes in data preprocessing

Data preprocessing is the set of steps that turn raw, dirty data into clean, consistent data that mining algorithms can use. Real data is usually incomplete, noisy and inconsistent, so poor data gives poor patterns ("garbage in, garbage out").

Major processes

1. Data cleaning: fill missing values, smooth noisy data (binning, regression, clustering), detect and remove outliers, and correct inconsistencies.
2. Data integration: combine data from several sources (databases, files, data cubes) into one store. It resolves entity identification, redundant attributes (found by correlation analysis) and conflicting values.
3. Data transformation: convert data into a form suitable for mining.
   • Normalization (min-max, z-score, decimal scaling)
   • Aggregation and generalization (day to month, city to country)
   • Attribute construction and smoothing
4. Data reduction: get a smaller representation that gives nearly the same results.
   • Dimensionality reduction (PCA, wavelets, feature selection)
   • Numerosity reduction (sampling, histograms, clustering)
   • Data compression
5. Data discretization: replace continuous values by intervals or concept labels (e.g. age 0-18, 19-60, 60+). It is also a form of reduction.

Raw data -> Cleaning -> Integration -> Transformation
         -> Reduction -> Discretization -> Clean data

These steps are not strictly in sequence; they are repeated as needed before mining.