Data cleaning: missing, noisy, outliers

Data cleaning (data cleansing) is the process of detecting and correcting errors and inconsistencies in data to improve its quality. Real data is incomplete (missing values), noisy (errors and outliers) and inconsistent.

1. Handling missing data

• Ignore/delete the record: used when the class label is missing or many attributes are missing; poor when percentage of missing values varies a lot.
• Fill manually: accurate but slow and infeasible for large data.
• Fill with a global constant such as "Unknown" (may mislead the miner).
• Fill with the attribute mean or median (median for skewed data), or the mean of the same class.
• Fill with the most probable value predicted by regression, a decision tree or Bayesian inference. This is the best method.

2. Handling noisy data

• Binning: sort the values, divide into bins, then smooth by bin means, bin medians or bin boundaries. Example: bin (4, 8, 15) becomes (9, 9, 9) by means.
• Regression: fit a function to the data and replace values by the fitted values.
• Clustering: values falling outside the clusters are noise or outliers.
• Combined computer and human inspection: the computer flags suspicious values, a person checks them.

3. Handling outliers

• Detect using statistics (z-score beyond ±3, IQR rule: below Q1 − 1.5·IQR or above Q3 + 1.5·IQR), clustering or distance-based methods, or graphs (box plot).
• Then remove them if they are errors, cap them (winsorising), transform them (log), or keep them if they are genuine and important (as in fraud detection).

Cleaning is iterative: discrepancy detection and data transformation steps are repeated until the data is acceptable.