Data mining as a step in KDD

Knowledge discovery in databases (KDD) is the overall process of turning raw data into useful knowledge. Data mining is the central step of this process, where intelligent methods are applied to extract patterns from the prepared data.

 Raw data -> [1 Data cleaning] -> [2 Data integration]
          -> [3 Selection] -> [4 Transformation]
          -> [5 Data mining] -> [6 Pattern evaluation]
          -> [7 Knowledge presentation]

1. Data cleaning: remove noise and inconsistent data, fill missing values.
2. Data integration: combine data from several sources (databases, files, web) into one store.
3. Data selection: retrieve only the data relevant to the analysis task.
4. Data transformation: convert data into forms suitable for mining (normalisation, aggregation, generalisation).
5. Data mining: apply algorithms (classification, clustering, association) to extract patterns.
6. Pattern evaluation: identify the truly interesting patterns using measures such as support and confidence.
7. Knowledge presentation: show the knowledge to the user with visualisation and reports.

Position of data mining

• Steps 1 to 4 prepare the data (cleaning, integration, selection, transformation). Good preparation decides the quality of mining.
• Step 5, data mining: choose the task (classification, clustering, association, prediction), choose the algorithm (decision tree, k-means, Apriori) and run it on the transformed data to produce patterns.
• Steps 6 and 7 judge the patterns and present knowledge.
• The process is iterative: results of mining may lead back to earlier steps.

Example: for a telecom company, the data (call records) is cleaned, integrated with customer data, and selected for the last six months. Data mining then applies a decision-tree classifier to find which customers will leave. The rules are then evaluated, and the useful ones given to the marketing team.