KDD phases with example

KDD (Knowledge Discovery in Databases) is the complete process of discovering useful knowledge from large data. Data mining is only one step (the core step) of this process, so KDD is not the same as data mining: data mining is a part of KDD, while KDD also includes preparation of data before mining and interpretation of results after mining.

Relation between KDD and data mining

• KDD is the overall, multi-step, iterative process; data mining is the step that applies algorithms to find patterns.
• KDD starts with raw data and ends with usable knowledge; data mining starts with prepared data and ends with patterns.
• In many books and in industry, the two terms are used loosely as synonyms, but strictly data mining is one phase of KDD.

Phases of KDD

 Raw data -> [1 Data cleaning] -> [2 Data integration]
          -> [3 Selection] -> [4 Transformation]
          -> [5 Data mining] -> [6 Pattern evaluation]
          -> [7 Knowledge presentation]

1. Data cleaning: remove noise and inconsistent data, fill missing values.
2. Data integration: combine data from several sources (databases, files, web) into one store.
3. Data selection: retrieve only the data relevant to the analysis task.
4. Data transformation: convert data into forms suitable for mining (normalisation, aggregation, generalisation).
5. Data mining: apply algorithms (classification, clustering, association) to extract patterns.
6. Pattern evaluation: identify the truly interesting patterns using measures such as support and confidence.
7. Knowledge presentation: show the knowledge to the user with visualisation and reports.

Example: supermarket

A supermarket chain wants to know which products are bought together.

1. Cleaning: remove duplicate bills and fix missing product codes.
2. Integration: merge billing data of all branches with the customer-card database.
3. Selection: choose only the last 12 months of sales transactions.
4. Transformation: convert each bill into a list of items bought (basket format).
5. Mining: run the Apriori algorithm to find frequent itemsets.
6. Evaluation: keep rules with support ≥ 2% and confidence ≥ 60%, e.g. {bread} → {butter} (80%).
7. Presentation: show the rules to the manager, who places butter near bread and runs combo offers.