Ten golden rules for KDD environment

A KDD environment is the set of people, data, tools and practices that supports knowledge discovery. The ten golden rules for setting up a reliable KDD environment (Adriaans and Zantinge) are:

1. Support extremely large data sets: data mining tools must handle millions of records (GBs to TBs) and scale as the data grows.
2. Support hybrid learning: no single algorithm solves every problem, so the environment should combine several techniques (decision trees, neural networks, clustering, association rules, genetic algorithms, statistics).
3. Establish a data warehouse: mining needs a clean, integrated, historical data source that is separate from the operational systems.
4. Introduce data cleaning facilities: dirty data (missing values, duplicates, inconsistent codes, pollution) must be detected and cleaned before mining, otherwise the results are misleading ("garbage in, garbage out").
5. Facilitate working with dynamic coding: KDD is iterative, so users must be able to easily recode and transform attributes (e.g. group ages into ranges, turn dates into durations), create several versions of the data set and keep track of them.
6. Integrate with decision support systems: mining results must flow into the DSS, OLAP and reporting tools that decision makers already use.
7. Choose an extendible architecture: it must be easy to add new algorithms, data sources and tools as needs change (modular, open design).
8. Support heterogeneous databases: the environment must access data from different DBMSs, legacy systems and flat files on different platforms.
9. Introduce client/server architecture: heavy mining computation runs on a powerful (possibly parallel) server, while users work with the interface and visualisation on client machines.
10. Introduce cache optimisation: learning algorithms scan the data many times, so frequently used data and intermediate results should be kept in memory or cache to speed up the repeated passes.

(Also important: the environment must be driven by a clear business goal and involve domain experts.)