Clustering, need and qualities

Clustering is the process of grouping a set of data objects into multiple groups (clusters) so that objects within a cluster have high similarity to each other and are dissimilar to objects in other clusters. It is unsupervised (learning by observation): no class labels are given. Example: grouping customers of a shop by their purchase pattern.

Why cluster analysis is needed

• Discover natural groups and the structure of unlabelled data.
• Summarise and compress large data sets with cluster representatives.
• Find outliers (e.g. fraud) and prepare data for other methods.
• Applications: market segmentation, document grouping, image segmentation, gene analysis, city planning.

Qualities of a good cluster / clustering method

1. High intra-cluster similarity (compactness) and low inter-cluster similarity (good separation).
2. Scalability: works for small and very large data.
3. Ability to handle different attribute types (numeric, categorical, mixed, binary).
4. Discovery of clusters with arbitrary shape, not only spherical ones.
5. Minimal domain knowledge required to fix input parameters (such as k).
6. Ability to deal with noisy data, outliers and missing values.
7. Insensitivity to the order of input records.
8. Handling high dimensionality.
9. Incorporation of user constraints.
10. Interpretability and usability: the clusters should be meaningful.

The quality is measured by a good similarity measure and by indices such as SSE, silhouette coefficient and cohesion/separation.