Dimensionality reduction techniques

Dimensionality reduction reduces the number of attributes (dimensions) so that the data become easier to store, visualise and mine, and the curse of dimensionality is avoided. There are two main families.

1. Feature extraction (projection) techniques

New attributes are built as combinations of the old ones.

• PCA (Principal Component Analysis): finds orthogonal directions (principal components) of maximum variance. Steps: normalise the data, compute the covariance matrix, find its eigenvectors/eigenvalues, sort eigenvalues in descending order, keep the top k eigenvectors, and project the data on them. Keeping components that explain, say, 95% of the variance gives a much smaller data set.
• SVD (Singular Value Decomposition): factors the data matrix A = UΣVᵀ and keeps the largest singular values.
• LDA (Linear Discriminant Analysis): supervised; finds directions that best separate classes.
• Wavelet transform (DWT): keeps only the strongest wavelet coefficients; good for signals and images.
• Non-linear methods: t-SNE, Isomap, autoencoders.

2. Feature selection techniques

A subset of the original attributes is kept.

• Filter: score each attribute independently (information gain, chi-square, correlation) and pick the best.
• Wrapper: evaluate subsets by the accuracy of a mining model (forward selection, backward elimination, decision-tree induction).
• Embedded: selection happens during training (L1/Lasso, tree importance).
• Removing attributes with low variance or high mutual correlation.

Result

Smaller data, faster algorithms, less noise and overfitting, and better visualisation. The price is some loss of information, so the reduction should keep the structure needed for the task.