Anomaly detection challenges and schemes

Challenges in anomaly detection

1. Defining normal behaviour: the boundary between normal and abnormal is not clear, and normal behaviour keeps changing.
2. Different domains: the notion of an anomaly depends on the application, so a method cannot easily be transferred.
3. Lack of labelled data: labelled anomalies are rare and costly to get; training data may be contaminated.
4. Noise that looks like anomalies, and adversaries making anomalies look normal (fraudsters).
5. Class imbalance: anomalies are only a tiny fraction (< 1%) of the data; accuracy is a poor measure and false alarms are costly (base-rate fallacy).
6. High dimensionality and different attribute types.
7. Choosing the right number or degree of outlierness (a threshold), and interpretability of results.
8. Real-time requirements for streaming data.

Types of anomaly detection schemes

1. By availability of labels
   • Supervised: both normal and anomaly labels available; a classifier is built (class imbalance is a problem).
   • Semi-supervised: only normal data labelled; model of normal; deviation = anomaly.
   • Unsupervised: no labels; assumes anomalies are rare and different.
2. By method
   • Statistical (model-based): assume a distribution (e.g. normal); objects with low probability are outliers.
   • Proximity (distance)-based: an object far from its neighbours is an outlier.
   • Density-based: object in a region of much lower density than its neighbours (LOF).
   • Clustering-based: objects not belonging to any large cluster, or far from the centroid.
   • Classification-based, and graphical (box plot, scatter plot) approaches.
3. By type of anomaly: point anomaly, contextual (conditional) anomaly (e.g. 35 °C in winter), and collective anomaly (a group of related records).