Short note: Base rate fallacy

The base rate fallacy is the error of ignoring the prior probability (base rate) of an event when judging a conditional probability. It matters in anomaly and intrusion detection, where the real anomalies are extremely rare, so even a very accurate detector produces mostly false alarms.

Explanation (Bayes' theorem)

P(Anomaly | Alarm) = P(Alarm | Anomaly)·P(Anomaly) / [P(Alarm | Anomaly)·P(Anomaly) + P(Alarm | Normal)·P(Normal)]

Example

An intrusion detector has a detection rate (true positive rate) of 99% and a false alarm rate of 1%. Only 1 in 1000 events is a real intrusion (base rate = 0.001).

P(Intrusion | Alarm) = (0.99 × 0.001) / (0.99 × 0.001 + 0.01 × 0.999) = 0.00099 / 0.01098 ≈ 0.09

So only about 9% of alarms are real intrusions; 91% are false alarms, although the detector looks 99% accurate. Out of 100,000 events: 100 intrusions (99 detected) and 99,900 normal (999 false alarms), so 99 of 1098 alarms are true.

Consequences

• The detection rate alone is misleading; the false alarm rate must be extremely low (e.g. 1 in 10⁵) to get a useful detector.
• Too many false alarms waste investigation effort and users start to ignore the system.
• Evaluate with precision, ROC curves and cost-based measures, not with accuracy.