ROC curve for model selection

A Receiver Operating Characteristic (ROC) curve is a graph that shows the trade-off between the true positive rate and the false positive rate of a classifier as its decision threshold is varied. It is used to compare classification models visually.

• Y-axis: TPR (sensitivity) = TP/(TP + FN)
• X-axis: FPR (1 − specificity) = FP/(FP + TN)

Construction

Rank the test tuples by the classifier's probability (or score) for the positive class, in decreasing order. Starting from (0, 0), move through the list: for each tuple that is truly positive move up, and for each negative move right (by the corresponding steps). Each threshold gives one point (FPR, TPR).

TPR
1.0 |        ,-----------*  perfect: (0,1)
    |     ,-'   model M1
    |   ,'    ..'' model M2
    |  / ..''    .  diagonal = random guess
    | /.'
0.0 +-----------------------> FPR
    0                       1.0

Role in model selection

• The diagonal line from (0, 0) to (1, 1) is a random classifier; a better model has a curve closer to the top-left corner (0, 1).
• The area under the curve (AUC) summarises accuracy: 0.5 = random, 1.0 = perfect. The model with the larger AUC is preferred. Rule of thumb: 0.9-1.0 excellent, 0.8-0.9 good, 0.7-0.8 fair.
• The shape of the curve helps in choosing a threshold matching the cost of errors (e.g. high TPR for cancer screening).
• It is independent of class distribution and costs, so is useful with imbalanced data.
• Two models can be compared across all thresholds, not only at one operating point.