Metrics That Matter

Accuracy is the fraction of all predictions that are correct. It breaks down on imbalanced data, where one class dominates, because predicting the majority class every time scores well while learning nothing.

$\text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{TP + FN}$

Precision answers: of everything I flagged as positive, how much was right? Recall answers: of everything that was truly positive, how much did I catch? There is a tension between them, you can usually raise one by sacrificing the other.

The F1 score is the harmonic mean of the two, giving a single number that stays low unless both are reasonable:

$F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$

A worked example

Take a concrete confusion matrix: TP = 40, FP = 10, FN = 5, TN = 45 (100 predictions total).

1. 1Accuracy(40 + 45) / 100 = 0.85, looks great in isolation.
2. 2Precision40 / (40 + 10) = 40/50 = 0.80-20% of flags were false alarms.
3. 3Recall40 / (40 + 5) = 40/45 ≈ 0.889, we caught ~89% of true positives.
4. 4F12 · (0.80 · 0.889) / (0.80 + 0.889) ≈ 0.842, balances both.

Here recall is greater than precision, so this model misses fewer positives than it raises false alarms. Whether that trade-off is good depends entirely on your problem.

ROC and AUC

Most classifiers output a probability, and you pick a threshold to convert it to a label. The ROC curve plots the true positive rate against the false positive rate across all thresholds, and the AUC (area under that curve) summarizes it as one number from 0.5 (random) to 1.0 (perfect). AUC is threshold-independent and useful for ranking quality, but on heavily imbalanced data a precision-recall curve often tells a more honest story.

Regression metrics

When the target is a continuous number, you measure the size of prediction errors instead of counting them:

• MAE, mean absolute error, the average error magnitude in the original units, robust to outliers: $\text{MAE} = \frac{1}{n}\sum_{i=1}^{n} |y_i - \hat{y}_i|$
• RMSE, root mean squared error, which squares errors before averaging and therefore punishes large mistakes harder: $\text{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2}$
• $R^2$, the coefficient of determination, the fraction of variance explained (1.0 is perfect, 0 means no better than predicting the mean): $R^2 = 1 - \frac{\sum_i (y_i - \hat{y}_i)^2}{\sum_i (y_i - \bar{y})^2}$

Use RMSE when large errors are especially costly; use MAE when every unit of error matters equally.

Further reading

• scikit-learn, Metrics and scoring
• Google ML Crash Course, Classification
• scikit-learn, Confusion matrix reference