Jaber Notes · 5 of 16

Evaluation & Validation

Precision/recall/F1, ROC/PR-AUC, calibration, imbalance, CV strategies.

Measuring a model honestly: the confusion matrix and its metrics, ROC vs PR-AUC, calibration and ECE, regression and ranking metrics, imbalance handling, and validation schemes that do not leak.

Visual reference

Confusion matrix

Predicted class along the top, actual class down the side. Precision looks at the predicted-positive column; recall looks at the actual-positive row.

Choosing the right metric is as important as choosing the right model. A metric that doesn't match your true objective will optimize the wrong thing.

Classification metrics
ROC curve and AUC
Calibration
Regression metrics
Ranking metrics
Multi-class metrics
Imbalanced data: strategies and metrics
Validation strategies
Statistical significance and comparison
Sklearn cheatsheet

1. Classification metrics

1.1 Confusion matrix

For binary classification with positive class 1:

	Predicted 0	Predicted 1
Actual 0	TN	FP
Actual 1	FN	TP

TP: True Positive (correctly predicted positive)
TN: True Negative (correctly predicted negative)
FP: False Positive (predicted positive, was negative) — Type I error
FN: False Negative (predicted negative, was positive) — Type II error

1.2 Core metrics (derivations)

Accuracy:

\text{Acc} = \frac{TP+TN}{TP+TN+FP+FN}.

When to use: classes roughly balanced. Useless when one class dominates (predicting all-majority gets high accuracy but is useless).

Precision (Positive Predictive Value):

P = \frac{TP}{TP+FP}.

"Of all I predicted positive, how many truly were?" High precision = few false alarms. Important when false positives are costly (e.g., spam filter).

Recall (Sensitivity / True Positive Rate):

R = \frac{TP}{TP+FN}.

"Of all actual positives, how many did I catch?" High recall = few missed positives. Important when false negatives are costly (e.g., cancer detection).

Specificity (True Negative Rate):

\text{Spec} = \frac{TN}{TN+FP}.

Recall for the negative class.

F1 score:

F_1 = \frac{2\cdot P \cdot R}{P + R} = \frac{2TP}{2TP+FP+FN}.

Harmonic mean of precision and recall. Harmonic mean penalizes extremes: a model with precision=1 and recall=0 gets F1=0, not 0.5.

F-beta score (generalized):

F_\beta = \frac{(1+\beta^2)\cdot P \cdot R}{\beta^2 P + R}.

$\beta > 1$ weights recall higher; $\beta < 1$ weights precision higher.

Matthews Correlation Coefficient (MCC):

\text{MCC} = \frac{TP\cdot TN - FP\cdot FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}.

Range: $[-1, 1]$ . Considered one of the best single metrics for imbalanced binary classification; accounts for all four cells of the confusion matrix.

1.3 Precision-recall tradeoff

Changing the classification threshold (default 0.5) trades precision for recall:

Higher threshold → higher precision, lower recall.
Lower threshold → lower precision, higher recall.

Plot the Precision-Recall (PR) curve and compute Average Precision (AP) = area under PR curve. Use PR curve (not ROC) when positive class is rare.

2. ROC curve and AUC

2.1 ROC curve

Receiver Operating Characteristic: plot True Positive Rate (Recall) vs False Positive Rate as decision threshold varies from 1 (predict nothing positive) to 0 (predict everything positive).

\text{TPR} = \frac{TP}{TP+FN}, \quad \text{FPR} = \frac{FP}{FP+TN}.

Diagonal line (FPR=TPR): random classifier.
Upper-left corner (TPR=1, FPR=0): perfect classifier.
Curve bowing toward upper-left = good.

2.2 AUC (Area Under ROC Curve)

\text{AUC} \in [0, 1].

AUC = 0.5: random (diagonal).
AUC = 1: perfect discrimination.
AUC = 0: perfectly wrong (flip predictions → AUC = 1).

Probabilistic interpretation (Mann-Whitney U statistic):

\text{AUC} = P(\hat{y}_{i_+} > \hat{y}_{i_-}),

where $i_+$ is a random positive sample and $i_-$ is a random negative sample. AUC measures ranking quality: how often does the model score positives higher than negatives?

When to use AUC: when you care about ordering/ranking rather than absolute thresholds. Scale-invariant (doesn't depend on threshold).

When NOT to use AUC: when you need to compare performance at a specific operating point, or when imbalance makes PR-AUC more informative.

2.3 PR-AUC (Average Precision)

For highly imbalanced problems, ROC-AUC can look deceptively good because TN is huge. PR-AUC focuses on the positive class: a random classifier on a problem with 1% positives gets AP ≈ 0.01, making improvements obvious.

3. Calibration

3.1 What is calibration?

A model is well-calibrated if among all predictions with score $p$ , the fraction of positives is $p$ .

Example: if a model says 70% probability of rain for 100 days, it should actually rain on ≈70 of those days.

Calibration is distinct from discrimination (AUC): a model can rank well but be poorly calibrated (e.g., logistic regression outputs near 0.99 or 0.01 when true probabilities are moderate).

3.2 Measuring calibration

Reliability diagram: bin predictions, plot mean predicted probability vs fraction of positives per bin. Well-calibrated = on the diagonal.

Expected Calibration Error (ECE):

\text{ECE} = \sum_{b=1}^B \frac{|B_b|}{n} |\text{acc}(B_b) - \text{conf}(B_b)|,

where bins $B_b$ group predictions by confidence.

3.3 Calibration methods

Platt scaling: fit a logistic regression on top of model scores. Effective for SVMs.
Isotonic regression: non-parametric monotone fit. Needs more data. sklearn.calibration.CalibratedClassifierCV.
Temperature scaling: divide logits by a scalar $T>0$ (from DL; used in production).

4. Regression metrics

Metric	Formula	Notes
MSE	$\frac{1}{n}\sum(y_i-\hat{y}_i)^2$	Penalizes large errors heavily
RMSE	$\sqrt{\text{MSE}}$	Same unit as $y$ ; interpretable
MAE	$\frac{1}{n}\sum\\|y_i-\hat{y}_i\\|_1$	Robust to outliers; less smooth
R²	$1 - \text{RSS}/\text{TSS}$	Fraction of variance explained
Adj. R²	$1-(1-R^2)\frac{n-1}{n-p-1}$	Penalizes extra features
MAPE	$\frac{100}{n}\sum\\|\frac{y_i-\hat{y}_i}{y_i}\\|$ %	Scale-free; undefined if $y_i=0$
Huber loss	MSE near zero, MAE far from zero	Balance of both
Log loss (regression)	Used when target is a probability

When to use each:

Large outliers present and you don't want them to dominate: MAE.
Standard regression with Gaussian noise: RMSE.
Need to compare across datasets with different scales: MAPE (but be careful with near-zero targets).

5. Ranking metrics

Used in information retrieval, recommender systems.

Precision@k: fraction of top- $k$ recommendations that are relevant.

Recall@k: fraction of all relevant items that appear in top $k$ .

NDCG@k (Normalized Discounted Cumulative Gain):

\text{DCG@k} = \sum_{i=1}^k \frac{2^{r_i}-1}{\log_2(i+1)}, \quad \text{NDCG@k} = \frac{\text{DCG@k}}{\text{IDCG@k}},

where $r_i$ = relevance of item at rank $i$ and IDCG = ideal (best possible) DCG. Discounts gains at lower ranks.

Mean Reciprocal Rank (MRR):

\text{MRR} = \frac{1}{|Q|}\sum_{q}\frac{1}{\text{rank of first relevant item for query }q}.

6. Multi-class metrics

6.1 Confusion matrix extension

$K \times K$ matrix. Entry $C_{ij}$ = number of samples with true class $i$ predicted as class $j$ .

6.2 Averaging strategies (for precision, recall, F1)

Strategy	Formula	Use case
Macro	Unweighted mean over classes	Equal importance to all classes, including rare ones
Weighted	Weighted by class frequency	Accounts for class imbalance
Micro	Aggregate TP/FP/FN over all classes, then compute	Dominated by large classes

Example for F1:

Macro F1 = mean of per-class F1 scores.
Weighted F1 = $\sum_k (n_k/n) \cdot F1_k$ .
Micro F1 = $\frac{2\sum_k TP_k}{2\sum_k TP_k + \sum_k FP_k + \sum_k FN_k}$ = accuracy for balanced classes.

6.3 Cohen's Kappa

Agreement beyond chance:

\kappa = \frac{P_o - P_e}{1 - P_e},

where $P_o$ = observed agreement (accuracy), $P_e$ = expected agreement by chance. $\kappa=1$ perfect, $\kappa=0$ no better than random.

7. Imbalanced data: strategies and metrics

7.1 The problem

Consider 99% negative class. A model that always predicts negative gets:

Accuracy: 99% (useless)
Recall on positive class: 0%
F1: 0%

Always examine per-class metrics and use appropriate measures.

7.2 Data-level strategies

Oversampling (minority class):

Random oversampling: duplicate minority samples. Risk: overfitting on copies.
SMOTE (Synthetic Minority Over-sampling Technique): generate synthetic samples by interpolating between nearest minority neighbors.

Undersampling (majority class):

Random undersampling: discard majority samples. Risk: losing information.
Tomek links: remove borderline majority samples adjacent to minority.

Combination: SMOTE + Tomek or SMOTE + ENN (Edited Nearest Neighbors).

7.3 Algorithm-level strategies

Class weights: multiply loss for minority class by a higher weight. In sklearn: class_weight='balanced' sets $w_k = n/(K \cdot n_k)$ .

Threshold tuning: instead of 0.5, choose threshold that maximizes a specific metric (e.g., F1 or maximizes recall subject to precision > X). Use PR curve or ROC curve to select.

Ensemble methods: BalancedBaggingClassifier, EasyEnsemble (train on balanced bootstraps).

7.4 What metrics to use for imbalanced data

Preferred	Avoid
F1, F-beta	Accuracy
PR-AUC	ROC-AUC alone
MCC
Per-class recall/precision	Micro averages

8. Validation strategies

8.1 Holdout (train/val/test)

Suitable when: large dataset, quick iteration needed.

────────────── DATASET ─────────────────
│────── Train (60-70%) ────│ Val (15%) │ Test (15-20%) │

8.2 k-fold cross-validation

Shuffle dataset (unless time series).
Split into $k$ folds.
For each fold: train on remaining $k-1$ , evaluate on fold.
Report mean ± std of scores.

Sklearn:

from sklearn.model_selection import cross_val_score, KFold
scores = cross_val_score(model, X, y, cv=5, scoring='f1')

8.3 Stratified k-fold

Preserves class proportion in each fold. Always use for classification.

from sklearn.model_selection import StratifiedKFold
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

8.4 Group k-fold

When samples within a group must not be split across train/test (e.g., multiple rows per patient, multiple frames per video).

from sklearn.model_selection import GroupKFold

8.5 Time series split

Forward-chaining validation: test always in the future relative to train.

Fold 1: ████ train │ ■ test
Fold 2: ████████ train │ ■■ test
Fold 3: ████████████ train │ ■■■ test

from sklearn.model_selection import TimeSeriesSplit

8.6 Nested CV

For unbiased estimation when tuning hyperparameters:

Outer loop (model evaluation):
  Fold 1 outer: ─────────────────────────────────────────────────────
                │ Inner CV for hyperparams │ Outer test fold │ ...

Report: outer test scores (unbiased estimate of generalization).

8.7 Bootstrap

Sample with replacement $B$ times; estimate variance of any statistic. Used in bagging (see ensemble note). Bootstrap confidence intervals for metrics.

9. Statistical significance and comparison

9.1 Is model A better than model B?

Paired t-test on k-fold scores: if A and B are evaluated on the same folds, use paired test.

McNemar's test: for comparing two classifiers on same test set; tests if disagreements are symmetric.

Multiple comparisons: testing many models on the same test set inflates false positive rate (multiple testing problem). Apply Bonferroni correction or Holm-Bonferroni.

9.2 Overfitting to the test set

Every time you look at test performance and change your model based on it, you effectively use the test set as a validation set. The more this happens, the more the test score is optimistic.

Best practice: look at test set at most once per project.

10. Sklearn cheatsheet

from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    confusion_matrix, classification_report,
    roc_auc_score, roc_curve, average_precision_score,
    mean_squared_error, mean_absolute_error, r2_score,
    matthews_corrcoef, cohen_kappa_score,
)

# Classification report (precision, recall, F1 per class)
print(classification_report(y_test, y_pred, target_names=['neg','pos']))

# ROC-AUC (needs probabilities or scores)
auc = roc_auc_score(y_test, model.predict_proba(X_test)[:,1])

# PR-AUC
ap = average_precision_score(y_test, model.predict_proba(X_test)[:,1])

# Threshold selection via ROC curve
fpr, tpr, thresholds = roc_curve(y_test, scores)
optimal_idx = np.argmax(tpr - fpr)  # maximize Youden's J = TPR - FPR
optimal_threshold = thresholds[optimal_idx]

# Class-weighted F1 for imbalanced multi-class
f1 = f1_score(y_test, y_pred, average='weighted')

# Cross-validation with multiple metrics
from sklearn.model_selection import cross_validate
results = cross_validate(
    model, X, y, cv=5,
    scoring=['accuracy','f1_macro','roc_auc'],
    return_train_score=True
)

*File: notes/05_evaluation_and_validation.md — next: notes/06_feature_engineering.md*