Machine Learning Model Evaluation

A "90% accurate" model can still lose money. Learn how to truly judge a model — confusion matrices, precision/recall, F1, regression metrics, the bias-variance tradeoff and cross-validation.

5.1 Classification Metrics & the Confusion Matrix

The Confusion Matrix

A 2×2 table comparing predictions against reality. Memorise these four cells:

• TP — correctly predicted positive.
• TN — correctly predicted negative.
• FP (Type I error) — predicted positive but actually negative (a "false alarm").
• FN (Type II error) — predicted negative but actually positive (a "miss").

Accuracy

5.2 Precision, Recall & F1 Score

Precision — "of what I flagged, how much was right?"

Use when false positives are costly. Example: spam detection — you must not send an important email to the spam folder, so high precision matters.

Recall (Sensitivity) — "of all real positives, how many did I catch?"

Use when false negatives are dangerous. Example: cancer diagnosis — you cannot afford to miss a sick patient, so high recall is critical.

F1 Score — the balance

from sklearn.metrics import (confusion_matrix, precision_score,
                             recall_score, f1_score, accuracy_score)

y_true = [1, 0, 1, 1, 0, 1, 0, 0]
y_pred = [1, 0, 0, 1, 0, 1, 1, 0]

print("Confusion matrix:\n", confusion_matrix(y_true, y_pred))
print("Accuracy :", accuracy_score(y_true, y_pred))
print("Precision:", round(precision_score(y_true, y_pred), 3))
print("Recall   :", round(recall_score(y_true, y_pred), 3))
print("F1 Score :", round(f1_score(y_true, y_pred), 3))

5.3 Regression Metrics

For regression, error is the residual = actual − predicted. The metrics summarise these residuals.

MAE — Mean Absolute Error

Pro: easy to interpret, robust to outliers. Con: treats a big mistake linearly — does not punish large errors extra.

MSE — Mean Squared Error

RMSE — Root Mean Squared Error

RMSE keeps MSE's heavy penalty for big errors but is interpretable like MAE.

R² — Coefficient of Determination

Adjusted R²

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np

y_true = [100, 150, 200, 250, 500]
y_pred = [90, 170, 180, 240, 400]

mae  = mean_absolute_error(y_true, y_pred)
mse  = mean_squared_error(y_true, y_pred)
rmse = np.sqrt(mse)
r2   = r2_score(y_true, y_pred)

print(f"MAE  = {mae}")
print(f"MSE  = {mse}")
print(f"RMSE = {rmse:.1f}")
print(f"R2   = {r2:.2f}")

5.4 Bias & Variance

High Bias → Underfitting

Signs: poor performance on both training and test data. Causes: model too simple, or missing useful features.

High Variance → Overfitting

Signs: excellent on training data but poor on test data (a big gap). Causes: model too complex/flexible, or too little training data.

5.5 The Bias-Variance Tradeoff

You cannot minimise both to zero at once — reducing bias tends to raise variance and vice versa. The goal is the sweet spot that minimises total error on unseen data.

Fixing overfitting

Get more data · data augmentation · reduce model complexity · early stopping · regularisation.

5.6 Cross-Validation

K-Fold Cross-Validation

Split the data into K equal folds. Train on K−1 folds, test on the remaining 1. Repeat K times so every fold is used as the test set once. Average the K scores for a reliable estimate.

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
# cv=5  ->  5-fold cross-validation
scores = cross_val_score(model, X, y, cv=5)

print("Scores per fold:", scores.round(3))
print("Average score  :", scores.mean().round(3))

Evaluation metrics are heavily tested. Work through every question.

import numpy as np
from sklearn.metrics import mean_absolute_error, mean_squared_error

y_true = [200, 300, 400]
y_pred = [210, 290, 440]

mae  = mean_absolute_error(y_true, y_pred)
mse  = mean_squared_error(y_true, y_pred)
rmse = np.sqrt(mse)

print("MAE :", mae)
print("MSE :", mse)
print("RMSE:", round(rmse, 2))