Practical ML

The gap between "getting a model to run" and "getting a model to work well in the real world." This is where most of the ML engineering effort goes.

Table of contents

1. The ML project workflow
2. Model selection and comparison
3. Hyperparameter optimization
4. Sklearn pipelines (full example)
5. Model interpretability
6. Calibration and threshold selection
7. Handling time series data
8. Production considerations
9. Common failure modes and debugging
10. Experiment tracking

1. The ML project workflow

1. Define the problem
   - What is the business metric? (revenue, retention, safety)
   - What ML proxy metric? (AUC, RMSE, F1)
   - What is the decision that will change based on predictions?

2. Data understanding
   - Data types, volume, label availability
   - Label quality / noise
   - Class balance, temporal structure

3. Establish a baseline
   - Simplest possible model (rules, mean predictor, logistic)
   - Human performance if possible

4. Feature engineering (iterative)
   - See Note 06

5. Model training + validation (iterative)
   - Choose algorithm family
   - Hyperparameter search
   - Evaluate on validation set

6. Error analysis
   - Where is the model failing?
   - Bias vs variance diagnosis
   - Slice performance by segment

7. Production deployment
   - Latency, throughput requirements
   - Monitoring setup
   - Rollback plan

8. Post-deployment monitoring
   - Data drift, concept drift
   - Performance regression alerts

2. Model selection and comparison

2.1 The information criterion approach

For in-sample model comparison (use with caution; better to use CV):

AIC (Akaike Information Criterion):

where $k$ = number of parameters, $\hat{\ell}$ = maximized log-likelihood. Lower = better. Penalizes complexity.

BIC (Bayesian Information Criterion):

BIC penalizes complexity more for large $n$; tends to select simpler models. Consistent (selects true model as $n\to\infty$ if true model is in the set).

Both: minimize to trade off goodness-of-fit against complexity.

2.2 Cross-validated comparison

Preferred over AIC/BIC for most ML tasks:

from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline

models = {
    'logistic': Pipeline([('scale', StandardScaler()), ('clf', LogisticRegression())]),
    'rf': RandomForestClassifier(n_estimators=100),
    'gbm': GradientBoostingClassifier(n_estimators=100),
}

cv_results = {}
for name, model in models.items():
    scores = cross_val_score(model, X, y, cv=StratifiedKFold(5), scoring='roc_auc')
    cv_results[name] = scores
    print(f"{name}: {scores.mean():.3f} ± {scores.std():.3f}")

2.3 Corrected paired t-test for CV

To test if model A is significantly better than model B using $k$-fold CV:

Differences $d_i = \text{score}_{A,i} - \text{score}_{B,i}$ for each fold $i$.

Nadeau-Bengio correction (accounts for CV overlap in train sets):

Simpler: just use the uncorrected paired t-test; it is slightly anti-conservative but common in practice.

from scipy.stats import ttest_rel
t_stat, p_val = ttest_rel(cv_results['rf'], cv_results['gbm'])

3. Hyperparameter optimization

3.1 What are hyperparameters?

Parameters set before training that control the learning process (not learned from data):

• Regularization strength ($\lambda$, C).
• Tree structure (max_depth, n_estimators).
• Learning rate, batch size.
• Architecture choices (network size, kernel type).

3.2 Grid search

Exhaustively try all combinations.

from sklearn.model_selection import GridSearchCV

param_grid = {
    'max_depth': [3, 5, 7],
    'learning_rate': [0.01, 0.05, 0.1],
    'n_estimators': [100, 200, 300],
}
gs = GridSearchCV(
    GradientBoostingClassifier(),
    param_grid, cv=5, scoring='roc_auc', n_jobs=-1
)
gs.fit(X_train, y_train)
print(gs.best_params_, gs.best_score_)

Cost: $\prod_i |\text{grid}_i| \times k$ fits. Grows exponentially — impractical for > 3–4 hyperparameters.

3.3 Random search (Bergstra & Bengio, 2012)

Sample randomly from distributions over hyperparameter space. Key finding: for the same budget, random search finds comparable or better results than grid search because:

• In high dimensions, most budget in grid search is wasted on unimportant parameters.
• Random search naturally samples all dimensions.

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform, randint

param_dist = {
    'max_depth': randint(3, 10),
    'learning_rate': loguniform(0.005, 0.2),  # sample from log-uniform
    'n_estimators': randint(50, 500),
    'min_samples_leaf': randint(1, 20),
}
rs = RandomizedSearchCV(
    GradientBoostingClassifier(),
    param_dist, n_iter=50, cv=5, scoring='roc_auc',
    n_jobs=-1, random_state=42
)
rs.fit(X_train, y_train)

3.4 Bayesian optimization

Model the objective function $f(\boldsymbol\theta) = \text{validation score}(\boldsymbol\theta)$ with a surrogate model (typically a Gaussian process), then use an acquisition function to decide where to evaluate next.

Acquisition functions:

• Expected Improvement (EI): $\mathbb{E}[\max(f(\boldsymbol\theta) - f^\star, 0)]$. Balances exploration and exploitation.
• Upper Confidence Bound (UCB): $\mu(\boldsymbol\theta) + \kappa\sigma(\boldsymbol\theta)$.

Libraries: optuna, scikit-optimize (skopt), hyperopt, ray[tune].

import optuna
optuna.logging.set_verbosity(optuna.logging.WARNING)

def objective(trial):
    params = {
        'max_depth': trial.suggest_int('max_depth', 3, 10),
        'learning_rate': trial.suggest_float('learning_rate', 0.005, 0.2, log=True),
        'n_estimators': trial.suggest_int('n_estimators', 50, 500),
        'subsample': trial.suggest_float('subsample', 0.5, 1.0),
    }
    model = GradientBoostingClassifier(**params, random_state=42)
    scores = cross_val_score(model, X, y, cv=5, scoring='roc_auc')
    return scores.mean()

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)
print(study.best_params, study.best_value)

3.5 Early stopping as a hyperparameter

For boosted models: tune n_estimators automatically using a validation set.

model = xgb.XGBClassifier(n_estimators=2000, learning_rate=0.02, ...)
model.fit(X_train, y_train,
          eval_set=[(X_val, y_val)],
          early_stopping_rounds=50)
# model.best_iteration gives optimal n_estimators

3.6 Nested CV for unbiased evaluation

from sklearn.model_selection import cross_val_score, StratifiedKFold, GridSearchCV

outer_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
inner_cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=0)

gs = GridSearchCV(model, param_grid, cv=inner_cv, scoring='roc_auc')
# outer loop uses gs (which internally does inner CV)
outer_scores = cross_val_score(gs, X, y, cv=outer_cv, scoring='roc_auc')
print("Unbiased estimate:", outer_scores.mean())

4. Sklearn pipelines (full example)

4.1 Why pipelines are essential

1. Prevent leakage: transformers are fit only on training fold.
2. Simplify code: one .fit(), one .predict(), one .score().
3. Grid search over preprocessing: can include preprocessing parameters in hyperparameter search.
4. Deployment: serialize the whole pipeline including preprocessing.

4.2 Full production-style example

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer, KNNImputer
from sklearn.preprocessing import (
    StandardScaler, OneHotEncoder, OrdinalEncoder,
    PolynomialFeatures, FunctionTransformer
)
from sklearn.feature_selection import SelectFromModel
from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.calibration import CalibratedClassifierCV

# --- Feature groups ---
numeric_cols   = ['age', 'income', 'credit_score']
ordinal_cols   = ['education']        # ['high school', 'bachelor', 'graduate']
categorical_cols = ['city', 'gender', 'occupation']

# --- Numeric pipeline ---
num_pipe = Pipeline([
    ('impute',   SimpleImputer(strategy='median')),
    ('scale',    StandardScaler()),
])

# --- Ordinal categorical pipeline ---
ord_pipe = Pipeline([
    ('impute', SimpleImputer(strategy='most_frequent')),
    ('enc',    OrdinalEncoder(categories=[['high school','bachelor','graduate']])),
])

# --- Nominal categorical pipeline ---
cat_pipe = Pipeline([
    ('impute', SimpleImputer(strategy='most_frequent')),
    ('ohe',    OneHotEncoder(handle_unknown='ignore', sparse_output=False)),
])

preprocessor = ColumnTransformer([
    ('num', num_pipe, numeric_cols),
    ('ord', ord_pipe, ordinal_cols),
    ('cat', cat_pipe, categorical_cols),
], remainder='drop')

# --- Full pipeline ---
full_pipe = Pipeline([
    ('preprocess', preprocessor),
    ('select',     SelectFromModel(RandomForestClassifier(n_estimators=100, random_state=0),
                                   threshold='median')),
    ('model',      GradientBoostingClassifier(n_estimators=200, learning_rate=0.05,
                                               max_depth=4, random_state=0)),
])

# Calibrate probabilities:
calibrated = CalibratedClassifierCV(full_pipe, cv=5, method='isotonic')
scores = cross_val_score(calibrated, X, y, cv=StratifiedKFold(5), scoring='roc_auc')
print("AUC:", scores.mean())

5. Model interpretability

5.1 Why interpretability matters

• Debugging: understand why the model makes specific predictions.
• Trust and adoption: stakeholders need to understand recommendations.
• Regulatory compliance: some domains require explainability (GDPR, finance).
• Discovering data issues (a model leaning on a leaky feature shows up in explanations).

5.2 Global interpretability (model-level)

Linear model weights: directly interpretable if features are scaled. Sign = direction; magnitude = importance.

Tree-based feature importance:

• MDI (mean decrease in impurity): fast but biased toward high-cardinality features.
• Permutation importance: model-agnostic, correct interpretation.

Partial Dependence Plots (PDP):

Marginal effect of feature $j$ on prediction, averaging over all other features:

from sklearn.inspection import PartialDependenceDisplay
PartialDependenceDisplay.from_estimator(model, X, features=[0, 1, (0,1)], kind='average')

ICE (Individual Conditional Expectation): like PDP but shows one line per sample (heterogeneity).

5.3 Local interpretability (prediction-level)

LIME (Local Interpretable Model-agnostic Explanations):

1. Perturb input around the instance of interest.
2. Get model predictions on perturbed samples (weighted by proximity).
3. Fit a simple linear model to this locally weighted dataset.
4. Use linear model coefficients as explanation.

Key idea: explain any black-box model locally by a simpler model.

import lime.lime_tabular
explainer = lime.lime_tabular.LimeTabularExplainer(X_train, feature_names=feature_names, class_names=['no','yes'])
exp = explainer.explain_instance(X_test[0], model.predict_proba, num_features=10)
exp.show_in_notebook()

SHAP (SHapley Additive exPlanations):

Based on Shapley values from cooperative game theory: the fair attribution of a prediction to each feature.

Shapley value for feature $j$ and instance $\mathbf{x}$:

where the sum is over all subsets $S$ of features not containing $j$, and $f(S)$ is the model output when only features in $S$ are included (others marginalized).

Axioms satisfied:

• Efficiency: $\sum_j \phi_j = f(\mathbf{x}) - \mathbb{E}[f]$. Shapley values sum to the prediction minus baseline.
• Symmetry: interchangeable features get equal credit.
• Dummy: zero-contribution feature gets zero.
• Additivity: for ensemble of models.

In practice — TreeSHAP (fast, exact):

import shap

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)

# Summary plot (global feature importance)
shap.summary_plot(shap_values, X_test, feature_names=feature_names)

# Force plot (local explanation)
shap.force_plot(explainer.expected_value, shap_values[0], X_test[0], feature_names=feature_names)

# Dependence plot (feature effect)
shap.dependence_plot('age', shap_values, X_test)

SHAP advantages over LIME:

• Mathematically grounded (unique satisfying 4 axioms).
• Globally consistent: same values regardless of reference.
• Fast exact computation for trees (TreeSHAP).

5.4 Model cards and documentation

Document: model purpose, training data, evaluation metrics by segment, known limitations, fairness analysis. Best practice for responsible deployment.

6. Calibration and threshold selection

6.1 When and how to calibrate

Use sklearn.calibration.CalibratedClassifierCV after fitting.

Methods:

• Platt scaling (method='sigmoid'): fast, needs few examples. Good for SVMs.
• Isotonic regression (method='isotonic'): non-parametric, more data needed.

from sklearn.calibration import CalibratedClassifierCV, CalibrationDisplay

cal_model = CalibratedClassifierCV(base_model, cv=5, method='isotonic')
cal_model.fit(X_train, y_train)

# Plot calibration curve
CalibrationDisplay.from_estimator(cal_model, X_val, y_val, n_bins=10)

6.2 Threshold selection

Default 0.5 is rarely optimal. Choose threshold based on:

from sklearn.metrics import precision_recall_curve, roc_curve

# Maximize F1
prec, rec, thresh = precision_recall_curve(y_val, model.predict_proba(X_val)[:,1])
f1_scores = 2*prec*rec/(prec+rec+1e-9)
best_thresh = thresh[np.argmax(f1_scores)]

# Maximize recall subject to precision >= 0.9
valid = prec >= 0.9
best_thresh = thresh[np.argmax(rec[:-1][valid[:-1]])] if valid.any() else 0.5

7. Handling time series data

7.1 Unique challenges

• Temporal leakage: future data cannot inform past predictions.
• Non-stationarity: distribution drifts over time.
• Autocorrelation: samples are not i.i.d.

7.2 Proper train/test splitting

Always split by time, never shuffle.

from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5, gap=0)
for train_idx, test_idx in tscv.split(X):
    X_tr, X_te = X[train_idx], X[test_idx]

Gap parameter: skip $g$ rows between train and test to prevent leakage in features that use rolling windows.

7.3 Feature engineering for time series

• Lag features: $x_{t-1}, x_{t-2}, \ldots, x_{t-k}$.
• Rolling statistics: $\text{mean}(x_{t-W:t}), \text{std}(x_{t-W:t})$.
• Calendar features: hour, day of week, month, is_holiday.
• Expanding statistics: cumulative mean (no leakage if computed on past only).

8. Production considerations

8.1 Serialization

import joblib

# Save
joblib.dump(full_pipeline, 'model.pkl')

# Load
pipeline = joblib.load('model.pkl')
predictions = pipeline.predict(new_data)

8.2 Data/concept drift

• Data drift (covariate shift): input distribution $P(\mathbf{x})$ changes. Detect with KS test, population stability index (PSI).
• Concept drift: $P(y|\mathbf{x})$ changes (relationship changes). Harder to detect; monitor model performance.

PSI (Population Stability Index):

where $A_k, B_k$ = fraction of samples in bin $k$ for reference vs production. PSI < 0.1: stable; 0.1–0.25: slight shift; > 0.25: significant drift.

8.3 Serving patterns

8.4 Feature stores

Centralize feature computation and serving. Prevent train/serve skew (same feature logic in both). Tools: Feast, Tecton, Hopsworks.

9. Common failure modes and debugging

9.1 Sanity checks to always run

# 1. Overfit a tiny dataset
model.fit(X[:20], y[:20])
assert model.score(X[:20], y[:20]) > 0.95, "Model cannot even overfit 20 samples"

# 2. Check class distribution in splits
from sklearn.model_selection import train_test_split
X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y)
print("Train label dist:", np.bincount(y_tr)/len(y_tr))
print("Test  label dist:", np.bincount(y_te)/len(y_te))

# 3. Verify no leaking features
corr_with_target = pd.DataFrame(X).corrwith(pd.Series(y))
print(corr_with_target.abs().sort_values(ascending=False).head(10))
# Very high correlations deserve investigation

# 4. Baseline comparison
from sklearn.dummy import DummyClassifier
dummy = DummyClassifier(strategy='stratified').fit(X_tr, y_tr)
print("Dummy score:", dummy.score(X_te, y_te))

10. Experiment tracking

10.1 What to track

• Hyperparameters.
• Data version (hash, version tag).
• Feature set (list of features used).
• Preprocessing choices.
• Training and validation metrics.
• Artifacts: model weights, feature importance, calibration curves.

10.2 MLflow (lightweight)

import mlflow
import mlflow.sklearn

mlflow.set_experiment("fraud_detection")

with mlflow.start_run():
    mlflow.log_params({"max_depth": 5, "learning_rate": 0.05, "n_estimators": 200})
    model = GradientBoostingClassifier(max_depth=5, learning_rate=0.05, n_estimators=200)
    model.fit(X_train, y_train)
    
    auc = roc_auc_score(y_val, model.predict_proba(X_val)[:,1])
    mlflow.log_metric("val_auc", auc)
    
    mlflow.sklearn.log_model(model, "model")
    print("AUC:", auc)

10.3 Reproducibility checklist

import random, numpy as np

SEED = 42
random.seed(SEED)
np.random.seed(SEED)
# sklearn uses random_state=SEED for stochastic algorithms

Also: pin dependency versions (requirements.txt), version your data, log git commit hash.

*File: notes/10_practical_ml.md — all ML notes complete.*