Machine Learning Model Training & Evaluation: The Complete Guide

Key Techniques:

• Train-Validation-Test Split: 60-20-20 typical ratio
• Cross-Validation: k-fold (k=5 or 10) for small datasets
• Early Stopping: Monitor validation loss
• Learning Rate Scheduling: Reduce on plateau

Python Implementation:

from sklearn.model_selection import train_test_split, KFold
from sklearn.ensemble import RandomForestClassifier

# Basic split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Cross-validation
kf = KFold(n_splits=5)
for train_index, val_index in kf.split(X):
    X_train, X_val = X[train_index], X[val_index]
    y_train, y_val = y[train_index], y[val_index]
    model = RandomForestClassifier().fit(X_train, y_train)
    # Evaluate on X_val, y_val

Pro Tip:

Use stratified splits for imbalanced datasets to maintain class distribution

Tuning Methods:

Method	Description	When to Use
Grid Search	Exhaustive search	Small parameter spaces
Random Search	Random sampling	Medium parameter spaces
Bayesian Optimization	Probabilistic model	Expensive evaluations

Optuna Example:

import optuna

def objective(trial):
    params = {
        'n_estimators': trial.suggest_int('n_estimators', 50, 500),
        'max_depth': trial.suggest_int('max_depth', 3, 10),
        'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3)
    }
    model = XGBClassifier(**params)
    return cross_val_score(model, X, y, cv=5).mean()

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)
 

Key Metrics by Task:

Classification Report:

from sklearn.metrics import classification_report, roc_auc_score

y_pred = model.predict(X_test)
print(classification_report(y_test, y_pred))

# For probabilities
roc_auc = roc_auc_score(y_test, model.predict_proba(X_test)[:,1])
        

Specialized Techniques:

• Time Series: Walk-forward validation
• Geospatial: Spatial cross-validation
• Grouped: Leave-one-group-out
• Bootstrapping: Confidence intervals

Time Series Example:

from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits=5)
for train_index, test_index in tscv.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    # Train and evaluate
        

Production Tip:

Implement continuous evaluation in production with A/B testing and monitoring