3. Model evaluation and validation
Any model can be made to fit the data it was trained on. What matters is how it performs on data it has never seen. This module makes evaluation a first-class skill: how to estimate out-of-sample error honestly, how to use it to choose models, and the traps that make it easy to fool yourself, especially with small or dependent datasets.
Objectives
- Distinguish in-sample from out-of-sample error and see why training error is optimistic.
- Split data into training, validation, and test sets and know the role of each.
- Estimate generalization error with k-fold cross-validation.
- Use validation to select models and hyperparameters without contaminating the test set.
- Avoid data leakage and look-ahead bias, and validate dependent data with time-series or grouped schemes.
3.1 In-sample versus out-of-sample error
The quantity we care about is the generalization error, the expected loss on a fresh draw from the same population:
\[\boxed{ R(h) = \mathbb{E}_{(x, y)}\left[ L\!\left(h(x), y\right) \right] }\]We cannot observe it, so we estimate it. The tempting estimate is the training error, the average loss on the data used to fit \(h\). It is biased downward: the model has already adapted to that particular sample, so it scores itself too kindly.
\[\boxed{ \hat{R}_{\text{train}}(h) = \frac{1}{m}\sum_{i=1}^{m} L\!\left(h(x^{(i)}), y^{(i)}\right) \;\le\; R(h) \ \text{(in expectation)} }\]Remark: a flexible model driven to near-zero training error has usually memorized noise. That is overfitting, the high-variance end of the bias-variance trade-off introduced in General concepts.
3.2 Training, validation, and test sets
The fix is to keep data the model never touched during fitting. The standard split has three disjoint roles:
| Set | Used for | Touched |
|---|---|---|
| Training | fitting the model parameters | every fit |
| Validation | choosing the model and its hyperparameters | many times |
| Test | reporting one honest final estimate | exactly once |
Remark: the test set is sacred. Every time a choice is guided by test performance, the test set quietly becomes part of training and its estimate turns optimistic.
3.3 Cross-validation
Samples are often small, and a single train/validation split both wastes data and gives a noisy estimate. k-fold cross-validation reuses the data: partition it into \(K\) folds, and for each fold train on the other \(K-1\) and validate on the held-out fold. The cross-validation error averages the \(K\) rounds:
\[\boxed{ \text{CV}_K = \frac{1}{K}\sum_{k=1}^{K} \frac{1}{|F_k|}\sum_{i \in F_k} L\!\left(h^{(-k)}(x^{(i)}), y^{(i)}\right) }\]
where \(h^{(-k)}\) is trained on all folds except \(F_k\). Taking \(K = m\) gives leave-one-out cross-validation. Common choices are \(K = 5\) or \(K = 10\), trading computation against a lower-variance estimate.
Each round holds out one fold for validation and trains on the rest, and the reported score is the average across folds.
3.4 Model and hyperparameter selection
Cross-validation is how we tune. Fit each candidate (a model family, a tree depth, or the penalty \(\lambda\) of the next module) and keep the one with the lowest validation or CV error. Only then, once the choice is frozen, do we touch the test set to report a final number.
Remark: choosing the winner on the test set inflates the estimate. With enough candidates one will look good by chance alone, the winner's curse, so selection and final evaluation must use different data.
3.5 Common validation pitfalls
Honest validation is harder than it looks, and real data often breaks the usual assumptions in three ways.
- Data leakage. Information about the target leaks into the features. Standardizing with statistics computed on the full sample, or including a variable realized after the outcome, lets the model peek at the answer. Any preprocessing must be fit on the training folds only.
- Look-ahead bias. Using information that was not yet available at the moment of prediction, which arises whenever the data is time-ordered, produces backtests that cannot be reproduced live.
- Dependence. Many datasets are serially correlated (time series) or grouped (several observations that share a unit). Shuffling them into random folds mixes near-identical neighbours across train and validation, so the estimate is far too optimistic.
For time series, use a rolling-origin (blocked) scheme so the model is only ever tested on data that comes after its training window. For grouped data, hold out whole units (grouped cross-validation) so no unit appears on both sides.
In a rolling-origin scheme the training window grows forward in time and the model is validated on the next block, never on shuffled data.
Remark: the honest question behind every split is the same. Would this have been knowable at the time, from data the model actually had?
With a way to measure generalization in hand, the next module fits our first models, and the one after controls their complexity with regularization tuned by exactly this cross-validation.