7. Regularization and high-dimensional inference
You often do not have a handful of clean regressors. There can be many candidate predictors, sometimes more than observations, and they are correlated. Ordinary least squares overfits or breaks down in that regime. Regularization tames it by shrinking the coefficients, and this is where regularized regression meets classical statistics most directly. It also carries a warning: selecting variables and then doing inference on the same data invalidates the classical standard errors, which matters whenever the goal is a causal estimate rather than a prediction.
Throughout we write the regression coefficients as \(\beta\), the parameters \(\theta\) of the linear model from the linear regression module.
Objectives
- See why ordinary least squares fails with many correlated regressors.
- Define ridge (L2) and lasso (L1) regression and the role of the penalty \(\lambda\).
- Understand why the lasso produces sparse, variable-selecting solutions.
- Choose the penalty \(\lambda\) by cross-validation.
- Recognize why naive post-selection inference is invalid, and know the standard corrections.
7.1 Why regularize
When the number of regressors \(p\) is large relative to the sample size \(n\), the least-squares fit chases noise and its coefficients have huge variance. With correlated regressors the matrix \(X^T X\) is nearly singular, so small data changes swing the estimates wildly, and when \(p > n\) it is singular and OLS has no unique solution at all. Regularization accepts a little bias in exchange for a large cut in variance, the trade-off from General concepts.
7.2 Ridge regression (L2)
Ridge adds a squared-norm penalty on the coefficients to the least-squares objective:
\[\boxed{ \hat{\beta}_{\text{ridge}} = \arg\min_{\beta} \; \|y - X\beta\|_2^2 + \lambda \|\beta\|_2^2 }\]
It has a closed form that is always invertible for \(\lambda > 0\), which is exactly what rescues the collinear and \(p > n\) cases:
\[\boxed{ \hat{\beta}_{\text{ridge}} = \left(X^T X + \lambda I\right)^{-1} X^T y }\]Ridge shrinks all coefficients smoothly toward zero but never sets them exactly to zero, so it stabilizes rather than selects.
7.3 Lasso regression (L1)
The lasso replaces the squared penalty with an absolute-value penalty:
\[\boxed{ \hat{\beta}_{\text{lasso}} = \arg\min_{\beta} \; \|y - X\beta\|_2^2 + \lambda \|\beta\|_1 }\]
This small change has a large consequence: the lasso drives some coefficients to exactly zero, so it performs variable selection while it fits. The reason is geometric. The constraint region \(\|\beta\|_1 \le t\) is a diamond with corners on the axes, and the elliptical loss contours tend to first touch it at a corner, where one coordinate is zero.
The rounded L2 ball is touched off the axes, keeping every coefficient nonzero, while the L1 diamond is touched at a corner, setting a coefficient to exactly zero.
As the penalty grows, more coefficients cross to zero, tracing the regularization path from the full model to the empty one.
Each coefficient shrinks as \(\lambda\) increases and then hits exactly zero, so the lasso yields a compact, interpretable subset of regressors.
7.4 Elastic net
The elastic net blends the two penalties, keeping the lasso's selection while borrowing the ridge's stability with correlated regressors:
\[\boxed{ \hat{\beta}_{\text{en}} = \arg\min_{\beta} \; \|y - X\beta\|_2^2 + \lambda\left(\alpha \|\beta\|_1 + (1 - \alpha)\|\beta\|_2^2\right) }\]
with \(\alpha \in [0, 1]\) mixing selection (\(\alpha = 1\), lasso) and shrinkage (\(\alpha = 0\), ridge).
7.5 Choosing the penalty
The penalty \(\lambda\) is a hyperparameter, so it is chosen by cross-validation from the previous module: fit over a grid of \(\lambda\) values and keep the one with the lowest cross-validated error, or the largest \(\lambda\) within one standard error of the best for a simpler model. Larger \(\lambda\) means more shrinkage, more bias, and less variance.
7.6 The inference caveat
Prediction is not inference, and this is the point that is easy to miss. Suppose you select regressors with the lasso and then run ordinary least squares on the chosen subset and report textbook standard errors. Those standard errors are wrong. They ignore that the data was already used to pick the variables, so the confidence intervals are too narrow and the p-values are not valid, a form of the winner's curse. Three corrections are standard:
- Sample splitting. Select the variables on one part of the data and estimate and do inference on another, so the selection does not contaminate the standard errors.
- Debiased (desparsified) lasso. Add a correction term to the lasso estimate that removes the shrinkage bias and restores an asymptotically valid confidence interval for each coefficient.
- Post-double-selection (Belloni, Chernozhukov, and Hansen). To estimate the effect of a treatment with many controls, select the controls that predict the outcome and the controls that predict the treatment, then estimate the effect on the union of both sets.
Remark: these ideas are the doorway to causal machine learning, where flexible learners estimate nuisance functions while a correction preserves valid inference on the parameter of interest. Regularization is superb for prediction, but for a causal parameter you need one of these corrections, not the raw penalized coefficients.
With shrinkage and selection covered, the next module takes a different route to a good decision boundary, the maximum-margin classifier, before we turn to trees and ensembles.