5. Linear regression
Linear regression predicts a continuous target from a linear score \(\theta^T x\). This module presents it probabilistically, building on the Probabilistic formulation module: the model (extended to polynomial features), fitting by maximum likelihood, which turns out to be ordinary least squares, and fitting by maximum a posteriori, which adds a prior and yields a regularized fit.
Objectives
- Write the linear and polynomial regression model and fit it by least squares.
- Give regression a probabilistic formulation with Gaussian noise.
- See that maximum likelihood under that model is exactly least squares.
- Add a prior and fit by maximum a posteriori, recovering a regularized fit.
5.1 The linear and polynomial model
The hypothesis is linear in the augmented input \(x \in \mathbb{R}^{n+1}\) with \(x_0 = 1\) and parameters \(\theta\):
\[\boxed{ h_\theta(x) = \theta^T x }\]Polynomial regression is the same model applied to a feature map. Replacing \(x\) by \(\phi(x) = (1, x, x^2, \dots, x^d)\) fits a degree-\(d\) polynomial while staying linear in the parameters:
\[\boxed{ h_\theta(x) = \theta^T \phi(x) = \sum_{j=0}^{d} \theta_j\, x^{j} }\]so everything below applies unchanged once the design matrix \(X\) stacks the transformed inputs \(\phi(x^{(i)})\) as its rows.
5.2 Least squares
The cost is half the sum of squared residuals over the \(m\) examples:
\[\boxed{ J(\theta) = \tfrac{1}{2}\sum_{i=1}^{m}\left(h_\theta(x^{(i)}) - y^{(i)}\right)^2 }\]Setting \(\nabla_\theta J = 0\) gives the closed-form normal equation, and gradient descent gives the equivalent iterative update:
\[\boxed{ \theta = (X^T X)^{-1}X^T y \qquad \theta_j \leftarrow \theta_j + \alpha\left(y^{(i)} - h_\theta(x^{(i)})\right)x_j^{(i)} }\]
Least squares fits the curve that minimizes the squared residuals (grey segments).
5.3 Probabilistic formulation: maximum likelihood
Give the data a generative story: each target is the linear prediction plus independent Gaussian noise,
\[\boxed{ y^{(i)} = \theta^T x^{(i)} + \varepsilon^{(i)}, \quad \varepsilon^{(i)} \sim \mathcal{N}(0, \sigma^2) }\]so \(p(y^{(i)} \mid x^{(i)}; \theta) = \mathcal{N}(\theta^T x^{(i)}, \sigma^2)\). Maximizing the log-likelihood over the \(m\) i.i.d. examples drops every term that does not depend on \(\theta\) and leaves the least-squares cost:
\[\boxed{ \arg\max_\theta \ell(\theta) = \arg\min_\theta \sum_{i=1}^{m}\left(y^{(i)} - \theta^T x^{(i)}\right)^2 }\]Remark: this is why least squares is a principled objective and not merely a convenient one. Ordinary least squares is the maximum-likelihood estimate under Gaussian noise, exactly the maximum-likelihood principle from the previous module.
5.4 Maximum a posteriori
Maximum likelihood can overfit, especially at high polynomial degree. Placing a zero-mean Gaussian prior on the parameters, \(\theta \sim \mathcal{N}(0, \tau^2 I)\), and maximizing the posterior instead adds a penalty on their size:
\[\boxed{ \theta_{\mathrm{MAP}} = \arg\min_\theta \; \sum_{i=1}^{m}\left(y^{(i)} - \theta^T x^{(i)}\right)^2 + \lambda \lVert \theta \rVert_2^2, \quad \lambda = \frac{\sigma^2}{\tau^2} }\]This is regularized (ridge) regression: the Gaussian prior becomes an L2 penalty, exactly the prior-to-penalty link noted in the previous module.
Remark: a stronger prior (small \(\tau\)) means a larger \(\lambda\) and more shrinkage toward zero. With abundant data the likelihood dominates the prior and the maximum-a-posteriori fit approaches the maximum-likelihood one. Choosing the degree \(d\) and the penalty \(\lambda\) is a model-selection problem, settled by cross-validation from the evaluation module, and taken further in the regularization module.
The same linear score, passed through a squashing function instead of read directly, turns regression into classification, the subject of the next module.
Next: Linear classification · Course overview