Linear regression predicts a continuous target from a linear score $\theta^T x$. This module presents it probabilistically, building on the [Probabilistic formulation](/en/Machine%20Learning/04%20Probabilistic%20formulation) module: the model (extended to polynomial features), fitting by maximum likelihood, which turns out to be ordinary least squares, and fitting by maximum a posteriori, which adds a prior and yields a regularized fit.

- Write the linear and polynomial regression model and fit it by least squares.

- Give regression a probabilistic formulation with Gaussian noise.

- See that maximum likelihood under that model is exactly least squares.

- Add a prior and fit by maximum a posteriori, recovering a regularized fit.

The hypothesis is linear in the augmented input $x \in \mathbb{R}^{n+1}$ with $x_0 = 1$ and parameters $\theta$:

$$\boxed{ h_\theta(x) = \theta^T x }$$

Polynomial regression is the same model applied to a feature map. Replacing $x$ by $\phi(x) = (1, x, x^2, \dots, x^d)$ fits a degree-$d$ polynomial while staying linear in the parameters:

$$\boxed{ h_\theta(x) = \theta^T \phi(x) = \sum_{j=0}^{d} \theta_j\, x^{j} }$$

so everything below applies unchanged once the design matrix $X$ stacks the transformed inputs $\phi(x^{(i)})$ as its rows.

The cost is half the sum of squared residuals over the $m$ examples:

$$\boxed{ J(\theta) = \tfrac{1}{2}\sum_{i=1}^{m}\left(h_\theta(x^{(i)}) - y^{(i)}\right)^2 }$$

Setting $\nabla_\theta J = 0$ gives the closed-form normal equation, and gradient descent gives the equivalent iterative update:

$$\boxed{ \theta = (X^T X)^{-1}X^T y \qquad \theta_j \leftarrow \theta_j + \alpha\left(y^{(i)} - h_\theta(x^{(i)})\right)x_j^{(i)} }$$


Give the data a generative story: each target is the linear prediction plus independent Gaussian noise,

$$\boxed{ y^{(i)} = \theta^T x^{(i)} + \varepsilon^{(i)}, \quad \varepsilon^{(i)} \sim \mathcal{N}(0, \sigma^2) }$$

so $p(y^{(i)} \mid x^{(i)}; \theta) = \mathcal{N}(\theta^T x^{(i)}, \sigma^2)$. Maximizing the log-likelihood over the $m$ i.i.d. examples drops every term that does not depend on $\theta$ and leaves the least-squares cost:

$$\boxed{ \arg\max_\theta \ell(\theta) = \arg\min_\theta \sum_{i=1}^{m}\left(y^{(i)} - \theta^T x^{(i)}\right)^2 }$$

*Remark:* this is why least squares is a principled objective and not merely a convenient one. Ordinary least squares is the maximum-likelihood estimate under Gaussian noise, exactly the maximum-likelihood principle from the previous module.

Maximum likelihood can overfit, especially at high polynomial degree. Placing a zero-mean Gaussian prior on the parameters, $\theta \sim \mathcal{N}(0, \tau^2 I)$, and maximizing the posterior instead adds a penalty on their size:

$$\boxed{ \theta_{\mathrm{MAP}} = \arg\min_\theta \; \sum_{i=1}^{m}\left(y^{(i)} - \theta^T x^{(i)}\right)^2 + \lambda \lVert \theta \rVert_2^2, \quad \lambda = \frac{\sigma^2}{\tau^2} }$$

This is regularized (ridge) regression: the Gaussian prior becomes an L2 penalty, exactly the prior-to-penalty link noted in the previous module.

*Remark:* a stronger prior (small $\tau$) means a larger $\lambda$ and more shrinkage toward zero. With abundant data the likelihood dominates the prior and the maximum-a-posteriori fit approaches the maximum-likelihood one. Choosing the degree $d$ and the penalty $\lambda$ is a model-selection problem, settled by cross-validation from the [evaluation module](/en/Machine%20Learning/03%20Model%20evaluation%20and%20validation), and taken further in the [regularization module](/en/Machine%20Learning/07%20Regularization%20and%20high-dimensional%20inference).

---

Next: [Linear classification](/en/Machine%20Learning/06%20Linear%20classification) · [Course overview](/en/Machine%20Learning)