$$\boxed{ h_w(x) = w^T x = w_0 + w_1 x_1 + \dots + w_n x_n }$$

$w_0$ is the bias (the intercept) and the remaining coordinates are the weights, and folding the bias into the dot product is exactly what the $x_0 = 1$ convention buys. Geometrically, the prediction is a line for $n = 1$, a plane for $n = 2$, and a hyperplane beyond.

Given the training set $\{(x^{(i)}, y^{(i)})\}_{i=1}^{m}$, ideally we would have $h_w(x^{(i)}) = y^{(i)}$ at every point. Real targets are noisy (measurement error, unmodelled factors), so no line passes through them all, and the goal becomes to make the smallest total error. Least squares takes the squared residual as the error and sums it over the training set:

$$\boxed{ w^{*} = \arg\min_w \; \sum_{i=1}^{m}\left(w^T x^{(i)} - y^{(i)}\right)^2 }$$

$$\boxed{ y^{(i)} = w^T x^{(i)} + \varepsilon^{(i)}, \quad \varepsilon^{(i)} \sim \mathcal{N}(0, \sigma^2) }$$

<details class="proof">

The examples are i.i.d., so the likelihood of the whole training set factorizes into a product of Gaussian densities:

$$p(y \mid X; w) = \prod_{i=1}^{m} p(y^{(i)} \mid x^{(i)}; w) = \prod_{i=1}^{m} \frac{1}{\sqrt{2\pi\sigma^2}}\, \exp\!\left(-\frac{\left(y^{(i)} - w^T x^{(i)}\right)^2}{2\sigma^2}\right)$$

The logarithm is increasing, so it preserves the argmax and turns the product into a sum, the log-likelihood, which separates into a constant and the sum of squares:

$$\ell(w) = \sum_{i=1}^{m} \log \mathcal{N}\!\left(y^{(i)} \mid w^T x^{(i)}, \sigma^2\right) = -\frac{m}{2}\log(2\pi\sigma^2) \;-\; \frac{1}{2\sigma^2}\sum_{i=1}^{m}\left(y^{(i)} - w^T x^{(i)}\right)^2$$

The first term does not depend on $w$ and the factor $\tfrac{1}{2\sigma^2}$ is a positive constant, so neither moves the argmax. Maximizing $\ell$ is therefore minimizing the sum of squared errors. $\blacksquare$

</details>

<details class="proof">

With the design matrix, the sum of squares is the quadratic $\lVert Xw - y \rVert^2$, a convex function of $w$, so its global minimum is the point of zero gradient:

$$w_{\mathrm{MAP}} = \arg\max_w \; p(y \mid X, w)\, p(w), \qquad w \sim \mathcal{N}(0, \tau^2 I)$$

Two results again. The Gaussian prior turns into an L2 penalty added to least squares:

$$\boxed{ w_{\mathrm{MAP}} = \arg\min_w \; \sum_{i=1}^{m}\left(y^{(i)} - w^T x^{(i)}\right)^2 + \lambda \lVert w \rVert_2^2, \quad \lambda = \frac{\sigma^2}{\tau^2} }$$

and the penalized minimizer keeps a closed form:

$$\boxed{ w_{\mathrm{MAP}} = (X^T X + \lambda I)^{-1}X^T y }$$

*Remark:* $\lambda \to 0$ recovers maximum likelihood, and a growing $\lambda$ shrinks $w$ toward zero and fights overfitting. A stronger prior (small $\tau$) means a larger $\lambda$. Note also that $X^T X + \lambda I$ is always invertible for $\lambda > 0$, which rescues least squares exactly where it breaks down: strongly correlated features, or more features than examples.

<details class="proof">

By Bayes' rule the posterior is

and the denominator does not depend on $w$, so maximizing the posterior is maximizing the likelihood times the prior. The prior covariance is generally unknown, so it is assumed isotropic, $\tau^2 I$, which gives the density

$$p(w) = \frac{1}{(2\pi\tau^2)^{(n+1)/2}}\, \exp\!\left(-\frac{\lVert w \rVert^2}{2\tau^2}\right)$$

Taking logarithms and reusing the log-likelihood $\ell(w)$ from the previous proof,

$$\log p(y \mid X, w) + \log p(w) = \mathrm{const} \;-\; \frac{1}{2\sigma^2}\sum_{i=1}^{m}\left(y^{(i)} - w^T x^{(i)}\right)^2 \;-\; \frac{1}{2\tau^2}\lVert w \rVert^2$$

where the constant gathers every term independent of $w$. Multiplying by $-2\sigma^2$, a negative constant that flips the argmax into an argmin, leaves

$$\boxed{ w_{\mathrm{lasso}} = \arg\min_w \; \sum_{i=1}^{m}\left(y^{(i)} - w^T x^{(i)}\right)^2 + \lambda \lVert w \rVert_1 }$$

The change looks small, its consequences are not:

- **It selects.** The lasso drives some coefficients to exactly zero, performing variable selection while it fits. Ridge only shrinks and never zeroes: ridge stabilizes, the lasso selects.

$$\boxed{ h_w(x) = w^T \phi(x) = \sum_{j=0}^{M-1} w_j\, \phi_j(x), \qquad \phi_0(x) = 1 }$$

The $\phi_j$ are basis functions, fixed before training. With $\phi(x) = (1, x, x^2, \dots, x^d)$ they give polynomial regression, the running example of [General concepts](/en/Machine%20Learning/02%20General%20concepts), and the identity $\phi(x) = x$ recovers everything above. The model can now be wildly nonlinear in $x$ yet stays linear in $w$, so nothing changes in the fit: stack the $\phi(x^{(i)})^T$ as the rows of the design matrix $\Phi \in \mathbb{R}^{m \times M}$ and the two closed forms return verbatim:

$$\boxed{ w_{\mathrm{MLE}} = (\Phi^T \Phi)^{-1}\Phi^T y, \qquad w_{\mathrm{MAP}} = (\Phi^T \Phi + \lambda I)^{-1}\Phi^T y }$$

*Remark:* the basis (its family and its size $M$) is a hyperparameter, chosen before training, while $w$ is learned. Choosing $M$ and $\lambda$ is the model-selection problem settled by the cross-validation of [General concepts](/en/Machine%20Learning/02%20General%20concepts).

| Model | $h_w(x) = w^T \phi(x)$ |