Probability is the language machine learning uses to handle uncertainty. This module sets out the rules for discrete and continuous variables, takes a first look at information theory, shows the Bayesian way of turning probabilities into decisions, and defines the two estimation principles the course returns to again and again: maximum likelihood and maximum a posteriori.

- State the rules of probability for discrete and continuous variables.

- Relate joint, conditional, and marginal probabilities by the sum and product rules and by Bayes' rule.

- Measure uncertainty with entropy, cross-entropy, and the Kullback-Leibler divergence.

- Make the decision that minimizes expected loss, and recover the maximum-a-posteriori classifier.

- Define the maximum-likelihood and maximum-a-posteriori estimators.

A random variable takes values with probabilities that are non-negative and sum or integrate to one. A discrete variable has a probability mass function, a continuous one a probability density function:

$$\boxed{ \sum_x p(x) = 1 \qquad \int p(x)\, dx = 1, \quad p(x) \ge 0 }$$

For a continuous variable, probability attaches to intervals through an integral, $P(a \le X \le b) = \int_a^b p(x)\, dx$, not to single points.

Two variables have a joint distribution $p(x, y)$. Summing (or integrating) out one variable gives the marginal, the sum rule, and the joint factors into a conditional times a marginal, the product rule:

$$\boxed{ p(x) = \sum_y p(x, y) \qquad p(x, y) = p(y \mid x)\, p(x) }$$

Rearranging the product rule both ways gives Bayes' rule, which flips a conditional:

$$\boxed{ p(y \mid x) = \frac{p(x \mid y)\, p(y)}{p(x)} }$$

Two variables are independent when the joint is the product of the marginals, $p(x, y) = p(x)\, p(y)$.

The entropy of a distribution measures its uncertainty, the average number of bits needed to describe an outcome:

$$\boxed{ H(X) = -\sum_x p(x)\log p(x) }$$


The cross-entropy measures the cost of using a model $q$ when the truth is $p$, and the Kullback-Leibler divergence measures how far $q$ sits from $p$:

$$\boxed{ H(p, q) = -\sum_x p(x)\log q(x) \qquad D_{\mathrm{KL}}(p \,\|\, q) = \sum_x p(x)\log\frac{p(x)}{q(x)} \ge 0 }$$

*Remark:* minimizing the cross-entropy between the true labels and a model's predictions is the same as maximizing the likelihood of those labels. This is why classification networks minimize cross-entropy, a thread picked up in later modules.

To classify an input $x$, the Bayesian rule uses the posterior over classes. Under the 0-1 loss, the decision that minimizes the expected loss is simply the most probable class, and because the posterior is proportional to the class-conditional density times the prior, it can be computed either way:

$$\boxed{ \hat{y} = \arg\max_y \; p(y \mid x) = \arg\max_y \; p(x \mid y)\, p(y) }$$


*Remark:* this is the optimal classifier, called the Bayes classifier. Every method later in the course is, in effect, an attempt to approximate these posteriors from data.

We rarely know the true distribution, so we estimate its parameters $\theta$ from data. Maximum likelihood picks the $\theta$ that makes the observed data most probable, usually maximized as a sum of log-likelihoods over the $m$ examples:

$$\boxed{ \theta_{\mathrm{MLE}} = \arg\max_\theta \sum_{i=1}^{m} \log p(x^{(i)} \mid \theta) }$$

Maximum a posteriori instead maximizes the posterior, which multiplies the likelihood by a prior on $\theta$:

$$\boxed{ \theta_{\mathrm{MAP}} = \arg\max_\theta \; p(D \mid \theta)\, p(\theta) }$$

*Remark:* maximum a posteriori is maximum likelihood plus a prior. A Gaussian prior on $\theta$ becomes an L2 penalty and a Laplace prior an L1 penalty, which is exactly the regularization of a later module. With abundant data the prior washes out and the two estimators agree.

---

Next: [Linear regression](/en/Machine%20Learning/05%20Linear%20regression) · [Course overview](/en/Machine%20Learning)