Probability is the language machine learning uses to handle uncertainty. This module sets out the rules for discrete and continuous variables, takes a first look at information theory, shows the Bayesian way of turning probabilities into decisions, and defines the two estimation principles the course returns to again and again: maximum likelihood and maximum a posteriori.

A random variable takes values with probabilities that are non-negative and sum or integrate to one. A discrete variable has a probability mass function, a continuous one a probability density function:

$$\boxed{ \sum_x p(x) = 1 \qquad \int p(x)\, dx = 1, \quad p(x) \ge 0 }$$

For a continuous variable, probability attaches to intervals through an integral, $P(a \le X \le b) = \int_a^b p(x)\, dx$, not to single points.

Two variables have a joint distribution $p(x, y)$. Summing (or integrating) out one variable gives the marginal, the sum rule, and the joint factors into a conditional times a marginal, the product rule:

$$\boxed{ p(x) = \sum_y p(x, y) \qquad p(x, y) = p(y \mid x)\, p(x) }$$

Rearranging the product rule both ways gives Bayes' rule, which flips a conditional:

$$\boxed{ p(y \mid x) = \frac{p(x \mid y)\, p(y)}{p(x)} }$$

Two variables are independent when the joint is the product of the marginals, $p(x, y) = p(x)\, p(y)$.

The entropy of a distribution measures its uncertainty, the average number of bits needed to describe an outcome:

$$\boxed{ H(X) = -\sum_x p(x)\log p(x) }$$

The cross-entropy measures the cost of using a model $q$ when the truth is $p$, and the Kullback-Leibler divergence measures how far $q$ sits from $p$:

$$\boxed{ H(p, q) = -\sum_x p(x)\log q(x) \qquad D_{\mathrm{KL}}(p \,\|\, q) = \sum_x p(x)\log\frac{p(x)}{q(x)} \ge 0 }$$

*Remark:* minimizing the cross-entropy between the true labels and a model's predictions is the same as maximizing the likelihood of those labels. This is why classification networks minimize cross-entropy, a thread picked up in later modules.

To classify an input $x$, the Bayesian rule uses the posterior over classes. Under the 0-1 loss, the decision that minimizes the expected loss is simply the most probable class, and because the posterior is proportional to the class-conditional density times the prior, it can be computed either way:

$$\boxed{ \hat{y} = \arg\max_y \; p(y \mid x) = \arg\max_y \; p(x \mid y)\, p(y) }$$

*Remark:* this is the optimal classifier, called the Bayes classifier. Every method later in the course is, in effect, an attempt to approximate these posteriors from data.

---