Home A - Z
Page Index

An Otter Wiki

Blame

36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
1 
# 1. Introduction
2
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
3
This course continues directly from the Machine Learning course, whose [Multilayer neural networks](/en/Machine%20Learning/06%20Multilayer%20neural%20networks) module built the network itself: the model, its losses, and backpropagation. This lesson re-anchors that bridge. It recalls what one unit can do, shows the concrete task (XOR) where a single unit fails, fixes the notation used throughout the rest of the course, then restates the multilayer perceptron and its training loop in that notation.
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
4
5
## 1.1 The perceptron, recalled
6
6b31d5 lugonthier 2026-07-15 12:37:13
feat: Update "Decision trees and ensemble methods" module with new content and visuals - Revamped the introduction to ensemble methods, emphasizing the benefits of combining models. - Expanded sections on decision trees, bagging, and boosting, including detailed explanations and formulas. - Added new SVG diagrams illustrating the bagging process, the transition from stumps to trees, and variance reduction. - Introduced new images for AdaBoost rounds and variance reduction to enhance understanding.
7
The perceptron from the Machine Learning course is a single computational unit. It scores an input with a linear combination of its features and passes that score through a hard threshold. With parameters $w$ and the step activation $g$, its hypothesis is:
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
8
6b31d5 lugonthier 2026-07-15 12:37:13
feat: Update "Decision trees and ensemble methods" module with new content and visuals - Revamped the introduction to ensemble methods, emphasizing the benefits of combining models. - Expanded sections on decision trees, bagging, and boosting, including detailed explanations and formulas. - Added new SVG diagrams illustrating the bagging process, the transition from stumps to trees, and variance reduction. - Introduced new images for AdaBoost rounds and variance reduction to enhance understanding.
9
$$\boxed{ h(x) = g(w^T x), \quad g(z) = \begin{cases} 1 & \text{if } z \ge 0 \\ 0 & \text{otherwise} \end{cases} }$$
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
10
6b31d5 lugonthier 2026-07-15 12:37:13
feat: Update "Decision trees and ensemble methods" module with new content and visuals - Revamped the introduction to ensemble methods, emphasizing the benefits of combining models. - Expanded sections on decision trees, bagging, and boosting, including detailed explanations and formulas. - Added new SVG diagrams illustrating the bagging process, the transition from stumps to trees, and variance reduction. - Introduced new images for AdaBoost rounds and variance reduction to enhance understanding.
11
The equation $w^T x = 0$ is a hyperplane, so the perceptron splits the input space with a single flat boundary. Points on one side are labelled $1$, points on the other are labelled $0$.
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
12
6b31d5 lugonthier 2026-07-15 12:37:13
feat: Update "Decision trees and ensemble methods" module with new content and visuals - Revamped the introduction to ensemble methods, emphasizing the benefits of combining models. - Expanded sections on decision trees, bagging, and boosting, including detailed explanations and formulas. - Added new SVG diagrams illustrating the bagging process, the transition from stumps to trees, and variance reduction. - Introduced new images for AdaBoost rounds and variance reduction to enhance understanding.
13
*Remark:* the boundary is linear because the score $w^T x$ is linear in $x$. The threshold only chooses a side, it does not bend the boundary.
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
14
15
## 1.2 Why one unit is not enough
16
17
A single linear boundary can only solve problems whose classes are **linearly separable**, that is, separable by one straight cut. Many simple problems are, but not all. The classic counterexample is the exclusive-or (XOR) function of two binary inputs.
18
19
The truth tables below compare AND, OR, and XOR:
20
21
| $x_1$ | $x_2$ | AND | OR | XOR |
22
| --- | --- | --- | --- | --- |
23
| 0 | 0 | 0 | 0 | 0 |
24
| 0 | 1 | 0 | 1 | 1 |
25
| 1 | 0 | 0 | 1 | 1 |
26
| 1 | 1 | 1 | 1 | 0 |
27
28
![AND, OR, and XOR points with separating lines](/en/Deep%20Learning/01%20Introduction/a/xor-problem.png)
29
30
*AND and OR are separable by a single straight line, but XOR is not, which is why one unit cannot solve it.*
31
32
For AND and OR the two output classes can be separated by a single line, so a perceptron solves them. For XOR the positive points $(0,1)$ and $(1,0)$ sit on one diagonal and the negative points $(0,0)$ and $(1,1)$ sit on the other. No single straight line can separate them.
33
34
*Remark:* XOR is not a special curiosity. It shows that some patterns are inherently nonlinear, so any model built from one linear boundary is fundamentally limited. The fix is to combine several units.
35
36
If we place a layer of units between the input and the output, the first units can carve the space with several boundaries and a later unit can combine their outputs. Two lines can isolate the XOR pattern where one cannot. That intermediate layer is a **hidden layer**, and it is what turns a single unit into a network.
37
38
## 1.3 From units to networks
39
40
Stacking units into layers, and layers into a pipeline, gives a **neural network**. A network is **deep** when it has more than one hidden layer between the input and the output. Each layer applies a linear map followed by a nonlinear activation, and the layers are composed so the output of one feeds the input of the next.
41
42
The payoff is **representation learning**. In classical machine learning we hand-craft features, then feed them to a linear model. In a deep network the hidden layers learn their own features from raw input: early layers capture simple patterns and later layers combine them into more abstract ones. We specify the architecture and the objective, and the network discovers the intermediate representations by training.
43
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
44
*Remark:* stacking linear maps alone would collapse back to a single linear map, so the nonlinear activation $g$ between layers is essential. Without it, no depth would add expressive power. Activation functions are covered in the next lesson.
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
45
46
## 1.4 Notation for this course
47
6b31d5 lugonthier 2026-07-15 12:37:13
feat: Update "Decision trees and ensemble methods" module with new content and visuals - Revamped the introduction to ensemble methods, emphasizing the benefits of combining models. - Expanded sections on decision trees, bagging, and boosting, including detailed explanations and formulas. - Added new SVG diagrams illustrating the bagging process, the transition from stumps to trees, and variance reduction. - Introduced new images for AdaBoost rounds and variance reduction to enhance understanding.
48
The Machine Learning course folded the bias into the score with the intercept convention $x_0 = 1$, so a single dot product $w^T x$ carried the constant term. This course keeps the bias **explicit** and uses a separate weight matrix per layer. This is the seam between the two courses: from here on, no augmented input and no folded bias.
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
49
50
### 1.4.1 A single unit
51
52
With explicit bias, one unit has a weight vector $w$ and a scalar bias $b$. Its activation is:
53
54
$$\boxed{ a = g(w^T x + b) }$$
55
6b31d5 lugonthier 2026-07-15 12:37:13
feat: Update "Decision trees and ensemble methods" module with new content and visuals - Revamped the introduction to ensemble methods, emphasizing the benefits of combining models. - Expanded sections on decision trees, bagging, and boosting, including detailed explanations and formulas. - Added new SVG diagrams illustrating the bagging process, the transition from stumps to trees, and variance reduction. - Introduced new images for AdaBoost rounds and variance reduction to enhance understanding.
56
The score $w^T x + b$ is the same affine function as before, only now the bias $b$ is written out instead of hidden inside $w$.
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
57
58
### 1.4.2 A layer and a network
59
60
Group the units of layer $l$ into a weight matrix $W^{[l]}$ and a bias vector $b^{[l]}$. The layer computes a pre-activation $z^{[l]}$, then an activation $a^{[l]}$:
61
62
$$\boxed{ z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}, \quad a^{[l]} = g^{[l]}(z^{[l]}) }$$
63
64
The input feeds the first layer as $a^{[0]} = x$, and for an $L$-layer network the prediction is the last activation:
65
66
$$\boxed{ a^{[0]} = x, \quad \hat{y} = a^{[L]} }$$
67
68
So the network is a composition of layer maps: $x = a^{[0]} \mapsto a^{[1]} \mapsto \cdots \mapsto a^{[L]} = \hat{y}$.
69
70
### 1.4.3 Symbol table
71
72
| Symbol | Meaning | Shape |
73
| --- | --- | --- |
74
| $L$ | number of layers | scalar |
75
| $n_l$ | number of units in layer $l$ | scalar |
76
| $W^{[l]}$ | weight matrix of layer $l$ | $n_l \times n_{l-1}$ |
77
| $b^{[l]}$ | bias vector of layer $l$ | $n_l$ |
78
| $z^{[l]}$ | pre-activation of layer $l$ | $n_l$ |
79
| $a^{[l]}$ | activation of layer $l$ | $n_l$ |
80
| $g^{[l]}$ | activation function of layer $l$ | applied elementwise |
81
| $\hat{y}$ | prediction, equal to $a^{[L]}$ | $n_L$ |
82
83
*Remark:* the activation $g^{[l]}$ acts componentwise, so an elementwise product later on is written with the Hadamard symbol $\odot$. The superscript in brackets, $[l]$, indexes the layer, not an exponent.
84
85
The following diagram shows the smallest useful network: an input layer, one hidden layer, and an output layer.
86
87
![A single-hidden-layer neural network](/en/Deep%20Learning/01%20Introduction/a/network-single-hidden.svg)
88
89
*A neural network: an input layer, one hidden layer, and an output. Each edge carries a weight and each unit adds a bias then applies an activation g.*
90
91
Each arrow carries a weight from $W^{[l]}$, and every hidden and output unit adds its bias from $b^{[l]}$ before applying its activation. This two-unit hidden layer is exactly what lets the network solve XOR, the task that defeated a single unit.
92
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
93
## 1.5 The multilayer perceptron, recalled
94
95
The network these symbols describe is the multilayer perceptron (MLP), built step by step in [Multilayer neural networks](/en/Machine%20Learning/06%20Multilayer%20neural%20networks): take logistic regression, insert hidden layers of the same dot-product units, and read the composition of section 1.4.2 from left to right. Training was settled there too, and one full step translates into the new notation in one pass.
96
97
**Forward.** Propagate the input through $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$ and $a^{[l]} = g^{[l]}(z^{[l]})$, caching every $z^{[l]}$ and $a^{[l]}$ along the way.
98
99
**Loss.** Compare $\hat{y} = a^{[L]}$ to the label with the loss matched to the task: squared error behind an identity output for regression, cross-entropy behind a sigmoid or softmax for classification.
100
101
**Backward.** Send the loss gradient back through the same wiring with the chain rule, layer by layer:
102
103
$$\boxed{ \delta^{[l]} = \left((W^{[l+1]})^T \delta^{[l+1]}\right) \odot g'^{[l]}\!\left(z^{[l]}\right), \qquad \frac{\partial L}{\partial W^{[l]}} = \delta^{[l]} (a^{[l-1]})^T, \qquad \frac{\partial L}{\partial b^{[l]}} = \delta^{[l]} }$$
104
105
The only novelty is the last equality: the bias gradient gets its own line now, because $b^{[l]}$ is no longer a column of $W^{[l]}$ fed by a constant neuron.
106
107
**Update.** Take a gradient step on a mini-batch (lesson 3 improves this step with momentum, RMSProp, and Adam).
108
109
If any step feels foggy, the module has it in full: [the model](/en/Machine%20Learning/06%20Multilayer%20neural%20networks#62-make-logistic-regression-deep) with a worked example on the graph, [the losses](/en/Machine%20Learning/06%20Multilayer%20neural%20networks#63-the-loss-function), and [backpropagation](/en/Machine%20Learning/06%20Multilayer%20neural%20networks#64-how-to-optimize-the-parameters) with a complete numeric training step. This course takes those as given and owns everything that follows: the activation functions (next lesson), the optimizers and their good practices (lesson 3), and the training toolkit of initialization, normalization, and regularization (lesson 4).
110
111
*The model, its losses, and its training were built in the Machine Learning course. The next lesson picks the story up at the choice that makes depth worthwhile: the activation functions.*
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
112
113
---
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
114
Next: [Activation functions](/en/Deep%20Learning/02%20Activation%20functions) · [Course overview](/en/Deep%20Learning)