Home A - Z
Page Index

An Otter Wiki

Blame

12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
1 
# 3. Optimization
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
2
3
Backpropagation returns the gradient of the cost with respect to every parameter. An optimizer is the rule that turns those gradients into updates. This module covers the gradient-descent variants and the adaptive optimizers (momentum, RMSProp, Adam) that make deep networks trainable, plus the learning-rate schedules that shape the run.
4
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
5
## 3.1 Gradient descent
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
6
6b31d5 lugonthier 2026-07-15 12:37:13
feat: Update "Decision trees and ensemble methods" module with new content and visuals - Revamped the introduction to ensemble methods, emphasizing the benefits of combining models. - Expanded sections on decision trees, bagging, and boosting, including detailed explanations and formulas. - Added new SVG diagrams illustrating the bagging process, the transition from stumps to trees, and variance reduction. - Introduced new images for AdaBoost rounds and variance reduction to enhance understanding.
7
Let $w$ collect all parameters (every $W^{[l]}$ and $b^{[l]}$) and let $J(w)$ be the cost, the average of the per-example loss $L$. Write $g = \nabla_w J(w)$ for the gradient of the cost with respect to the parameters, as returned by backpropagation. The base update moves $w$ downhill:
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
8
6b31d5 lugonthier 2026-07-15 12:37:13
feat: Update "Decision trees and ensemble methods" module with new content and visuals - Revamped the introduction to ensemble methods, emphasizing the benefits of combining models. - Expanded sections on decision trees, bagging, and boosting, including detailed explanations and formulas. - Added new SVG diagrams illustrating the bagging process, the transition from stumps to trees, and variance reduction. - Introduced new images for AdaBoost rounds and variance reduction to enhance understanding.
9
$$\boxed{ w \leftarrow w - \alpha\, g }$$
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
10
11
with learning rate $\alpha > 0$. This is the LMS update from the Machine Learning course, written for the full parameter vector instead of one coordinate.
12
13
*Remark:* bias is explicit here. The gradient $g$ has one block per $W^{[l]}$ and one per $b^{[l]}$, and the update applies to each block with the same $\alpha$.
14
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
15
### 3.1.1 Batch, mini-batch, stochastic
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
16
17
The variants differ only in how many examples enter the gradient $g$ at each step.
18
19
| variant | examples per step | update noise | per step | use when |
20
| --- | --- | --- | --- | --- |
21
| Batch GD | all $m$ | none | $O(m)$ passes | $m$ small, exact gradient wanted |
22
| Mini-batch GD | a batch of $B$ | moderate | $O(B)$ | the default for deep nets |
23
| Stochastic GD (SGD) | one example | high | $O(1)$ | streaming, very large $m$ |
24
25
*Remark:* one pass over the whole dataset is an epoch. Mini-batch is the standard choice: batches of $32$ to $512$ fit the accelerator, exploit vectorized matrix products, and the residual noise in $g$ helps escape shallow local minima. In deep learning "SGD" is used loosely to mean mini-batch gradient descent.
26
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
27
## 3.2 Momentum
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
28
29
Plain SGD zig-zags across narrow valleys because the gradient points across the valley more than along it. Momentum accumulates an exponentially weighted average of past gradients in a velocity vector $v$, then steps in that averaged direction:
30
6b31d5 lugonthier 2026-07-15 12:37:13
feat: Update "Decision trees and ensemble methods" module with new content and visuals - Revamped the introduction to ensemble methods, emphasizing the benefits of combining models. - Expanded sections on decision trees, bagging, and boosting, including detailed explanations and formulas. - Added new SVG diagrams illustrating the bagging process, the transition from stumps to trees, and variance reduction. - Introduced new images for AdaBoost rounds and variance reduction to enhance understanding.
31
$$\boxed{ v \leftarrow \beta\, v + g, \qquad w \leftarrow w - \alpha\, v }$$
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
32
33
with momentum coefficient $\beta \in [0, 1)$, typically $\beta = 0.9$. Components of $g$ that keep the same sign reinforce each other, so $v$ grows and the step accelerates along consistent directions. Components that flip sign cancel in the average, so oscillations across the valley are damped.
34
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
35
![Plain gradient descent against momentum in a ravine](/en/Deep%20Learning/03%20Optimization/a/ravine-momentum.svg)
36
37
*The curves are level lines of the loss, the dot is the minimum. The gradient is perpendicular to the level line it sits on, so in a ravine it points mostly across the valley, and plain gradient descent bounces. Momentum keeps a memory of the previous steps, the bounces cancel and the valley direction accumulates.*
38
39
### 3.2.1 Nesterov momentum
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
40
6b31d5 lugonthier 2026-07-15 12:37:13
feat: Update "Decision trees and ensemble methods" module with new content and visuals - Revamped the introduction to ensemble methods, emphasizing the benefits of combining models. - Expanded sections on decision trees, bagging, and boosting, including detailed explanations and formulas. - Added new SVG diagrams illustrating the bagging process, the transition from stumps to trees, and variance reduction. - Introduced new images for AdaBoost rounds and variance reduction to enhance understanding.
41
Nesterov accelerated gradient evaluates the gradient at a look-ahead point, after the momentum step has been provisionally applied, rather than at the current $w$. This anticipatory correction reacts sooner when the slope changes:
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
42
6b31d5 lugonthier 2026-07-15 12:37:13
feat: Update "Decision trees and ensemble methods" module with new content and visuals - Revamped the introduction to ensemble methods, emphasizing the benefits of combining models. - Expanded sections on decision trees, bagging, and boosting, including detailed explanations and formulas. - Added new SVG diagrams illustrating the bagging process, the transition from stumps to trees, and variance reduction. - Introduced new images for AdaBoost rounds and variance reduction to enhance understanding.
43
$$\boxed{ v \leftarrow \beta\, v + \nabla_w J(w - \alpha \beta\, v), \qquad w \leftarrow w - \alpha\, v }$$
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
44
45
*Remark:* think of $\beta \approx 0.9$ as averaging over roughly the last $\tfrac{1}{1 - \beta} = 10$ gradients. Nesterov usually converges slightly faster than plain momentum for the same $\alpha$ and $\beta$.
46
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
47
## 3.3 RMSProp
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
48
49
Different parameters can need very different step sizes, and one global $\alpha$ cannot serve them all. RMSProp keeps a per-coordinate running average $s$ of squared gradients, then divides the step by $\sqrt{s}$, so coordinates with large recent gradients take smaller steps and quiet coordinates take larger ones:
50
6b31d5 lugonthier 2026-07-15 12:37:13
feat: Update "Decision trees and ensemble methods" module with new content and visuals - Revamped the introduction to ensemble methods, emphasizing the benefits of combining models. - Expanded sections on decision trees, bagging, and boosting, including detailed explanations and formulas. - Added new SVG diagrams illustrating the bagging process, the transition from stumps to trees, and variance reduction. - Introduced new images for AdaBoost rounds and variance reduction to enhance understanding.
51
$$\boxed{ s \leftarrow \rho\, s + (1 - \rho)\, g^2, \qquad w \leftarrow w - \alpha\, \frac{g}{\sqrt{s} + \epsilon} }$$
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
52
53
with decay $\rho \approx 0.9$ and a small $\epsilon \approx 10^{-8}$ for numerical safety. Here $g^2 = g \odot g$ is the Hadamard (elementwise) square and the division is elementwise, so each coordinate is normalized by its own recent gradient scale.
54
55
*Remark:* $s$ estimates the uncentered second moment of each coordinate of $g$, so $\sqrt{s}$ is roughly its recent root-mean-square magnitude. RMSProp suits non-stationary objectives, which is exactly what a moving mini-batch gradient is.
56
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
57
## 3.4 Adam
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
58
59
Adam (adaptive moment estimation) combines momentum and RMSProp: it keeps a first-moment estimate $m$ (the mean of the gradient) and a second-moment estimate $v$ (the mean of the squared gradient).
60
61
$$\boxed{ m \leftarrow \beta_1\, m + (1 - \beta_1)\, g, \qquad v \leftarrow \beta_2\, v + (1 - \beta_2)\, g^2 }$$
62
63
Both $m$ and $v$ start at zero, so early in training they are biased toward zero. Dividing by $1 - \beta_1^t$ and $1 - \beta_2^t$ at step $t$ removes that bias:
64
65
$$\boxed{ \hat m = \frac{m}{1 - \beta_1^{\,t}}, \qquad \hat v = \frac{v}{1 - \beta_2^{\,t}} }$$
66
67
The update then steps in the momentum direction, rescaled per coordinate by the second moment:
68
6b31d5 lugonthier 2026-07-15 12:37:13
feat: Update "Decision trees and ensemble methods" module with new content and visuals - Revamped the introduction to ensemble methods, emphasizing the benefits of combining models. - Expanded sections on decision trees, bagging, and boosting, including detailed explanations and formulas. - Added new SVG diagrams illustrating the bagging process, the transition from stumps to trees, and variance reduction. - Introduced new images for AdaBoost rounds and variance reduction to enhance understanding.
69
$$\boxed{ w \leftarrow w - \alpha\, \frac{\hat m}{\sqrt{\hat v} + \epsilon} }$$
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
70
71
Common defaults are $\beta_1 = 0.9$, $\beta_2 = 0.999$, and $\epsilon = 10^{-8}$. As before the square, square root, and division are elementwise.
72
73
*Remark:* the bias correction matters most in the first few dozen steps, when $t$ is small and $\beta_2^t$ is still close to $1$. Without it, $\hat v$ would be far too small and the early steps far too large. AdamW, a common variant, decouples weight decay from this update.
74
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
75
![Optimizer family: gradient feeds momentum and RMSProp, which combine into Adam and the parameter update](/en/Deep%20Learning/03%20Optimization/a/optimizer-family.svg)
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
76
77
*Adam combines the momentum of averaged gradients with the per-parameter scaling of RMSProp.*
78
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
79
## 3.5 Learning-rate schedules
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
80
81
The learning rate $\alpha$ is the single most important hyperparameter, and holding it fixed is rarely optimal. A large $\alpha$ speeds early progress but prevents settling into a minimum, so schedules typically decrease $\alpha$ over training. Here $\alpha_0$ is the initial rate and $t$ indexes the step or epoch.
82
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
83
### 3.5.1 Step decay
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
84
85
Multiply $\alpha$ by a factor $\gamma \in (0, 1)$ every $s$ epochs, so it drops in discrete stages:
86
87
$$\boxed{ \alpha_t = \alpha_0\, \gamma^{\lfloor t / s \rfloor} }$$
88
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
89
### 3.5.2 Cosine decay
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
90
91
Anneal $\alpha$ smoothly from $\alpha_0$ toward a floor of zero along a half cosine over $T$ total steps:
92
93
$$\boxed{ \alpha_t = \tfrac{1}{2}\,\alpha_0\left(1 + \cos\frac{\pi t}{T}\right) }$$
94
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
95
### 3.5.3 Warmup
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
96
97
Warmup ramps $\alpha$ up linearly from a small value over the first few hundred to few thousand steps, then hands off to a decay schedule. It prevents the large, poorly conditioned updates a cold start with a big $\alpha$ would produce, and it is standard for deep networks such as transformers.
98
99
| schedule | shape | main use |
100
| --- | --- | --- |
101
| Step decay | staircase drops | classic vision training |
102
| Cosine | smooth anneal to zero | modern default, often with warmup |
103
| Warmup | linear ramp up, then decay | stabilize early steps, large models |
104
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
105
![Three learning-rate schedules over training steps: step decay, cosine decay, and warmup then decay](/en/Deep%20Learning/03%20Optimization/a/lr-schedules.png)
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
106
107
*Common learning-rate schedules: step decay, cosine decay, and a warmup followed by decay.*
108
109
*Remark:* warmup and a decay are usually composed, warmup for the first phase and cosine or step decay afterward.
110
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
111
## 3.6 Choosing an optimizer
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
112
113
| optimizer | what it adds | tracks | typical use |
114
| --- | --- | --- | --- |
115
| SGD | nothing, base rule | none | strong baseline, best final accuracy with tuning |
116
| Momentum | velocity, damps oscillation | first moment $v$ | vision models, with a schedule |
117
| RMSProp | per-coordinate scaling | second moment $s$ | RNNs, non-stationary objectives |
118
| Adam | momentum plus scaling, bias-corrected | first and second moments | the default first choice for most nets |
119
120
*Remark:* Adam is the safe default and converges fast with little tuning. Well-tuned SGD with momentum and a schedule often reaches slightly better final test accuracy on large vision models, which is why both remain in wide use.
121
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
122
![Optimization paths of SGD, momentum, and Adam on an elongated quadratic bowl](/en/Deep%20Learning/03%20Optimization/a/optimizer-paths.png)
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
123
124
*On an elongated loss surface, momentum and Adam reach the minimum far faster than plain gradient descent.*
125
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
126
## 3.7 Good practices
127
128
Two of the habits that make training behave, careful initialization and dropout, live in the next lesson ([Training deep networks](/en/Deep%20Learning/04%20Training%20deep%20networks)). Two more belong right here.
129
130
**Center and normalize the inputs.** Standardize each feature (subtract its mean, divide by its standard deviation), so no feature dominates the first dot products and the first layer's gradients start well scaled.
131
132
**Sanity-check before training long.** A freshly initialized $K$-class classifier should start near the loss $\ln K$ (about $2.3$ for $K = 10$). A tiny training set should be easy to overfit: if the network cannot, the code is broken. Watch the training and validation curves. And since backpropagation is error-prone, check its analytic gradient against a numerical finite-difference estimate:
133
134
$$\boxed{ \frac{\partial L}{\partial w} \approx \frac{L(w + \varepsilon) - L(w - \varepsilon)}{2\varepsilon} }$$
135
36084c lugonthier 2026-07-02 14:39:19
Add new content and images for Linear Models, Regularization, SVMs, and Decision Trees - Added images for linear regression, logistic regression, and perceptron. - Introduced a new section on Regularization and High-Dimensional Inference with detailed explanations and images. - Added content on Support Vector Machines, including definitions, loss functions, and kernel methods. - Created a new section on Decision Trees and Ensemble Methods, covering CART, bagging, random forests, and boosting. - Included relevant images to illustrate concepts in Decision Trees and Ensemble Methods.
136
*Every optimizer here scales the raw gradient, so its behaviour depends on how large those gradients are to begin with. The next part studies how the initial weights and the network depth set that scale, and how poor choices make gradients vanish or explode.*
137
138
---
12d21f lugonthier 2026-07-24 11:49:19
Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.
139
Next: [Training deep networks](/en/Deep%20Learning/04%20Training%20deep%20networks) · [Course overview](/en/Deep%20Learning)