Backpropagation returns the gradient of the cost with respect to every parameter. An optimizer is the rule that turns those gradients into updates. This module covers the gradient-descent variants and the adaptive optimizers (momentum, RMSProp, Adam) that make deep networks trainable, plus the learning-rate schedules that shape the run.

- Reuse the gradient-descent update from the Machine Learning course and name its batch, mini-batch, and stochastic variants.

- Add momentum to damp oscillations and accelerate along consistent directions.

- Rescale each coordinate by its recent gradient magnitude with RMSProp.

- Combine both ideas into Adam and understand its bias correction.

- Pick a learning-rate schedule: step decay, cosine, or warmup.

- Compare the optimizers and know when to reach for each.

with learning rate $\alpha > 0$. This is the LMS update from the Machine Learning course, written for the full parameter vector instead of one coordinate.

*Remark:* bias is explicit here. The gradient $g$ has one block per $W^{[l]}$ and one per $b^{[l]}$, and the update applies to each block with the same $\alpha$.

The variants differ only in how many examples enter the gradient $g$ at each step.

| variant | examples per step | update noise | per step | use when |

| --- | --- | --- | --- | --- |

| Batch GD | all $m$ | none | $O(m)$ passes | $m$ small, exact gradient wanted |

| Mini-batch GD | a batch of $B$ | moderate | $O(B)$ | the default for deep nets |

| Stochastic GD (SGD) | one example | high | $O(1)$ | streaming, very large $m$ |

*Remark:* one pass over the whole dataset is an epoch. Mini-batch is the standard choice: batches of $32$ to $512$ fit the accelerator, exploit vectorized matrix products, and the residual noise in $g$ helps escape shallow local minima. In deep learning "SGD" is used loosely to mean mini-batch gradient descent.

Plain SGD zig-zags across narrow valleys because the gradient points across the valley more than along it. Momentum accumulates an exponentially weighted average of past gradients in a velocity vector $v$, then steps in that averaged direction:

with momentum coefficient $\beta \in [0, 1)$, typically $\beta = 0.9$. Components of $g$ that keep the same sign reinforce each other, so $v$ grows and the step accelerates along consistent directions. Components that flip sign cancel in the average, so oscillations across the valley are damped.


The learning rate $\alpha$ is the single most important hyperparameter, and holding it fixed is rarely optimal. A large $\alpha$ speeds early progress but prevents settling into a minimum, so schedules typically decrease $\alpha$ over training. Here $\alpha_0$ is the initial rate and $t$ indexes the step or epoch.

Multiply $\alpha$ by a factor $\gamma \in (0, 1)$ every $s$ epochs, so it drops in discrete stages:

$$\boxed{ \alpha_t = \alpha_0\, \gamma^{\lfloor t / s \rfloor} }$$

Anneal $\alpha$ smoothly from $\alpha_0$ toward a floor of zero along a half cosine over $T$ total steps:

$$\boxed{ \alpha_t = \tfrac{1}{2}\,\alpha_0\left(1 + \cos\frac{\pi t}{T}\right) }$$

Warmup ramps $\alpha$ up linearly from a small value over the first few hundred to few thousand steps, then hands off to a decay schedule. It prevents the large, poorly conditioned updates a cold start with a big $\alpha$ would produce, and it is standard for deep networks such as transformers.

| schedule | shape | main use |

| --- | --- | --- |

| Step decay | staircase drops | classic vision training |

| Cosine | smooth anneal to zero | modern default, often with warmup |

| Warmup | linear ramp up, then decay | stabilize early steps, large models |


*Remark:* warmup and a decay are usually composed, warmup for the first phase and cosine or step decay afterward.

| optimizer | what it adds | tracks | typical use |

| --- | --- | --- | --- |

| SGD | nothing, base rule | none | strong baseline, best final accuracy with tuning |

| Momentum | velocity, damps oscillation | first moment $v$ | vision models, with a schedule |

| RMSProp | per-coordinate scaling | second moment $s$ | RNNs, non-stationary objectives |

| Adam | momentum plus scaling, bias-corrected | first and second moments | the default first choice for most nets |

*Remark:* Adam is the safe default and converges fast with little tuning. Well-tuned SGD with momentum and a schedule often reaches slightly better final test accuracy on large vision models, which is why both remain in wide use.


---

Next: [Initialization and vanishing gradients](/en/Deep%20Learning/07%20Initialization%20and%20vanishing%20gradients) · [Course overview](/en/Deep%20Learning)