Commit 12d21f
2026-07-24 11:49:19 lugonthier: Remove unused SVG files and update Markdown content for clarity and accuracy in MLOps and Machine Learning modules. Adjust references to optimization techniques and activation functions, and enhance explanations in the mathematical refresher section.| en/Deep Learning.md .. | |
| @@ 2,25 2,20 @@ | |
| Neural networks from the single perceptron to modern transformers: how depth, the right activations, and gradient-based training let a model learn its own features instead of hand-crafted ones. | |
| - | **Prerequisites:** the [Machine Learning](/en/Machine%20Learning) course (especially the perceptron in [Linear classification](/en/Machine%20Learning/05%20Linear%20classification)), basic Python, calculus, and linear algebra. |
| + | **Prerequisites:** the [Machine Learning](/en/Machine%20Learning) course, especially [Multilayer neural networks](/en/Machine%20Learning/06%20Multilayer%20neural%20networks), which builds the model and covers the losses and backpropagation used throughout this course. Basic Python, calculus, and linear algebra. |
| ## Syllabus | |
| 1. [Introduction](/en/Deep%20Learning/01%20Introduction) | |
| - | 2. [Multilayer perceptron](/en/Deep%20Learning/02%20Multilayer%20perceptron) |
| - | 3. [Activation functions](/en/Deep%20Learning/03%20Activation%20functions) |
| - | 4. [Loss functions and output layers](/en/Deep%20Learning/04%20Loss%20functions%20and%20output%20layers) |
| - | 5. [Backpropagation](/en/Deep%20Learning/05%20Backpropagation) |
| - | 6. [Optimization](/en/Deep%20Learning/06%20Optimization) |
| - | 7. [Initialization and vanishing gradients](/en/Deep%20Learning/07%20Initialization%20and%20vanishing%20gradients) |
| - | 8. [Normalization](/en/Deep%20Learning/08%20Normalization) |
| - | 9. [Regularization and dropout](/en/Deep%20Learning/09%20Regularization%20and%20dropout) |
| - | 10. [Convolutional networks](/en/Deep%20Learning/10%20Convolutional%20networks) |
| - | 11. [Embeddings and representation learning](/en/Deep%20Learning/11%20Embeddings%20and%20representation%20learning) |
| - | 12. [Recurrent networks](/en/Deep%20Learning/12%20Recurrent%20networks) |
| - | 13. [LSTM and GRU](/en/Deep%20Learning/13%20LSTM%20and%20GRU) |
| - | 14. [Attention](/en/Deep%20Learning/14%20Attention) |
| - | 15. [Transformers](/en/Deep%20Learning/15%20Transformers) |
| + | 2. [Activation functions](/en/Deep%20Learning/02%20Activation%20functions) |
| + | 3. [Optimization](/en/Deep%20Learning/03%20Optimization) |
| + | 4. [Training deep networks](/en/Deep%20Learning/04%20Training%20deep%20networks) |
| + | 5. [Convolutional networks](/en/Deep%20Learning/05%20Convolutional%20networks) |
| + | 6. [Embeddings and representation learning](/en/Deep%20Learning/06%20Embeddings%20and%20representation%20learning) |
| + | 7. [Recurrent networks](/en/Deep%20Learning/07%20Recurrent%20networks) |
| + | 8. [LSTM and GRU](/en/Deep%20Learning/08%20LSTM%20and%20GRU) |
| + | 9. [Attention](/en/Deep%20Learning/09%20Attention) |
| + | 10. [Transformers](/en/Deep%20Learning/10%20Transformers) |
| --- | |
| [Machine Learning](/en/Machine%20Learning) · [MLOps](/en/MLOps) · [Home](/en) | |
| en/Deep Learning/01 Introduction.md .. | |
| @@ 1,13 1,6 @@ | |
| # 1. Introduction | |
| - | This course continues directly from the Machine Learning course, which closed the [Linear classification](/en/Machine%20Learning/05%20Linear%20classification) part with a key remark: a perceptron is a single unit, and stacked into layers it becomes a neural network. This lesson makes that bridge explicit. It recalls what one unit can do, shows the concrete task (XOR) where a single unit fails, and fixes the notation used throughout the rest of the course. |
| - | |
| - | **Objectives** |
| - | - Recall the perceptron as a single unit with a step activation and a linear boundary. |
| - | - See why one unit cannot solve XOR, motivating hidden layers. |
| - | - Understand what "deep" means and why hidden layers learn features. |
| - | - Adopt the explicit-bias, per-layer notation used across this course. |
| - | - Read a network as a composition of layer maps from input to prediction. |
| + | This course continues directly from the Machine Learning course, whose [Multilayer neural networks](/en/Machine%20Learning/06%20Multilayer%20neural%20networks) module built the network itself: the model, its losses, and backpropagation. This lesson re-anchors that bridge. It recalls what one unit can do, shows the concrete task (XOR) where a single unit fails, fixes the notation used throughout the rest of the course, then restates the multilayer perceptron and its training loop in that notation. |
| ## 1.1 The perceptron, recalled | |
| @@ 48,7 41,7 @@ | |
| The payoff is **representation learning**. In classical machine learning we hand-craft features, then feed them to a linear model. In a deep network the hidden layers learn their own features from raw input: early layers capture simple patterns and later layers combine them into more abstract ones. We specify the architecture and the objective, and the network discovers the intermediate representations by training. | |
| - | *Remark:* stacking linear maps alone would collapse back to a single linear map, so the nonlinear activation $g$ between layers is essential. Without it, no depth would add expressive power. Activation functions are covered in the next lessons. |
| + | *Remark:* stacking linear maps alone would collapse back to a single linear map, so the nonlinear activation $g$ between layers is essential. Without it, no depth would add expressive power. Activation functions are covered in the next lesson. |
| ## 1.4 Notation for this course | |
| @@ 97,7 90,25 @@ | |
| Each arrow carries a weight from $W^{[l]}$, and every hidden and output unit adds its bias from $b^{[l]}$ before applying its activation. This two-unit hidden layer is exactly what lets the network solve XOR, the task that defeated a single unit. | |
| - | *The next lesson formalizes this picture as the multilayer perceptron, writing the full forward pass layer by layer and choosing the activation functions.* |
| + | ## 1.5 The multilayer perceptron, recalled |
| + | |
| + | The network these symbols describe is the multilayer perceptron (MLP), built step by step in [Multilayer neural networks](/en/Machine%20Learning/06%20Multilayer%20neural%20networks): take logistic regression, insert hidden layers of the same dot-product units, and read the composition of section 1.4.2 from left to right. Training was settled there too, and one full step translates into the new notation in one pass. |
| + | |
| + | **Forward.** Propagate the input through $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$ and $a^{[l]} = g^{[l]}(z^{[l]})$, caching every $z^{[l]}$ and $a^{[l]}$ along the way. |
| + | |
| + | **Loss.** Compare $\hat{y} = a^{[L]}$ to the label with the loss matched to the task: squared error behind an identity output for regression, cross-entropy behind a sigmoid or softmax for classification. |
| + | |
| + | **Backward.** Send the loss gradient back through the same wiring with the chain rule, layer by layer: |
| + | |
| + | $$\boxed{ \delta^{[l]} = \left((W^{[l+1]})^T \delta^{[l+1]}\right) \odot g'^{[l]}\!\left(z^{[l]}\right), \qquad \frac{\partial L}{\partial W^{[l]}} = \delta^{[l]} (a^{[l-1]})^T, \qquad \frac{\partial L}{\partial b^{[l]}} = \delta^{[l]} }$$ |
| + | |
| + | The only novelty is the last equality: the bias gradient gets its own line now, because $b^{[l]}$ is no longer a column of $W^{[l]}$ fed by a constant neuron. |
| + | |
| + | **Update.** Take a gradient step on a mini-batch (lesson 3 improves this step with momentum, RMSProp, and Adam). |
| + | |
| + | If any step feels foggy, the module has it in full: [the model](/en/Machine%20Learning/06%20Multilayer%20neural%20networks#62-make-logistic-regression-deep) with a worked example on the graph, [the losses](/en/Machine%20Learning/06%20Multilayer%20neural%20networks#63-the-loss-function), and [backpropagation](/en/Machine%20Learning/06%20Multilayer%20neural%20networks#64-how-to-optimize-the-parameters) with a complete numeric training step. This course takes those as given and owns everything that follows: the activation functions (next lesson), the optimizers and their good practices (lesson 3), and the training toolkit of initialization, normalization, and regularization (lesson 4). |
| + | |
| + | *The model, its losses, and its training were built in the Machine Learning course. The next lesson picks the story up at the choice that makes depth worthwhile: the activation functions.* |
| --- | |
| - | Next: [Multilayer perceptron](/en/Deep%20Learning/02%20Multilayer%20perceptron) · [Course overview](/en/Deep%20Learning) |
| + | Next: [Activation functions](/en/Deep%20Learning/02%20Activation%20functions) · [Course overview](/en/Deep%20Learning) |
| en/Deep Learning/03 Activation functions.md .. en/Deep Learning/02 Activation functions.md | |
| @@ 1,15 1,8 @@ | |
| - | # 3. Activation functions |
| + | # 2. Activation functions |
| Each layer computes a pre-activation $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$ and then an activation $a^{[l]} = g^{[l]}(z^{[l]})$. The choice of the nonlinearity $g^{[l]}$ is what makes depth worthwhile. This lesson explains why a nonlinear $g$ is required, surveys the sigmoid, tanh, and ReLU families, introduces the softmax used at the output, and gives practical guidance on which activation to pick. | |
| - | **Objectives** |
| - | - Show that a stack of purely linear layers collapses to a single linear map. |
| - | - Define the sigmoid and tanh, derive their derivatives, and explain saturation. |
| - | - Survey the ReLU family (ReLU, leaky ReLU, PReLU, ELU, GELU) and the dead-unit problem. |
| - | - Define the softmax and place it at the output rather than in hidden layers. |
| - | - Give a short rule of thumb for choosing an activation per layer. |
| - | |
| - | ## 3.1 Why nonlinearity is required |
| + | ## 2.1 Why nonlinearity is required |
| Suppose every activation were the identity, $g^{[l]}(z) = z$. Then each layer is just $a^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$, and composing two of them gives $W^{[2]}(W^{[1]} x + b^{[1]}) + b^{[2]} = (W^{[2]} W^{[1]}) x + (W^{[2]} b^{[1]} + b^{[2]})$. That is again of the form $W x + b$. By induction the whole $L$-layer network reduces to a single affine map: | |
| @@ 19,13 12,13 @@ | |
| *Remark:* the bias is kept explicit here as $b^{[l]}$, unlike the Machine Learning course where the intercept was folded into $w^T x$ via the augmented input $x_0 = 1$. In this Deep Learning course each layer has its own weight matrix $W^{[l]}$ and its own bias vector $b^{[l]}$. | |
| - | ## 3.2 Sigmoid and tanh |
| + | ## 2.2 Sigmoid and tanh |
| - |  |
| + |  |
| *Common activation functions: the bounded sigmoid and tanh saturate in their tails, while ReLU and its variants stay linear for positive inputs.* | |
| - | ### 3.2.1 Sigmoid |
| + | ### 2.2.1 Sigmoid |
| The sigmoid squashes any real pre-activation into the open interval $(0, 1)$: | |
| @@ 35,7 28,7 @@ | |
| $$\boxed{ \sigma'(z) = \sigma(z)\left(1 - \sigma(z)\right) }$$ | |
| - | ### 3.2.2 Tanh |
| + | ### 2.2.2 Tanh |
| The hyperbolic tangent is a rescaled sigmoid centred at zero, with output in $(-1, 1)$. Its derivative is likewise expressible from the forward value: | |
| @@ 43,15 36,42 @@ | |
| *Remark:* $\tanh$ is zero-centred while $\sigma$ is not, so $\tanh$ often trains a little better as a hidden activation. The two are related by $\tanh(z) = 2\sigma(2z) - 1$. | |
| - | ### 3.2.3 Saturation |
| + | ### 2.2.3 Saturation |
| - | Both curves flatten in their tails. For large $|z|$ the output is close to a constant ($0$ or $1$ for $\sigma$, $\pm 1$ for $\tanh$), so the derivative is close to zero: $\sigma'(z) \to 0$ and $\tanh'(z) \to 0$. A unit sitting in that flat region is said to saturate, and it passes almost no gradient backward. When many such factors multiply through a deep stack the gradient shrinks toward zero, the vanishing-gradient problem revisited in [Initialization and vanishing gradients](/en/Deep%20Learning/07%20Initialization%20and%20vanishing%20gradients). |
| + | Both curves flatten in their tails. For large $|z|$ the output is close to a constant ($0$ or $1$ for $\sigma$, $\pm 1$ for $\tanh$), so the derivative is close to zero: $\sigma'(z) \to 0$ and $\tanh'(z) \to 0$. A unit sitting in that flat region is said to saturate, and it passes almost no gradient backward. When many such factors multiply through a deep stack the gradient shrinks toward zero: the vanishing gradient, made precise in the next section. |
| - |  |
| + |  |
| *Activation derivatives: sigmoid and tanh gradients vanish in the tails, whereas the ReLU gradient is 1 wherever the unit is active.* | |
| - | ## 3.3 The ReLU family |
| + | ## 2.3 The vanishing gradient |
| + | |
| + | Saturation is not a cosmetic flaw, it decides whether a deep stack can train at all. Backpropagation multiplies the error by the local slope $g'(z^{[l]})$ at every layer it crosses, so the gradient reaching layer 1 contains one such factor per layer. With sigmoid activations those factors are small by construction: section 2.2.1 gave $\sigma'(z) = \sigma(z)(1 - \sigma(z))$, and that product never exceeds |
| + | |
| + | $$\boxed{ \sigma'(z) = \sigma(z)\left(1 - \sigma(z)\right) \le \tfrac{1}{4} }$$ |
| + | |
| + | The result is the vanishing gradient: the layers near the output learn, the layers near the input receive almost nothing and barely move. Deep sigmoid networks stall, and the fix is not a better optimizer, it is a better activation, the ReLU of the next section. |
| + | |
| + | <details class="proof"> |
| + | <summary>Proof: the gradient shrinks geometrically with depth</summary> |
| + | |
| + | **Step 1: the sigmoid's slope never exceeds $1/4$.** Write $s = \sigma(z) \in (0, 1)$. The product $s(1 - s)$ is a downward parabola in $s$, largest at $s = \tfrac{1}{2}$ where it equals $\tfrac{1}{4}$. So the bound holds, with equality only at $z = 0$, and saturation makes it far worse: a unit sitting at $\sigma(2) \approx 0.88$ already has slope $0.88 \cdot 0.12 \approx 0.10$. |
| + | |
| + | **Step 2: backpropagation multiplies those slopes.** Take the simplest deep network, a chain of $L$ layers with one unit each, so every quantity is a scalar. Each layer crossed contributes the factor $\partial z^{[l]} / \partial z^{[l-1]} = w^{[l]}\, \sigma'(z^{[l-1]})$: |
| + | |
| + | $$\frac{\partial L}{\partial z^{[1]}} = \frac{\partial L}{\partial z^{[L]}} \prod_{l=2}^{L} w^{[l]}\, \sigma'(z^{[l-1]})$$ |
| + | |
| + | With weights of typical size $|w^{[l]}| \le 1$, every factor is at most $\tfrac{1}{4}$ in absolute value, so |
| + | |
| + | $$\boxed{ \left|\frac{\partial L}{\partial z^{[1]}}\right| \le \left(\tfrac{1}{4}\right)^{L-1} \left|\frac{\partial L}{\partial z^{[L]}}\right| }$$ |
| + | |
| + | Ten layers already shrink the gradient by about $10^{-6}$. $\blacksquare$ |
| + | |
| + | </details> |
| + | |
| + | Weights much larger than $1$ only trade the problem for its mirror image, the exploding gradient. [Training deep networks](/en/Deep%20Learning/04%20Training%20deep%20networks) picks the weight scale so this product stays near $1$. |
| + | |
| + | ## 2.4 The ReLU family |
| The rectified linear unit keeps the positive part of its input and zeroes the rest: | |
| @@ 69,7 89,7 @@ | |
| *Remark:* leaky ReLU and PReLU add a small slope $\alpha$ on the negative side so a unit is never fully switched off. GELU weights the input by the probability $\Phi(z)$ that a standard normal is below $z$, giving a smooth curve that behaves like ReLU for large $|z|$. It is the standard choice inside Transformers. | |
| - | ## 3.4 Softmax for multiclass outputs |
| + | ## 2.5 Softmax for multiclass outputs |
| For a classification with $K$ classes the final layer outputs a vector $z \in \mathbb{R}^K$ of scores, and the softmax turns it into a probability distribution over the classes: | |
| @@ 77,13 97,13 @@ | |
| Each component lies in $(0, 1)$ and the components sum to $1$, so $\text{softmax}(z)_k$ reads as the predicted probability of class $k$. The largest score becomes the most likely class. | |
| - | *Remark:* softmax belongs at the output layer, not in a hidden layer. It couples every unit through the shared denominator (a normalization across the whole vector), which is exactly what a probability output needs but is not a useful per-unit hidden nonlinearity. For a single output ($K = 1$ vs its complement) softmax reduces to the sigmoid. The pairing of softmax with its loss is the subject of [Loss functions and output layers](/en/Deep%20Learning/04%20Loss%20functions%20and%20output%20layers). |
| + | *Remark:* softmax belongs at the output layer, not in a hidden layer. It couples every unit through the shared denominator (a normalization across the whole vector), which is exactly what a probability output needs but is not a useful per-unit hidden nonlinearity. For a single output ($K = 1$ vs its complement) softmax reduces to the sigmoid. The pairing of softmax with its loss was settled in section 6.3 of [Multilayer neural networks](/en/Machine%20Learning/06%20Multilayer%20neural%20networks). |
| - | ## 3.5 Choosing an activation |
| + | ## 2.6 Choosing an activation |
| A useful default: use ReLU or GELU in the hidden layers, and choose the output activation from the task. The diagram and table below summarize the decision. | |
| - |  |
| + |  |
| *Choosing an activation: ReLU or GELU for hidden layers, and an output activation matched to the task.* | |
| @@ 97,7 117,7 @@ | |
| *Remark:* sigmoid and tanh are now rarely used as hidden activations in deep feed-forward networks precisely because of the saturation in Section 3.2.3. They survive at the output (sigmoid) and inside gated recurrent units, where their bounded range is the point. | |
| - | *With the per-layer nonlinearities fixed, the next lesson pairs the output activation with a matching loss so the network has something to minimize.* |
| + | *With the per-layer nonlinearities fixed, the next lesson turns the gradients into updates: the optimizers that make deep training practical.* |
| --- | |
| - | Next: [Loss functions and output layers](/en/Deep%20Learning/04%20Loss%20functions%20and%20output%20layers) · [Course overview](/en/Deep%20Learning) |
| + | Next: [Optimization](/en/Deep%20Learning/03%20Optimization) · [Course overview](/en/Deep%20Learning) |
| en/Deep Learning/03 Activation functions/activation-choice.svg .. en/Deep Learning/02 Activation functions/activation-choice.svg | |
| en/Deep Learning/03 Activation functions/activation-derivatives.png .. en/Deep Learning/02 Activation functions/activation-derivatives.png | |
| en/Deep Learning/03 Activation functions/activation-functions.png .. en/Deep Learning/02 Activation functions/activation-functions.png | |
| en/Deep Learning/02 Multilayer perceptron.md .. /dev/null | |
| @@ 1,100 0,0 @@ | |
| - | # 2. Multilayer perceptron |
| - | |
| - | A perceptron is one unit, $a = g(w^T x + b)$. Stack many units that read the same input and you get a layer, stack layers and you get a multilayer perceptron (MLP). This module builds the MLP from units, writes forward propagation for one example and for a mini-batch, tracks the shapes and the parameter count, and states the universal approximation theorem. |
| - | |
| - | **Objectives** |
| - | - Build a layer as a stack of perceptron-like units reading a shared input. |
| - | - Write forward propagation for one example with explicit per-layer weights and bias. |
| - | - Vectorize the forward pass over a mini-batch with broadcast bias. |
| - | - Track the shape of every $W^{[l]}$ and $b^{[l]}$ and count the parameters. |
| - | - State the universal approximation theorem and contrast width against depth. |
| - | |
| - | ## 2.1 From a unit to a layer |
| - | |
| - | ### 2.1.1 A single unit |
| - | |
| - | A unit takes an input vector $x \in \mathbb{R}^{n_0}$, forms a weighted sum with a weight vector $w$ and a bias scalar $b$, then applies a nonlinear activation $g$: |
| - | |
| - | $$\boxed{ a = g\left(w^T x + b\right) }$$ |
| - | |
| - | This is the perceptron of the previous course, except the hard step is now a smooth activation such as the sigmoid or ReLU. The activation is named here and defined fully in the [next lesson](/en/Deep%20Learning/03%20Activation%20functions). |
| - | |
| - | ### 2.1.2 A layer of units |
| - | |
| - | Now place $n_1$ units side by side, all reading the same input $x$. Unit $i$ has its own weight vector $w_i$ and bias $b_i$, producing $a_i = g(w_i^T x + b_i)$. Collect the weight vectors as the rows of a matrix $W^{[1]}$ and the biases into a vector $b^{[1]}$: |
| - | |
| - | $$\boxed{ W^{[1]} = \begin{bmatrix} w_1^{T} \\ \vdots \\ w_{n_1}^{T} \end{bmatrix}, \quad b^{[1]} = \begin{bmatrix} b_1 \\ \vdots \\ b_{n_1} \end{bmatrix} }$$ |
| - | |
| - | The whole layer then computes a pre-activation vector and an activation vector in one matrix expression, $z^{[1]} = W^{[1]} x + b^{[1]}$ and $a^{[1]} = g^{[1]}(z^{[1]})$, where $g^{[1]}$ is applied elementwise. |
| - | |
| - | *Remark:* the rows of $W^{[1]}$ are exactly the individual unit weight vectors, so a layer is just many units packed into one matrix. Bias stays explicit here: unlike the Machine Learning course, which folded the intercept into $w$ via the augmented input $x_0 = 1$, this course keeps $b^{[l]}$ as its own vector. |
| - | |
| - | ## 2.2 Forward propagation |
| - | |
| - | Stacking $L$ such layers gives the MLP. Layer $l$ reads the activation of the layer below, $a^{[l-1]}$, and produces $a^{[l]}$. The input is $a^{[0]} = x$ and the prediction is the output of the last layer. |
| - | |
| - | ### 2.2.1 One example |
| - | |
| - | For $l = 1, \dots, L$: |
| - | |
| - | $$\boxed{ a^{[0]} = x, \quad z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}, \quad a^{[l]} = g^{[l]}\!\left(z^{[l]}\right), \quad \hat{y} = a^{[L]} }$$ |
| - | |
| - | Each layer may use its own activation $g^{[l]}$: hidden layers typically use ReLU, while the output layer uses sigmoid or softmax for classification and the identity for regression. |
| - | |
| - | *Remark:* the composition $\hat{y} = g^{[L]}(W^{[L]} g^{[L-1]}(\cdots g^{[1]}(W^{[1]} x + b^{[1]}) \cdots) + b^{[L]})$ is what makes the network expressive. Without the nonlinear $g^{[l]}$ the whole stack would collapse to a single linear map $W x + b$. |
| - | |
| - | ### 2.2.2 Vectorized over a mini-batch |
| - | |
| - | Training runs on batches, not single examples. Place $m$ examples as the columns of a matrix, so $A^{[0]} = X \in \mathbb{R}^{n_0 \times m}$, and the forward pass becomes a matrix product with the bias broadcast across all columns: |
| - | |
| - | $$\boxed{ Z^{[l]} = W^{[l]} A^{[l-1]} + b^{[l]}, \quad A^{[l]} = g^{[l]}\!\left(Z^{[l]}\right) }$$ |
| - | |
| - | Here $Z^{[l]}$ and $A^{[l]}$ have shape $n_l \times m$, one column per example. The bias $b^{[l]} \in \mathbb{R}^{n_l}$ is added to every column, an operation known as broadcasting. |
| - | |
| - | *Remark:* the only change from the single-example form is that the vector $a^{[l-1]}$ becomes the matrix $A^{[l-1]}$. Processing a batch as one matrix multiply is what lets a GPU run the pass efficiently. |
| - | |
| - | ## 2.3 Shapes and parameter count |
| - | |
| - | The shapes follow from one rule: to compute $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$, the matrix $W^{[l]}$ must map an $n_{l-1}$-vector to an $n_l$-vector. |
| - | |
| - | $$\boxed{ W^{[l]} \in \mathbb{R}^{n_l \times n_{l-1}}, \quad b^{[l]} \in \mathbb{R}^{n_l} }$$ |
| - | |
| - | Layer $l$ therefore holds $n_l \, n_{l-1}$ weights plus $n_l$ biases. Consider a small network with $n_0 = 4$ inputs, two hidden layers of $5$ and $3$ units, and a single output unit. |
| - | |
| - | | Layer $l$ | $W^{[l]}$ shape | $b^{[l]}$ shape | Parameters | |
| - | | --- | --- | --- | --- | |
| - | | 1 | $5 \times 4$ | $5$ | $25$ | |
| - | | 2 | $3 \times 5$ | $3$ | $18$ | |
| - | | 3 | $1 \times 3$ | $1$ | $4$ | |
| - | | Total | | | $47$ | |
| - | |
| - | *Remark:* the input layer holds no parameters, it is just the data $a^{[0]} = x$. When counting layers we count the layers that carry weights, so this network has $L = 3$. |
| - | |
| - | ## 2.4 A multi-layer network |
| - | |
| - | The diagram below shows the same $4$-$5$-$3$-$1$ network as a flow of activations. Each arrow group is a full weight matrix, and each box applies its activation to the pre-activation. |
| - | |
| - |  |
| - | |
| - | *A multilayer perceptron: each layer computes z = W a + b then a = g(z), composing input a0 into the prediction aL.* |
| - | |
| - | Information flows strictly left to right during the forward pass, which is why this is a feedforward network. Nothing loops back. The reverse direction, used to compute gradients, is the subject of a later lesson. |
| - | |
| - | ## 2.5 Universal approximation |
| - | |
| - | How expressive is an MLP? The universal approximation theorem gives a strong answer. Let $f$ be any continuous function on a compact set $K \subset \mathbb{R}^{n_0}$, and let $\varepsilon > 0$. Then there exists a network with a single hidden layer of finite width, using a suitable nonlinear activation, whose output $F$ satisfies: |
| - | |
| - | $$\boxed{ \sup_{x \in K} \left| F(x) - f(x) \right| < \varepsilon }$$ |
| - | |
| - | In words, one hidden layer with enough units can approximate any continuous function on a bounded region to any desired accuracy $\varepsilon$. This is an existence result, not a recipe: it promises that such weights exist, but says nothing about how many units are needed or how to find them. |
| - | |
| - | *Remark:* the catch is width. Matching a target to accuracy $\varepsilon$ with one hidden layer can demand an enormous number of units, growing fast as $\varepsilon$ shrinks. Depth is usually far more parameter-efficient: stacking several narrow layers can represent functions that a single layer would need exponentially many units to match. This efficiency of depth over width is the practical reason the field is called deep learning. |
| - | |
| - |  |
| - | |
| - | *A network with one hidden layer approximates a target function by summing many simple activated units.* |
| - | |
| - | *The network is only defined once the activations $g^{[l]}$ are fixed. The next lesson defines them, sigmoid, tanh, ReLU and its variants, and explains how each shapes learning.* |
| - | |
| - | --- |
| - | Next: [Activation functions](/en/Deep%20Learning/03%20Activation%20functions) · [Course overview](/en/Deep%20Learning) |
| en/Deep Learning/02 Multilayer perceptron/mlp-forward.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 760 392" width="760" height="392" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="760" height="392" fill="#ffffff"/><text x="380.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">Forward propagation through a 4-5-3-1 network</text><line x1="136.0" y1="125.0" x2="274.0" y2="100.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="125.0" x2="274.0" y2="150.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="125.0" x2="274.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="125.0" x2="274.0" y2="250.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="125.0" x2="274.0" y2="300.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="175.0" x2="274.0" y2="100.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="175.0" x2="274.0" y2="150.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="175.0" x2="274.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="175.0" x2="274.0" y2="250.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="175.0" x2="274.0" y2="300.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="225.0" x2="274.0" y2="100.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="225.0" x2="274.0" y2="150.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="225.0" x2="274.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="225.0" x2="274.0" y2="250.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="225.0" x2="274.0" y2="300.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="275.0" x2="274.0" y2="100.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="275.0" x2="274.0" y2="150.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="275.0" x2="274.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="275.0" x2="274.0" y2="250.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="275.0" x2="274.0" y2="300.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="100.0" x2="444.0" y2="150.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="100.0" x2="444.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="100.0" x2="444.0" y2="250.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="150.0" x2="444.0" y2="150.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="150.0" x2="444.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="150.0" x2="444.0" y2="250.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="200.0" x2="444.0" y2="150.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="200.0" x2="444.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="200.0" x2="444.0" y2="250.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="250.0" x2="444.0" y2="150.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="250.0" x2="444.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="250.0" x2="444.0" y2="250.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="300.0" x2="444.0" y2="150.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="300.0" x2="444.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="300.0" x2="444.0" y2="250.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="476.0" y1="150.0" x2="614.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="476.0" y1="200.0" x2="614.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="476.0" y1="250.0" x2="614.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><circle cx="120.0" cy="125.0" r="16.0" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><circle cx="120.0" cy="175.0" r="16.0" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><circle cx="120.0" cy="225.0" r="16.0" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><circle cx="120.0" cy="275.0" r="16.0" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><circle cx="290.0" cy="100.0" r="16.0" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><circle cx="290.0" cy="150.0" r="16.0" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><circle cx="290.0" cy="200.0" r="16.0" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><circle cx="290.0" cy="250.0" r="16.0" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><circle cx="290.0" cy="300.0" r="16.0" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><circle cx="460.0" cy="150.0" r="16.0" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><circle cx="460.0" cy="200.0" r="16.0" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><circle cx="460.0" cy="250.0" r="16.0" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><circle cx="630.0" cy="200.0" r="16.0" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="120.0" y="355.0" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#5b6b7b" text-anchor="middle">a<tspan baseline-shift="sub" font-size="9px">0</tspan> (input)</text><text x="290.0" y="355.0" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#5b6b7b" text-anchor="middle">a<tspan baseline-shift="sub" font-size="9px">1</tspan></text><text x="460.0" y="355.0" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#5b6b7b" text-anchor="middle">a<tspan baseline-shift="sub" font-size="9px">2</tspan></text><text x="630.0" y="355.0" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#5b6b7b" text-anchor="middle">a<tspan baseline-shift="sub" font-size="9px">3</tspan> = ŷ</text><text x="205.0" y="90.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">W a + b</text><text x="380.0" y="372.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">each box computes z = W a + b then a = g(z), information flows left to right</text></svg> |
| \ | No newline at end of file |
| en/Deep Learning/02 Multilayer perceptron/universal-approximation.png .. /dev/null | |
| en/Deep Learning/06 Optimization.md .. en/Deep Learning/03 Optimization.md | |
| @@ 1,16 1,8 @@ | |
| - | # 6. Optimization |
| + | # 3. Optimization |
| Backpropagation returns the gradient of the cost with respect to every parameter. An optimizer is the rule that turns those gradients into updates. This module covers the gradient-descent variants and the adaptive optimizers (momentum, RMSProp, Adam) that make deep networks trainable, plus the learning-rate schedules that shape the run. | |
| - | **Objectives** |
| - | - Reuse the gradient-descent update from the Machine Learning course and name its batch, mini-batch, and stochastic variants. |
| - | - Add momentum to damp oscillations and accelerate along consistent directions. |
| - | - Rescale each coordinate by its recent gradient magnitude with RMSProp. |
| - | - Combine both ideas into Adam and understand its bias correction. |
| - | - Pick a learning-rate schedule: step decay, cosine, or warmup. |
| - | - Compare the optimizers and know when to reach for each. |
| - | |
| - | ## 6.1 Gradient descent |
| + | ## 3.1 Gradient descent |
| Let $w$ collect all parameters (every $W^{[l]}$ and $b^{[l]}$) and let $J(w)$ be the cost, the average of the per-example loss $L$. Write $g = \nabla_w J(w)$ for the gradient of the cost with respect to the parameters, as returned by backpropagation. The base update moves $w$ downhill: | |
| @@ 20,7 12,7 @@ | |
| *Remark:* bias is explicit here. The gradient $g$ has one block per $W^{[l]}$ and one per $b^{[l]}$, and the update applies to each block with the same $\alpha$. | |
| - | ### 6.1.1 Batch, mini-batch, stochastic |
| + | ### 3.1.1 Batch, mini-batch, stochastic |
| The variants differ only in how many examples enter the gradient $g$ at each step. | |
| @@ 32,7 24,7 @@ | |
| *Remark:* one pass over the whole dataset is an epoch. Mini-batch is the standard choice: batches of $32$ to $512$ fit the accelerator, exploit vectorized matrix products, and the residual noise in $g$ helps escape shallow local minima. In deep learning "SGD" is used loosely to mean mini-batch gradient descent. | |
| - | ## 6.2 Momentum |
| + | ## 3.2 Momentum |
| Plain SGD zig-zags across narrow valleys because the gradient points across the valley more than along it. Momentum accumulates an exponentially weighted average of past gradients in a velocity vector $v$, then steps in that averaged direction: | |
| @@ 40,7 32,11 @@ | |
| with momentum coefficient $\beta \in [0, 1)$, typically $\beta = 0.9$. Components of $g$ that keep the same sign reinforce each other, so $v$ grows and the step accelerates along consistent directions. Components that flip sign cancel in the average, so oscillations across the valley are damped. | |
| - | ### 6.2.1 Nesterov momentum |
| + |  |
| + | |
| + | *The curves are level lines of the loss, the dot is the minimum. The gradient is perpendicular to the level line it sits on, so in a ravine it points mostly across the valley, and plain gradient descent bounces. Momentum keeps a memory of the previous steps, the bounces cancel and the valley direction accumulates.* |
| + | |
| + | ### 3.2.1 Nesterov momentum |
| Nesterov accelerated gradient evaluates the gradient at a look-ahead point, after the momentum step has been provisionally applied, rather than at the current $w$. This anticipatory correction reacts sooner when the slope changes: | |
| @@ 48,7 44,7 @@ | |
| *Remark:* think of $\beta \approx 0.9$ as averaging over roughly the last $\tfrac{1}{1 - \beta} = 10$ gradients. Nesterov usually converges slightly faster than plain momentum for the same $\alpha$ and $\beta$. | |
| - | ## 6.3 RMSProp |
| + | ## 3.3 RMSProp |
| Different parameters can need very different step sizes, and one global $\alpha$ cannot serve them all. RMSProp keeps a per-coordinate running average $s$ of squared gradients, then divides the step by $\sqrt{s}$, so coordinates with large recent gradients take smaller steps and quiet coordinates take larger ones: | |
| @@ 58,7 54,7 @@ | |
| *Remark:* $s$ estimates the uncentered second moment of each coordinate of $g$, so $\sqrt{s}$ is roughly its recent root-mean-square magnitude. RMSProp suits non-stationary objectives, which is exactly what a moving mini-batch gradient is. | |
| - | ## 6.4 Adam |
| + | ## 3.4 Adam |
| Adam (adaptive moment estimation) combines momentum and RMSProp: it keeps a first-moment estimate $m$ (the mean of the gradient) and a second-moment estimate $v$ (the mean of the squared gradient). | |
| @@ 76,27 72,27 @@ | |
| *Remark:* the bias correction matters most in the first few dozen steps, when $t$ is small and $\beta_2^t$ is still close to $1$. Without it, $\hat v$ would be far too small and the early steps far too large. AdamW, a common variant, decouples weight decay from this update. | |
| - |  |
| + |  |
| *Adam combines the momentum of averaged gradients with the per-parameter scaling of RMSProp.* | |
| - | ## 6.5 Learning-rate schedules |
| + | ## 3.5 Learning-rate schedules |
| The learning rate $\alpha$ is the single most important hyperparameter, and holding it fixed is rarely optimal. A large $\alpha$ speeds early progress but prevents settling into a minimum, so schedules typically decrease $\alpha$ over training. Here $\alpha_0$ is the initial rate and $t$ indexes the step or epoch. | |
| - | ### 6.5.1 Step decay |
| + | ### 3.5.1 Step decay |
| Multiply $\alpha$ by a factor $\gamma \in (0, 1)$ every $s$ epochs, so it drops in discrete stages: | |
| $$\boxed{ \alpha_t = \alpha_0\, \gamma^{\lfloor t / s \rfloor} }$$ | |
| - | ### 6.5.2 Cosine decay |
| + | ### 3.5.2 Cosine decay |
| Anneal $\alpha$ smoothly from $\alpha_0$ toward a floor of zero along a half cosine over $T$ total steps: | |
| $$\boxed{ \alpha_t = \tfrac{1}{2}\,\alpha_0\left(1 + \cos\frac{\pi t}{T}\right) }$$ | |
| - | ### 6.5.3 Warmup |
| + | ### 3.5.3 Warmup |
| Warmup ramps $\alpha$ up linearly from a small value over the first few hundred to few thousand steps, then hands off to a decay schedule. It prevents the large, poorly conditioned updates a cold start with a big $\alpha$ would produce, and it is standard for deep networks such as transformers. | |
| @@ 106,13 102,13 @@ | |
| | Cosine | smooth anneal to zero | modern default, often with warmup | | |
| | Warmup | linear ramp up, then decay | stabilize early steps, large models | | |
| - |  |
| + |  |
| *Common learning-rate schedules: step decay, cosine decay, and a warmup followed by decay.* | |
| *Remark:* warmup and a decay are usually composed, warmup for the first phase and cosine or step decay afterward. | |
| - | ## 6.6 Choosing an optimizer |
| + | ## 3.6 Choosing an optimizer |
| | optimizer | what it adds | tracks | typical use | | |
| | --- | --- | --- | --- | | |
| @@ 123,11 119,21 @@ | |
| *Remark:* Adam is the safe default and converges fast with little tuning. Well-tuned SGD with momentum and a schedule often reaches slightly better final test accuracy on large vision models, which is why both remain in wide use. | |
| - |  |
| + |  |
| *On an elongated loss surface, momentum and Adam reach the minimum far faster than plain gradient descent.* | |
| + | ## 3.7 Good practices |
| + | |
| + | Two of the habits that make training behave, careful initialization and dropout, live in the next lesson ([Training deep networks](/en/Deep%20Learning/04%20Training%20deep%20networks)). Two more belong right here. |
| + | |
| + | **Center and normalize the inputs.** Standardize each feature (subtract its mean, divide by its standard deviation), so no feature dominates the first dot products and the first layer's gradients start well scaled. |
| + | |
| + | **Sanity-check before training long.** A freshly initialized $K$-class classifier should start near the loss $\ln K$ (about $2.3$ for $K = 10$). A tiny training set should be easy to overfit: if the network cannot, the code is broken. Watch the training and validation curves. And since backpropagation is error-prone, check its analytic gradient against a numerical finite-difference estimate: |
| + | |
| + | $$\boxed{ \frac{\partial L}{\partial w} \approx \frac{L(w + \varepsilon) - L(w - \varepsilon)}{2\varepsilon} }$$ |
| + | |
| *Every optimizer here scales the raw gradient, so its behaviour depends on how large those gradients are to begin with. The next part studies how the initial weights and the network depth set that scale, and how poor choices make gradients vanish or explode.* | |
| --- | |
| - | Next: [Initialization and vanishing gradients](/en/Deep%20Learning/07%20Initialization%20and%20vanishing%20gradients) · [Course overview](/en/Deep%20Learning) |
| + | Next: [Training deep networks](/en/Deep%20Learning/04%20Training%20deep%20networks) · [Course overview](/en/Deep%20Learning) |
| en/Deep Learning/06 Optimization/lr-schedules.png .. en/Deep Learning/03 Optimization/lr-schedules.png | |
| en/Deep Learning/06 Optimization/optimizer-family.svg .. en/Deep Learning/03 Optimization/optimizer-family.svg | |
| en/Deep Learning/06 Optimization/optimizer-paths.png .. en/Deep Learning/03 Optimization/optimizer-paths.png | |
| /dev/null .. en/Deep Learning/03 Optimization/ravine-momentum.svg | |
| @@ 0,0 1,51 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 370" width="900" height="370" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs> |
| + | <marker id="arrowred" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#d1495b"/></marker> |
| + | <marker id="arrowblue" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#3b6fb6"/></marker> |
| + | </defs> |
| + | <rect width="900" height="370" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">A ravine in the loss: steep across, shallow along</text> |
| + | |
| + | <!-- Panel 1: plain gradient descent --> |
| + | <text x="225" y="56" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">plain gradient descent</text> |
| + | <ellipse cx="225" cy="195" rx="170" ry="70" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <ellipse cx="225" cy="195" rx="135" ry="55" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <ellipse cx="225" cy="195" rx="100" ry="40" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <ellipse cx="225" cy="195" rx="65" ry="26" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <ellipse cx="225" cy="195" rx="30" ry="12" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <circle cx="225" cy="195" r="3.5" fill="#1f2933"/> |
| + | <polyline points="70,168 86,236 101,163 116,231 130,167 143,226 156,171 168,221 179,175 190,217 200,179 209,212 217,183 224,208 230,187 235,204 239,192" fill="none" stroke="#d1495b" stroke-width="1.6" marker-end="url(#arrowred)"/> |
| + | <circle cx="70" cy="168" r="2.5" fill="#d1495b"/> |
| + | <circle cx="86" cy="236" r="1.8" fill="#d1495b"/> |
| + | <circle cx="101" cy="163" r="1.8" fill="#d1495b"/> |
| + | <circle cx="116" cy="231" r="1.8" fill="#d1495b"/> |
| + | <circle cx="130" cy="167" r="1.8" fill="#d1495b"/> |
| + | <circle cx="143" cy="226" r="1.8" fill="#d1495b"/> |
| + | <circle cx="156" cy="171" r="1.8" fill="#d1495b"/> |
| + | <circle cx="168" cy="221" r="1.8" fill="#d1495b"/> |
| + | <circle cx="179" cy="175" r="1.8" fill="#d1495b"/> |
| + | <circle cx="190" cy="217" r="1.8" fill="#d1495b"/> |
| + | <circle cx="200" cy="179" r="1.8" fill="#d1495b"/> |
| + | <circle cx="209" cy="212" r="1.8" fill="#d1495b"/> |
| + | <circle cx="217" cy="183" r="1.8" fill="#d1495b"/> |
| + | <circle cx="224" cy="208" r="1.8" fill="#d1495b"/> |
| + | <circle cx="230" cy="187" r="1.8" fill="#d1495b"/> |
| + | <circle cx="235" cy="204" r="1.8" fill="#d1495b"/> |
| + | <text x="62" y="156" font-size="10" fill="#5b6b7b" text-anchor="middle">start</text> |
| + | <text x="225" y="328" font-size="11" fill="#5b6b7b" text-anchor="middle">the gradient points mostly across the valley:</text> |
| + | <text x="225" y="344" font-size="11" fill="#5b6b7b" text-anchor="middle">the step bounces between the walls, crawls toward the minimum (dot)</text> |
| + | |
| + | <!-- Panel 2: momentum --> |
| + | <text x="675" y="56" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">with momentum</text> |
| + | <ellipse cx="675" cy="195" rx="170" ry="70" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <ellipse cx="675" cy="195" rx="135" ry="55" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <ellipse cx="675" cy="195" rx="100" ry="40" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <ellipse cx="675" cy="195" rx="65" ry="26" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <ellipse cx="675" cy="195" rx="30" ry="12" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <circle cx="675" cy="195" r="3.5" fill="#1f2933"/> |
| + | <path d="M520,168 C545,210 566,229 592,227 C620,225 646,211 668,203 C688,196 702,193 709,194 C707,198 698,198 690,197" fill="none" stroke="#3b6fb6" stroke-width="1.8" marker-end="url(#arrowblue)"/> |
| + | <circle cx="520" cy="168" r="2.5" fill="#3b6fb6"/> |
| + | <text x="512" y="156" font-size="10" fill="#5b6b7b" text-anchor="middle">start</text> |
| + | <text x="675" y="328" font-size="11" fill="#5b6b7b" text-anchor="middle">the crossings cancel while the along-the-valley component builds speed:</text> |
| + | <text x="675" y="344" font-size="11" fill="#5b6b7b" text-anchor="middle">the path glides to the minimum, with one small overshoot</text> |
| + | </svg> |
| en/Deep Learning/04 Loss functions and output layers.md .. /dev/null | |
| @@ 1,92 0,0 @@ | |
| - | # 4. Loss functions and output layers |
| - | |
| - | Before a network can learn it needs a target to descend toward. The output layer turns the last activation $a^{[L]}$ into a prediction, and the loss measures how far that prediction is from the true label. This module fixes both choices per task, because backpropagation in the next module differentiates a concrete loss. The output activation and the loss are not picked independently: matching them to the task shape is what makes the training signal clean. |
| - | |
| - | **Objectives** |
| - | - Go from a per-example loss $L$ to the cost $J$ averaged over the batch. |
| - | - Choose a linear output with mean squared error for regression. |
| - | - Choose a sigmoid output with binary cross-entropy for two-class problems. |
| - | - Choose a softmax output with categorical cross-entropy for multiclass problems. |
| - | - Derive the clean logit gradient of the softmax and cross-entropy pair. |
| - | - Map any task to its output activation and loss with a single lookup table. |
| - | |
| - | ## 4.1 From per-example loss to cost |
| - | |
| - | The network predicts $\hat{y} = a^{[L]}$ from input $a^{[0]} = x$. For a single example the loss $L(\hat{y}, y)$ scores that prediction against the target $y$. Training minimizes the cost $J$, defined as the average of $L$ over the $m$ examples in the batch or dataset: |
| - | |
| - | $$\boxed{ J = \frac{1}{m}\sum_{i=1}^{m} L\!\left(\hat{y}^{(i)}, y^{(i)}\right) }$$ |
| - | |
| - | *Remark:* the loss $L$ scores one prediction, the cost $J$ is what the optimizer actually reduces. Averaging (rather than summing) keeps the gradient scale independent of the batch size, so the learning rate does not have to be retuned when $m$ changes. |
| - | |
| - | The three tasks below reuse the losses introduced in the Machine Learning course. The cross-entropy row of the loss table at [General concepts](/en/Machine%20Learning/02%20General%20concepts), labelled "Neural networks", is exactly the objective a classification network minimizes. The novelty here is pairing each loss with the output activation $g^{[L]}$ that produces $\hat{y}$. |
| - | |
| - | ## 4.2 Regression: linear output and mean squared error |
| - | |
| - | For a continuous target $y \in \mathbb{R}^{n_L}$ the output layer uses no activation, so it is linear (the identity) and the prediction can take any real value: |
| - | |
| - | $$\boxed{ \hat{y} = a^{[L]} = z^{[L]} = W^{[L]} a^{[L-1]} + b^{[L]} }$$ |
| - | |
| - | The per-example loss is the squared Euclidean distance between prediction and target, scaled by one half: |
| - | |
| - | $$\boxed{ L(\hat{y}, y) = \tfrac{1}{2}\,\lVert \hat{y} - y \rVert^2 }$$ |
| - | |
| - | *Remark:* the factor $\tfrac{1}{2}$ cancels the $2$ that appears on differentiating the square, leaving the tidy residual gradient $\partial L / \partial \hat{y} = \hat{y} - y$. This is the same mean-squared-error objective used for linear regression, now sitting on top of a deep network instead of a single linear score. |
| - | |
| - | ## 4.3 Binary classification: sigmoid output and binary cross-entropy |
| - | |
| - | For a two-class label $y \in \{0, 1\}$ the output layer has a single unit whose activation is the sigmoid, squashing the logit $z^{[L]}$ into a probability: |
| - | |
| - | $$\hat{y} = a^{[L]} = \sigma\!\left(z^{[L]}\right) = \frac{1}{1 + e^{-z^{[L]}}} \in (0, 1)$$ |
| - | |
| - | Here $\hat{y}$ is read as $p(y = 1 \mid x)$. The matching loss is the binary cross-entropy, the negative log-likelihood of the Bernoulli label: |
| - | |
| - | $$\boxed{ L(\hat{y}, y) = -\Big[\, y \log \hat{y} + (1 - y)\log(1 - \hat{y}) \,\Big] }$$ |
| - | |
| - | *Remark:* only one of the two terms is active for any given label. When $y = 1$ the loss is $-\log \hat{y}$, penalizing a small predicted probability, and when $y = 0$ it is $-\log(1 - \hat{y})$. Cross-entropy is preferred over squared error here because it keeps the gradient large when the prediction is confidently wrong, so learning does not stall. |
| - | |
| - |  |
| - | |
| - | *Cross-entropy loss grows without bound as the predicted probability moves away from the true label.* |
| - | |
| - | ## 4.4 Multiclass classification: softmax output and categorical cross-entropy |
| - | |
| - | For a $K$-class label the output layer has $K$ units and the softmax activation turns the logit vector $z^{[L]} \in \mathbb{R}^{K}$ into a probability distribution over the classes: |
| - | |
| - | $$\boxed{ \hat{y}_k = \frac{e^{z^{[L]}_k}}{\sum_{j=1}^{K} e^{z^{[L]}_j}} }$$ |
| - | |
| - | The outputs are positive and sum to one, so $\hat{y}$ is a proper distribution and $\hat{y}_k = p(y = k \mid x)$. The target $y$ is one-hot: $y_k = 1$ for the true class and $0$ otherwise. The matching loss is the categorical cross-entropy: |
| - | |
| - | $$\boxed{ L(\hat{y}, y) = -\sum_{k=1}^{K} y_k \log \hat{y}_k }$$ |
| - | |
| - | *Remark:* because $y$ is one-hot the sum collapses to a single term, $-\log \hat{y}_{k^\star}$, where $k^\star$ is the true class. The loss therefore rewards putting probability mass on the correct class and ignores how the remaining mass is split. Binary cross-entropy is the special case $K = 2$. |
| - | |
| - | ## 4.5 The softmax and cross-entropy gradient |
| - | |
| - | The softmax output and the categorical cross-entropy loss are used together because their composition has a remarkably clean derivative at the logits $z^{[L]}$. Differentiating $L$ with respect to a single logit $z^{[L]}_k$ gives: |
| - | |
| - | $$\boxed{ \frac{\partial L}{\partial z^{[L]}_k} = \hat{y}_k - y_k }$$ |
| - | |
| - | The gradient at the output layer is just the prediction minus the target, a plain residual with no awkward sigmoid or softmax factor left over. The same identity holds for the sigmoid and binary cross-entropy pair, which is its $K = 2$ instance. This is precisely why each activation is coupled to its matching loss rather than mixed with, say, squared error. |
| - | |
| - | *Remark:* the elementwise form $\partial L / \partial z^{[L]} = \hat{y} - y$ is what seeds backpropagation. The next module starts the backward pass from this vector and then repeatedly applies the chain rule and the Hadamard product $\odot$ to push it back through the hidden layers. |
| - | |
| - | ## 4.6 Task to output to loss |
| - | |
| - | The three cases collapse into one lookup. Fix the task, and the output activation and loss follow. |
| - | |
| - | | Task | Output activation $g^{[L]}$ | Per-example loss $L$ | Logit gradient $\partial L / \partial z^{[L]}$ | |
| - | | --- | --- | --- | --- | |
| - | | Regression | linear (identity) | mean squared error | $\hat{y} - y$ | |
| - | | Binary classification | sigmoid | binary cross-entropy | $\hat{y} - y$ | |
| - | | Multiclass classification | softmax | categorical cross-entropy | $\hat{y} - y$ | |
| - | |
| - | *Remark:* the last column is identical across all three rows. Matching the output activation to its natural loss makes the network start its backward pass from the same simple residual regardless of the task. |
| - | |
| - |  |
| - | |
| - | *The output activation and loss are chosen together per task, and the matched pairs share the clean logit gradient yhat minus y.* |
| - | |
| - | *With a concrete loss chosen and its output-layer gradient in hand, the next module runs the chain rule backward through every layer: backpropagation.* |
| - | |
| - | --- |
| - | Next: [Backpropagation](/en/Deep%20Learning/05%20Backpropagation) · [Course overview](/en/Deep%20Learning) |
| en/Deep Learning/04 Loss functions and output layers/loss-curves.png .. /dev/null | |
| en/Deep Learning/04 Loss functions and output layers/output-loss-map.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 780 386" width="780" height="386" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="780" height="386" fill="#ffffff"/><text x="390.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">Task, output activation, and loss are matched per task</text><text x="135.0" y="58.0" font-family="Helvetica, Arial, sans-serif" font-size="12" font-weight="600" fill="#5b6b7b" text-anchor="middle">task</text><text x="325.0" y="58.0" font-family="Helvetica, Arial, sans-serif" font-size="12" font-weight="600" fill="#5b6b7b" text-anchor="middle">output activation</text><text x="545.0" y="58.0" font-family="Helvetica, Arial, sans-serif" font-size="12" font-weight="600" fill="#5b6b7b" text-anchor="middle">loss</text><rect x="640.0" y="156.0" width="118.0" height="68.0" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="699.0" y="179.1" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">gradient at</text><text x="699.0" y="194.4" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">logits = ŷ</text><text x="699.0" y="209.8" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">- y</text><rect x="60.0" y="72.0" width="150.0" height="46.0" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="135.0" y="99.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">regression</text><rect x="250.0" y="72.0" width="150.0" height="46.0" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="325.0" y="99.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">linear output</text><rect x="470.0" y="72.0" width="150.0" height="46.0" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="545.0" y="99.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">mean squared error</text><line x1="210.0" y1="95.0" x2="250.0" y2="95.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="400.0" y1="95.0" x2="470.0" y2="95.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="620.0" y1="95.0" x2="640.0" y2="190.0" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><rect x="60.0" y="167.0" width="150.0" height="46.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="135.0" y="194.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">binary</text><rect x="250.0" y="167.0" width="150.0" height="46.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="325.0" y="194.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">sigmoid output</text><rect x="470.0" y="167.0" width="150.0" height="46.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="545.0" y="186.8" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">binary</text><text x="545.0" y="202.1" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">cross-entropy</text><line x1="210.0" y1="190.0" x2="250.0" y2="190.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="400.0" y1="190.0" x2="470.0" y2="190.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="620.0" y1="190.0" x2="640.0" y2="190.0" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><rect x="60.0" y="262.0" width="150.0" height="46.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="135.0" y="289.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">multiclass</text><rect x="250.0" y="262.0" width="150.0" height="46.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="325.0" y="289.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">softmax output</text><rect x="470.0" y="262.0" width="150.0" height="46.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="545.0" y="281.8" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">categorical</text><text x="545.0" y="297.1" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">cross-entropy</text><line x1="210.0" y1="285.0" x2="250.0" y2="285.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="400.0" y1="285.0" x2="470.0" y2="285.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="620.0" y1="285.0" x2="640.0" y2="190.0" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><text x="390.0" y="366.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">the matched pairs all seed the backward pass from the same residual</text></svg> |
| \ | No newline at end of file |
| /dev/null .. en/Deep Learning/04 Training deep networks.md | |
| @@ 0,0 1,72 @@ | |
| + | # 4. Training deep networks |
| + | |
| + | Lesson 3 chose the optimizer. This lesson collects the three tools that make that optimization behave on a deep stack: start the weights at the right scale (initialization), keep the activations well-scaled while they move (normalization), and stop the network from memorizing the training set (regularization). Each deserves a chapter of its own. Here we keep what matters most in practice. |
| + | |
| + | ## 4.1 Initialization |
| + | |
| + | **Break the symmetry.** Setting $W^{[l]} = 0$, or any value that makes every unit of a layer identical, breaks learning: two units with the same weights and the same input compute the same activation, receive the same gradient, and stay identical forever. The layer behaves like a single unit no matter how wide it is. Weights therefore start random. Biases can start at zero: the random weights already differ, and a zero bias keeps each unit in the responsive region of its activation. |
| + | |
| + | **Pick the right scale.** For a unit $z = \sum_{j=1}^{n_{\text{in}}} W_j a_j$ with independent zero-mean weights and inputs, the variance is a sum of $n_{\text{in}}$ terms: |
| + | |
| + | $$\boxed{ \operatorname{Var}(z) = n_{\text{in}} \cdot \operatorname{Var}(W) \cdot \operatorname{Var}(a) }$$ |
| + | |
| + | If $n_{\text{in}} \operatorname{Var}(W)$ drifts below $1$ the signal shrinks layer after layer, and lesson 2 proved where that ends: a vanished gradient. Above $1$ it explodes instead. The fix is to hold $n_{\text{in}} \operatorname{Var}(W) \approx 1$, which pins the weight variance to the layer's size: |
| + | |
| + | $$\boxed{ \text{Xavier: } \operatorname{Var}(W^{[l]}) = \frac{2}{n_{\text{in}} + n_{\text{out}}}, \qquad \text{He: } \operatorname{Var}(W^{[l]}) = \frac{2}{n_{\text{in}}} }$$ |
| + | |
| + | Xavier (which balances the forward and backward passes) suits $\tanh$ and the sigmoid. He doubles the variance because ReLU zeroes half of its inputs on average, so it is the right target for the ReLU family. |
| + | |
| + |  |
| + | |
| + | *Gradient magnitude across depth: poorly scaled weights make it vanish or explode, while variance-preserving initialization keeps it near one.* |
| + | |
| + | **Clip what still explodes.** Initialization sets the scale once, at step zero. If gradients still blow up during training (frequent in recurrent networks, lesson 7), rescale the gradient so its norm never exceeds a threshold $\tau$, keeping its direction: |
| + | |
| + | $$\boxed{ g \leftarrow g \cdot \min\!\left(1, \frac{\tau}{\lVert g \rVert}\right) }$$ |
| + | |
| + | ## 4.2 Normalization |
| + | |
| + | Initialization only positions the network at step zero: as training moves the weights, the distribution of every layer's input drifts, and later layers keep chasing a moving target. A normalization layer fixes the statistics on the fly: standardize, then let the network learn its scale back. For a feature $x$ over a mini-batch of size $m$, batch normalization computes |
| + | |
| + | $$\boxed{ \mu_\mathcal{B} = \frac{1}{m}\sum_{i=1}^{m} x^{(i)}, \qquad \sigma_\mathcal{B}^2 = \frac{1}{m}\sum_{i=1}^{m}\left(x^{(i)} - \mu_\mathcal{B}\right)^2 }$$ |
| + | |
| + | $$\boxed{ \hat{x}^{(i)} = \frac{x^{(i)} - \mu_\mathcal{B}}{\sqrt{\sigma_\mathcal{B}^2 + \epsilon}}, \qquad y^{(i)} = \gamma\, \hat{x}^{(i)} + \beta }$$ |
| + | |
| + | The scale $\gamma$ and shift $\beta$ are learned like any weight, so normalization removes no capacity: the network can even learn to undo it. The payoff is a smoother loss surface, higher usable learning rates, and less sensitivity to initialization. Two practical consequences: at inference, where there is no batch, BatchNorm switches to running averages of $\mu$ and $\sigma^2$ accumulated during training (forgetting that switch is the classic BatchNorm bug), and the shift $\beta$ makes the layer bias $b^{[l]}$ redundant. |
| + | |
| + | Layer normalization keeps the same standardize-scale-shift recipe but averages over the features of a single example instead of over the batch. Its statistics no longer depend on the batch, so it behaves identically in training and inference and handles variable-length sequences, which makes it the choice for recurrent networks and Transformers (lesson 10). |
| + | |
| + | | Aspect | Batch normalization | Layer normalization | |
| + | | --- | --- | --- | |
| + | | Normalization axis | across the batch, per feature | across the features, per example | |
| + | | Train vs inference | batch statistics vs running statistics | identical in both | |
| + | | Typical use | CNNs and feedforward vision models | RNNs and Transformers | |
| + | |
| + |  |
| + | |
| + | *Batch normalization computes statistics down a feature column across the batch, layer normalization across the features of a single example.* |
| + | |
| + | ## 4.3 Regularization and dropout |
| + | |
| + | A deep network usually has more parameters than training examples, so it can memorize the training set, noise included. Regularization trades a little training accuracy for generalization, the bias-variance story of [General concepts](/en/Machine%20Learning/02%20General%20concepts). |
| + | |
| + | **Weight decay ($L_2$).** Add a penalty on the squared weights to the cost, with strength $\lambda$: |
| + | |
| + | $$\boxed{ J_{\text{reg}} = J + \frac{\lambda}{2}\sum_{l=1}^{L}\lVert W^{[l]} \rVert_F^2 \quad\Longrightarrow\quad W^{[l]} \leftarrow (1 - \alpha\lambda)\, W^{[l]} - \alpha\,\frac{\partial J}{\partial W^{[l]}} }$$ |
| + | |
| + | The factor $(1 - \alpha\lambda)$ shrinks every weight at every step, hence the name. Its $L_1$ cousin penalizes absolute values instead and drives many weights to exactly zero. Biases are left out of the penalty. |
| + | |
| + | **Dropout.** On each training pass, keep every unit with probability $p$ and zero it otherwise, then divide by $p$ so the expected signal is unchanged: |
| + | |
| + | $$\boxed{ \tilde{a}^{[l]} = \frac{m \odot a^{[l]}}{p}, \qquad m_i \sim \text{Bernoulli}(p) }$$ |
| + | |
| + | No unit can rely on its neighbours, so the representation spreads out. Each step trains one of $2^k$ thinned subnetworks that share their weights, and inference, with dropout off, approximates their average prediction for free (that is what the $1/p$ buys). Typical keep probabilities: around $0.8$ at the input, $0.5$ in hidden layers. |
| + | |
| + |  |
| + | |
| + | *Dropout trains a different thinned subnetwork on each step by randomly removing units, and averages them at inference.* |
| + | |
| + | *Initialized at the right scale, normalized in flight, and regularized against memorizing, the network is ready for architecture. The next lesson builds the convolutional network, whose weight sharing is itself a form of regularization.* |
| + | |
| + | --- |
| + | Next: [Convolutional networks](/en/Deep%20Learning/05%20Convolutional%20networks) · [Course overview](/en/Deep%20Learning) |
| en/Deep Learning/08 Normalization/batchnorm-vs-layernorm.svg .. en/Deep Learning/04 Training deep networks/batchnorm-vs-layernorm.svg | |
| en/Deep Learning/09 Regularization and dropout/dropout-network.svg .. en/Deep Learning/04 Training deep networks/dropout-network.svg | |
| en/Deep Learning/07 Initialization and vanishing gradients/gradient-flow.png .. en/Deep Learning/04 Training deep networks/gradient-flow.png | |
| en/Deep Learning/05 Backpropagation.md .. /dev/null | |
| @@ 1,100 0,0 @@ | |
| - | # 5. Backpropagation |
| - | |
| - | Backpropagation is the algorithm that computes the gradient of the cost with respect to every parameter of a network. It is nothing more than the chain rule applied in a careful, reverse order over the computational graph, reusing the quantities cached during the forward pass. This module derives it layer by layer using the error signal $\delta^{[l]} = \partial L / \partial z^{[l]}$. |
| - | |
| - | **Objectives** |
| - | - Read a network as a composition of functions and see why gradients flow backward by the chain rule. |
| - | - Define the layer error $\delta^{[l]}$ and compute the output-layer error $\delta^{[L]}$. |
| - | - Establish the backward recursion that carries $\delta$ from layer $L$ down to layer $1$. |
| - | - Turn each $\delta^{[l]}$ into the parameter gradients for $W^{[l]}$ and $b^{[l]}$. |
| - | - Assemble the full forward-and-backward algorithm and connect it to the parameter update. |
| - | |
| - | ## 5.1 The chain rule over a computational graph |
| - | |
| - | A feedforward network is a composition of functions. Each layer $l$ takes the previous activation $a^{[l-1]}$ and produces a pre-activation and an activation: |
| - | |
| - | $$\boxed{ z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}, \quad a^{[l]} = g^{[l]}\!\left(z^{[l]}\right) }$$ |
| - | |
| - | with $a^{[0]} = x$ and prediction $\hat{y} = a^{[L]}$. The scalar loss $L$ sits at the end of this chain. Because the cost is a composition, its derivative with respect to any intermediate quantity is a product of local derivatives, one per link in the graph. The chain rule tells us to accumulate those products. |
| - | |
| - | The efficient way to do this is to walk the graph backward. A single reverse traversal computes, for every node, the derivative of the final loss with respect to that node, and each step reuses the derivative already computed for the node just downstream. This reuse is what makes backpropagation cost about the same as one forward pass, rather than one pass per parameter. |
| - | |
| - |  |
| - | |
| - | *Backpropagation traverses the computational graph in reverse: the forward pass (solid) caches values, the backward pass (dashed) propagates the error delta.* |
| - | |
| - | *Remark:* the solid arrows are the forward pass (data flowing to the loss) and the dashed arrows are the backward pass (gradients flowing from the loss). The two passes traverse the same graph in opposite directions. |
| - | |
| - | ## 5.2 The layer error |
| - | |
| - | The central object is the error of layer $l$, the sensitivity of the loss to the pre-activation $z^{[l]}$: |
| - | |
| - | $$\boxed{ \delta^{[l]} = \frac{\partial L}{\partial z^{[l]}} \in \mathbb{R}^{n_l} }$$ |
| - | |
| - | Once we know $\delta^{[l]}$ at every layer, all parameter gradients follow immediately (Section 5.5). The whole algorithm reduces to computing these vectors, first at the output layer, then recursively backward. |
| - | |
| - | *Remark:* placing $\delta$ at the pre-activation $z^{[l]}$ rather than at the activation $a^{[l]}$ is a deliberate choice. It makes the activation derivative $g'^{[l]}$ appear exactly once per layer and keeps the recursion clean. |
| - | |
| - | ## 5.3 Output-layer error |
| - | |
| - | At the output layer the chain rule has two links: the loss depends on $a^{[L]} = \hat{y}$, and $a^{[L]}$ depends on $z^{[L]}$ through the activation $g^{[L]}$. Multiplying the two local derivatives elementwise gives the output error: |
| - | |
| - | $$\boxed{ \delta^{[L]} = \nabla_{a^{[L]}} L \;\odot\; g'^{[L]}\!\left(z^{[L]}\right) }$$ |
| - | |
| - | The Hadamard product $\odot$ appears because $g^{[L]}$ acts elementwise, so component $j$ of $z^{[L]}$ influences only component $j$ of $a^{[L]}$. |
| - | |
| - | ### 5.3.1 The softmax and cross-entropy shortcut |
| - | |
| - | For multiclass classification the natural pairing is a softmax output with the cross-entropy loss (introduced in [Loss functions and output layers](/en/Deep%20Learning/04%20Loss%20functions%20and%20output%20layers)). The two derivatives combine and cancel, leaving a strikingly simple result: |
| - | |
| - | $$\boxed{ \delta^{[L]} = \hat{y} - y }$$ |
| - | |
| - | *Remark:* the same clean form appears for a sigmoid output with binary cross-entropy, and for a linear output with squared error. In each case the output activation is the matched inverse link of the loss, so the messy factors cancel and the error is just the residual $\hat{y} - y$. |
| - | |
| - | ## 5.4 The backward recursion |
| - | |
| - | Given the error at layer $l+1$, we obtain the error at layer $l$. The loss depends on $z^{[l]}$ only through $z^{[l+1]} = W^{[l+1]} a^{[l]} + b^{[l]}$, and $a^{[l]} = g^{[l]}(z^{[l]})$. Propagating the sensitivity back through the weight matrix and then through the activation gives: |
| - | |
| - | $$\boxed{ \delta^{[l]} = \left( \left(W^{[l+1]}\right)^{T} \delta^{[l+1]} \right) \odot g'^{[l]}\!\left(z^{[l]}\right) }$$ |
| - | |
| - | Two operations happen here. The transpose $\left(W^{[l+1]}\right)^{T}$ sends the downstream error back across the linear map, spreading each downstream component onto the units that fed it. The elementwise product with $g'^{[l]}(z^{[l]})$ then filters it by how sensitive each activation was at its operating point. |
| - | |
| - | | Symbol | Meaning | Shape | |
| - | | --- | --- | --- | |
| - | | $\delta^{[l]}$ | error at layer $l$ | $(n_l)$ | |
| - | | $W^{[l+1]}$ | weights into layer $l+1$ | $(n_{l+1} \times n_l)$ | |
| - | | $\left(W^{[l+1]}\right)^{T}\delta^{[l+1]}$ | error pushed back to layer $l$ | $(n_l)$ | |
| - | | $g'^{[l]}(z^{[l]})$ | local activation slope | $(n_l)$ | |
| - | |
| - | *Remark:* the forward pass uses $W^{[l+1]}$ and the backward pass uses its transpose. This is the same linear map read in reverse, which is why the backward pass has the same cost as the forward pass. |
| - | |
| - | ## 5.5 Parameter gradients |
| - | |
| - | The error $\delta^{[l]}$ is all we need for the parameters of layer $l$. Since $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$ is linear in $W^{[l]}$ and $b^{[l]}$, the last chain-rule link is easy. The weight gradient is the outer product of the layer error with the cached input activation: |
| - | |
| - | $$\boxed{ \frac{\partial L}{\partial W^{[l]}} = \delta^{[l]} \left(a^{[l-1]}\right)^{T}, \qquad \frac{\partial L}{\partial b^{[l]}} = \delta^{[l]} }$$ |
| - | |
| - | The weight gradient has shape $(n_l \times n_{l-1})$, matching $W^{[l]}$, and the bias gradient has shape $(n_l)$, matching $b^{[l]}$. The bias gradient is exactly $\delta^{[l]}$ because $\partial z^{[l]} / \partial b^{[l]}$ is the identity. |
| - | |
| - | *Remark:* the cached activation $a^{[l-1]}$ from the forward pass is reused verbatim in the weight gradient. This is the concrete payoff of caching: nothing from the forward pass is recomputed. |
| - | |
| - | ## 5.6 The full algorithm |
| - | |
| - | Backpropagation runs one forward pass to fill a cache, one backward pass to propagate $\delta$, and then a parameter update. |
| - | |
| - | 1. **Forward pass.** Set $a^{[0]} = x$. For $l = 1, \dots, L$ compute $z^{[l]}$ and $a^{[l]}$, caching each. Evaluate the loss $L$ at $\hat{y} = a^{[L]}$. |
| - | 2. **Output error.** Compute $\delta^{[L]}$ from Section 5.3. |
| - | 3. **Backward pass.** For $l = L-1, \dots, 1$ apply the recursion of Section 5.4 to get $\delta^{[l]}$. |
| - | 4. **Gradients.** For each layer form $\partial L / \partial W^{[l]}$ and $\partial L / \partial b^{[l]}$ from Section 5.5. |
| - | 5. **Update.** Over a batch, average the per-example gradients into the cost gradient $\nabla J$ and take one gradient-descent step (detailed in [Optimization](/en/Deep%20Learning/06%20Optimization)). |
| - | |
| - |  |
| - | |
| - | *The backpropagation algorithm as a pipeline from a cached forward pass to the parameter update.* |
| - | |
| - | *Remark:* backpropagation gives the gradient, not the step. It answers which direction lowers the cost, and by how much per unit of each parameter. Turning that gradient into an actual weight change is the job of the optimizer. |
| - | |
| - | In short, backpropagation is an ordered, single-pass application of the chain rule that reuses cached forward quantities to compute every gradient at the price of roughly one extra forward pass. *With the gradient in hand, the next lesson studies how to use it well: learning rates, momentum, and the adaptive methods that make deep networks trainable.* |
| - | |
| - | --- |
| - | Next: [Optimization](/en/Deep%20Learning/06%20Optimization) · [Course overview](/en/Deep%20Learning) |
| en/Deep Learning/05 Backpropagation/backprop-steps.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1120 320" width="1120" height="320" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="1120" height="320" fill="#ffffff"/><text x="560.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">The backpropagation algorithm as a pipeline</text><rect x="25.0" y="130.0" width="150.0" height="92.0" rx="8" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><text x="100.0" y="172.8" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">1. Forward pass:</text><text x="100.0" y="188.1" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">cache z, a</text><rect x="209.0" y="130.0" width="150.0" height="92.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="284.0" y="180.4" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">2. Evaluate loss L</text><rect x="393.0" y="130.0" width="150.0" height="92.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="468.0" y="172.8" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">3. Output error</text><text x="468.0" y="188.1" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">δ<tspan baseline-shift="super" font-size="9px">[L]</tspan></text><rect x="577.0" y="130.0" width="150.0" height="92.0" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="652.0" y="157.4" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">4. Backward</text><text x="652.0" y="172.8" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">recursion</text><text x="652.0" y="188.1" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">δ<tspan baseline-shift="super" font-size="9px">[L]</tspan> to</text><text x="652.0" y="203.4" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">δ<tspan baseline-shift="super" font-size="9px">[1]</tspan></text><rect x="761.0" y="130.0" width="150.0" height="92.0" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="836.0" y="165.1" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">5. Parameter</text><text x="836.0" y="180.4" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">gradients ∇ W,</text><text x="836.0" y="195.8" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">∇ b</text><rect x="945.0" y="130.0" width="150.0" height="92.0" rx="8" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="1020.0" y="165.1" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">6.</text><text x="1020.0" y="180.4" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">Gradient-descent</text><text x="1020.0" y="195.8" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">step</text><line x1="178.0" y1="176.0" x2="206.0" y2="176.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="362.0" y1="176.0" x2="390.0" y2="176.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="546.0" y1="176.0" x2="574.0" y2="176.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="730.0" y1="176.0" x2="758.0" y2="176.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="914.0" y1="176.0" x2="942.0" y2="176.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><text x="100.0" y="256.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">forward</text><text x="652.0" y="256.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">backward</text><text x="1020.0" y="256.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">update</text></svg> |
| \ | No newline at end of file |
| en/Deep Learning/05 Backpropagation/computational-graph.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1080 380" width="1080" height="380" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="1080" height="380" fill="#ffffff"/><text x="540.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">Forward and backward passes over the computational graph</text><rect x="40.0" y="150.0" width="118.0" height="54.0" rx="8" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><text x="99.0" y="182.1" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">x = a<tspan baseline-shift="super" font-size="11px">[0]</tspan></text><rect x="212.0" y="150.0" width="118.0" height="54.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="271.0" y="182.1" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">z<tspan baseline-shift="super" font-size="11px">[1]</tspan></text><rect x="384.0" y="150.0" width="118.0" height="54.0" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="443.0" y="182.1" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">a<tspan baseline-shift="super" font-size="11px">[1]</tspan></text><rect x="556.0" y="150.0" width="118.0" height="54.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="615.0" y="182.1" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">z<tspan baseline-shift="super" font-size="11px">[2]</tspan></text><rect x="728.0" y="150.0" width="118.0" height="54.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="787.0" y="182.1" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">ŷ</text><rect x="900.0" y="150.0" width="118.0" height="54.0" rx="8" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="959.0" y="182.1" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">loss L</text><text x="99.0" y="124.0" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#5b6b7b" text-anchor="start">forward pass (solid): cache z and a</text><line x1="162.0" y1="171.0" x2="208.0" y2="171.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="334.0" y1="171.0" x2="380.0" y2="171.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="506.0" y1="171.0" x2="552.0" y2="171.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="678.0" y1="171.0" x2="724.0" y2="171.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="850.0" y1="171.0" x2="896.0" y2="171.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="959.0" y1="250.0" x2="787.0" y2="250.0" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)" stroke-dasharray="5 4"/><text x="873.0" y="242.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">δ<tspan baseline-shift="super" font-size="9px">[L]</tspan></text><line x1="787.0" y1="250.0" x2="615.0" y2="250.0" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)" stroke-dasharray="5 4"/><text x="701.0" y="242.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">δ<tspan baseline-shift="super" font-size="9px">[2]</tspan></text><line x1="615.0" y1="250.0" x2="443.0" y2="250.0" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)" stroke-dasharray="5 4"/><text x="529.0" y="242.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">∇ W<tspan baseline-shift="super" font-size="9px">[2]</tspan>, ∇ b<tspan baseline-shift="super" font-size="9px">[2]</tspan></text><line x1="443.0" y1="250.0" x2="271.0" y2="250.0" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)" stroke-dasharray="5 4"/><text x="357.0" y="242.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">δ<tspan baseline-shift="super" font-size="9px">[1]</tspan></text><line x1="271.0" y1="250.0" x2="99.0" y2="250.0" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)" stroke-dasharray="5 4"/><text x="185.0" y="242.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">∇ W<tspan baseline-shift="super" font-size="9px">[1]</tspan>, ∇ b<tspan baseline-shift="super" font-size="9px">[1]</tspan></text><text x="99.0" y="280.0" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#5b6b7b" text-anchor="start">backward pass (dashed): propagate the error δ</text></svg> |
| \ | No newline at end of file |
| en/Deep Learning/10 Convolutional networks.md .. en/Deep Learning/05 Convolutional networks.md | |
| @@ 1,17 1,8 @@ | |
| - | # 10. Convolutional networks |
| + | # 5. Convolutional networks |
| A dense layer treats an image as a flat vector, so it must learn a separate weight for every pixel and forgets that nearby pixels belong together. Convolutional networks replace that dense connectivity with a small filter that slides across the grid, reusing the same weights everywhere. This module introduces the convolution as a structured layer for grid data, then builds up stride, padding, channels, and pooling. | |
| - | **Objectives** |
| - | - Motivate convolution from locality, translation equivariance, and parameter sharing. |
| - | - Define the 2D convolution (cross-correlation) used in deep learning. |
| - | - Compute the output size from input size, kernel, padding, and stride. |
| - | - Extend a filter to multiple input and output channels (feature maps). |
| - | - Use max and average pooling to downsample and add small translation invariance. |
| - | - Compare the parameter count of a convolution against an equivalent dense layer. |
| - | - Recognize the landmark architectures, LeNet to ResNet, and the one idea each contributed. |
| - | |
| - | ## 10.1 Why not a dense layer |
| + | ## 5.1 Why not a dense layer |
| Consider a modest $224 \times 224$ RGB image. Flattened it has $224 \times 224 \times 3 \approx 150{,}000$ inputs, so a single dense layer with even $1{,}000$ units carries about $150$ million weights. Three facts about images make almost all of them wasteful. | |
| @@ 21,11 12,24 @@ | |
| A convolutional layer bakes all three in. It uses a small filter (the shared weights) applied at every location (locality and equivariance), which is why it needs orders of magnitude fewer parameters than the dense layer above. | |
| + |  |
| + | |
| + | *The dense layer flattens the image and pays one private weight per pixel and per unit, about $150$ million in this example. The convolutional filter carries $27$ weights and one bias, and is simply reused at every position.* |
| + | |
| + | The payoff in numbers, this time on a smaller $32 \times 32 \times 3$ input with a layer of $16$ filters (filter banks and channels are made precise in section 5.4, only the counting matters here): |
| + | |
| + | | Layer | Weights | Biases | Total parameters | |
| + | | --- | --- | --- | --- | |
| + | | Convolution ($3\times3$, $16$ filters) | $3 \cdot 3 \cdot 3 \cdot 16 = 432$ | $16$ | $448$ | |
| + | | Equivalent dense layer | $(32\cdot32\cdot3)\cdot(32\cdot32\cdot16) \approx 5.0\times10^{10}$ | $16{,}384$ | $\approx 5.0\times10^{10}$ | |
| + | |
| + | A few hundred parameters against about fifty billion, and the convolution generalizes better, because the same feature detector is reused everywhere rather than relearned per position. |
| + | |
| *Remark:* recall the notation from the Introduction. A layer $l$ computes $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$ and $a^{[l]} = g^{[l]}(z^{[l]})$, with explicit bias $b^{[l]}$. A convolution is just a structured $W^{[l]}$ whose entries are tied together and mostly zero, so the same layer equation still holds. | |
| - | ## 10.2 The 2D convolution |
| + | ## 5.2 The 2D convolution |
| - | ### 10.2.1 Cross-correlation |
| + | ### 5.2.1 Cross-correlation |
| Let $I$ be a 2D input (one channel of an image) and $K$ a kernel of size $k \times k$. The operation used in deep learning slides $K$ over $I$ and takes, at each position $(i, j)$, the sum of elementwise products between the kernel and the patch it covers: | |
| @@ 33,13 37,13 @@ | |
| Each output value is one dot product between the kernel and a local window of the input, so a small $3 \times 3$ kernel looks at nine pixels regardless of image size. | |
| - |  |
| + |  |
| *A convolution slides a small kernel across the input, and each position produces one cell of the output feature map.* | |
| *Remark:* this is technically cross-correlation. The mathematical convolution flips the kernel first, but deep learning libraries do not flip and still call it convolution, because the learned kernel simply absorbs the flip. We follow that convention throughout. | |
| - | ### 10.2.2 The layer output |
| + | ### 5.2.2 The layer output |
| A convolutional layer applies this operation, adds the explicit bias $b$, and passes the result through the activation $g$: | |
| @@ 47,7 51,7 @@ | |
| The bias is a single scalar shared across every position of the output, exactly one more instance of parameter sharing. | |
| - | ## 10.3 Stride, padding, and output size |
| + | ## 5.3 Stride, padding, and output size |
| Two hyperparameters control how the kernel sweeps the input. | |
| @@ 62,7 66,11 @@ | |
| For example, with $n = 32$, $k = 5$, $p = 0$, $s = 1$ the output is $\lfloor (32 - 5)/1 \rfloor + 1 = 28$. Adding $p = 2$ ("same") gives $\lfloor (32 + 4 - 5)/1 \rfloor + 1 = 32$. | |
| - | ## 10.4 Channels and feature maps |
| + |  |
| + | |
| + | *The same $3 \times 3$ kernel on a $6 \times 6$ input, three ways. Stride $1$ visits four positions per row, stride $2$ skips every other one, and one ring of zero padding ($p = 1$) lets the kernel cover the border so the output keeps the input size.* |
| + | |
| + | ## 5.4 Channels and feature maps |
| Real images have channels (three for RGB), and a kernel spans all of them. A filter for an input with $C_\text{in}$ channels has shape $k \times k \times C_\text{in}$, and its convolution sums over spatial positions and channels to produce one 2D output, called a **feature map**. | |
| @@ 72,7 80,7 @@ | |
| *Remark:* the output channel count $C_\text{out}$ of one layer becomes the input channel count $C_\text{in}$ of the next, so depth grows as spatial size shrinks. There is one bias per output channel, which is why $b^{[l]}$ has $C_\text{out}$ entries. | |
| - | ## 10.5 Pooling |
| + | ## 5.5 Pooling |
| Pooling downsamples a feature map by summarising each small window with a single number, using a fixed rule and no learned weights. The two common rules are the maximum and the average over each $k \times k$ window: | |
| @@ 80,38 88,27 @@ | |
| Pooling with stride $s = k$ (non-overlapping windows) shrinks each spatial dimension by a factor of $k$, which cuts computation for later layers. It also grants small **translation invariance**: a max over a window returns the same value if the strong response shifts within that window. | |
| - |  |
| + |  |
| *Max pooling downsamples each region to its largest value, shrinking the feature map and adding small translation invariance.* | |
| *Remark:* pooling has no parameters and reduces resolution, which is why modern architectures often replace it with strided convolutions instead. Convolution is equivariant to translation (the response moves with the input), whereas pooling adds a little invariance (the response ignores small moves). | |
| - | ## 10.6 The parameter payoff |
| - | |
| - | The point of parameter sharing is size. Take an input of $32 \times 32 \times 3$ and a layer producing a $32 \times 32 \times 16$ output with a $3 \times 3$ kernel ("same" padding). The convolution shares one small filter bank across all positions, while a dense layer connecting every input to every output does not. |
| - | |
| - | | Layer | Weights | Biases | Total parameters | |
| - | | --- | --- | --- | --- | |
| - | | Convolution ($3\times3$, $16$ filters) | $3 \cdot 3 \cdot 3 \cdot 16 = 432$ | $16$ | $448$ | |
| - | | Equivalent dense layer | $(32\cdot32\cdot3)\cdot(32\cdot32\cdot16) \approx 5.0\times10^{10}$ | $16{,}384$ | $\approx 5.0\times10^{10}$ | |
| - | |
| - | The convolution uses a few hundred parameters against about fifty billion for the dense layer, and it generalizes better because the same feature detector is reused everywhere rather than relearned per position. |
| - | |
| - | ## 10.7 A convolutional stage |
| + | ## 5.6 A convolutional stage |
| A typical stage chains convolution, activation, and pooling, turning the raw image into a stack of feature maps that later stages refine. | |
| - |  |
| + |  |
| *A convolutional stage: convolution, activation, then pooling, repeated to build feature maps.* | |
| *Remark:* stacking such stages makes the receptive field (the input region that influences one output value) grow with depth, so early layers see edges and deep layers see whole objects, all built from the same local operation. | |
| - | ## 10.8 From layers to architectures |
| + | ## 5.7 From layers to architectures |
| The landmark convolutional networks all share one shape: a stack of convolution and pooling stages that extracts features, then a small fully connected head that classifies them. | |
| - |  |
| + |  |
| *A deep CNN progressively reduces spatial size while increasing channel depth, then flattens into fully connected layers.* | |
| @@ 121,15 118,7 @@ | |
| - **AlexNet** scaled that skeleton to large images and GPUs, made trainable by ReLU activations and dropout. | |
| - **VGG** made every convolution $3 \times 3$ and got its depth by stacking: two $3 \times 3$ layers see the same region as one $5 \times 5$ with fewer parameters ($18c^2$ against $25c^2$) and one more nonlinearity. | |
| - **Inception** runs branches of several filter sizes in parallel and concatenates them, kept affordable by $1 \times 1$ convolutions, per-position channel maps that squeeze a thick feature map down before the expensive filters. | |
| - | - **ResNet** lets each block learn a correction around an identity skip connection: |
| - | |
| - | $$\boxed{\ y = F(x, W) + x, \qquad \frac{\partial y}{\partial x} = \frac{\partial F}{\partial x} + I\ }$$ |
| - | |
| - | The $+I$ gives the gradient a backward route that never shrinks, the direct remedy to the [vanishing gradient](/en/Deep%20Learning/07%20Initialization%20and%20vanishing%20gradients) of lesson 7, and networks of hundreds of layers train reliably. |
| - | |
| - |  |
| - | |
| - | *A residual block adds an identity skip connection around the convolution path, so the layer only has to learn a correction F(x).* |
| + | - **ResNet** lets each block learn a correction around an identity skip connection: the skip gives the gradient a backward route that never shrinks, the direct remedy to the [vanishing gradient](/en/Deep%20Learning/02%20Activation%20functions) of lesson 2, and networks of hundreds of layers train reliably. |
| | Architecture | Approx. depth | Key idea | | |
| | --- | --- | --- | | |
| @@ 144,4 133,4 @@ | |
| *These deep stacks learn feature maps whose deeper activations behave as reusable representations, the entry point of the next module on embeddings and representation learning.* | |
| --- | |
| - | Next: [Embeddings and representation learning](/en/Deep%20Learning/11%20Embeddings%20and%20representation%20learning) · [Course overview](/en/Deep%20Learning) |
| + | Next: [Embeddings and representation learning](/en/Deep%20Learning/06%20Embeddings%20and%20representation%20learning) · [Course overview](/en/Deep%20Learning) |
| en/Deep Learning/10 Convolutional networks/cnn-stack.svg .. en/Deep Learning/05 Convolutional networks/cnn-stack.svg | |
| en/Deep Learning/10 Convolutional networks/conv-pipeline.svg .. en/Deep Learning/05 Convolutional networks/conv-pipeline.svg | |
| en/Deep Learning/10 Convolutional networks/convolution.svg .. en/Deep Learning/05 Convolutional networks/convolution.svg | |
| /dev/null .. en/Deep Learning/05 Convolutional networks/dense-vs-conv.svg | |
| @@ 0,0 1,107 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 360" width="900" height="360" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs> |
| + | <marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | </defs> |
| + | <rect width="900" height="360" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">The same image, two layers</text> |
| + | <text x="225" y="56" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">dense layer on the flattened image</text> |
| + | <line x1="60" y1="80" x2="60" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="73" y1="80" x2="73" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="86" y1="80" x2="86" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="99" y1="80" x2="99" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="112" y1="80" x2="112" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="125" y1="80" x2="125" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="138" y1="80" x2="138" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="60" y1="80" x2="138" y2="80" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="60" y1="93" x2="138" y2="93" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="60" y1="106" x2="138" y2="106" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="60" y1="119" x2="138" y2="119" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="60" y1="132" x2="138" y2="132" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="60" y1="145" x2="138" y2="145" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="60" y1="158" x2="138" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <text x="99" y="178" font-size="10" fill="#5b6b7b" text-anchor="middle">224 × 224 × 3 image</text> |
| + | <line x1="145" y1="119" x2="178" y2="119" stroke="#1f2933" stroke-width="1.4" marker-end="url(#arrow)"/> |
| + | <text x="161" y="107" font-size="10" fill="#5b6b7b" text-anchor="middle">flatten</text> |
| + | <rect x="190" y="70" width="12" height="12" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1"/> |
| + | <rect x="190" y="84" width="12" height="12" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1"/> |
| + | <rect x="190" y="98" width="12" height="12" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1"/> |
| + | <rect x="190" y="132" width="12" height="12" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1"/> |
| + | <rect x="190" y="146" width="12" height="12" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1"/> |
| + | <rect x="190" y="160" width="12" height="12" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1"/> |
| + | <text x="196" y="124" font-size="11" fill="#5b6b7b" text-anchor="middle">⋮</text> |
| + | <text x="196" y="192" font-size="10" fill="#5b6b7b" text-anchor="middle">150,000</text> |
| + | <text x="196" y="204" font-size="10" fill="#5b6b7b" text-anchor="middle">inputs</text> |
| + | <line x1="204" y1="76" x2="321" y2="92" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="76" x2="321" y2="120" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="76" x2="321" y2="148" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="76" x2="321" y2="176" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="76" x2="321" y2="204" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="90" x2="321" y2="92" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="90" x2="321" y2="120" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="90" x2="321" y2="148" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="90" x2="321" y2="176" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="90" x2="321" y2="204" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="104" x2="321" y2="92" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="104" x2="321" y2="120" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="104" x2="321" y2="148" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="104" x2="321" y2="176" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="104" x2="321" y2="204" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="138" x2="321" y2="92" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="138" x2="321" y2="120" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="138" x2="321" y2="148" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="138" x2="321" y2="176" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="138" x2="321" y2="204" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="152" x2="321" y2="92" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="152" x2="321" y2="120" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="152" x2="321" y2="148" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="152" x2="321" y2="176" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="152" x2="321" y2="204" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="166" x2="321" y2="92" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="166" x2="321" y2="120" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="166" x2="321" y2="148" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="166" x2="321" y2="176" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="166" x2="321" y2="204" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <circle cx="330" cy="92" r="9" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.4"/> |
| + | <circle cx="330" cy="120" r="9" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.4"/> |
| + | <circle cx="330" cy="148" r="9" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.4"/> |
| + | <circle cx="330" cy="176" r="9" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.4"/> |
| + | <circle cx="330" cy="204" r="9" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.4"/> |
| + | <text x="330" y="228" font-size="10" fill="#5b6b7b" text-anchor="middle">1,000 units</text> |
| + | <text x="225" y="272" font-size="13" font-weight="600" fill="#d1495b" text-anchor="middle">≈ 150,000,000 weights</text> |
| + | <text x="225" y="292" font-size="11" fill="#5b6b7b" text-anchor="middle">one private weight per pixel and per unit, position baked in</text> |
| + | <text x="675" y="56" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">convolutional layer</text> |
| + | <line x1="560" y1="80" x2="560" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="573" y1="80" x2="573" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="586" y1="80" x2="586" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="599" y1="80" x2="599" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="612" y1="80" x2="612" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="625" y1="80" x2="625" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="638" y1="80" x2="638" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="560" y1="80" x2="638" y2="80" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="560" y1="93" x2="638" y2="93" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="560" y1="106" x2="638" y2="106" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="560" y1="119" x2="638" y2="119" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="560" y1="132" x2="638" y2="132" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="560" y1="145" x2="638" y2="145" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="560" y1="158" x2="638" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <rect x="560" y="80" width="39" height="39" fill="#3b6fb6" fill-opacity="0.15" stroke="#3b6fb6" stroke-width="1.8"/> |
| + | <rect x="599" y="119" width="39" height="39" fill="none" stroke="#3b6fb6" stroke-width="1.4" stroke-dasharray="4 3" opacity="0.75"/> |
| + | <text x="599" y="178" font-size="10" fill="#5b6b7b" text-anchor="middle">the same 3 × 3 filter</text> |
| + | <text x="599" y="190" font-size="10" fill="#5b6b7b" text-anchor="middle">slides everywhere</text> |
| + | <line x1="648" y1="119" x2="682" y2="119" stroke="#1f2933" stroke-width="1.4" marker-end="url(#arrow)"/> |
| + | <rect x="692" y="93" width="52" height="52" fill="#e7f5ea"/> |
| + | <line x1="692" y1="93" x2="692" y2="145" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="705" y1="93" x2="705" y2="145" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="718" y1="93" x2="718" y2="145" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="731" y1="93" x2="731" y2="145" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="744" y1="93" x2="744" y2="145" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="692" y1="93" x2="744" y2="93" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="692" y1="106" x2="744" y2="106" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="692" y1="119" x2="744" y2="119" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="692" y1="132" x2="744" y2="132" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="692" y1="145" x2="744" y2="145" stroke="#38a05a" stroke-width="1.0"/> |
| + | <text x="718" y="166" font-size="10" fill="#5b6b7b" text-anchor="middle">feature map</text> |
| + | <text x="675" y="272" font-size="13" font-weight="600" fill="#38a05a" text-anchor="middle">27 weights + 1 bias, shared</text> |
| + | <text x="675" y="292" font-size="11" fill="#5b6b7b" text-anchor="middle">one 3 × 3 × 3 filter reused at every position</text> |
| + | <text x="450" y="334" font-size="11" fill="#5b6b7b" text-anchor="middle">same image, over a million times fewer parameters</text> |
| + | </svg> |
| \ | No newline at end of file |
| en/Deep Learning/10 Convolutional networks/pooling.svg .. en/Deep Learning/05 Convolutional networks/pooling.svg | |
| /dev/null .. en/Deep Learning/05 Convolutional networks/stride-padding.svg | |
| @@ 0,0 1,121 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 360" width="900" height="360" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs> |
| + | <marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | </defs> |
| + | <rect width="900" height="360" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">Stride and padding: a 3 × 3 kernel on a 6 × 6 input</text> |
| + | <text x="165" y="56" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">valid (p = 0), stride 1</text> |
| + | <line x1="126" y1="80" x2="126" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="139" y1="80" x2="139" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="152" y1="80" x2="152" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="165" y1="80" x2="165" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="178" y1="80" x2="178" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="191" y1="80" x2="191" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="204" y1="80" x2="204" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="126" y1="80" x2="204" y2="80" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="126" y1="93" x2="204" y2="93" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="126" y1="106" x2="204" y2="106" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="126" y1="119" x2="204" y2="119" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="126" y1="132" x2="204" y2="132" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="126" y1="145" x2="204" y2="145" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="126" y1="158" x2="204" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <rect x="126" y="80" width="39" height="39" fill="#3b6fb6" fill-opacity="0.15" stroke="#3b6fb6" stroke-width="1.8"/> |
| + | <rect x="139" y="80" width="39" height="39" fill="none" stroke="#3b6fb6" stroke-width="1.3" stroke-dasharray="4 3" opacity="0.7"/> |
| + | <line x1="141" y1="72" x2="153" y2="72" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/> |
| + | <text x="133" y="68" font-size="10" fill="#5b6b7b" text-anchor="middle">+1</text> |
| + | <line x1="165" y1="168" x2="165" y2="196" stroke="#1f2933" stroke-width="1.4" marker-end="url(#arrow)"/> |
| + | <rect x="139" y="204" width="52" height="52" fill="#e7f5ea"/> |
| + | <line x1="139" y1="204" x2="139" y2="256" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="152" y1="204" x2="152" y2="256" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="165" y1="204" x2="165" y2="256" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="178" y1="204" x2="178" y2="256" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="191" y1="204" x2="191" y2="256" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="139" y1="204" x2="191" y2="204" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="139" y1="217" x2="191" y2="217" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="139" y1="230" x2="191" y2="230" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="139" y1="243" x2="191" y2="243" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="139" y1="256" x2="191" y2="256" stroke="#38a05a" stroke-width="1.0"/> |
| + | <text x="165" y="290" font-size="12" fill="#1f2933" text-anchor="middle">o = (6 − 3)/1 + 1 = 4</text> |
| + | <text x="165" y="310" font-size="11" fill="#5b6b7b" text-anchor="middle">the kernel visits 4 positions per row</text> |
| + | <text x="450" y="56" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">stride 2</text> |
| + | <line x1="411" y1="80" x2="411" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="424" y1="80" x2="424" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="437" y1="80" x2="437" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="450" y1="80" x2="450" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="463" y1="80" x2="463" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="476" y1="80" x2="476" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="489" y1="80" x2="489" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="411" y1="80" x2="489" y2="80" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="411" y1="93" x2="489" y2="93" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="411" y1="106" x2="489" y2="106" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="411" y1="119" x2="489" y2="119" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="411" y1="132" x2="489" y2="132" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="411" y1="145" x2="489" y2="145" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="411" y1="158" x2="489" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <rect x="411" y="80" width="39" height="39" fill="#3b6fb6" fill-opacity="0.15" stroke="#3b6fb6" stroke-width="1.8"/> |
| + | <rect x="437" y="80" width="39" height="39" fill="none" stroke="#3b6fb6" stroke-width="1.3" stroke-dasharray="4 3" opacity="0.7"/> |
| + | <line x1="426" y1="72" x2="452" y2="72" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/> |
| + | <text x="416" y="68" font-size="10" fill="#5b6b7b" text-anchor="middle">+2</text> |
| + | <line x1="450" y1="168" x2="450" y2="196" stroke="#1f2933" stroke-width="1.4" marker-end="url(#arrow)"/> |
| + | <rect x="437" y="204" width="26" height="26" fill="#e7f5ea"/> |
| + | <line x1="437" y1="204" x2="437" y2="230" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="450" y1="204" x2="450" y2="230" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="463" y1="204" x2="463" y2="230" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="437" y1="204" x2="463" y2="204" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="437" y1="217" x2="463" y2="217" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="437" y1="230" x2="463" y2="230" stroke="#38a05a" stroke-width="1.0"/> |
| + | <text x="450" y="290" font-size="12" fill="#1f2933" text-anchor="middle">o = ⌊(6 − 3)/2⌋ + 1 = 2</text> |
| + | <text x="450" y="310" font-size="11" fill="#5b6b7b" text-anchor="middle">it skips every other position</text> |
| + | <text x="735" y="56" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">“same” padding, p = 1</text> |
| + | <line x1="683" y1="68" x2="683" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="696" y1="68" x2="696" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="709" y1="68" x2="709" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="722" y1="68" x2="722" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="735" y1="68" x2="735" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="748" y1="68" x2="748" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="761" y1="68" x2="761" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="774" y1="68" x2="774" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="787" y1="68" x2="787" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="683" y1="68" x2="787" y2="68" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="683" y1="81" x2="787" y2="81" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="683" y1="94" x2="787" y2="94" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="683" y1="107" x2="787" y2="107" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="683" y1="120" x2="787" y2="120" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="683" y1="133" x2="787" y2="133" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="683" y1="146" x2="787" y2="146" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="683" y1="159" x2="787" y2="159" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="683" y1="172" x2="787" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="696" y1="81" x2="696" y2="159" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="709" y1="81" x2="709" y2="159" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="722" y1="81" x2="722" y2="159" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="735" y1="81" x2="735" y2="159" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="748" y1="81" x2="748" y2="159" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="761" y1="81" x2="761" y2="159" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="774" y1="81" x2="774" y2="159" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="696" y1="81" x2="774" y2="81" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="696" y1="94" x2="774" y2="94" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="696" y1="107" x2="774" y2="107" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="696" y1="120" x2="774" y2="120" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="696" y1="133" x2="774" y2="133" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="696" y1="146" x2="774" y2="146" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="696" y1="159" x2="774" y2="159" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <rect x="683" y="68" width="39" height="39" fill="#3b6fb6" fill-opacity="0.15" stroke="#3b6fb6" stroke-width="1.8"/> |
| + | <line x1="735" y1="176" x2="735" y2="196" stroke="#1f2933" stroke-width="1.4" marker-end="url(#arrow)"/> |
| + | <rect x="696" y="204" width="78" height="78" fill="#e7f5ea"/> |
| + | <line x1="696" y1="204" x2="696" y2="282" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="709" y1="204" x2="709" y2="282" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="722" y1="204" x2="722" y2="282" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="735" y1="204" x2="735" y2="282" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="748" y1="204" x2="748" y2="282" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="761" y1="204" x2="761" y2="282" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="774" y1="204" x2="774" y2="282" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="696" y1="204" x2="774" y2="204" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="696" y1="217" x2="774" y2="217" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="696" y1="230" x2="774" y2="230" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="696" y1="243" x2="774" y2="243" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="696" y1="256" x2="774" y2="256" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="696" y1="269" x2="774" y2="269" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="696" y1="282" x2="774" y2="282" stroke="#38a05a" stroke-width="1.0"/> |
| + | <text x="735" y="302" font-size="12" fill="#1f2933" text-anchor="middle">o = (6 + 2 − 3)/1 + 1 = 6</text> |
| + | <text x="735" y="322" font-size="11" fill="#5b6b7b" text-anchor="middle">one ring of zeros keeps the output size</text> |
| + | </svg> |
| \ | No newline at end of file |
| en/Deep Learning/11 Embeddings and representation learning.md .. en/Deep Learning/06 Embeddings and representation learning.md | |
| @@ 1,18 1,10 @@ | |
| - | # 11. Embeddings and representation learning |
| + | # 6. Embeddings and representation learning |
| Neural networks turn raw inputs into useful features by learning them rather than hand-crafting them. For discrete symbols (words, product IDs, user IDs, categories) the natural representation is a learned dense vector called an embedding. This lesson shows why one-hot codes are a poor input, how an embedding matrix maps each symbol to a compact vector, how word2vec learns such vectors from co-occurrence, and why embeddings are the standard input to the sequence models and Transformers that follow. | |
| - | **Objectives** |
| - | - Explain why one-hot encodings are large, sparse, and blind to similarity. |
| - | - Define an embedding as a lookup into a learned matrix $E$ and treat its rows as parameters. |
| - | - State the word2vec skip-gram objective and the role of negative sampling. |
| - | - Measure semantic closeness with cosine similarity. |
| - | - See how the same idea covers items, users, and categorical features. |
| - | - Connect embeddings to recurrent networks and Transformers as the input layer. |
| + | ## 6.1 From one-hot to dense vectors |
| - | ## 11.1 From one-hot to dense vectors |
| - | |
| - | ### 11.1.1 The one-hot representation |
| + | ### 6.1.1 The one-hot representation |
| Suppose the vocabulary has $V$ distinct symbols. The classic way to feed symbol $i$ to a network is the one-hot vector $x_{\text{onehot}} \in \{0, 1\}^V$, which is all zeros except for a single $1$ at position $i$. It carries no structure: every pair of distinct symbols is exactly as far apart as every other pair, so the code holds no notion of similarity. It is also enormous, a modern vocabulary has $V$ in the tens or hundreds of thousands, and it is almost entirely zeros. | |
| @@ 26,7 18,7 @@ | |
| *Remark:* feeding a one-hot vector into a linear layer $W x_{\text{onehot}}$ simply selects one column of $W$. The embedding lookup below makes that selection explicit and cheap. | |
| - | ### 11.1.2 The embedding lookup |
| + | ### 6.1.2 The embedding lookup |
| An embedding matrix $E \in \mathbb{R}^{V \times d}$ stores one $d$-dimensional row per symbol. The embedding of a one-hot input is the matrix-vector product | |
| @@ 36,11 28,11 @@ | |
| *Remark:* the rows of $E$ are ordinary parameters. They start random and are updated by backpropagation together with the rest of the network, so the geometry of the space is shaped by whatever task the network is trained on. | |
| - | ## 11.2 Learning word embeddings with word2vec |
| + | ## 6.2 Learning word embeddings with word2vec |
| Embeddings can be learned end to end inside any task, but they can also be learned on their own from unlabelled text. The word2vec skip-gram model does exactly this: it learns a vector per word by predicting the surrounding context words from a centre word. | |
| - | ### 11.2.1 Skip-gram objective |
| + | ### 6.2.1 Skip-gram objective |
| Each word $w$ has an input vector $v_w$ (its row in the embedding matrix). Given a centre word $w_I$, the model scores each candidate output word $w_O$ by a dot product and normalizes over the whole vocabulary with a softmax: | |
| @@ 48,17 40,17 @@ | |
| Training maximizes this probability for the (centre, context) pairs that actually co-occur in a sliding window over the text. Words that appear in similar contexts are pushed to have large dot products, so their vectors end up close together. | |
| - | ### 11.2.2 Negative sampling |
| + | ### 6.2.2 Negative sampling |
| The denominator sums over all $V$ words, which is far too expensive to compute for every training pair. Negative sampling replaces the full softmax with a cheap binary problem: for each real (centre, context) pair, draw a few random words as negatives and train the model to tell the true context word from the fakes. This turns one $V$-way normalization into a handful of logistic updates per step and is what makes word2vec fast enough to train on billions of words. | |
| - |  |
| + |  |
| *The skip-gram model learns embeddings by predicting a word context from a center word.* | |
| *Remark:* the learned space has a striking linear structure. Directions in it encode consistent relations, so analogies show up as vector arithmetic, the classic example being that the vector for "king" minus "man" plus "woman" lands near "queen". | |
| - | ## 11.3 Measuring similarity |
| + | ## 6.3 Measuring similarity |
| Once symbols are dense vectors, "how related are two symbols" becomes a geometric question. The standard answer is cosine similarity, the cosine of the angle between two vectors $u$ and $v$: | |
| @@ 66,13 58,13 @@ | |
| It lies in $[-1, 1]$: a value near $1$ means the vectors point the same way (very similar), near $0$ means unrelated, and near $-1$ means opposite. Cosine ignores vector length and looks only at direction, which is usually what we want, since a word's meaning should not depend on how often it appears. | |
| - |  |
| + |  |
| *Learned embeddings place related words near each other, and consistent directions in the space capture analogies.* | |
| *Remark:* nearest-neighbour search under cosine similarity is how embeddings power retrieval and recommendation. Find the stored vectors whose direction is closest to a query vector and you have the most relevant items. | |
| - | ## 11.4 Embeddings beyond words |
| + | ## 6.4 Embeddings beyond words |
| Nothing in the construction is specific to language. Any set of discrete symbols can be embedded by giving it a matrix $E$ and learning its rows. | |
| @@ 91,11 83,11 @@ | |
| *Remark:* embeddings are also a form of dimensionality reduction. They compress a $V$-way symbol into $d$ numbers while keeping the information a downstream task needs, which is the essence of representation learning. | |
| - | ## 11.5 Embeddings as the input to sequence models |
| + | ## 6.5 Embeddings as the input to sequence models |
| - | A sequence of symbols becomes a sequence of vectors by looking each one up in $E$. That matrix of embeddings is exactly the input a recurrent network reads step by step (lesson [Recurrent networks](/en/Deep%20Learning/12%20Recurrent%20networks)) and the input a Transformer attends over (lesson [Transformers](/en/Deep%20Learning/15%20Transformers)). In both cases the embedding table is learned jointly with the rest of the model, so the representations are tuned to the end task rather than fixed in advance. |
| + | A sequence of symbols becomes a sequence of vectors by looking each one up in $E$. That matrix of embeddings is exactly the input a recurrent network reads step by step (lesson [Recurrent networks](/en/Deep%20Learning/07%20Recurrent%20networks)) and the input a Transformer attends over (lesson [Transformers](/en/Deep%20Learning/10%20Transformers)). In both cases the embedding table is learned jointly with the rest of the model, so the representations are tuned to the end task rather than fixed in advance. |
| - |  |
| + |  |
| *An embedding lookup selects one row of the matrix E, mapping a sparse one-hot token to a dense learned vector.* | |
| @@ 104,4 96,4 @@ | |
| *Dense vectors give us a compact, similarity-aware input. The next lesson feeds such a sequence of vectors, one step at a time, into a recurrent network that carries a hidden state through time.* | |
| --- | |
| - | Next: [Recurrent networks](/en/Deep%20Learning/12%20Recurrent%20networks) · [Course overview](/en/Deep%20Learning) |
| \ | No newline at end of file |
| + | Next: [Recurrent networks](/en/Deep%20Learning/07%20Recurrent%20networks) · [Course overview](/en/Deep%20Learning) |
| \ | No newline at end of file |
| en/Deep Learning/11 Embeddings and representation learning/embedding-lookup.svg .. en/Deep Learning/06 Embeddings and representation learning/embedding-lookup.svg | |
| en/Deep Learning/11 Embeddings and representation learning/embedding-space.png .. en/Deep Learning/06 Embeddings and representation learning/embedding-space.png | |
| en/Deep Learning/11 Embeddings and representation learning/skipgram.svg .. en/Deep Learning/06 Embeddings and representation learning/skipgram.svg | |
| en/Deep Learning/07 Initialization and vanishing gradients.md .. /dev/null | |
| @@ 1,117 0,0 @@ | |
| - | # 7. Initialization and vanishing gradients |
| - | |
| - | Deep networks are hard to train because backpropagation multiplies one Jacobian per layer, so a signal can shrink or blow up geometrically with depth. This module explains where that instability comes from, why naive weight initialization makes it worse, and the two fixes that make deep training routine: variance-preserving initialization (Xavier and He) and gradient clipping. |
| - | |
| - | **Objectives** |
| - | - Write backpropagation as a product of per-layer Jacobians and see when it vanishes or explodes. |
| - | - Connect the effect to activation saturation from lesson 3. |
| - | - Explain why all-zeros and badly scaled initializations fail. |
| - | - Derive the variance target that Xavier and He initialization satisfy. |
| - | - Apply gradient clipping to tame exploding gradients. |
| - | - Pick an initializer from the activation function. |
| - | |
| - | ## 7.1 Why depth is unstable |
| - | |
| - | ### 7.1.1 The Jacobian product |
| - | |
| - | Recall the forward pass of lesson 6: layer $l$ computes $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$ and $a^{[l]} = g^{[l]}(z^{[l]})$, with $a^{[0]} = x$ and $\hat{y} = a^{[L]}$. Backpropagation sends the loss gradient from the output back to layer $l$ by the chain rule. The error signal $\delta^{[l]} = \partial L / \partial z^{[l]}$ obeys the recurrence $\delta^{[l]} = (W^{[l+1]})^T \delta^{[l+1]} \odot g'^{[l]}(z^{[l]})$, so unrolling it from the top layer $L$ down to layer $l$ gives a product: |
| - | |
| - | $$\boxed{ \frac{\partial L}{\partial z^{[l]}} = \left( \prod_{k=l+1}^{L} \operatorname{diag}\!\left(g'^{[k-1]}(z^{[k-1]})\right) (W^{[k]})^T \right) \frac{\partial L}{\partial z^{[L]}} }$$ |
| - | |
| - | Each factor is a layer Jacobian: a weight matrix $W^{[k]}$ combined with the diagonal matrix $\operatorname{diag}(g'(z))$ of activation slopes. The gradient reaching layer $l$ is this whole product acting on the top-layer error. |
| - | |
| - | ### 7.1.2 Vanishing and exploding |
| - | |
| - | A product of many factors is governed by their typical magnitude. Write $\rho$ for the typical size (a spectral norm) of one factor $W^{[k]} \odot \operatorname{diag}(g')$. Across $L - l$ layers the signal scales roughly as $\rho^{\,L-l}$: |
| - | |
| - | $$\boxed{ \left\| \frac{\partial L}{\partial z^{[l]}} \right\| \;\approx\; \rho^{\,L-l} \left\| \frac{\partial L}{\partial z^{[L]}} \right\| }$$ |
| - | |
| - | If $\rho < 1$ consistently the gradient shrinks toward zero as it travels back (the **vanishing gradient**), so early layers barely update and effectively stop learning. If $\rho > 1$ it grows without bound (the **exploding gradient**), so updates overshoot and the loss diverges to `NaN`. Only $\rho \approx 1$ keeps the signal alive across depth. |
| - | |
| - |  |
| - | |
| - | *Gradient magnitude across depth: poorly scaled weights make it vanish or explode, while variance-preserving initialization keeps it near one.* |
| - | |
| - | *Remark:* the same product runs forward for the activations themselves. If layer outputs shrink or grow geometrically, the network cannot represent anything useful even before a gradient is computed, so we want both the forward signal and the backward gradient near unit scale. |
| - | |
| - | ### 7.1.3 The saturation connection |
| - | |
| - | The factor $g'(z)$ links this directly to the saturation seen in lesson 3. The sigmoid and $\tanh$ flatten for large $|z|$, so their derivatives fall to near zero there. |
| - | |
| - | | Activation | $g'(z)$ | max slope | slope when saturated | |
| - | | --- | --- | --- | --- | |
| - | | sigmoid | $g(z)(1-g(z))$ | $0.25$ | $\to 0$ | |
| - | | $\tanh$ | $1 - \tanh^2(z)$ | $1$ | $\to 0$ | |
| - | | ReLU | $1$ for $z>0$, else $0$ | $1$ | $0$ on the dead side | |
| - | |
| - | *Remark:* the sigmoid slope never exceeds $0.25$, so each layer multiplies the backward signal by at most a quarter. Stack ten sigmoid layers and the gradient is scaled by at most $0.25^{10} \approx 10^{-6}$ before any weight is considered. This is why deep stacks of saturating units train poorly, and why ReLU (slope $1$ on the active side) became the default. |
| - | |
| - | ## 7.2 Bad initializations |
| - | |
| - | ### 7.2.1 All zeros |
| - | |
| - | Setting $W^{[l]} = 0$ (or any value that makes every unit in a layer identical) breaks learning through **symmetry**. If two units in a layer start with the same weights and see the same input, they compute the same activation and receive the same gradient, so they update identically and stay identical forever. The layer then behaves like a single unit no matter how wide it is. Random initialization exists precisely to break this symmetry so units can specialize. |
| - | |
| - | ### 7.2.2 Wrong scale |
| - | |
| - | Even with random, symmetry-breaking weights the **variance** matters. Consider a linear unit $z = \sum_{j=1}^{n_{\text{in}}} W_j a_j$ with independent zero-mean weights and inputs. Its variance is a sum of $n_{\text{in}}$ independent terms: |
| - | |
| - | $$\boxed{ \operatorname{Var}(z) = n_{\text{in}} \cdot \operatorname{Var}(W) \cdot \operatorname{Var}(a) }$$ |
| - | |
| - | If $n_{\text{in}} \cdot \operatorname{Var}(W) > 1$ the signal variance grows layer after layer and explodes, and if it is $< 1$ the variance decays and vanishes. To keep $\operatorname{Var}(z) \approx \operatorname{Var}(a)$ from layer to layer we need $n_{\text{in}} \cdot \operatorname{Var}(W) \approx 1$, which fixes the weight variance to roughly $1 / n_{\text{in}}$. That single condition is the seed of both initializers below. |
| - | |
| - | ## 7.3 Variance-preserving initialization |
| - | |
| - | ### 7.3.1 Xavier / Glorot |
| - | |
| - | Glorot and Bengio balance the forward pass ($\operatorname{Var}(W) = 1/n_{\text{in}}$) against the backward pass ($\operatorname{Var}(W) = 1/n_{\text{out}}$) by averaging the two, giving the Xavier initialization: |
| - | |
| - | $$\boxed{ \operatorname{Var}(W^{[l]}) = \frac{2}{n_{\text{in}} + n_{\text{out}}} }$$ |
| - | |
| - | Here $n_{\text{in}} = n_{l-1}$ is the fan-in and $n_{\text{out}} = n_l$ is the fan-out of the layer. Xavier is derived assuming the activation is roughly linear near the origin, so it suits **symmetric, unit-slope** activations like $\tanh$ and the sigmoid. |
| - | |
| - | ### 7.3.2 He |
| - | |
| - | ReLU zeros out half of its inputs on average, so it halves the variance of what passes through. He initialization compensates with a factor of two: |
| - | |
| - | $$\boxed{ \operatorname{Var}(W^{[l]}) = \frac{2}{n_{\text{in}}} }$$ |
| - | |
| - | This is the right target for **ReLU and its variants** (leaky ReLU, ELU, GELU). In practice you draw weights from a Gaussian with this variance, or from a uniform distribution with the matching range, and set the bias $b^{[l]}$ to zero. |
| - | |
| - | *Remark:* the bias starts at zero, not the weights. A zero bias does not create symmetry (the weights already differ), and it keeps the initial pre-activation centred so the activation starts in its responsive region rather than saturated. |
| - | |
| - | ### 7.3.3 The mechanism |
| - | |
| - |  |
| - | |
| - | *Variance-preserving initialization keeps the signal and the gradient near unit scale through depth.* |
| - | |
| - | Choosing the variance is a one-time fix at the start of training. It positions the network so the Jacobian product of section 7.1 has factors near $1$, but nothing keeps it there as the weights move during training. That is what the next module addresses. |
| - | |
| - | ## 7.4 Exploding gradients and clipping |
| - | |
| - | Good initialization tames vanishing gradients and greatly reduces explosions, but explosions can still appear during training, especially in recurrent networks where the same weight matrix is reused at every time step. The standard remedy is **gradient clipping**: rescale the whole gradient vector $g$ so its norm never exceeds a threshold $\tau$. |
| - | |
| - | $$\boxed{ g \leftarrow g \cdot \min\!\left(1, \frac{\tau}{\lVert g \rVert}\right) }$$ |
| - | |
| - | When $\lVert g \rVert \le \tau$ the factor is $1$ and the gradient is untouched. When $\lVert g \rVert > \tau$ the gradient is shrunk back to norm exactly $\tau$ while keeping its direction, so a single huge step cannot blow up the weights. |
| - | |
| - | *Remark:* clipping by global norm (rescaling the whole vector together) preserves the update direction, whereas clipping each coordinate independently to $[-\tau, \tau]$ can bend the direction. Global-norm clipping is the usual default. |
| - | |
| - | ## 7.5 Choosing an initializer |
| - | |
| - | Match the initializer to the activation of the layer it feeds. |
| - | |
| - | | Activation | Recommended initializer | Weight variance | |
| - | | --- | --- | --- | |
| - | | ReLU, leaky ReLU, ELU, GELU | He | $2 / n_{\text{in}}$ | |
| - | | $\tanh$ | Xavier / Glorot | $2 / (n_{\text{in}} + n_{\text{out}})$ | |
| - | | sigmoid | Xavier / Glorot | $2 / (n_{\text{in}} + n_{\text{out}})$ | |
| - | | softmax / linear output | Xavier / Glorot | $2 / (n_{\text{in}} + n_{\text{out}})$ | |
| - | |
| - | *Remark:* initialization and clipping only manage the signal at the endpoints of training and during large steps. Two structural remedies keep it controlled throughout: **normalization** re-centres and rescales activations at every layer, and **residual connections** add a shortcut that lets the gradient skip past the Jacobian product entirely. |
| - | |
| - | *Good initialization keeps the signal well scaled at step zero, but the statistics drift as training proceeds. The next module keeps them in check at every step with normalization.* |
| - | |
| - | --- |
| - | Next: [Normalization](/en/Deep%20Learning/08%20Normalization) · [Course overview](/en/Deep%20Learning) |
| en/Deep Learning/07 Initialization and vanishing gradients/init-reasoning.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1080 190" width="1080" height="190" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="1080" height="190" fill="#ffffff"/><text x="540.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">Why variance-preserving initialization stabilizes depth</text><rect x="24.0" y="78.0" width="176.0" height="66.0" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="112.0" y="107.8" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">random W break</text><text x="112.0" y="123.1" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">symmetry</text><rect x="224.0" y="78.0" width="176.0" height="66.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="312.0" y="107.8" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">set Var(W) near</text><text x="312.0" y="123.1" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">1/n<tspan baseline-shift="sub" font-size="9px">in</tspan></text><rect x="424.0" y="78.0" width="176.0" height="66.0" rx="8" fill="#e4f5f4" stroke="#2a9d9a" stroke-width="1.6"/><text x="512.0" y="107.8" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">signal Var(z) near</text><text x="512.0" y="123.1" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">Var(a)</text><rect x="624.0" y="78.0" width="176.0" height="66.0" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="712.0" y="107.8" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">gradient factor ρ</text><text x="712.0" y="123.1" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">near 1</text><rect x="824.0" y="78.0" width="176.0" height="66.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="912.0" y="107.8" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">deep network trains</text><text x="912.0" y="123.1" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">stably</text><line x1="200.0" y1="111.0" x2="224.0" y2="111.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="400.0" y1="111.0" x2="424.0" y2="111.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="600.0" y1="111.0" x2="624.0" y2="111.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="800.0" y1="111.0" x2="824.0" y2="111.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/></svg> |
| \ | No newline at end of file |
| en/Deep Learning/12 Recurrent networks.md .. en/Deep Learning/07 Recurrent networks.md | |
| @@ 1,21 1,23 @@ | |
| - | # 12. Recurrent networks |
| + | # 7. Recurrent networks |
| Feedforward and convolutional networks map a fixed-size input to an output in one pass, but many problems come as sequences whose length varies and whose order matters (text, audio, time series). A recurrent neural network (RNN) processes a sequence one step at a time and carries a hidden state forward, so past inputs influence the current output. This module introduces recurrence, the vanilla RNN cell, how it is trained by backpropagation through time, and why long-range gradients tend to vanish or explode. | |
| - | **Objectives** |
| - | - Explain why sequence data needs memory and weight sharing across time steps. |
| - | - Write the vanilla RNN recurrence for the hidden state and the output. |
| - | - Unroll a recurrent cell across time and read off the shared parameters. |
| - | - Derive how backpropagation through time (BPTT) accumulates the gradient over all steps. |
| - | - Diagnose vanishing and exploding gradients from the product of Jacobians over time. |
| + | ## 7.1 Why not an MLP or a CNN? |
| - | ## 12.1 Sequence data and memory |
| + | A sequence is an ordered list of inputs $x_1, x_2, \dots, x_T$: the words of a review, the samples of an audio clip, the daily values of a time series. Two things make it awkward for the networks we already have. The length $T$ changes from one example to the next, and the order carries the meaning: "not good, actually great" and "not great, actually good" contain exactly the same words. |
| - | A sequence is an ordered list of inputs $x_1, x_2, \dots, x_T$, where $T$ can differ from one example to the next. A feedforward network of the kind seen in earlier lessons expects a single fixed-size vector $a^{[0]} = x$, so it has no natural way to consume a variable-length input or to remember what came before the current element. |
| + | Try the tools already on the table: |
| - | Two ideas fix this. First, the network keeps a **hidden state** (or memory) $h_t$ that summarizes everything relevant seen up to step $t$. Second, the network **shares** one set of parameters across every step, so the same transformation applies whether the sequence has 5 elements or 500. Sharing keeps the parameter count independent of $T$ and lets a pattern learned at one position generalize to any other. |
| + | - **An MLP** expects one fixed-size vector $a^{[0]} = x$, so a longer sequence simply does not fit its input layer. Worse, each position gets its own private weights: a pattern learned on the 2nd word teaches nothing about the same pattern on the 9th, the dense-layer problem of lesson 5 all over again, in time instead of space. |
| + | - **A 1D CNN** fixes the sharing: one kernel slides along the sequence, the same weights at every position. But its window is $k$ steps wide, so two inputs only ever meet if they fall inside the same receptive field. The end of a long sequence cannot see its beginning without stacking many layers, and the reach is still fixed at build time. |
| - | *Remark:* weight sharing across time is the sequential analogue of weight sharing across space in a convolutional network. Both encode a prior that the same feature can appear anywhere. |
| + | What sequences call for is three things at once: accept any length, apply the same weights at every step, and carry a memory of what has been seen so far, however far back. |
| + | |
| + |  |
| + | |
| + | *The same six-step sequence, twice. The MLP has fixed slots and private weights per position. The 1D convolution shares its weights, but only relates inputs that fall inside its window.* |
| + | |
| + | Whatever meets all three requirements at once will be the subject of this lesson. Before building it, note the shapes sequence tasks come in: |
| | Setup | Input | Output | Example | | |
| | --- | --- | --- | --- | | |
| @@ 24,9 26,13 @@ | |
| | Many to many (seq2seq) | sequence | sequence, other length | machine translation | | |
| | One to many | single vector | sequence | image captioning | | |
| - | ## 12.2 The vanilla RNN cell |
| + | ## 7.2 The vanilla RNN cell |
| + | |
| + | The recurrent network meets all three requirements with one move. It keeps a **hidden state** (or memory) $h_t$ that summarizes everything relevant seen up to step $t$, and it **shares** one set of parameters across every step, so the same transformation applies whether the sequence has 5 elements or 500. Sharing keeps the parameter count independent of $T$ and lets a pattern learned at one position generalize to any other. |
| + | |
| + | *Remark:* weight sharing across time is the sequential analogue of weight sharing across space in a convolutional network. Both encode a prior that the same feature can appear anywhere. |
| - | ### 12.2.1 Recurrence |
| + | ### 7.2.1 Recurrence |
| At step $t$ the cell reads the current input $x_t$ and the previous hidden state $h_{t-1}$, then produces a new hidden state through an activation $g$ (usually $\tanh$): | |
| @@ 40,7 46,7 @@ | |
| *Remark:* this keeps the explicit-bias convention of the whole Deep Learning course. The bias $b_h$ is a separate additive term, never folded into the weight matrices the way the Machine Learning course folded the intercept into $w^T x$ with $x_0 = 1$. | |
| - | ### 12.2.2 Shared weights |
| + | ### 7.2.2 Shared weights |
| The crucial point is that $W_{hh}$, $W_{xh}$, $W_{hy}$, $b_h$, and $b_y$ do **not** depend on $t$. The same five parameters are reused at every step: | |
| @@ 48,17 54,17 @@ | |
| So an RNN is not a very deep network with distinct layers, it is one small cell applied repeatedly, feeding its own output back as input. | |
| - | ## 12.3 Unrolling in time |
| + | ## 7.3 Unrolling in time |
| Because the same cell is reused, we can **unroll** the recurrence into a chain: draw one copy of the cell per time step and connect the hidden state of each copy to the next. The unrolled view is an ordinary feedforward graph (with tied weights), which is exactly what makes gradient computation possible. | |
| - |  |
| + |  |
| *Unrolled in time, a recurrent network reuses the same weights at every step and passes the hidden state forward.* | |
| *Remark:* the horizontal arrows between hidden states are the only path along which information from the past reaches the present. Every one of them multiplies by the same matrix $W_{hh}$, which is the source of both the model's power and its training difficulty. | |
| - | ## 12.4 Backpropagation through time |
| + | ## 7.4 Backpropagation through time |
| Training minimizes a total cost that sums the per-step loss over the sequence. With per-step loss $L_t$ comparing $\hat{y}_t$ to the target $y_t$, the cost for one sequence is: | |
| @@ 74,7 80,7 @@ | |
| *Remark:* in practice the sum over $k$ is cut off after a fixed window, which is called truncated BPTT. It bounds memory and compute per update at the cost of ignoring dependencies longer than the window. | |
| - | ## 12.5 Vanishing and exploding gradients |
| + | ## 7.5 Vanishing and exploding gradients |
| The inner product $\prod_{i=k+1}^{t} \frac{\partial h_i}{\partial h_{i-1}}$ is what carries gradient information from step $t$ back to step $k$. From the recurrence $h_i = g(W_{hh} h_{i-1} + W_{xh} x_i + b_h)$, each factor is: | |
| @@ 86,7 92,7 @@ | |
| This product of $t - k$ near-identical factors behaves roughly like a matrix raised to the power $t - k$. If the relevant magnitude (informally, the largest singular value of $\operatorname{diag}(g'(z_i)) W_{hh}$) is below $1$, the product shrinks geometrically toward zero as the gap grows, so distant gradients **vanish**. If it is above $1$, the product blows up and gradients **explode**. | |
| - |  |
| + |  |
| *Through many time steps the gradient shrinks or grows geometrically, so long-range dependencies are hard for a plain RNN to learn.* | |
| @@ 103,4 109,4 @@ | |
| *The next lesson introduces the LSTM and GRU, gated architectures that carry a cell state through additive updates so gradients can travel across long spans without vanishing.* | |
| --- | |
| - | Next: [LSTM and GRU](/en/Deep%20Learning/13%20LSTM%20and%20GRU) · [Course overview](/en/Deep%20Learning) |
| + | Next: [LSTM and GRU](/en/Deep%20Learning/08%20LSTM%20and%20GRU) · [Course overview](/en/Deep%20Learning) |
| en/Deep Learning/12 Recurrent networks/bptt-decay.png .. en/Deep Learning/07 Recurrent networks/bptt-decay.png | |
| /dev/null .. en/Deep Learning/07 Recurrent networks/rnn-motivation.svg | |
| @@ 0,0 1,37 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 280" width="900" height="280" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs> |
| + | <marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | </defs> |
| + | <rect width="900" height="280" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">One sequence, two networks that fall short</text> |
| + | <text x="40" y="62" font-size="13" font-weight="600" fill="#1f2933">an MLP: flatten into fixed slots</text> |
| + | <rect x="40" y="80" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="50" y="94" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">1</tspan></text> |
| + | <rect x="66" y="80" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="76" y="94" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">2</tspan></text> |
| + | <rect x="92" y="80" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="102" y="94" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">3</tspan></text> |
| + | <rect x="118" y="80" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="128" y="94" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">4</tspan></text> |
| + | <rect x="144" y="80" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="154" y="94" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">5</tspan></text> |
| + | <rect x="170" y="80" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="180" y="94" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">6</tspan></text> |
| + | <rect x="202" y="80" width="20" height="20" fill="#eef1f4" stroke="#d1495b" stroke-width="1.3" stroke-dasharray="4 3"/><text x="212" y="94" font-size="10" fill="#d1495b" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">7</tspan></text> |
| + | <text x="212" y="118" font-size="9" fill="#d1495b" text-anchor="middle">does not fit</text> |
| + | <text x="115" y="118" font-size="10" fill="#5b6b7b" text-anchor="middle">fixed input width</text> |
| + | <line x1="242" y1="90" x2="272" y2="90" stroke="#1f2933" stroke-width="1.4" marker-end="url(#arrow)"/> |
| + | <rect x="282" y="74" width="92" height="32" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/> |
| + | <text x="328" y="94" font-size="10" fill="#1f2933" text-anchor="middle">dense layer</text> |
| + | <text x="328" y="122" font-size="10" fill="#5b6b7b" text-anchor="middle">one private weight per slot</text> |
| + | <text x="560" y="84" font-size="12" font-weight="600" fill="#d1495b">✗</text><text x="578" y="84" font-size="11" fill="#1f2933">fixed length only</text> |
| + | <text x="560" y="104" font-size="12" font-weight="600" fill="#d1495b">✗</text><text x="578" y="104" font-size="11" fill="#1f2933">nothing transfers across positions</text> |
| + | <text x="40" y="168" font-size="13" font-weight="600" fill="#1f2933">a 1D CNN: slide a shared window</text> |
| + | <rect x="40" y="186" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="50" y="200" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">1</tspan></text> |
| + | <rect x="66" y="186" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="76" y="200" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">2</tspan></text> |
| + | <rect x="92" y="186" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="102" y="200" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">3</tspan></text> |
| + | <rect x="118" y="186" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="128" y="200" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">4</tspan></text> |
| + | <rect x="144" y="186" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="154" y="200" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">5</tspan></text> |
| + | <rect x="170" y="186" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="180" y="200" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">6</tspan></text> |
| + | <rect x="38" y="184" width="76" height="24" fill="#3b6fb6" fill-opacity="0.15" stroke="#3b6fb6" stroke-width="1.8"/> |
| + | <rect x="116" y="184" width="76" height="24" fill="none" stroke="#3b6fb6" stroke-width="1.2" stroke-dasharray="4 3" opacity="0.7"/> |
| + | <path d="M180,210 C158,240 72,240 50,210" fill="none" stroke="#d1495b" stroke-width="1.2" stroke-dasharray="4 3"/> |
| + | <text x="115" y="238" font-size="11" font-weight="600" fill="#d1495b" text-anchor="middle">?</text> |
| + | <text x="118" y="258" font-size="10" fill="#5b6b7b" text-anchor="middle">window of 3: distant steps never meet</text> |
| + | <text x="560" y="190" font-size="12" font-weight="600" fill="#38a05a">✓</text><text x="578" y="190" font-size="11" fill="#1f2933">weights shared across positions</text> |
| + | <text x="560" y="210" font-size="12" font-weight="600" fill="#d1495b">✗</text><text x="578" y="210" font-size="11" fill="#1f2933">reach limited to the window</text> |
| + | </svg> |
| \ | No newline at end of file |
| en/Deep Learning/12 Recurrent networks/rnn-unrolled.svg .. en/Deep Learning/07 Recurrent networks/rnn-unrolled.svg | |
| en/Deep Learning/13 LSTM and GRU.md .. en/Deep Learning/08 LSTM and GRU.md | |
| @@ 1,15 1,8 @@ | |
| - | # 13. LSTM and GRU |
| + | # 8. LSTM and GRU |
| A plain recurrent network struggles to carry information across many time steps because repeated multiplication by the same weight matrix makes gradients vanish or explode. Gated recurrent cells fix this by adding a state that flows through time with mostly additive updates, controlled by learned gates. This module builds the long short-term memory (LSTM) cell and the lighter gated recurrent unit (GRU), and contrasts when to reach for each. | |
| - | **Objectives** |
| - | - Explain why a gated cell state preserves long-range gradient flow (the constant error carousel). |
| - | - Write the three LSTM gates as sigmoids of an affine map of the concatenated input. |
| - | - Derive the LSTM candidate, cell update, and hidden state. |
| - | - Write the GRU reset and update gates and its interpolated hidden state. |
| - | - Compare LSTM and GRU on gate count, cell state, parameter count, and typical use. |
| - | |
| - | ## 13.1 The gating idea |
| + | ## 8.1 The gating idea |
| A vanilla recurrent layer updates its hidden state by $h_t = g(W_h h_{t-1} + W_x x_t + b)$. Backpropagating the loss through $T$ steps multiplies many Jacobians of this map together, so the gradient magnitude scales roughly like the $T$-th power of the recurrent weight's spectral radius. Below one it vanishes, above one it explodes, and in both cases the network cannot learn dependencies that span many steps. | |
| @@ 17,11 10,11 @@ | |
| *Remark:* the key word is additive. Multiplicative recurrence compounds a factor at every step, while an additive path lets the state persist by default and change only when a gate opens. | |
| - | ## 13.2 The LSTM cell |
| + | ## 8.2 The LSTM cell |
| Throughout, $[h_{t-1}, x_t]$ denotes the concatenation of the previous hidden state and the current input into one vector. Each gate is a vector in $(0, 1)$ produced by a sigmoid $\sigma$ applied to an affine map of that concatenation, so a gate value near $1$ lets information through and a value near $0$ blocks it. | |
| - | ### 13.2.1 The three gates |
| + | ### 8.2.1 The three gates |
| The **forget** gate $f_t$ decides how much of the old cell state to keep, the **input** gate $i_t$ decides how much of the new candidate to write, and the **output** gate $o_t$ decides how much of the cell state to expose as the hidden state: | |
| @@ 29,7 22,11 @@ | |
| *Remark:* the gates share the same functional form and differ only in their learned parameters. The bias is explicit here, exactly as with the feedforward layers of earlier modules, and is never folded into the weight matrix. | |
| - | ### 13.2.2 Candidate and cell update |
| + |  |
| + | |
| + | *The cell so far: three sigmoid gates reading $[h_{t-1}, x_t]$ from the input rail. The paths they will control are still grayed out.* |
| + | |
| + | ### 8.2.2 Candidate and cell update |
| A $\tanh$ layer proposes a **candidate** update $\tilde{c}_t$, the new content the cell could store: | |
| @@ 41,7 38,11 @@ | |
| When $f_t \approx 1$ and $i_t \approx 0$ the cell simply copies $c_{t-1}$, which is the constant error carousel: $\partial c_t / \partial c_{t-1} \approx \mathrm{diag}(f_t)$, so gradients pass through nearly unattenuated. | |
| - | ### 13.2.3 Hidden state |
| + |  |
| + | |
| + | *Step 2 lights up the carousel: the candidate proposes content, the forget multiply and the write add edit the cell state as it crosses the top. Only the output side remains gray.* |
| + | |
| + | ### 8.2.3 Hidden state |
| The hidden state is the squashed cell state, gated by the output gate: | |
| @@ 49,21 50,27 @@ | |
| *Remark:* the cell state $c_t$ is the long-term memory that flows along the carousel, while the hidden state $h_t$ is the filtered view exposed to the next layer and to the output at this step. Keeping them separate is what distinguishes the LSTM from the GRU below. | |
| - | ## 13.3 The GRU |
| + | The whole cell, assembled: |
| - | The GRU merges the cell and hidden state into a single $h_t$ and uses only two gates, so it has fewer parameters while keeping the additive-update benefit. |
| + |  |
| + | |
| + | *One LSTM step. Along the top runs the carousel: the cell state crosses the cell touched only by the forget multiply and the write add, never by a matrix multiply. Below, the four blocks read $[h_{t-1}, x_t]$ and decide what to forget ($f_t$), what to write ($i_t \odot \tilde{c}_t$), and what to expose ($h_t = o_t \odot \tanh(c_t)$).* |
| - |  |
| + | ## 8.3 The GRU |
| - | *The GRU merges the cell and hidden state and uses just a reset and an update gate.* |
| + | The GRU merges the cell and hidden state into a single $h_t$ and uses only two gates, so it has fewer parameters while keeping the additive-update benefit. |
| - | ### 13.3.1 Reset and update gates |
| + | ### 8.3.1 Reset and update gates |
| The **reset** gate $r_t$ controls how much past state feeds the candidate, and the **update** gate $z_t$ controls how much of the state to refresh: | |
| $$\boxed{ r_t = \sigma\!\left(W_r\,[h_{t-1}, x_t] + b_r\right), \quad z_t = \sigma\!\left(W_z\,[h_{t-1}, x_t] + b_z\right) }$$ | |
| - | ### 13.3.2 Candidate and interpolated state |
| + |  |
| + | |
| + | *The cell so far: just two gates on the input rail, the state path still grayed out.* |
| + | |
| + | ### 8.3.2 Candidate and interpolated state |
| The candidate uses a reset-gated version of the previous hidden state, and the new state is a gated interpolation between the old state and the candidate: | |
| @@ 71,7 78,11 @@ | |
| *Remark:* the interpolation form ties the keep and write fractions together with a single gate: whatever weight $z_t$ gives the candidate, $1 - z_t$ is left for the past. The LSTM sets its keep fraction $f_t$ and write fraction $i_t$ independently, which is one more gate and one more matrix. | |
| - | ## 13.4 LSTM versus GRU |
| + |  |
| + | |
| + | *One GRU step. The reset gate $r_t$ filters how much of the past feeds the candidate, then the update gate splits the state between keeping ($1 - z_t$) and refreshing ($z_t$). One state line, two gates, same additive path.* |
| + | |
| + | ## 8.4 LSTM versus GRU |
| Both cells solve the vanishing-gradient problem with an additive state path. They differ in how many gates carry that path and whether the long-term memory is kept separate from the exposed state. | |
| @@ 85,17 96,7 @@ | |
| *Remark:* in practice the two often reach comparable accuracy. The GRU trains faster and generalizes well on smaller datasets, while the extra capacity of the LSTM can help on very long sequences. Treat the choice as a tunable hyperparameter rather than a settled rule. | |
| - | ## 13.5 Anatomy of a gated cell |
| - | |
| - | The diagram traces one LSTM step: the previous cell state enters on the additive path, the gates modulate what is forgotten, written, and exposed, and the outputs feed the next step. |
| - | |
| - |  |
| - | |
| - | *The LSTM cell carries a cell state along the top, edited by a forget multiply and an input add, with sigmoid gates controlling the flow.* |
| - | |
| - | *Remark:* the horizontal path from previous cell state to new cell state is the carousel, and it carries no full matrix multiply, only the elementwise gate products. |
| - | |
| *Gates let a recurrent state persist over long spans, but they still read one step at a time. The next part lets every position attend directly to every other, removing the sequential bottleneck.* | |
| --- | |
| - | Next: [Attention](/en/Deep%20Learning/14%20Attention) · [Course overview](/en/Deep%20Learning) |
| + | Next: [Attention](/en/Deep%20Learning/09%20Attention) · [Course overview](/en/Deep%20Learning) |
| /dev/null .. en/Deep Learning/08 LSTM and GRU/gru-cell.svg | |
| @@ 0,0 1,24 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 400" width="900" height="400" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker></defs> |
| + | <rect width="900" height="400" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">Inside the GRU cell</text> |
| + | <rect x="50" y="46" width="720" height="290" rx="14" fill="#f9fafb" stroke="#c7d0d9" stroke-width="1.2"/> |
| + | <line x1="20" y1="95" x2="247" y2="95" stroke="#1f2933" stroke-width="2.2"/> |
| + | <line x1="273" y1="95" x2="547" y2="95" stroke="#1f2933" stroke-width="2.2"/> |
| + | <line x1="573" y1="95" x2="795" y2="95" stroke="#1f2933" stroke-width="2.2"/> |
| + | <line x1="780" y1="95" x2="800" y2="95" stroke="#1f2933" stroke-width="1.8" marker-end="url(#arrow)"/> |
| + | <text x="28" y="80" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text><text x="822" y="99" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <circle cx="260" cy="95" r="13" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="260" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><text x="260" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">keep</text> |
| + | <circle cx="560" cy="95" r="13" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="560" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">+</text><text x="560" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">update</text> |
| + | <line x1="120" y1="95" x2="120" y2="165" stroke="#1f2933" stroke-width="1.5"/><line x1="120" y1="165" x2="155" y2="165" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><circle cx="170" cy="165" r="12" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="170" y="170" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text> |
| + | <line x1="183" y1="165" x2="300" y2="165" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="242" y="156" font-size="9" fill="#5b6b7b" text-anchor="middle">r<tspan baseline-shift="sub" font-size="9px">t</tspan> ⊙ h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text> |
| + | <rect x="300" y="148" width="96" height="34" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="348.0" y="163" font-size="12" fill="#1f2933" text-anchor="middle">tanh</text><text x="348.0" y="176" font-size="9" fill="#5b6b7b" text-anchor="middle">candidate</text><line x1="396" y1="165" x2="465" y2="165" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="430" y="156" font-size="11" fill="#1f2933" text-anchor="middle">h̃<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <circle cx="480" cy="165" r="12" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="480" y="170" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><line x1="489" y1="155" x2="551" y2="105" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <path d="M440,231 L260,231 L260,110" fill="none" stroke="#1f2933" stroke-width="1.5"/><line x1="260" y1="120" x2="260" y2="110" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="272" y="195" font-size="11" fill="#1f2933" text-anchor="start">1 − z<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="130" y="215" width="80" height="32" rx="8" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="170.0" y="230" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">r</tspan></text><text x="170.0" y="243" font-size="9" fill="#5b6b7b" text-anchor="middle">reset gate</text><line x1="170" y1="215" x2="170" y2="180" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="182" y="200" font-size="11" fill="#1f2933" text-anchor="start">r<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="440" y="215" width="80" height="32" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="480.0" y="230" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">z</tspan></text><text x="480.0" y="243" font-size="9" fill="#5b6b7b" text-anchor="middle">update gate</text><line x1="480" y1="215" x2="480" y2="180" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="492" y="202" font-size="11" fill="#1f2933" text-anchor="start">z<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <line x1="20" y1="288" x2="620" y2="288" stroke="#5b6b7b" stroke-width="1.6"/><line x1="170" y1="288" x2="170" y2="251" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="480" y1="288" x2="480" y2="251" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><text x="90" y="279" font-size="11" fill="#1f2933" text-anchor="middle">[h<tspan baseline-shift="sub" font-size="9px">t-1</tspan>, x<tspan baseline-shift="sub" font-size="9px">t</tspan>]</text> |
| + | <line x1="348" y1="288" x2="348" y2="186" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/> |
| + | <line x1="560" y1="356" x2="560" y2="292" stroke="#1f2933" stroke-width="1.6" marker-end="url(#arrow)"/><text x="560" y="372" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">x<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <text x="450" y="392" font-size="11" fill="#5b6b7b" text-anchor="middle">σ: gate in (0, 1) · ⊙: elementwise product · +: elementwise add</text> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. en/Deep Learning/08 LSTM and GRU/gru-step1.svg | |
| @@ 0,0 1,24 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 400" width="900" height="400" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker></defs> |
| + | <rect width="900" height="400" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">The GRU cell, step 1: the gates</text> |
| + | <rect x="50" y="46" width="720" height="290" rx="14" fill="#f9fafb" stroke="#c7d0d9" stroke-width="1.2"/> |
| + | <g opacity="0.22"><line x1="20" y1="95" x2="247" y2="95" stroke="#1f2933" stroke-width="2.2"/></g> |
| + | <g opacity="0.22"><line x1="273" y1="95" x2="547" y2="95" stroke="#1f2933" stroke-width="2.2"/></g> |
| + | <g opacity="0.22"><line x1="573" y1="95" x2="795" y2="95" stroke="#1f2933" stroke-width="2.2"/></g> |
| + | <g opacity="0.22"><line x1="780" y1="95" x2="800" y2="95" stroke="#1f2933" stroke-width="1.8" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><text x="28" y="80" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text><text x="822" y="99" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t</tspan></text></g> |
| + | <g opacity="0.22"><circle cx="260" cy="95" r="13" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="260" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><text x="260" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">keep</text></g> |
| + | <g opacity="0.22"><circle cx="560" cy="95" r="13" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="560" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">+</text><text x="560" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">update</text></g> |
| + | <g opacity="0.22"><line x1="120" y1="95" x2="120" y2="165" stroke="#1f2933" stroke-width="1.5"/><line x1="120" y1="165" x2="155" y2="165" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><circle cx="170" cy="165" r="12" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="170" y="170" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text></g> |
| + | <g opacity="0.22"><line x1="183" y1="165" x2="300" y2="165" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="242" y="156" font-size="9" fill="#5b6b7b" text-anchor="middle">r<tspan baseline-shift="sub" font-size="9px">t</tspan> ⊙ h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text></g> |
| + | <g opacity="0.22"><rect x="300" y="148" width="96" height="34" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="348.0" y="163" font-size="12" fill="#1f2933" text-anchor="middle">tanh</text><text x="348.0" y="176" font-size="9" fill="#5b6b7b" text-anchor="middle">candidate</text><line x1="396" y1="165" x2="465" y2="165" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="430" y="156" font-size="11" fill="#1f2933" text-anchor="middle">h̃<tspan baseline-shift="sub" font-size="9px">t</tspan></text></g> |
| + | <g opacity="0.22"><circle cx="480" cy="165" r="12" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="480" y="170" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><line x1="489" y1="155" x2="551" y2="105" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><path d="M440,231 L260,231 L260,110" fill="none" stroke="#1f2933" stroke-width="1.5"/><line x1="260" y1="120" x2="260" y2="110" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="272" y="195" font-size="11" fill="#1f2933" text-anchor="start">1 − z<tspan baseline-shift="sub" font-size="9px">t</tspan></text></g> |
| + | <rect x="130" y="215" width="80" height="32" rx="8" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="170.0" y="230" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">r</tspan></text><text x="170.0" y="243" font-size="9" fill="#5b6b7b" text-anchor="middle">reset gate</text><line x1="170" y1="215" x2="170" y2="180" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="182" y="200" font-size="11" fill="#1f2933" text-anchor="start">r<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="440" y="215" width="80" height="32" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="480.0" y="230" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">z</tspan></text><text x="480.0" y="243" font-size="9" fill="#5b6b7b" text-anchor="middle">update gate</text><line x1="480" y1="215" x2="480" y2="180" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="492" y="202" font-size="11" fill="#1f2933" text-anchor="start">z<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <line x1="20" y1="288" x2="620" y2="288" stroke="#5b6b7b" stroke-width="1.6"/><line x1="170" y1="288" x2="170" y2="251" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="480" y1="288" x2="480" y2="251" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><text x="90" y="279" font-size="11" fill="#1f2933" text-anchor="middle">[h<tspan baseline-shift="sub" font-size="9px">t-1</tspan>, x<tspan baseline-shift="sub" font-size="9px">t</tspan>]</text> |
| + | <g opacity="0.22"><line x1="348" y1="288" x2="348" y2="186" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/></g> |
| + | <line x1="560" y1="356" x2="560" y2="292" stroke="#1f2933" stroke-width="1.6" marker-end="url(#arrow)"/><text x="560" y="372" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">x<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <text x="450" y="392" font-size="11" fill="#5b6b7b" text-anchor="middle">σ: gate in (0, 1) · ⊙: elementwise product · +: elementwise add</text> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. en/Deep Learning/08 LSTM and GRU/lstm-cell.svg | |
| @@ 0,0 1,26 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 410" width="900" height="410" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker></defs> |
| + | <rect width="900" height="410" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">Inside the LSTM cell</text> |
| + | <rect x="50" y="46" width="720" height="296" rx="14" fill="#f9fafb" stroke="#c7d0d9" stroke-width="1.2"/> |
| + | <line x1="20" y1="95" x2="237" y2="95" stroke="#1f2933" stroke-width="2.2"/> |
| + | <line x1="263" y1="95" x2="477" y2="95" stroke="#1f2933" stroke-width="2.2"/> |
| + | <line x1="503" y1="95" x2="795" y2="95" stroke="#1f2933" stroke-width="2.2"/> |
| + | <line x1="780" y1="95" x2="800" y2="95" stroke="#1f2933" stroke-width="1.8" marker-end="url(#arrow)"/> |
| + | <text x="28" y="80" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">c<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text> |
| + | <text x="822" y="99" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">c<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <circle cx="250" cy="95" r="13" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="250" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><text x="250" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">forget</text> |
| + | <circle cx="490" cy="95" r="13" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="490" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">+</text><text x="490" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">write</text> |
| + | <circle cx="490" cy="160" r="12" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="490" y="165" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><line x1="490" y1="148" x2="490" y2="110" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <rect x="470" y="200" width="90" height="34" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="515.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">tanh</text><text x="515.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">candidate</text><line x1="515" y1="200" x2="497" y2="174" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="526" y="190" font-size="11" fill="#1f2933" text-anchor="start">c̃<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <line x1="600" y1="95" x2="600" y2="138" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><rect x="558" y="140" width="84" height="34" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="600.0" y="155" font-size="12" fill="#1f2933" text-anchor="middle">tanh</text><text x="600.0" y="168" font-size="9" fill="#5b6b7b" text-anchor="middle"></text> |
| + | <line x1="642" y1="157" x2="685" y2="157" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><circle cx="700" cy="157" r="13" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="700" y="162" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><line x1="713" y1="157" x2="795" y2="157" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="822" y="161" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="210" y="200" width="80" height="34" rx="8" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="250.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">f</tspan></text><text x="250.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">forget gate</text><line x1="250" y1="200" x2="250" y2="110" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="240" y="152" font-size="11" fill="#1f2933" text-anchor="end">f<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="348" y="200" width="80" height="34" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="388.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">i</tspan></text><text x="388.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">input gate</text><line x1="388" y1="198" x2="477" y2="167" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="420" y="175" font-size="11" fill="#1f2933" text-anchor="end">i<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="660" y="200" width="80" height="34" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="700.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">o</tspan></text><text x="700.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">output gate</text><line x1="700" y1="200" x2="700" y2="172" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="712" y="190" font-size="11" fill="#1f2933" text-anchor="start">o<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <line x1="70" y1="292" x2="700" y2="292" stroke="#5b6b7b" stroke-width="1.6"/><line x1="250" y1="292" x2="250" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="388" y1="292" x2="388" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="700" y1="292" x2="700" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><text x="150" y="283" font-size="11" fill="#1f2933" text-anchor="middle">[h<tspan baseline-shift="sub" font-size="9px">t-1</tspan>, x<tspan baseline-shift="sub" font-size="9px">t</tspan>]</text> |
| + | <line x1="515" y1="292" x2="515" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/> |
| + | <line x1="20" y1="292" x2="70" y2="292" stroke="#5b6b7b" stroke-width="1.6"/><text x="28" y="277" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text> |
| + | <line x1="130" y1="360" x2="130" y2="296" stroke="#1f2933" stroke-width="1.6" marker-end="url(#arrow)"/><text x="130" y="376" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">x<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <text x="450" y="398" font-size="11" fill="#5b6b7b" text-anchor="middle">σ: gate in (0, 1) · ⊙: elementwise product · +: elementwise add</text> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. en/Deep Learning/08 LSTM and GRU/lstm-step1.svg | |
| @@ 0,0 1,26 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 410" width="900" height="410" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker></defs> |
| + | <rect width="900" height="410" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">The LSTM cell, step 1: the gates</text> |
| + | <rect x="50" y="46" width="720" height="296" rx="14" fill="#f9fafb" stroke="#c7d0d9" stroke-width="1.2"/> |
| + | <g opacity="0.22"><line x1="20" y1="95" x2="237" y2="95" stroke="#1f2933" stroke-width="2.2"/></g> |
| + | <g opacity="0.22"><line x1="263" y1="95" x2="477" y2="95" stroke="#1f2933" stroke-width="2.2"/></g> |
| + | <g opacity="0.22"><line x1="503" y1="95" x2="795" y2="95" stroke="#1f2933" stroke-width="2.2"/></g> |
| + | <g opacity="0.22"><line x1="780" y1="95" x2="800" y2="95" stroke="#1f2933" stroke-width="1.8" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><text x="28" y="80" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">c<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text></g> |
| + | <g opacity="0.22"><text x="822" y="99" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">c<tspan baseline-shift="sub" font-size="9px">t</tspan></text></g> |
| + | <g opacity="0.22"><circle cx="250" cy="95" r="13" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="250" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><text x="250" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">forget</text></g> |
| + | <g opacity="0.22"><circle cx="490" cy="95" r="13" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="490" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">+</text><text x="490" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">write</text></g> |
| + | <g opacity="0.22"><circle cx="490" cy="160" r="12" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="490" y="165" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><line x1="490" y1="148" x2="490" y2="110" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><rect x="470" y="200" width="90" height="34" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="515.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">tanh</text><text x="515.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">candidate</text><line x1="515" y1="200" x2="497" y2="174" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="526" y="190" font-size="11" fill="#1f2933" text-anchor="start">c̃<tspan baseline-shift="sub" font-size="9px">t</tspan></text></g> |
| + | <g opacity="0.22"><line x1="600" y1="95" x2="600" y2="138" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><rect x="558" y="140" width="84" height="34" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="600.0" y="155" font-size="12" fill="#1f2933" text-anchor="middle">tanh</text><text x="600.0" y="168" font-size="9" fill="#5b6b7b" text-anchor="middle"></text></g> |
| + | <g opacity="0.22"><line x1="642" y1="157" x2="685" y2="157" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><circle cx="700" cy="157" r="13" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="700" y="162" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><line x1="713" y1="157" x2="795" y2="157" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="822" y="161" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t</tspan></text></g> |
| + | <rect x="210" y="200" width="80" height="34" rx="8" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="250.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">f</tspan></text><text x="250.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">forget gate</text><line x1="250" y1="200" x2="250" y2="110" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="240" y="152" font-size="11" fill="#1f2933" text-anchor="end">f<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="348" y="200" width="80" height="34" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="388.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">i</tspan></text><text x="388.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">input gate</text><line x1="388" y1="198" x2="477" y2="167" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="420" y="175" font-size="11" fill="#1f2933" text-anchor="end">i<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="660" y="200" width="80" height="34" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="700.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">o</tspan></text><text x="700.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">output gate</text><line x1="700" y1="200" x2="700" y2="172" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="712" y="190" font-size="11" fill="#1f2933" text-anchor="start">o<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <line x1="70" y1="292" x2="700" y2="292" stroke="#5b6b7b" stroke-width="1.6"/><line x1="250" y1="292" x2="250" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="388" y1="292" x2="388" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="700" y1="292" x2="700" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><text x="150" y="283" font-size="11" fill="#1f2933" text-anchor="middle">[h<tspan baseline-shift="sub" font-size="9px">t-1</tspan>, x<tspan baseline-shift="sub" font-size="9px">t</tspan>]</text> |
| + | <g opacity="0.22"><line x1="515" y1="292" x2="515" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/></g> |
| + | <line x1="20" y1="292" x2="70" y2="292" stroke="#5b6b7b" stroke-width="1.6"/><text x="28" y="277" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text> |
| + | <line x1="130" y1="360" x2="130" y2="296" stroke="#1f2933" stroke-width="1.6" marker-end="url(#arrow)"/><text x="130" y="376" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">x<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <text x="450" y="398" font-size="11" fill="#5b6b7b" text-anchor="middle">σ: gate in (0, 1) · ⊙: elementwise product · +: elementwise add</text> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. en/Deep Learning/08 LSTM and GRU/lstm-step2.svg | |
| @@ 0,0 1,26 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 410" width="900" height="410" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker></defs> |
| + | <rect width="900" height="410" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">The LSTM cell, step 2: editing the carousel</text> |
| + | <rect x="50" y="46" width="720" height="296" rx="14" fill="#f9fafb" stroke="#c7d0d9" stroke-width="1.2"/> |
| + | <line x1="20" y1="95" x2="237" y2="95" stroke="#1f2933" stroke-width="2.2"/> |
| + | <line x1="263" y1="95" x2="477" y2="95" stroke="#1f2933" stroke-width="2.2"/> |
| + | <line x1="503" y1="95" x2="795" y2="95" stroke="#1f2933" stroke-width="2.2"/> |
| + | <line x1="780" y1="95" x2="800" y2="95" stroke="#1f2933" stroke-width="1.8" marker-end="url(#arrow)"/> |
| + | <text x="28" y="80" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">c<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text> |
| + | <text x="822" y="99" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">c<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <circle cx="250" cy="95" r="13" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="250" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><text x="250" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">forget</text> |
| + | <circle cx="490" cy="95" r="13" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="490" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">+</text><text x="490" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">write</text> |
| + | <circle cx="490" cy="160" r="12" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="490" y="165" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><line x1="490" y1="148" x2="490" y2="110" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <rect x="470" y="200" width="90" height="34" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="515.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">tanh</text><text x="515.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">candidate</text><line x1="515" y1="200" x2="497" y2="174" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="526" y="190" font-size="11" fill="#1f2933" text-anchor="start">c̃<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <g opacity="0.22"><line x1="600" y1="95" x2="600" y2="138" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><rect x="558" y="140" width="84" height="34" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="600.0" y="155" font-size="12" fill="#1f2933" text-anchor="middle">tanh</text><text x="600.0" y="168" font-size="9" fill="#5b6b7b" text-anchor="middle"></text></g> |
| + | <g opacity="0.22"><line x1="642" y1="157" x2="685" y2="157" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><circle cx="700" cy="157" r="13" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="700" y="162" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><line x1="713" y1="157" x2="795" y2="157" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="822" y="161" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t</tspan></text></g> |
| + | <rect x="210" y="200" width="80" height="34" rx="8" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="250.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">f</tspan></text><text x="250.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">forget gate</text><line x1="250" y1="200" x2="250" y2="110" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="240" y="152" font-size="11" fill="#1f2933" text-anchor="end">f<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="348" y="200" width="80" height="34" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="388.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">i</tspan></text><text x="388.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">input gate</text><line x1="388" y1="198" x2="477" y2="167" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="420" y="175" font-size="11" fill="#1f2933" text-anchor="end">i<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="660" y="200" width="80" height="34" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="700.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">o</tspan></text><text x="700.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">output gate</text><line x1="700" y1="200" x2="700" y2="172" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="712" y="190" font-size="11" fill="#1f2933" text-anchor="start">o<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <line x1="70" y1="292" x2="700" y2="292" stroke="#5b6b7b" stroke-width="1.6"/><line x1="250" y1="292" x2="250" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="388" y1="292" x2="388" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="700" y1="292" x2="700" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><text x="150" y="283" font-size="11" fill="#1f2933" text-anchor="middle">[h<tspan baseline-shift="sub" font-size="9px">t-1</tspan>, x<tspan baseline-shift="sub" font-size="9px">t</tspan>]</text> |
| + | <line x1="515" y1="292" x2="515" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/> |
| + | <line x1="20" y1="292" x2="70" y2="292" stroke="#5b6b7b" stroke-width="1.6"/><text x="28" y="277" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text> |
| + | <line x1="130" y1="360" x2="130" y2="296" stroke="#1f2933" stroke-width="1.6" marker-end="url(#arrow)"/><text x="130" y="376" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">x<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <text x="450" y="398" font-size="11" fill="#5b6b7b" text-anchor="middle">σ: gate in (0, 1) · ⊙: elementwise product · +: elementwise add</text> |
| + | </svg> |
| \ | No newline at end of file |
| en/Deep Learning/08 Normalization.md .. /dev/null | |
| @@ 1,122 0,0 @@ | |
| - | # 8. Normalization |
| - | |
| - | Deep networks train faster and more reliably when the activations flowing between layers stay well-scaled. This lesson introduces normalization layers, which standardize a layer's inputs on the fly, then learn to rescale them. We cover batch normalization and layer normalization, where each computes its statistics, how they behave at inference, and where to place them. |
| - | |
| - | **Objectives** |
| - | - Explain why normalizing activations inside the network stabilizes and accelerates training. |
| - | - Derive the batch normalization transform: normalize, then scale and shift with learned $\gamma, \beta$. |
| - | - Understand why running (moving-average) statistics replace batch statistics at inference. |
| - | - Define layer normalization and see why it suits recurrent networks and Transformers. |
| - | - Decide where to place a normalization layer relative to the activation $g^{[l]}$. |
| - | - Compare batch and layer normalization along their normalization axis and use cases. |
| - | |
| - | ## 8.1 Why normalize inside the network |
| - | |
| - | Recall a layer computes $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$ and $a^{[l]} = g^{[l]}(z^{[l]})$. As training updates every $W^{[l]}$, the distribution of each layer's input $a^{[l-1]}$ keeps shifting. This moving target, sometimes called internal covariate shift, forces later layers to constantly re-adapt and slows the whole network down. |
| - | |
| - | Normalizing the activations at each layer keeps their mean and variance stable across updates. The immediate benefits: |
| - | |
| - | - The loss surface becomes smoother, so we can use a higher learning rate without diverging. |
| - | - Training converges in fewer epochs and is less sensitive to the weight initialization. |
| - | - The learned scale and shift give the network back the freedom to undo the normalization if that helps. |
| - | |
| - | *Remark:* normalization is applied to the pre-activation $z^{[l]}$ or the activation $a^{[l]}$, not to the parameters. It is a layer inserted into the forward pass, with its own learnable parameters. |
| - | |
| - | ## 8.2 Batch normalization |
| - | |
| - | Batch normalization (BatchNorm) standardizes each feature across the examples of a mini-batch, then applies a learned affine transform. It operates per feature, so every feature keeps its own statistics. |
| - | |
| - |  |
| - | |
| - | *Normalization recenters and rescales a layer input to zero mean and unit variance before the learned scale and shift.* |
| - | |
| - | ### 8.2.1 Batch statistics |
| - | |
| - | For a feature $x$ over a mini-batch $\mathcal{B} = \{x^{(1)}, \dots, x^{(m)}\}$ of size $m$, compute the batch mean and variance: |
| - | |
| - | $$\boxed{ \mu_\mathcal{B} = \frac{1}{m}\sum_{i=1}^{m} x^{(i)}, \qquad \sigma_\mathcal{B}^2 = \frac{1}{m}\sum_{i=1}^{m}\left(x^{(i)} - \mu_\mathcal{B}\right)^2 }$$ |
| - | |
| - | ### 8.2.2 Normalize, scale, and shift |
| - | |
| - | Standardize each value to zero mean and unit variance, using a small constant $\epsilon > 0$ for numerical stability: |
| - | |
| - | $$\boxed{ \hat{x}^{(i)} = \frac{x^{(i)} - \mu_\mathcal{B}}{\sqrt{\sigma_\mathcal{B}^2 + \epsilon}} }$$ |
| - | |
| - | Then rescale with two learned parameters per feature, a scale $\gamma$ and a shift $\beta$: |
| - | |
| - | $$\boxed{ y^{(i)} = \gamma\, \hat{x}^{(i)} + \beta }$$ |
| - | |
| - | *Remark:* $\gamma$ and $\beta$ are learned by gradient descent like any weight. If the optimal behaviour is the raw input, the network can recover it by learning $\gamma = \sqrt{\sigma_\mathcal{B}^2 + \epsilon}$ and $\beta = \mu_\mathcal{B}$. Normalization never removes capacity, it only reparameterizes it. |
| - | |
| - | ### 8.2.3 Inference with running statistics |
| - | |
| - | At inference we often score a single example, so a batch mean and variance are undefined or meaningless. Instead BatchNorm uses population estimates accumulated during training as exponential moving averages, with momentum $\alpha \in [0, 1)$: |
| - | |
| - | $$\boxed{ \mu \leftarrow \alpha\, \mu + (1 - \alpha)\, \mu_\mathcal{B}, \qquad \sigma^2 \leftarrow \alpha\, \sigma^2 + (1 - \alpha)\, \sigma_\mathcal{B}^2 }$$ |
| - | |
| - | At test time the transform is fixed and deterministic, using these running statistics in place of the batch ones: |
| - | |
| - | $$\boxed{ y = \gamma\, \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta }$$ |
| - | |
| - | *Remark:* this train/inference split is the source of most BatchNorm bugs. Forgetting to switch the layer to evaluation mode leaves it computing batch statistics at test time, which corrupts predictions. |
| - | |
| - | ## 8.3 Layer normalization |
| - | |
| - | Layer normalization (LayerNorm) keeps the same normalize-scale-shift recipe but changes the axis it averages over. Rather than pooling across the batch, it computes the statistics over the features of a single example. Each example is therefore normalized on its own, independent of the others in the batch. |
| - | |
| - | ### 8.3.1 Per-example statistics |
| - | |
| - | For one example with feature vector $a \in \mathbb{R}^{H}$ (its $H$ activations in a layer), average over the features: |
| - | |
| - | $$\boxed{ \mu = \frac{1}{H}\sum_{k=1}^{H} a_k, \qquad \sigma^2 = \frac{1}{H}\sum_{k=1}^{H}\left(a_k - \mu\right)^2 }$$ |
| - | |
| - | The normalization, scale, and shift are identical in form to BatchNorm, applied per example: |
| - | |
| - | $$\boxed{ \hat{a}_k = \frac{a_k - \mu}{\sqrt{\sigma^2 + \epsilon}}, \qquad y_k = \gamma_k\, \hat{a}_k + \beta_k }$$ |
| - | |
| - | ### 8.3.2 Why LayerNorm for sequences |
| - | |
| - | Because the statistics come from a single example, LayerNorm behaves the same in training and inference, and it does not depend on the batch size. This matters when the batch is tiny or when examples have variable length, as in text. LayerNorm is the normalization of choice for recurrent networks and Transformers, where the sequence length varies and a per-timestep batch mean would be ill-defined. |
| - | |
| - | *Remark:* LayerNorm needs no running statistics, so there is no train/inference discrepancy to manage. This alone makes it simpler to deploy than BatchNorm. |
| - | |
| - | ## 8.4 Placement and practical effects |
| - | |
| - | A normalization layer sits between the linear step $W^{[l]} a^{[l-1]} + b^{[l]}$ and the nonlinearity $g^{[l]}$. Two orderings are common. |
| - | |
| - |  |
| - | |
| - | *Normalization is inserted between the linear map and the activation inside each layer.* |
| - | |
| - | - **Before activation** (normalize $z^{[l]}$, then apply $g^{[l]}$): the original and most common placement. It keeps the input to the nonlinearity centred, which is where saturation hurts most. |
| - | - **After activation** (normalize $a^{[l]}$): sometimes used and occasionally better in practice, though it is less standard. |
| - | |
| - | Two more practical points: |
| - | |
| - | - **Bias becomes redundant.** The shift $\beta$ replaces the layer bias, since normalization subtracts the mean and would cancel $b^{[l]}$ anyway. Layers followed by normalization are often written without their own bias. |
| - | - **BatchNorm depends on batch size.** Its statistics are noisier with small batches, which acts as a mild regularizer but degrades badly when the batch is very small. LayerNorm is immune to this, which is another reason sequence models prefer it. |
| - | |
| - | *Remark:* the batch-dependent noise in BatchNorm can partly substitute for other regularizers, so networks using it sometimes need less dropout. |
| - | |
| - | ## 8.5 BatchNorm versus LayerNorm |
| - | |
| - | The two layers share the normalize-scale-shift transform and differ only in the axis of the statistics and the consequences that follow. |
| - | |
| - | | Aspect | Batch normalization | Layer normalization | |
| - | | --- | --- | --- | |
| - | | Normalization axis | across the batch, per feature | across the features, per example | |
| - | | Depends on batch size | yes | no | |
| - | | Train vs inference | batch statistics vs running statistics | identical in both | |
| - | | Running statistics needed | yes | no | |
| - | | Typical use | CNNs and feedforward vision models | RNNs and Transformers | |
| - | |
| - |  |
| - | |
| - | *Batch normalization computes statistics down a feature column across the batch, layer normalization across the features of a single example.* |
| - | |
| - | *Remark:* the Transformer block in lesson 16 places a LayerNorm before or after each sublayer, precisely because it removes the batch dependence that would otherwise couple examples of different lengths. |
| - | |
| - | *With activations kept well-scaled, the network trains stably at higher learning rates. The next lesson turns to controlling overfitting through regularization and dropout.* |
| - | |
| - | --- |
| - | Next: [Regularization and dropout](/en/Deep%20Learning/09%20Regularization%20and%20dropout) · [Course overview](/en/Deep%20Learning) |
| en/Deep Learning/08 Normalization/norm-placement.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 760 200" width="760" height="200" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="760" height="200" fill="#ffffff"/><text x="380.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">Normalization inside a layer</text><rect x="40.0" y="90.0" width="150.0" height="60.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="115.0" y="124.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">linear W a + b</text><rect x="210.0" y="90.0" width="150.0" height="60.0" rx="8" fill="#e4f5f4" stroke="#2a9d9a" stroke-width="1.6"/><text x="285.0" y="124.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">normalization</text><rect x="380.0" y="90.0" width="150.0" height="60.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="455.0" y="124.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">activation g</text><rect x="550.0" y="90.0" width="150.0" height="60.0" rx="8" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><text x="625.0" y="124.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">next layer</text><line x1="190.0" y1="120.0" x2="210.0" y2="120.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="360.0" y1="120.0" x2="380.0" y2="120.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="530.0" y1="120.0" x2="550.0" y2="120.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><text x="285.0" y="176.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">recenter and rescale z</text></svg> |
| \ | No newline at end of file |
| en/Deep Learning/08 Normalization/normalization-effect.png .. /dev/null | |
| en/Deep Learning/14 Attention.md .. en/Deep Learning/09 Attention.md | |
| @@ 1,17 1,10 @@ | |
| - | # 14. Attention |
| + | # 9. Attention |
| Recurrent encoder-decoder models push a whole input sequence through a single fixed context vector, which caps how much they can remember for long inputs. Attention removes that bottleneck by letting the decoder read every encoder state directly, weighting each one by how relevant it is to the current output step. This lesson builds the mechanism from alignment scores to the query-key-value view, which is the foundation the Transformer will generalize. | |
| - | **Objectives** |
| - | - Explain why the fixed context vector is a bottleneck in sequence-to-sequence models. |
| - | - Define alignment scores, attention weights, and the context vector. |
| - | - Contrast the additive (Bahdanau) and multiplicative (Luong) score functions. |
| - | - Recast attention as a query attending over keys and values. |
| - | - Connect this framing to self-attention and the Transformer. |
| + | ## 9.1 The seq2seq bottleneck |
| - | ## 14.1 The seq2seq bottleneck |
| - | |
| - | A sequence-to-sequence model uses an encoder recurrent network to read the input tokens $x_1, \dots, x_T$ into hidden states $h_1, \dots, h_T$, then a decoder recurrent network to emit the output tokens. In the vanilla design the decoder is initialised from a single context vector, the encoder's last hidden state: |
| + | Take a running example: translating the French "nous mangeons du pain" into "we are eating bread". A sequence-to-sequence model uses an encoder recurrent network to read the input tokens $x_1, \dots, x_T$ into hidden states $h_1, \dots, h_T$, then a decoder recurrent network to emit the output tokens, one per step. In the vanilla design the decoder is initialised from a single context vector, the encoder's last hidden state: |
| $$\boxed{ c = h_T }$$ | |
| @@ 19,21 12,17 @@ | |
| *Remark:* this is a genuine information bottleneck. For a short sentence $h_T$ can hold enough, but as $T$ grows the early tokens are overwritten and translation or summarisation quality drops sharply on long inputs. | |
| - |  |
| + |  |
| *Plain sequence-to-sequence squeezes the whole input into one fixed context vector, a bottleneck for long sequences.* | |
| The fix is to keep all encoder states $h_1, \dots, h_T$ available and let the decoder decide, at each step, which of them to read. | |
| - | ## 14.2 The attention mechanism |
| + | ## 9.2 The attention mechanism |
| Instead of one context vector shared across all steps, attention builds a fresh context vector $c_i$ for each decoder step $i$. It does this in three stages: score, normalise, combine. | |
| - |  |
| - | |
| - | *Attention scores each encoder state against the decoder query, then forms the context as a weighted sum of all states.* |
| - | |
| - | ### 14.2.1 Alignment scores |
| + | ### 9.2.1 Alignment scores |
| For decoder step $i$ with state $s_i$, a score function measures how well that state aligns with each encoder state $h_j$: | |
| @@ 43,7 32,11 @@ | |
| *Remark:* $s_i$ is usually the decoder state just before emitting token $i$, so the model chooses what to look at using what it has produced so far. | |
| - | ### 14.2.2 Attention weights |
| + |  |
| + | |
| + | *Step 1: the decoder state $s_i$ is scored against every encoder state $h_j$, one score per input position. Everything downstream is still grayed out.* |
| + | |
| + | ### 9.2.2 Attention weights |
| The scores are turned into a probability distribution over input positions with a softmax across $j$: | |
| @@ 51,7 44,11 @@ | |
| Each $\alpha_{i,j} \in (0,1)$ and $\sum_j \alpha_{i,j} = 1$, so the weights say how much of the decoder's attention at step $i$ goes to input position $j$. | |
| - | ### 14.2.3 Context vector |
| + |  |
| + | |
| + | *Step 2: the softmax turns the $T$ scores into weights $\alpha_{i,j}$ that sum to one, one bar per input position.* |
| + | |
| + | ### 9.2.3 Context vector |
| The context vector for step $i$ is the weighted average of the encoder states, using the attention weights: | |
| @@ 59,13 56,17 @@ | |
| This $c_i$ is recomputed at every decoder step, so the model reads a different mixture of the input for each output token. The decoder then combines $c_i$ with its state $s_i$ to predict the token, and the alignment weights $\alpha_{i,j}$ can be visualised as a soft matrix that shows which input words each output word attends to. | |
| - | *Remark:* because every step averages over all $h_j$, no single fixed vector has to carry the whole input. The bottleneck of 15.1 is gone, and long inputs no longer degrade so quickly. |
| + | *Remark:* because every step averages over all $h_j$, no single fixed vector has to carry the whole input. The bottleneck of section 9.1 is gone, and long inputs no longer degrade so quickly. |
| + | |
| + |  |
| + | |
| + | *The full mechanism: score, normalise, combine. The context $c_i$ is the $\alpha$-weighted average of the encoder states, rebuilt at every decoder step.* |
| - | ## 14.3 Score functions |
| + | ## 9.3 Score functions |
| - | The score function in 15.2.1 is a design choice. Two forms dominate the early attention literature. |
| + | The score function in section 9.2.1 is a design choice. Two forms dominate the early attention literature. |
| - | ### 14.3.1 Additive (Bahdanau) score |
| + | ### 9.3.1 Additive (Bahdanau) score |
| The additive score, from Bahdanau and co-authors, feeds the two states through a small one-hidden-layer network with learned matrices $W_1$ and $W_2$ and a learned vector $v$: | |
| @@ 73,7 74,7 @@ | |
| It works even when $s_i$ and $h_j$ have different dimensions, since $W_1$ and $W_2$ project both into a shared space before the $\tanh$. | |
| - | ### 14.3.2 Multiplicative (Luong) score |
| + | ### 9.3.2 Multiplicative (Luong) score |
| The multiplicative score, from Luong and co-authors, is a plain dot product between the two states: | |
| @@ 81,7 82,7 @@ | |
| It has no extra parameters in its simplest form and is far cheaper to compute, since a whole matrix of scores is a single matrix multiplication. A general variant inserts a learned matrix $W$ as $s_i^{\top} W h_j$ to handle mismatched dimensions. | |
| - | ### 14.3.3 Which to use |
| + | ### 9.3.3 Which to use |
| | Aspect | Additive (Bahdanau) | Multiplicative (Luong) | | |
| | --- | --- | --- | | |
| @@ 93,11 94,11 @@ | |
| *Remark:* the dot product grows with the dimension of the states, so at large widths its variance gets big and pushes the softmax into flat regions. Scaling the score by $1/\sqrt{d}$ fixes this, and that scaled dot product is exactly what the Transformer will adopt. | |
| - | ## 14.4 Query, key, value |
| + | ## 9.4 Query, key, value |
| Attention has a cleaner reading that drops the encoder-decoder framing. Rename the pieces: the state that does the looking is a query, and each thing that can be looked at contributes a key (used for scoring) and a value (used in the sum). | |
| - |  |
| + |  |
| *An attention weight matrix: each output token draws mostly from a few input tokens.* | |
| @@ 114,4 115,4 @@ | |
| *Building the query, key, and value from one sequence with learned projections turns attention into a general sequence layer, which is exactly where the next lesson on Transformers begins.* | |
| --- | |
| - | Next: [Transformers](/en/Deep%20Learning/15%20Transformers) · [Course overview](/en/Deep%20Learning) |
| + | Next: [Transformers](/en/Deep%20Learning/10%20Transformers) · [Course overview](/en/Deep%20Learning) |
| en/Deep Learning/14 Attention/attention-heatmap.png .. en/Deep Learning/09 Attention/attention-heatmap.png | |
| /dev/null .. en/Deep Learning/09 Attention/attention-step1.svg | |
| @@ 0,0 1,12 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 400" width="900" height="400" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | <marker id="arrowmut" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs> |
| + | <rect width="900" height="400" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">Attention, step 1: score the query against every state</text> |
| + | <text x="255" y="56" font-size="11" fill="#5b6b7b" text-anchor="middle">encoder states h<tspan baseline-shift="sub" font-size="8px">j</tspan></text><rect x="80" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="108" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">1</tspan></text><rect x="180" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="208" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">2</tspan></text><rect x="280" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="308" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">3</tspan></text><rect x="380" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="408" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">4</tspan></text> |
| + | <rect x="690" y="72" width="64" height="28" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="722" y="91" font-size="12" fill="#1f2933" text-anchor="middle">s<tspan baseline-shift="sub" font-size="8px">i</tspan></text><text x="722" y="56" font-size="11" fill="#5b6b7b" text-anchor="middle">decoder state (the query)</text> |
| + | <line x1="108" y1="100" x2="108" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="208" y1="100" x2="208" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="308" y1="100" x2="308" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="408" y1="100" x2="408" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="122" y2="170" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="221" y2="168" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="319" y2="166" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="417" y2="164" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><circle cx="108" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="108" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><circle cx="208" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="208" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><circle cx="308" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="308" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><circle cx="408" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="408" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><text x="470" y="152" font-size="10" fill="#5b6b7b" text-anchor="middle">score(s<tspan baseline-shift="sub" font-size="8px">i</tspan>, h<tspan baseline-shift="sub" font-size="8px">j</tspan>)</text> |
| + | <g opacity="0.22"><rect x="72" y="214" width="372" height="30" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="258" y="233" font-size="11" fill="#1f2933" text-anchor="middle">softmax over positions j</text><line x1="108" y1="192" x2="108" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="208" y1="192" x2="208" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="308" y1="192" x2="308" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="408" y1="192" x2="408" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="108" y1="244" x2="108" y2="311" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="96" y="315" width="24" height="7" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="208" y1="244" x2="208" y2="262" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="196" y="266" width="24" height="56" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="308" y1="244" x2="308" y2="298" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="296" y="302" width="24" height="20" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="408" y1="244" x2="408" y2="311" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="396" y="315" width="24" height="7" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="80" y1="322" x2="436" y2="322" stroke="#5b6b7b" stroke-width="1"/><text x="108" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,1</tspan></text><text x="208" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,2</tspan></text><text x="308" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,3</tspan></text><text x="408" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,4</tspan></text><text x="258" y="356" font-size="10" fill="#5b6b7b" text-anchor="middle">weights, sum to 1</text></g> |
| + | <g opacity="0.22"><line x1="108" y1="346" x2="108" y2="356" stroke="none"/><line x1="208" y1="346" x2="208" y2="356" stroke="none"/><line x1="308" y1="346" x2="308" y2="356" stroke="none"/><line x1="408" y1="346" x2="408" y2="356" stroke="none"/><line x1="118" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="218" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="318" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="418" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><path d="M436,86 C580,90 550,230 522,288" fill="none" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><text x="612" y="182" font-size="10" fill="#5b6b7b" text-anchor="middle">values h<tspan baseline-shift="sub" font-size="8px">j</tspan></text><circle cx="516" cy="300" r="14" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="516" y="305" font-size="14" fill="#1f2933" text-anchor="middle" font-weight="600">+</text><line x1="530" y1="300" x2="558" y2="300" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="560" y="286" width="64" height="28" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="592" y="305" font-size="12" fill="#1f2933" text-anchor="middle">c<tspan baseline-shift="sub" font-size="8px">i</tspan></text><text x="592" y="330" font-size="9" fill="#5b6b7b" text-anchor="middle">context for step i</text><line x1="624" y1="300" x2="676" y2="300" stroke="#5b6b7b" stroke-width="1.4" marker-end="url(#arrowmut)"/><text x="745" y="304" font-size="11" fill="#5b6b7b" text-anchor="middle">to the decoder step</text></g> |
| + | <text x="450" y="388" font-size="11" fill="#5b6b7b" text-anchor="middle">c<tspan baseline-shift="sub" font-size="8px">i</tspan> = Σ α<tspan baseline-shift="sub" font-size="8px">i,j</tspan> h<tspan baseline-shift="sub" font-size="8px">j</tspan> · score, normalise, combine</text> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. en/Deep Learning/09 Attention/attention-step2.svg | |
| @@ 0,0 1,12 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 400" width="900" height="400" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | <marker id="arrowmut" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs> |
| + | <rect width="900" height="400" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">Attention, step 2: normalise the scores into weights</text> |
| + | <text x="255" y="56" font-size="11" fill="#5b6b7b" text-anchor="middle">encoder states h<tspan baseline-shift="sub" font-size="8px">j</tspan></text><rect x="80" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="108" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">1</tspan></text><rect x="180" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="208" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">2</tspan></text><rect x="280" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="308" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">3</tspan></text><rect x="380" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="408" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">4</tspan></text> |
| + | <rect x="690" y="72" width="64" height="28" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="722" y="91" font-size="12" fill="#1f2933" text-anchor="middle">s<tspan baseline-shift="sub" font-size="8px">i</tspan></text><text x="722" y="56" font-size="11" fill="#5b6b7b" text-anchor="middle">decoder state (the query)</text> |
| + | <line x1="108" y1="100" x2="108" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="208" y1="100" x2="208" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="308" y1="100" x2="308" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="408" y1="100" x2="408" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="122" y2="170" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="221" y2="168" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="319" y2="166" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="417" y2="164" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><circle cx="108" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="108" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><circle cx="208" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="208" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><circle cx="308" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="308" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><circle cx="408" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="408" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><text x="470" y="152" font-size="10" fill="#5b6b7b" text-anchor="middle">score(s<tspan baseline-shift="sub" font-size="8px">i</tspan>, h<tspan baseline-shift="sub" font-size="8px">j</tspan>)</text> |
| + | <rect x="72" y="214" width="372" height="30" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="258" y="233" font-size="11" fill="#1f2933" text-anchor="middle">softmax over positions j</text><line x1="108" y1="192" x2="108" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="208" y1="192" x2="208" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="308" y1="192" x2="308" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="408" y1="192" x2="408" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="108" y1="244" x2="108" y2="311" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="96" y="315" width="24" height="7" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="208" y1="244" x2="208" y2="262" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="196" y="266" width="24" height="56" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="308" y1="244" x2="308" y2="298" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="296" y="302" width="24" height="20" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="408" y1="244" x2="408" y2="311" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="396" y="315" width="24" height="7" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="80" y1="322" x2="436" y2="322" stroke="#5b6b7b" stroke-width="1"/><text x="108" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,1</tspan></text><text x="208" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,2</tspan></text><text x="308" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,3</tspan></text><text x="408" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,4</tspan></text><text x="258" y="356" font-size="10" fill="#5b6b7b" text-anchor="middle">weights, sum to 1</text> |
| + | <g opacity="0.22"><line x1="108" y1="346" x2="108" y2="356" stroke="none"/><line x1="208" y1="346" x2="208" y2="356" stroke="none"/><line x1="308" y1="346" x2="308" y2="356" stroke="none"/><line x1="408" y1="346" x2="408" y2="356" stroke="none"/><line x1="118" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="218" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="318" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="418" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><path d="M436,86 C580,90 550,230 522,288" fill="none" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><text x="612" y="182" font-size="10" fill="#5b6b7b" text-anchor="middle">values h<tspan baseline-shift="sub" font-size="8px">j</tspan></text><circle cx="516" cy="300" r="14" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="516" y="305" font-size="14" fill="#1f2933" text-anchor="middle" font-weight="600">+</text><line x1="530" y1="300" x2="558" y2="300" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="560" y="286" width="64" height="28" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="592" y="305" font-size="12" fill="#1f2933" text-anchor="middle">c<tspan baseline-shift="sub" font-size="8px">i</tspan></text><text x="592" y="330" font-size="9" fill="#5b6b7b" text-anchor="middle">context for step i</text><line x1="624" y1="300" x2="676" y2="300" stroke="#5b6b7b" stroke-width="1.4" marker-end="url(#arrowmut)"/><text x="745" y="304" font-size="11" fill="#5b6b7b" text-anchor="middle">to the decoder step</text></g> |
| + | <text x="450" y="388" font-size="11" fill="#5b6b7b" text-anchor="middle">c<tspan baseline-shift="sub" font-size="8px">i</tspan> = Σ α<tspan baseline-shift="sub" font-size="8px">i,j</tspan> h<tspan baseline-shift="sub" font-size="8px">j</tspan> · score, normalise, combine</text> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. en/Deep Learning/09 Attention/attention-weights.svg | |
| @@ 0,0 1,12 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 400" width="900" height="400" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | <marker id="arrowmut" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs> |
| + | <rect width="900" height="400" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">Attention: score, normalise, combine</text> |
| + | <text x="255" y="56" font-size="11" fill="#5b6b7b" text-anchor="middle">encoder states h<tspan baseline-shift="sub" font-size="8px">j</tspan></text><rect x="80" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="108" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">1</tspan></text><rect x="180" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="208" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">2</tspan></text><rect x="280" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="308" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">3</tspan></text><rect x="380" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="408" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">4</tspan></text> |
| + | <rect x="690" y="72" width="64" height="28" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="722" y="91" font-size="12" fill="#1f2933" text-anchor="middle">s<tspan baseline-shift="sub" font-size="8px">i</tspan></text><text x="722" y="56" font-size="11" fill="#5b6b7b" text-anchor="middle">decoder state (the query)</text> |
| + | <line x1="108" y1="100" x2="108" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="208" y1="100" x2="208" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="308" y1="100" x2="308" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="408" y1="100" x2="408" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="122" y2="170" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="221" y2="168" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="319" y2="166" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="417" y2="164" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><circle cx="108" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="108" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><circle cx="208" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="208" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><circle cx="308" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="308" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><circle cx="408" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="408" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><text x="470" y="152" font-size="10" fill="#5b6b7b" text-anchor="middle">score(s<tspan baseline-shift="sub" font-size="8px">i</tspan>, h<tspan baseline-shift="sub" font-size="8px">j</tspan>)</text> |
| + | <rect x="72" y="214" width="372" height="30" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="258" y="233" font-size="11" fill="#1f2933" text-anchor="middle">softmax over positions j</text><line x1="108" y1="192" x2="108" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="208" y1="192" x2="208" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="308" y1="192" x2="308" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="408" y1="192" x2="408" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="108" y1="244" x2="108" y2="311" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="96" y="315" width="24" height="7" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="208" y1="244" x2="208" y2="262" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="196" y="266" width="24" height="56" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="308" y1="244" x2="308" y2="298" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="296" y="302" width="24" height="20" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="408" y1="244" x2="408" y2="311" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="396" y="315" width="24" height="7" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="80" y1="322" x2="436" y2="322" stroke="#5b6b7b" stroke-width="1"/><text x="108" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,1</tspan></text><text x="208" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,2</tspan></text><text x="308" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,3</tspan></text><text x="408" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,4</tspan></text><text x="258" y="356" font-size="10" fill="#5b6b7b" text-anchor="middle">weights, sum to 1</text> |
| + | <line x1="108" y1="346" x2="108" y2="356" stroke="none"/><line x1="208" y1="346" x2="208" y2="356" stroke="none"/><line x1="308" y1="346" x2="308" y2="356" stroke="none"/><line x1="408" y1="346" x2="408" y2="356" stroke="none"/><line x1="118" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="218" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="318" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="418" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><path d="M436,86 C580,90 550,230 522,288" fill="none" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><text x="612" y="182" font-size="10" fill="#5b6b7b" text-anchor="middle">values h<tspan baseline-shift="sub" font-size="8px">j</tspan></text><circle cx="516" cy="300" r="14" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="516" y="305" font-size="14" fill="#1f2933" text-anchor="middle" font-weight="600">+</text><line x1="530" y1="300" x2="558" y2="300" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="560" y="286" width="64" height="28" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="592" y="305" font-size="12" fill="#1f2933" text-anchor="middle">c<tspan baseline-shift="sub" font-size="8px">i</tspan></text><text x="592" y="330" font-size="9" fill="#5b6b7b" text-anchor="middle">context for step i</text><line x1="624" y1="300" x2="676" y2="300" stroke="#5b6b7b" stroke-width="1.4" marker-end="url(#arrowmut)"/><text x="745" y="304" font-size="11" fill="#5b6b7b" text-anchor="middle">to the decoder step</text> |
| + | <text x="450" y="388" font-size="11" fill="#5b6b7b" text-anchor="middle">c<tspan baseline-shift="sub" font-size="8px">i</tspan> = Σ α<tspan baseline-shift="sub" font-size="8px">i,j</tspan> h<tspan baseline-shift="sub" font-size="8px">j</tspan> · score, normalise, combine</text> |
| + | </svg> |
| \ | No newline at end of file |
| en/Deep Learning/14 Attention/seq2seq-bottleneck.svg .. en/Deep Learning/09 Attention/seq2seq-bottleneck.svg | |
| en/Deep Learning/09 Regularization and dropout.md .. /dev/null | |
| @@ 1,117 0,0 @@ | |
| - | # 9. Regularization and dropout |
| - | |
| - | A deep network has enough capacity to fit almost any training set, including its noise. Regularization is the set of techniques that trade a little training accuracy for better generalization. This module covers weight decay ($L_2$), its $L_1$ counterpart, dropout with the inverted-dropout rescaling, and the lighter-weight regularizers early stopping and data augmentation. |
| - | |
| - | **Objectives** |
| - | - Recall what overfitting is and why high-capacity networks are prone to it. |
| - | - Add an $L_2$ penalty to the cost and read off its effect on the gradient (weight decay). |
| - | - Contrast $L_2$ with $L_1$ and their different pressures on the weights. |
| - | - Apply inverted dropout as a Bernoulli mask with $1/p$ rescaling. |
| - | - Explain the ensemble view of dropout and why the rescaling leaves activations unbiased. |
| - | - Place early stopping and data augmentation in the same generalization toolbox. |
| - | |
| - | ## 9.1 Overfitting recap |
| - | |
| - | A model overfits when it drives its training cost $J$ toward zero by memorizing the training examples, including their noise, so it generalizes poorly to unseen data. The gap between training performance and test performance is the tell. Deep networks are especially exposed because their parameter count $\sum_l n_l\, n_{l-1}$ usually exceeds the number of training examples, so they have the capacity to memorize. |
| - | |
| - | *Remark:* the underlying bias-variance trade-off was introduced in the Machine Learning course, see [General concepts](/en/Machine%20Learning/02%20General%20concepts). Regularization pushes a high-variance model back toward the sweet spot. |
| - | |
| - | The remedy is to constrain the effective capacity so the network prefers simpler functions. Every technique below is one such constraint. |
| - | |
| - | ## 9.2 L2 regularization (weight decay) |
| - | |
| - | ### 9.2.1 The penalty |
| - | |
| - | $L_2$ regularization adds a penalty proportional to the squared magnitude of every weight matrix to the cost. With $\lambda \ge 0$ the regularization strength, the regularized cost is: |
| - | |
| - | $$\boxed{ J_{\text{reg}} = J + \frac{\lambda}{2}\sum_{l=1}^{L}\lVert W^{[l]} \rVert_F^2 }$$ |
| - | |
| - | where $\lVert W^{[l]} \rVert_F^2 = \sum_{i,j}\big(W^{[l]}_{ij}\big)^2$ is the squared Frobenius norm. The biases $b^{[l]}$ are normally left out of the penalty, as they add negligible capacity and penalizing them tends to underfit. |
| - | |
| - | ### 9.2.2 Effect on the gradient |
| - | |
| - | Differentiating the penalty is what gives the technique its second name. The extra term contributes $\lambda W^{[l]}$ to the gradient with respect to $W^{[l]}$: |
| - | |
| - | $$\boxed{ \frac{\partial J_{\text{reg}}}{\partial W^{[l]}} = \frac{\partial J}{\partial W^{[l]}} + \lambda\, W^{[l]} }$$ |
| - | |
| - | Plugging this into a gradient-descent step with learning rate $\alpha$ shrinks the weight before the data-driven update is applied: |
| - | |
| - | $$\boxed{ W^{[l]} \leftarrow (1 - \alpha\lambda)\, W^{[l]} - \alpha\,\frac{\partial J}{\partial W^{[l]}} }$$ |
| - | |
| - | *Remark:* the factor $(1 - \alpha\lambda) < 1$ multiplies every weight each step, which is literally a decay toward zero. That is why $L_2$ regularization is called weight decay. Smaller weights mean a smoother, lower-variance function. |
| - | |
| - | ### 9.2.3 Contrast with L1 |
| - | |
| - | Replacing the squared norm with the absolute-value norm gives $L_1$ regularization, penalizing $\lambda\sum_l \lVert W^{[l]} \rVert_1 = \lambda\sum_{l,i,j}\lvert W^{[l]}_{ij}\rvert$. Its gradient contribution is $\lambda\,\operatorname{sign}(W^{[l]})$, a constant pull toward zero regardless of magnitude. |
| - | |
| - | | Penalty | Added to cost | Gradient term | Pressure on weights | |
| - | | --- | --- | --- | --- | |
| - | | $L_2$ | $\tfrac{\lambda}{2}\lVert W \rVert_F^2$ | $\lambda W$ | shrinks all weights proportionally, rarely exactly zero | |
| - | | $L_1$ | $\lambda\lVert W \rVert_1$ | $\lambda\,\operatorname{sign}(W)$ | drives many weights to exactly zero (sparse) | |
| - | |
| - | *Remark:* $L_1$ produces sparse weight matrices and so doubles as feature selection. $L_2$ is the default in deep learning because it is smooth everywhere and pairs cleanly with gradient descent. |
| - | |
| - | ## 9.3 Dropout |
| - | |
| - | ### 9.3.1 The idea |
| - | |
| - | Dropout regularizes by injecting noise into the activations. On each training forward pass, every unit is kept with probability $p$ and zeroed with probability $1 - p$, independently. The network therefore cannot rely on any single unit, so it spreads the representation across many units and stops co-adapting them. |
| - | |
| - | ### 9.3.2 Inverted dropout |
| - | |
| - | Let $m$ be a Bernoulli$(p)$ mask with the same shape as the activation $a^{[l]}$, drawn fresh every step. Inverted dropout applies the mask and immediately divides by $p$: |
| - | |
| - | $$\boxed{ \tilde{a}^{[l]} = \frac{m \odot a^{[l]}}{p}, \qquad m_i \sim \text{Bernoulli}(p) }$$ |
| - | |
| - | The masked, rescaled $\tilde{a}^{[l]}$ then flows into layer $l+1$ in place of $a^{[l]}$. At inference time dropout is switched off and behaves as the identity, $\tilde{a}^{[l]} = a^{[l]}$, with no mask and no rescaling. |
| - | |
| - | *Remark:* keeping the $1/p$ rescaling at training time (hence inverted) is what lets inference stay a plain forward pass. The older, non-inverted form instead multiplied weights by $p$ at test time, which is easy to forget. |
| - | |
| - | ### 9.3.3 Why the rescaling |
| - | |
| - | Because $\mathbb{E}[m_i] = p$, the expected value of a kept, rescaled unit equals the original activation: |
| - | |
| - | $$\boxed{ \mathbb{E}\!\left[\tilde{a}^{[l]}_i\right] = \frac{p\cdot a^{[l]}_i + (1-p)\cdot 0}{p} = a^{[l]}_i }$$ |
| - | |
| - | So the expected input to the next layer is unchanged, and the network sees the same average signal with dropout on or off. This is exactly why no correction is needed at inference. |
| - | |
| - | ### 9.3.4 The ensemble view |
| - | |
| - | A network with $k$ droppable units defines $2^k$ possible thinned subnetworks, one per mask. Each training step samples one subnetwork and takes a gradient step on it, and all subnetworks share weights. At test time the full network with rescaled activations approximates the average prediction of this exponentially large ensemble, which is why dropout behaves like cheap model averaging. |
| - | |
| - |  |
| - | |
| - | *Dropout trains a different thinned subnetwork on each step by randomly removing units, and averages them at inference.* |
| - | |
| - | *Remark:* typical keep probabilities are $p$ around $0.8$ for input layers and $0.5$ for hidden layers. A smaller $p$ means stronger regularization. |
| - | |
| - | ## 9.4 Other regularizers |
| - | |
| - | ### 9.4.1 Early stopping |
| - | |
| - | Track the validation cost during training and stop at the epoch where it starts rising, even though the training cost is still falling. Halting early keeps the weights near their small initial values, so it acts like an implicit $L_2$ penalty without adding a term to the cost. |
| - | |
| - |  |
| - | |
| - | *Training loss keeps falling while validation loss turns upward, the gap is overfitting and its minimum is where early stopping halts training.* |
| - | |
| - | ### 9.4.2 Data augmentation |
| - | |
| - | Expand the training set with label-preserving transformations of the inputs (random crops, flips, small rotations, colour jitter for images, noise for audio). More effective variety in the data lowers variance directly, which is regularization applied to the dataset rather than to the weights. |
| - | |
| - | ### 9.4.3 Summary |
| - | |
| - | | Technique | Where it acts | Effect | |
| - | | --- | --- | --- | |
| - | | $L_2$ (weight decay) | cost via $\lambda W$ | shrinks weights, smoother function | |
| - | | $L_1$ | cost via $\lambda\,\operatorname{sign}(W)$ | sparse weights, feature selection | |
| - | | Dropout | activations at training | ensemble of thinned subnetworks | |
| - | | Early stopping | training loop | keeps weights near initialization | |
| - | | Data augmentation | training data | more variety, lower variance | |
| - | |
| - | *Remark:* these techniques compose. A convolutional network commonly uses weight decay, dropout, and heavy data augmentation together. |
| - | |
| - | *With overfitting under control, the next module builds an architecture whose weight sharing is itself a form of regularization: the convolutional network.* |
| - | |
| - | --- |
| - | Next: [Convolutional networks](/en/Deep%20Learning/10%20Convolutional%20networks) · [Course overview](/en/Deep%20Learning) |
| en/Deep Learning/09 Regularization and dropout/overfitting-curves.png .. /dev/null | |
| en/Deep Learning/10 Convolutional networks/residual-block.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 760 321" width="760" height="321" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="760" height="321" fill="#ffffff"/><text x="380.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">A residual block</text><rect x="40.0" y="144.0" width="90.0" height="52.0" rx="8" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><text x="85.0" y="174.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">input x</text><rect x="210.0" y="144.0" width="150.0" height="52.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="285.0" y="174.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">conv path F(x)</text><circle cx="470.0" cy="170.0" r="20.0" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="470.0" y="176.1" font-family="Helvetica, Arial, sans-serif" font-size="18" fill="#1f2933" text-anchor="middle">+</text><rect x="540.0" y="144.0" width="90.0" height="52.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="585.0" y="174.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">relu</text><rect x="660.0" y="144.0" width="80.0" height="52.0" rx="8" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><text x="700.0" y="174.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">output y</text><line x1="130.0" y1="170.0" x2="210.0" y2="170.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="360.0" y1="170.0" x2="450.0" y2="170.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="490.0" y1="170.0" x2="540.0" y2="170.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="630.0" y1="170.0" x2="660.0" y2="170.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><circle cx="160.0" cy="170.0" r="4.0" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><path d="M160.0 170.0 Q268.5 70.0 470.0 150.0" fill="none" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><text x="315.0" y="155.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">identity skip x</text><text x="285.0" y="222.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">learns a correction F(x)</text><text x="380.0" y="300.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">y = relu( F(x) + x )</text></svg> |
| \ | No newline at end of file |
| en/Deep Learning/15 Transformers.md .. en/Deep Learning/10 Transformers.md | |
| @@ 1,16 1,8 @@ | |
| - | # 15. Transformers |
| + | # 10. Transformers |
| - | The Transformer replaces recurrence with attention alone. It processes a whole sequence of token embeddings in parallel, letting every token attend to every other token through learned queries, keys, and values. This lesson builds the architecture from self-attention, assuming embeddings (lesson 12) and the attention mechanism (lesson 15), and reuses normalization (lesson 8) and residual connections (lesson 11). |
| + | The Transformer replaces recurrence with attention alone. It processes a whole sequence of token embeddings in parallel, letting every token attend to every other token through learned queries, keys, and values. This lesson builds the architecture from self-attention, assuming embeddings (lesson 6) and the attention mechanism (lesson 9), and reuses normalization (lesson 4) and residual connections (lesson 5). |
| - | **Objectives** |
| - | - Project token embeddings into queries $Q$, keys $K$, and values $V$ with learned matrices. |
| - | - Define scaled dot-product attention and explain the $1/\sqrt{d_k}$ scaling. |
| - | - Run several attention heads in parallel and combine them with multi-head attention. |
| - | - Inject order into a set-based operation with positional encodings. |
| - | - Assemble a Transformer block from residual connections and layer normalization. |
| - | - Place the block inside the encoder-decoder stack and name its encoder-only and decoder-only variants. |
| - | |
| - | ## 15.1 Self-attention and Q, K, V |
| + | ## 10.1 Self-attention and Q, K, V |
| A sequence of $n$ tokens is represented by an embedding matrix $X \in \mathbb{R}^{n \times d}$, one row per token. Self-attention lets each token gather information from the others by asking a question (a query), matching it against every token's label (a key), and reading out content (a value). | |
| @@ 20,7 12,11 @@ | |
| *Remark:* the projections are the only learned parameters here, and the same three matrices are shared across all positions. Because a token is compared against every other token, the operation captures long-range dependencies in a single step, unlike a recurrence that must carry information forward one position at a time. | |
| - | ## 15.2 Scaled dot-product attention |
| + |  |
| + | |
| + | *From one input matrix (here four tokens, "nous mangeons du pain"), three learned projections give every token its query, key, and value. What they feed is still grayed out.* |
| + | |
| + | ## 10.2 Scaled dot-product attention |
| Each query is compared against every key by a dot product, giving an $n \times n$ matrix of raw scores. The scores are scaled, turned into weights by a row-wise softmax, and used to average the values: | |
| @@ 28,7 24,11 @@ | |
| Row $i$ of the softmax is a probability distribution over all tokens, so output row $i$ is a weighted average of the value vectors, weighted by how relevant each token is to token $i$. | |
| - | ### 15.2.1 Why divide by $\sqrt{d_k}$ |
| + |  |
| + | |
| + | *The whole layer as matrix products, shapes included: $QK^T/\sqrt{d_k}$ compares every token with every other ($n \times n$), the row-wise softmax turns scores into weights, and multiplying by $V$ returns one output row per token.* |
| + | |
| + | ### 10.2.1 Why divide by $\sqrt{d_k}$ |
| If the entries of $q$ and $k$ are independent with zero mean and unit variance, the dot product $q^{T} k = \sum_{j=1}^{d_k} q_j k_j$ has variance $d_k$, so its typical magnitude grows like $\sqrt{d_k}$. | |
| @@ 36,7 36,11 @@ | |
| Large scores push the softmax into a saturated regime where one weight is near $1$ and the rest are near $0$, and the softmax gradient there is tiny. Dividing by $\sqrt{d_k}$ keeps the logits at a moderate scale, which keeps the softmax gradients healthy and stabilizes training. | |
| - | ## 15.3 Multi-head attention |
| + |  |
| + | |
| + | *The unit so far: one scaled dot-product attention head, sitting where it will live. The rest of the block is still grayed out.* |
| + | |
| + | ## 10.3 Multi-head attention |
| A single attention computation forces every relationship to be read through one $d_k$-dimensional subspace. Multi-head attention runs $h$ attention operations in parallel, each with its own projections, so different heads can specialize (one on syntax, another on coreference, and so on). | |
| @@ 50,7 54,11 @@ | |
| *Remark:* the per-head width is usually set to $d_k = d_v = d / h$, so the concatenation returns to width $d$ and the total cost matches a single full-width head. The heads are independent and computed in parallel, which is one reason Transformers train efficiently on modern hardware. | |
| - | ## 15.4 Positional encoding |
| + |  |
| + | |
| + | *Step 2: several heads run in parallel on their own projections, and their outputs are concatenated and mixed by $W^O$. The multi-head sublayer is complete.* |
| + | |
| + | ## 10.4 Positional encoding |
| Attention treats its input as a set: permuting the rows of $X$ permutes the output the same way, so the operation is order-agnostic. Language is not, therefore position must be supplied explicitly. The original Transformer adds a fixed sinusoidal encoding to the embeddings, using a different frequency per feature dimension: | |
| @@ 58,29 66,33 @@ | |
| Here $pos$ is the token position and $i$ indexes the feature dimension. Low dimensions vary quickly with position and high dimensions vary slowly, so the vector encodes position across many scales. The encoding is added to the token embedding before the first block. | |
| - |  |
| + |  |
| *Sinusoidal positional encodings vary quickly in low dimensions and slowly in high dimensions, giving each position a unique multi-scale signature.* | |
| *Remark:* sinusoids let a relative shift $PE_{pos+k}$ be written as a linear function of $PE_{pos}$, so the model can learn to attend by relative offset. The encodings are fixed (not learned) and extend to sequence lengths unseen during training. Many later models replace them with learned or relative position schemes. | |
| - | ## 15.5 The Transformer block |
| + |  |
| + | |
| + | *Step 3: the input side. Token embeddings enter through an addition with the positional encoding, which gives attention its sense of order.* |
| + | |
| + | ## 10.5 The Transformer block |
| Each sublayer is wrapped in a residual connection followed by layer normalization, which keeps gradients flowing through deep stacks and stabilizes the activation scale: | |
| $$\boxed{ x \leftarrow \mathrm{LayerNorm}\!\left(x + \mathrm{Sublayer}(x)\right) }$$ | |
| - |  |
| + |  |
| - | *A Transformer block wraps multi-head attention and a feed-forward network, each in a residual connection followed by layer normalization.* |
| + | *The assembled block: multi-head self-attention, then the position-wise feed-forward network, each wrapped in a residual connection and layer normalization. Stacked $N$ times, this is the Transformer.* |
| - | A block chains two sublayers in this pattern. The first is multi-head self-attention (tokens exchange information). The second is a position-wise feed-forward network, a two-layer MLP applied independently to each position, using the notation from lesson 12 onward: |
| + | A block chains two sublayers in this pattern. The first is multi-head self-attention (tokens exchange information). The second is a position-wise feed-forward network, a two-layer MLP applied independently to each position, using the notation of lesson 1: |
| $$\boxed{ \mathrm{FFN}(x) = g\!\left(x W_1 + b_1\right) W_2 + b_2 }$$ | |
| with a nonlinearity $g$ (ReLU or GELU) and an inner width several times larger than $d$. | |
| - | *Remark:* the residual reuses the identity shortcut of lesson 11, so the sublayer only has to learn a correction to its input. Layer normalization (lesson 8) normalizes across the feature dimension per token, which suits variable-length sequences better than batch normalization. The form above is the original post-norm placement. Many modern implementations use pre-norm, $x \leftarrow x + \mathrm{Sublayer}(\mathrm{LayerNorm}(x))$, which trains more stably at great depth. |
| + | *Remark:* the residual reuses the identity shortcut of lesson 5, so the sublayer only has to learn a correction to its input. Layer normalization (lesson 4) normalizes across the feature dimension per token, which suits variable-length sequences better than batch normalization. The form above is the original post-norm placement. Many modern implementations use pre-norm, $x \leftarrow x + \mathrm{Sublayer}(\mathrm{LayerNorm}(x))$, which trains more stably at great depth. |
| | Component | Role | Acts across | | |
| | --- | --- | --- | | |
| @@ 89,15 101,15 @@ | |
| | Residual connection | preserve a gradient path | the depth | | |
| | Layer normalization | stabilize the activation scale | the features per token | | |
| - | ## 15.6 The encoder-decoder architecture |
| + | ## 10.6 The encoder-decoder architecture |
| The full Transformer stacks $N$ identical blocks in an encoder and $N$ in a decoder. The encoder maps the input sequence to a set of context vectors. Each decoder block has three sublayers: masked self-attention over the tokens generated so far (the mask blocks attention to future positions), cross-attention whose queries come from the decoder and whose keys and values come from the encoder output, and a feed-forward network. A final linear layer plus softmax turns the top decoder states into a distribution over the vocabulary. | |
| - |  |
| + |  |
| *The full Transformer: a stack of encoder blocks and a stack of decoder blocks joined by cross-attention.* | |
| - | ### 15.6.1 Variants |
| + | ### 10.6.1 Variants |
| Not every task needs both halves. Two families dominate practice: | |
| en/Deep Learning/15 Transformers/positional-encoding.png .. en/Deep Learning/10 Transformers/positional-encoding.png | |
| /dev/null .. en/Deep Learning/10 Transformers/selfattention-matrices.svg | |
| @@ 0,0 1,10 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 330" width="900" height="330" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker></defs> |
| + | <rect width="900" height="330" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">Self-attention as matrix products</text> |
| + | <rect x="88" y="110" width="72" height="88" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="88" y1="132.0" x2="160" y2="132.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="88" y1="154.0" x2="160" y2="154.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="88" y1="176.0" x2="160" y2="176.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="100.0" y1="110" x2="100.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="112.0" y1="110" x2="112.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="124.0" y1="110" x2="124.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="136.0" y1="110" x2="136.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="148.0" y1="110" x2="148.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="124.0" y="102" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">X</text><text x="124.0" y="212" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d</text><text x="82" y="125" font-size="9" fill="#5b6b7b" text-anchor="end">nous</text><text x="82" y="147" font-size="9" fill="#5b6b7b" text-anchor="end">mangeons</text><text x="82" y="169" font-size="9" fill="#5b6b7b" text-anchor="end">du</text><text x="82" y="191" font-size="9" fill="#5b6b7b" text-anchor="end">pain</text> |
| + | <rect x="208" y="77" width="50" height="26" rx="7" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.5"/><text x="233" y="94" font-size="11" fill="#1f2933" text-anchor="middle">W<tspan baseline-shift="super" font-size="7px">Q</tspan></text><line x1="160" y1="154" x2="206" y2="90" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="208" y="153" width="50" height="26" rx="7" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.5"/><text x="233" y="170" font-size="11" fill="#1f2933" text-anchor="middle">W<tspan baseline-shift="super" font-size="7px">K</tspan></text><line x1="160" y1="154" x2="206" y2="166" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="208" y="229" width="50" height="26" rx="7" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.5"/><text x="233" y="246" font-size="11" fill="#1f2933" text-anchor="middle">W<tspan baseline-shift="super" font-size="7px">V</tspan></text><line x1="160" y1="154" x2="206" y2="242" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/> |
| + | <line x1="258" y1="90" x2="292" y2="90" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="294" y="70" width="44" height="40" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="294" y1="80.0" x2="338" y2="80.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="90.0" x2="338" y2="90.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="100.0" x2="338" y2="100.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="308.6666666666667" y1="70" x2="308.6666666666667" y2="110" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="323.3333333333333" y1="70" x2="323.3333333333333" y2="110" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="316.0" y="62" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">Q</text><text x="316.0" y="124" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d<tspan baseline-shift="sub" font-size="7px">k</tspan></text><line x1="258" y1="166" x2="292" y2="166" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="294" y="146" width="44" height="40" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="294" y1="156.0" x2="338" y2="156.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="166.0" x2="338" y2="166.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="176.0" x2="338" y2="176.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="308.6666666666667" y1="146" x2="308.6666666666667" y2="186" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="323.3333333333333" y1="146" x2="323.3333333333333" y2="186" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="316.0" y="138" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">K</text><text x="316.0" y="200" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d<tspan baseline-shift="sub" font-size="7px">k</tspan></text><line x1="258" y1="242" x2="292" y2="242" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="294" y="222" width="44" height="40" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="294" y1="232.0" x2="338" y2="232.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="242.0" x2="338" y2="242.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="252.0" x2="338" y2="252.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="308.6666666666667" y1="222" x2="308.6666666666667" y2="262" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="323.3333333333333" y1="222" x2="323.3333333333333" y2="262" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="316.0" y="214" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">V</text><text x="316.0" y="276" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d<tspan baseline-shift="sub" font-size="7px">v</tspan></text> |
| + | <line x1="338" y1="90" x2="396" y2="116" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="338" y1="166" x2="396" y2="134" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="398" y="96" width="56" height="56" fill="#fff1e0" stroke="#e0872e" stroke-width="1.5"/><line x1="398" y1="110.0" x2="454" y2="110.0" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="398" y1="124.0" x2="454" y2="124.0" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="398" y1="138.0" x2="454" y2="138.0" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="412.0" y1="96" x2="412.0" y2="152" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="426.0" y1="96" x2="426.0" y2="152" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="440.0" y1="96" x2="440.0" y2="152" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><text x="426.0" y="88" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">QK​<tspan baseline-shift="super" font-size="7px">T</tspan>/√dₖ</text><text x="426.0" y="166" font-size="9" fill="#5b6b7b" text-anchor="middle">scores · n × n</text><line x1="454" y1="124" x2="490" y2="124" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="492" y="108" width="88" height="32" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.5"/><text x="536" y="122" font-size="11" fill="#1f2933" text-anchor="middle">softmax</text><text x="536" y="134" font-size="8" fill="#5b6b7b" text-anchor="middle">per row</text><line x1="580" y1="124" x2="616" y2="124" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="618" y="96" width="56" height="56" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.5"/><line x1="618" y1="110.0" x2="674" y2="110.0" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="618" y1="124.0" x2="674" y2="124.0" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="618" y1="138.0" x2="674" y2="138.0" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="632.0" y1="96" x2="632.0" y2="152" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="646.0" y1="96" x2="646.0" y2="152" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="660.0" y1="96" x2="660.0" y2="152" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><text x="646.0" y="88" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">A</text><text x="646.0" y="166" font-size="9" fill="#5b6b7b" text-anchor="middle">weights · rows sum to 1</text><line x1="674" y1="124" x2="712" y2="124" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><circle cx="726" cy="124" r="12" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="726" y="129" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">×</text><line x1="338" y1="242" x2="726" y2="242" stroke="#1f2933" stroke-width="1.3"/><line x1="726" y1="242" x2="726" y2="138" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="766" y="100" width="48" height="48" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="766" y1="112.0" x2="814" y2="112.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="766" y1="124.0" x2="814" y2="124.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="766" y1="136.0" x2="814" y2="136.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="782.0" y1="100" x2="782.0" y2="148" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="798.0" y1="100" x2="798.0" y2="148" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="790.0" y="92" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">Y</text><text x="790.0" y="162" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d<tspan baseline-shift="sub" font-size="7px">v</tspan></text><line x1="738" y1="124" x2="764" y2="124" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/> |
| + | <text x="450" y="316" font-size="11" fill="#5b6b7b" text-anchor="middle">n tokens as rows · one matrix product per step, the whole sequence at once</text> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. en/Deep Learning/10 Transformers/selfattention-step1.svg | |
| @@ 0,0 1,10 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 330" width="900" height="330" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker></defs> |
| + | <rect width="900" height="330" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">Self-attention, step 1: three projections of the same input</text> |
| + | <rect x="88" y="110" width="72" height="88" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="88" y1="132.0" x2="160" y2="132.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="88" y1="154.0" x2="160" y2="154.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="88" y1="176.0" x2="160" y2="176.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="100.0" y1="110" x2="100.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="112.0" y1="110" x2="112.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="124.0" y1="110" x2="124.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="136.0" y1="110" x2="136.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="148.0" y1="110" x2="148.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="124.0" y="102" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">X</text><text x="124.0" y="212" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d</text><text x="82" y="125" font-size="9" fill="#5b6b7b" text-anchor="end">nous</text><text x="82" y="147" font-size="9" fill="#5b6b7b" text-anchor="end">mangeons</text><text x="82" y="169" font-size="9" fill="#5b6b7b" text-anchor="end">du</text><text x="82" y="191" font-size="9" fill="#5b6b7b" text-anchor="end">pain</text> |
| + | <rect x="208" y="77" width="50" height="26" rx="7" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.5"/><text x="233" y="94" font-size="11" fill="#1f2933" text-anchor="middle">W<tspan baseline-shift="super" font-size="7px">Q</tspan></text><line x1="160" y1="154" x2="206" y2="90" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="208" y="153" width="50" height="26" rx="7" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.5"/><text x="233" y="170" font-size="11" fill="#1f2933" text-anchor="middle">W<tspan baseline-shift="super" font-size="7px">K</tspan></text><line x1="160" y1="154" x2="206" y2="166" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="208" y="229" width="50" height="26" rx="7" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.5"/><text x="233" y="246" font-size="11" fill="#1f2933" text-anchor="middle">W<tspan baseline-shift="super" font-size="7px">V</tspan></text><line x1="160" y1="154" x2="206" y2="242" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/> |
| + | <line x1="258" y1="90" x2="292" y2="90" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="294" y="70" width="44" height="40" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="294" y1="80.0" x2="338" y2="80.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="90.0" x2="338" y2="90.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="100.0" x2="338" y2="100.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="308.6666666666667" y1="70" x2="308.6666666666667" y2="110" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="323.3333333333333" y1="70" x2="323.3333333333333" y2="110" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="316.0" y="62" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">Q</text><text x="316.0" y="124" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d<tspan baseline-shift="sub" font-size="7px">k</tspan></text><line x1="258" y1="166" x2="292" y2="166" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="294" y="146" width="44" height="40" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="294" y1="156.0" x2="338" y2="156.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="166.0" x2="338" y2="166.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="176.0" x2="338" y2="176.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="308.6666666666667" y1="146" x2="308.6666666666667" y2="186" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="323.3333333333333" y1="146" x2="323.3333333333333" y2="186" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="316.0" y="138" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">K</text><text x="316.0" y="200" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d<tspan baseline-shift="sub" font-size="7px">k</tspan></text><line x1="258" y1="242" x2="292" y2="242" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="294" y="222" width="44" height="40" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="294" y1="232.0" x2="338" y2="232.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="242.0" x2="338" y2="242.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="252.0" x2="338" y2="252.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="308.6666666666667" y1="222" x2="308.6666666666667" y2="262" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="323.3333333333333" y1="222" x2="323.3333333333333" y2="262" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="316.0" y="214" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">V</text><text x="316.0" y="276" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d<tspan baseline-shift="sub" font-size="7px">v</tspan></text> |
| + | <g opacity="0.22"><line x1="338" y1="90" x2="396" y2="116" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="338" y1="166" x2="396" y2="134" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="398" y="96" width="56" height="56" fill="#fff1e0" stroke="#e0872e" stroke-width="1.5"/><line x1="398" y1="110.0" x2="454" y2="110.0" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="398" y1="124.0" x2="454" y2="124.0" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="398" y1="138.0" x2="454" y2="138.0" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="412.0" y1="96" x2="412.0" y2="152" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="426.0" y1="96" x2="426.0" y2="152" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="440.0" y1="96" x2="440.0" y2="152" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><text x="426.0" y="88" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">QK​<tspan baseline-shift="super" font-size="7px">T</tspan>/√dₖ</text><text x="426.0" y="166" font-size="9" fill="#5b6b7b" text-anchor="middle">scores · n × n</text><line x1="454" y1="124" x2="490" y2="124" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="492" y="108" width="88" height="32" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.5"/><text x="536" y="122" font-size="11" fill="#1f2933" text-anchor="middle">softmax</text><text x="536" y="134" font-size="8" fill="#5b6b7b" text-anchor="middle">per row</text><line x1="580" y1="124" x2="616" y2="124" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="618" y="96" width="56" height="56" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.5"/><line x1="618" y1="110.0" x2="674" y2="110.0" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="618" y1="124.0" x2="674" y2="124.0" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="618" y1="138.0" x2="674" y2="138.0" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="632.0" y1="96" x2="632.0" y2="152" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="646.0" y1="96" x2="646.0" y2="152" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="660.0" y1="96" x2="660.0" y2="152" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><text x="646.0" y="88" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">A</text><text x="646.0" y="166" font-size="9" fill="#5b6b7b" text-anchor="middle">weights · rows sum to 1</text><line x1="674" y1="124" x2="712" y2="124" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><circle cx="726" cy="124" r="12" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="726" y="129" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">×</text><line x1="338" y1="242" x2="726" y2="242" stroke="#1f2933" stroke-width="1.3"/><line x1="726" y1="242" x2="726" y2="138" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="766" y="100" width="48" height="48" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="766" y1="112.0" x2="814" y2="112.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="766" y1="124.0" x2="814" y2="124.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="766" y1="136.0" x2="814" y2="136.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="782.0" y1="100" x2="782.0" y2="148" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="798.0" y1="100" x2="798.0" y2="148" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="790.0" y="92" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">Y</text><text x="790.0" y="162" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d<tspan baseline-shift="sub" font-size="7px">v</tspan></text><line x1="738" y1="124" x2="764" y2="124" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/></g> |
| + | <text x="450" y="316" font-size="11" fill="#5b6b7b" text-anchor="middle">n tokens as rows · one matrix product per step, the whole sequence at once</text> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. en/Deep Learning/10 Transformers/transformer-block.svg | |
| @@ 0,0 1,30 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 545" width="900" height="545" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | <marker id="arrowmut" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs> |
| + | <rect width="900" height="545" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">The Transformer block, assembled</text> |
| + | <rect x="280" y="110" width="340" height="330" rx="14" fill="#f9fafb" stroke="#c7d0d9" stroke-width="1.2"/> |
| + | <rect x="340" y="330" width="220" height="86" rx="10" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="450" y="345" font-size="10" fill="#1f2933" text-anchor="middle">multi-head self-attention</text> |
| + | <rect x="352" y="352" width="196" height="20" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.2"/><text x="450" y="366" font-size="9" fill="#1f2933" text-anchor="middle">concat, then W<tspan baseline-shift="super" font-size="7px">O</tspan></text> |
| + | <rect x="352" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="380" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="380" y1="382" x2="380" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <rect x="422" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="450" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="450" y1="382" x2="450" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <rect x="492" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="520" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="520" y1="382" x2="520" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <line x1="628" y1="392" x2="552" y2="392" stroke="#5b6b7b" stroke-width="1" stroke-dasharray="3 3"/><text x="634" y="389" font-size="10" fill="#5b6b7b" text-anchor="start">one head = scaled dot-product attention:</text><text x="634" y="403" font-size="10" fill="#5b6b7b" text-anchor="start">softmax(QK<tspan baseline-shift="super" font-size="7px">T</tspan>/√dₖ) V</text> |
| + | <rect x="365" y="492" width="170" height="30" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="450.0" y="511.0" font-size="11" fill="#1f2933" text-anchor="middle">token embeddings X</text> |
| + | <line x1="450" y1="492" x2="450" y2="470" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <circle cx="450" cy="455" r="12" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="450" y="460" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">+</text> |
| + | <rect x="560" y="441" width="170" height="28" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="645.0" y="459.0" font-size="10" fill="#1f2933" text-anchor="middle">positional encoding</text> |
| + | <line x1="560" y1="455" x2="465" y2="455" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <line x1="450" y1="443" x2="450" y2="418" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <rect x="350" y="286" width="200" height="26" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="450.0" y="303.0" font-size="10" fill="#1f2933" text-anchor="middle">add & norm</text> |
| + | <rect x="350" y="205" width="200" height="52" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="450.0" y="229.0" font-size="11" fill="#1f2933" text-anchor="middle">feed-forward network</text><text x="450.0" y="242.0" font-size="9" fill="#5b6b7b" text-anchor="middle">two-layer MLP, per position</text> |
| + | <rect x="350" y="150" width="200" height="26" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="450.0" y="167.0" font-size="10" fill="#1f2933" text-anchor="middle">add & norm</text> |
| + | <line x1="450" y1="330" x2="450" y2="314" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <line x1="450" y1="286" x2="450" y2="259" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <line x1="450" y1="205" x2="450" y2="178" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <line x1="450" y1="150" x2="450" y2="80" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="450" y="66" font-size="11" fill="#5b6b7b" text-anchor="middle">to the next block</text> |
| + | <path d="M452,430 C606,430 606,299 554,299" fill="none" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/> |
| + | <path d="M452,268 C606,268 606,163 554,163" fill="none" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/> |
| + | <text x="648" y="240" font-size="10" fill="#5b6b7b" text-anchor="start">residual shortcuts</text> |
| + | <text x="648" y="130" font-size="11" fill="#5b6b7b" text-anchor="start">× N identical blocks</text> |
| + | </svg> |
| \ | No newline at end of file |
| en/Deep Learning/15 Transformers/transformer-stack.svg .. en/Deep Learning/10 Transformers/transformer-stack.svg | |
| /dev/null .. en/Deep Learning/10 Transformers/transformer-step1.svg | |
| @@ 0,0 1,30 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 545" width="900" height="545" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | <marker id="arrowmut" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs> |
| + | <rect width="900" height="545" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">The Transformer block, step 1: one attention head</text> |
| + | <g opacity="0.22"><rect x="280" y="110" width="340" height="330" rx="14" fill="#f9fafb" stroke="#c7d0d9" stroke-width="1.2"/></g> |
| + | <g opacity="0.22"><rect x="340" y="330" width="220" height="86" rx="10" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="450" y="345" font-size="10" fill="#1f2933" text-anchor="middle">multi-head self-attention</text></g> |
| + | <g opacity="0.22"><rect x="352" y="352" width="196" height="20" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.2"/><text x="450" y="366" font-size="9" fill="#1f2933" text-anchor="middle">concat, then W<tspan baseline-shift="super" font-size="7px">O</tspan></text></g> |
| + | <g opacity="0.22"><rect x="352" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="380" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="380" y1="382" x2="380" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/></g> |
| + | <rect x="422" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="450" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="450" y1="382" x2="450" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <g opacity="0.22"><rect x="492" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="520" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="520" y1="382" x2="520" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/></g> |
| + | <line x1="628" y1="392" x2="552" y2="392" stroke="#5b6b7b" stroke-width="1" stroke-dasharray="3 3"/><text x="634" y="389" font-size="10" fill="#5b6b7b" text-anchor="start">one head = scaled dot-product attention:</text><text x="634" y="403" font-size="10" fill="#5b6b7b" text-anchor="start">softmax(QK<tspan baseline-shift="super" font-size="7px">T</tspan>/√dₖ) V</text> |
| + | <g opacity="0.22"><rect x="365" y="492" width="170" height="30" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="450.0" y="511.0" font-size="11" fill="#1f2933" text-anchor="middle">token embeddings X</text></g> |
| + | <g opacity="0.22"><line x1="450" y1="492" x2="450" y2="470" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><circle cx="450" cy="455" r="12" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="450" y="460" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">+</text></g> |
| + | <g opacity="0.22"><rect x="560" y="441" width="170" height="28" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="645.0" y="459.0" font-size="10" fill="#1f2933" text-anchor="middle">positional encoding</text></g> |
| + | <g opacity="0.22"><line x1="560" y1="455" x2="465" y2="455" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="443" x2="450" y2="418" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><rect x="350" y="286" width="200" height="26" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="450.0" y="303.0" font-size="10" fill="#1f2933" text-anchor="middle">add & norm</text></g> |
| + | <g opacity="0.22"><rect x="350" y="205" width="200" height="52" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="450.0" y="229.0" font-size="11" fill="#1f2933" text-anchor="middle">feed-forward network</text><text x="450.0" y="242.0" font-size="9" fill="#5b6b7b" text-anchor="middle">two-layer MLP, per position</text></g> |
| + | <g opacity="0.22"><rect x="350" y="150" width="200" height="26" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="450.0" y="167.0" font-size="10" fill="#1f2933" text-anchor="middle">add & norm</text></g> |
| + | <g opacity="0.22"><line x1="450" y1="330" x2="450" y2="314" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="286" x2="450" y2="259" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="205" x2="450" y2="178" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="150" x2="450" y2="80" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="450" y="66" font-size="11" fill="#5b6b7b" text-anchor="middle">to the next block</text></g> |
| + | <g opacity="0.22"><path d="M452,430 C606,430 606,299 554,299" fill="none" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><path d="M452,268 C606,268 606,163 554,163" fill="none" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><text x="648" y="240" font-size="10" fill="#5b6b7b" text-anchor="start">residual shortcuts</text></g> |
| + | <g opacity="0.22"><text x="648" y="130" font-size="11" fill="#5b6b7b" text-anchor="start">× N identical blocks</text></g> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. en/Deep Learning/10 Transformers/transformer-step2.svg | |
| @@ 0,0 1,30 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 545" width="900" height="545" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | <marker id="arrowmut" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs> |
| + | <rect width="900" height="545" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">The Transformer block, step 2: multi-head attention</text> |
| + | <g opacity="0.22"><rect x="280" y="110" width="340" height="330" rx="14" fill="#f9fafb" stroke="#c7d0d9" stroke-width="1.2"/></g> |
| + | <rect x="340" y="330" width="220" height="86" rx="10" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="450" y="345" font-size="10" fill="#1f2933" text-anchor="middle">multi-head self-attention</text> |
| + | <rect x="352" y="352" width="196" height="20" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.2"/><text x="450" y="366" font-size="9" fill="#1f2933" text-anchor="middle">concat, then W<tspan baseline-shift="super" font-size="7px">O</tspan></text> |
| + | <rect x="352" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="380" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="380" y1="382" x2="380" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <rect x="422" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="450" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="450" y1="382" x2="450" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <rect x="492" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="520" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="520" y1="382" x2="520" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <line x1="628" y1="392" x2="552" y2="392" stroke="#5b6b7b" stroke-width="1" stroke-dasharray="3 3"/><text x="634" y="389" font-size="10" fill="#5b6b7b" text-anchor="start">one head = scaled dot-product attention:</text><text x="634" y="403" font-size="10" fill="#5b6b7b" text-anchor="start">softmax(QK<tspan baseline-shift="super" font-size="7px">T</tspan>/√dₖ) V</text> |
| + | <g opacity="0.22"><rect x="365" y="492" width="170" height="30" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="450.0" y="511.0" font-size="11" fill="#1f2933" text-anchor="middle">token embeddings X</text></g> |
| + | <g opacity="0.22"><line x1="450" y1="492" x2="450" y2="470" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><circle cx="450" cy="455" r="12" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="450" y="460" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">+</text></g> |
| + | <g opacity="0.22"><rect x="560" y="441" width="170" height="28" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="645.0" y="459.0" font-size="10" fill="#1f2933" text-anchor="middle">positional encoding</text></g> |
| + | <g opacity="0.22"><line x1="560" y1="455" x2="465" y2="455" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="443" x2="450" y2="418" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><rect x="350" y="286" width="200" height="26" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="450.0" y="303.0" font-size="10" fill="#1f2933" text-anchor="middle">add & norm</text></g> |
| + | <g opacity="0.22"><rect x="350" y="205" width="200" height="52" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="450.0" y="229.0" font-size="11" fill="#1f2933" text-anchor="middle">feed-forward network</text><text x="450.0" y="242.0" font-size="9" fill="#5b6b7b" text-anchor="middle">two-layer MLP, per position</text></g> |
| + | <g opacity="0.22"><rect x="350" y="150" width="200" height="26" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="450.0" y="167.0" font-size="10" fill="#1f2933" text-anchor="middle">add & norm</text></g> |
| + | <g opacity="0.22"><line x1="450" y1="330" x2="450" y2="314" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="286" x2="450" y2="259" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="205" x2="450" y2="178" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="150" x2="450" y2="80" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="450" y="66" font-size="11" fill="#5b6b7b" text-anchor="middle">to the next block</text></g> |
| + | <g opacity="0.22"><path d="M452,430 C606,430 606,299 554,299" fill="none" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><path d="M452,268 C606,268 606,163 554,163" fill="none" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><text x="648" y="240" font-size="10" fill="#5b6b7b" text-anchor="start">residual shortcuts</text></g> |
| + | <g opacity="0.22"><text x="648" y="130" font-size="11" fill="#5b6b7b" text-anchor="start">× N identical blocks</text></g> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. en/Deep Learning/10 Transformers/transformer-step3.svg | |
| @@ 0,0 1,30 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 545" width="900" height="545" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | <marker id="arrowmut" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs> |
| + | <rect width="900" height="545" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">The Transformer block, step 3: embeddings and positions enter</text> |
| + | <g opacity="0.22"><rect x="280" y="110" width="340" height="330" rx="14" fill="#f9fafb" stroke="#c7d0d9" stroke-width="1.2"/></g> |
| + | <rect x="340" y="330" width="220" height="86" rx="10" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="450" y="345" font-size="10" fill="#1f2933" text-anchor="middle">multi-head self-attention</text> |
| + | <rect x="352" y="352" width="196" height="20" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.2"/><text x="450" y="366" font-size="9" fill="#1f2933" text-anchor="middle">concat, then W<tspan baseline-shift="super" font-size="7px">O</tspan></text> |
| + | <rect x="352" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="380" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="380" y1="382" x2="380" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <rect x="422" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="450" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="450" y1="382" x2="450" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <rect x="492" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="520" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="520" y1="382" x2="520" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <line x1="628" y1="392" x2="552" y2="392" stroke="#5b6b7b" stroke-width="1" stroke-dasharray="3 3"/><text x="634" y="389" font-size="10" fill="#5b6b7b" text-anchor="start">one head = scaled dot-product attention:</text><text x="634" y="403" font-size="10" fill="#5b6b7b" text-anchor="start">softmax(QK<tspan baseline-shift="super" font-size="7px">T</tspan>/√dₖ) V</text> |
| + | <rect x="365" y="492" width="170" height="30" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="450.0" y="511.0" font-size="11" fill="#1f2933" text-anchor="middle">token embeddings X</text> |
| + | <line x1="450" y1="492" x2="450" y2="470" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <circle cx="450" cy="455" r="12" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="450" y="460" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">+</text> |
| + | <rect x="560" y="441" width="170" height="28" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="645.0" y="459.0" font-size="10" fill="#1f2933" text-anchor="middle">positional encoding</text> |
| + | <line x1="560" y1="455" x2="465" y2="455" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <line x1="450" y1="443" x2="450" y2="418" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <g opacity="0.22"><rect x="350" y="286" width="200" height="26" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="450.0" y="303.0" font-size="10" fill="#1f2933" text-anchor="middle">add & norm</text></g> |
| + | <g opacity="0.22"><rect x="350" y="205" width="200" height="52" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="450.0" y="229.0" font-size="11" fill="#1f2933" text-anchor="middle">feed-forward network</text><text x="450.0" y="242.0" font-size="9" fill="#5b6b7b" text-anchor="middle">two-layer MLP, per position</text></g> |
| + | <g opacity="0.22"><rect x="350" y="150" width="200" height="26" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="450.0" y="167.0" font-size="10" fill="#1f2933" text-anchor="middle">add & norm</text></g> |
| + | <g opacity="0.22"><line x1="450" y1="330" x2="450" y2="314" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="286" x2="450" y2="259" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="205" x2="450" y2="178" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="150" x2="450" y2="80" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="450" y="66" font-size="11" fill="#5b6b7b" text-anchor="middle">to the next block</text></g> |
| + | <g opacity="0.22"><path d="M452,430 C606,430 606,299 554,299" fill="none" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><path d="M452,268 C606,268 606,163 554,163" fill="none" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><text x="648" y="240" font-size="10" fill="#5b6b7b" text-anchor="start">residual shortcuts</text></g> |
| + | <g opacity="0.22"><text x="648" y="130" font-size="11" fill="#5b6b7b" text-anchor="start">× N identical blocks</text></g> |
| + | </svg> |
| \ | No newline at end of file |
| en/Deep Learning/13 LSTM and GRU/gru-cell.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 760 446" width="760" height="446" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="760" height="446" fill="#ffffff"/><text x="380.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">Inside a GRU cell</text><text x="50.0" y="84.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text><line x1="30.0" y1="100.0" x2="690.0" y2="100.0" stroke="#1f2933" stroke-width="2.2"/><line x1="670.0" y1="100.0" x2="690.0" y2="100.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><text x="716.0" y="104.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="9px">t</tspan></text><circle cx="300.0" cy="100.0" r="15.0" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="300.0" y="105.0" font-family="Helvetica, Arial, sans-serif" font-size="16" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><text x="300.0" y="72.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">keep 1-z</text><circle cx="500.0" cy="100.0" r="15.0" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="500.0" y="105.0" font-family="Helvetica, Arial, sans-serif" font-size="16" font-weight="600" fill="#1f2933" text-anchor="middle">+</text><text x="500.0" y="72.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">combine</text><circle cx="500.0" cy="200.0" r="15.0" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="500.0" y="205.0" font-family="Helvetica, Arial, sans-serif" font-size="16" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><text x="500.0" y="176.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">write z</text><circle cx="200.0" cy="250.0" r="15.0" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="200.0" y="255.0" font-family="Helvetica, Arial, sans-serif" font-size="16" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><text x="158.0" y="254.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="end">reset r</text><rect x="60.0" y="320.0" width="122.0" height="52.0" rx="8" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="121.0" y="350.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">r</tspan></text><rect x="300.0" y="320.0" width="122.0" height="52.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="361.0" y="350.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">z</tspan></text><rect x="560.0" y="320.0" width="122.0" height="52.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="621.0" y="350.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">tanh cand</text><text x="55.0" y="425.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text><text x="135.0" y="425.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="9px">t</tspan></text><line x1="55.0" y1="413.0" x2="100.0" y2="403.0" stroke="#5b6b7b" stroke-width="1.4"/><line x1="135.0" y1="413.0" x2="100.0" y2="403.0" stroke="#5b6b7b" stroke-width="1.4"/><path d="M100.0 399.0 Q110.5 356.0 121.0 372.0" fill="none" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><path d="M100.0 399.0 Q230.5 356.0 361.0 372.0" fill="none" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><path d="M100.0 399.0 Q360.5 356.0 621.0 372.0" fill="none" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><line x1="121.0" y1="320.0" x2="200.0" y2="265.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><path d="M300.0 100.0 Q240.0 175.0 210.0 236.0" fill="none" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><path d="M200.0 250.0 Q360.0 300.0 601.0 320.0" fill="none" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><path d="M621.0 320.0 Q560.0 260.0 512.0 214.0" fill="none" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="361.0" y1="320.0" x2="490.0" y2="214.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><path d="M361.0 320.0 Q320.0 240.0 300.0 115.0" fill="none" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="500.0" y1="185.0" x2="500.0" y2="115.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/></svg> |
| \ | No newline at end of file |
| en/Deep Learning/13 LSTM and GRU/lstm-cell.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 840 473" width="840" height="473" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="840" height="473" fill="#ffffff"/><text x="420.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">Inside an LSTM cell</text><text x="50.0" y="79.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">c<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text><line x1="30.0" y1="95.0" x2="770.0" y2="95.0" stroke="#1f2933" stroke-width="2.2"/><line x1="750.0" y1="95.0" x2="770.0" y2="95.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><text x="796.0" y="99.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">c<tspan baseline-shift="sub" font-size="9px">t</tspan></text><circle cx="250.0" cy="95.0" r="15.0" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="250.0" y="100.0" font-family="Helvetica, Arial, sans-serif" font-size="16" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><text x="250.0" y="67.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">forget</text><circle cx="470.0" cy="95.0" r="15.0" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="470.0" y="100.0" font-family="Helvetica, Arial, sans-serif" font-size="16" font-weight="600" fill="#1f2933" text-anchor="middle">+</text><text x="470.0" y="67.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">input add</text><circle cx="470.0" cy="195.0" r="15.0" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="470.0" y="200.0" font-family="Helvetica, Arial, sans-serif" font-size="16" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><rect x="190.0" y="315.0" width="122.0" height="54.0" rx="8" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="251.0" y="346.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">f</tspan></text><rect x="342.0" y="315.0" width="122.0" height="54.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="403.0" y="346.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">i</tspan></text><rect x="590.0" y="315.0" width="122.0" height="54.0" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="651.0" y="346.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">o</tspan></text><rect x="342.0" y="398.0" width="122.0" height="48.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="403.0" y="426.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">tanh cand</text><text x="70.0" y="452.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text><text x="150.0" y="452.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="9px">t</tspan></text><line x1="70.0" y1="440.0" x2="118.0" y2="430.0" stroke="#5b6b7b" stroke-width="1.4"/><line x1="150.0" y1="440.0" x2="118.0" y2="430.0" stroke="#5b6b7b" stroke-width="1.4"/><path d="M118.0 426.0 Q184.5 351.0 251.0 369.0" fill="none" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><path d="M118.0 426.0 Q260.5 351.0 403.0 369.0" fill="none" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><path d="M118.0 426.0 Q384.5 351.0 651.0 369.0" fill="none" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><path d="M118.0 426.0 Q260.5 412.0 403.0 446.0" fill="none" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><line x1="251.0" y1="315.0" x2="250.0" y2="110.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="403.0" y1="315.0" x2="458.0" y2="210.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="403.0" y1="398.0" x2="482.0" y2="210.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="470.0" y1="180.0" x2="470.0" y2="110.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><circle cx="720.0" cy="153.0" r="15.0" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="720.0" y="158.0" font-family="Helvetica, Arial, sans-serif" font-size="16" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><rect x="590.0" y="200.0" width="122.0" height="46.0" rx="8" fill="#e4f5f4" stroke="#2a9d9a" stroke-width="1.6"/><text x="651.0" y="227.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">tanh</text><line x1="651.0" y1="95.0" x2="651.0" y2="200.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><path d="M651.0 200.0 Q675.0 187.0 707.0 159.0" fill="none" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="651.0" y1="315.0" x2="720.0" y2="168.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="720.0" y1="153.0" x2="770.0" y2="153.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><text x="796.0" y="157.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="9px">t</tspan></text></svg> |
| \ | No newline at end of file |
| en/Deep Learning/14 Attention/attention-weights.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 760 404" width="760" height="404" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="760" height="404" fill="#ffffff"/><text x="380.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">Attention: context is a weighted sum of encoder states</text><rect x="60.0" y="300.0" width="110.0" height="44.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="115.0" y="326.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="9px">1</tspan></text><rect x="230.0" y="300.0" width="110.0" height="44.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="285.0" y="326.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="9px">2</tspan></text><rect x="400.0" y="300.0" width="110.0" height="44.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="455.0" y="326.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="9px">3</tspan></text><rect x="570.0" y="300.0" width="110.0" height="44.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="625.0" y="326.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="9px">4</tspan></text><text x="370.0" y="362.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">encoder states</text><text x="370.0" y="384.0" font-family="Helvetica, Arial, sans-serif" font-size="11" font-style="italic" fill="#5b6b7b" text-anchor="middle">thicker line = larger weight</text><circle cx="380.0" cy="175.0" r="34.0" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="380.0" y="180.1" font-family="Helvetica, Arial, sans-serif" font-size="15" fill="#1f2933" text-anchor="middle">c</text><text x="454.0" y="175.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">context</text><circle cx="380.0" cy="60.0" r="30.0" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="380.0" y="65.1" font-family="Helvetica, Arial, sans-serif" font-size="15" fill="#1f2933" text-anchor="middle">s</text><text x="465.0" y="60.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">decoder query</text><line x1="380.0" y1="90.0" x2="380.0" y2="141.0" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><line x1="115.0" y1="300.0" x2="380.0" y2="209.0" stroke="#38a05a" stroke-width="2.2800000000000002"/><text x="173.3" y="272.0" font-family="Helvetica, Arial, sans-serif" font-size="12" font-weight="600" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="9px">1</tspan></text><line x1="285.0" y1="300.0" x2="380.0" y2="209.0" stroke="#38a05a" stroke-width="6.15"/><text x="305.9" y="272.0" font-family="Helvetica, Arial, sans-serif" font-size="12" font-weight="600" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="9px">2</tspan></text><line x1="455.0" y1="300.0" x2="380.0" y2="209.0" stroke="#38a05a" stroke-width="3.45"/><text x="438.5" y="272.0" font-family="Helvetica, Arial, sans-serif" font-size="12" font-weight="600" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="9px">3</tspan></text><line x1="625.0" y1="300.0" x2="380.0" y2="209.0" stroke="#38a05a" stroke-width="1.92"/><text x="571.1" y="272.0" font-family="Helvetica, Arial, sans-serif" font-size="12" font-weight="600" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="9px">4</tspan></text></svg> |
| \ | No newline at end of file |
| en/Deep Learning/15 Transformers/transformer-block.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 560 600" width="560" height="600" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="560" height="600" fill="#ffffff"/><text x="280.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">A Transformer block</text><rect x="130.0" y="50.0" width="240.0" height="46.0" rx="8" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><text x="250.0" y="77.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">output</text><rect x="130.0" y="140.0" width="240.0" height="46.0" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="250.0" y="167.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">Add and Norm</text><rect x="130.0" y="240.0" width="240.0" height="46.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="250.0" y="267.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">Feed Forward</text><rect x="130.0" y="340.0" width="240.0" height="46.0" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="250.0" y="367.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">Add and Norm</text><rect x="130.0" y="440.0" width="240.0" height="46.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="250.0" y="467.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">Multi-Head Attention</text><rect x="130.0" y="530.0" width="240.0" height="46.0" rx="8" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><text x="250.0" y="557.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">input</text><line x1="250.0" y1="530.0" x2="250.0" y2="486.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="250.0" y1="440.0" x2="250.0" y2="386.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="250.0" y1="340.0" x2="250.0" y2="286.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="250.0" y1="240.0" x2="250.0" y2="186.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="250.0" y1="140.0" x2="250.0" y2="96.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><path d="M370.0 494.0 Q450.0 428.5 370.0 363.0" fill="none" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><text x="468.0" y="413.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">residual</text><path d="M370.0 294.0 Q450.0 228.5 370.0 163.0" fill="none" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><text x="468.0" y="213.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">residual</text></svg> |
| \ | No newline at end of file |
| en/MLOps/01 Introduction.md .. | |
| @@ 2,11 2,6 @@ | |
| > Lesson 01 · MLOps | |
| - | **Objectives** |
| - | - Understand what MLOps is and why models need more than good accuracy. |
| - | - Recognize how ML systems differ from traditional software. |
| - | - See the lifecycle the rest of the course builds on. |
| - | |
| ## What is MLOps? | |
| MLOps applies engineering and operations practices to the machine learning lifecycle so models | |
| en/Machine Learning/05 Linear classification.md .. | |
| @@ 60,7 60,7 @@ | |
| The batch variant computes the gradient over the whole training set before each step, a smooth descent that reads every example every time. The stochastic variant (SGD) steps on one example at a time, cheap and noisy, and is the default on large datasets. If $\alpha$ is too large the iterates can diverge, if too small convergence crawls. | |
| - | *Remark:* fancier optimizers exist, momentum, Adam and their cousins, refinements of this same rule that matter for deep networks ([Optimization](/en/Deep%20Learning/06%20Optimization) in the Deep Learning course). Everything in this module needs only the plain version. |
| + | *Remark:* fancier optimizers exist, momentum, Adam and their cousins, refinements of this same rule that matter for deep networks ([Optimization](/en/Deep%20Learning/03%20Optimization) in the Deep Learning course). Everything in this module needs only the plain version. |
| On a single misclassified example the gradient of the criterion is $-y^{(i)} x^{(i)}$, so the stochastic step is the perceptron update: on a mistake, | |
| en/Machine Learning/06 Multilayer neural networks.md .. | |
| @@ 1,6 1,6 @@ | |
| # 6. Multilayer neural networks | |
| - | A single linear unit only draws a straight boundary. Stacking many simple units with a nonlinearity between them gives a multilayer neural network, which fits curved boundaries and learns its own features. This module builds that model the gentle way: take the logistic regression of the previous module, draw it as a graph, and make it deep, one step at a time. The recipe is the one every module has used: a model (layers, run by forward propagation), a loss function matched to the task, and gradient descent, now powered by backpropagation. The story then continues the way practice forced it to: gradients vanish in deep stacks, better activations revive them, good practices make training behave, and gradient descent itself gets an upgrade. This module is the gateway to the [Deep Learning](/en/Deep%20Learning) course, which develops every topic here in depth. |
| + | A single linear unit only draws a straight boundary. Stacking many simple units with a nonlinearity between them gives a multilayer neural network, which fits curved boundaries and learns its own features. This module builds that model the gentle way: take the logistic regression of the previous module, draw it as a graph, and make it deep, one step at a time. The recipe is the one every module has used: a model (layers, run by forward propagation), a loss function matched to the task, and gradient descent, now powered by backpropagation. The story then continues in the [Deep Learning](/en/Deep%20Learning) course, which picks up exactly here: why gradients vanish in deep stacks, the activations that revive them, the good practices that make training behave, and the upgrades to gradient descent. |
| ## 6.1 Linear versus nonlinear | |
| @@ 138,7 138,9 @@ | |
|  | |
| - | *Forward propagation computes and caches the activations, backpropagation sends the loss gradient back through the same edges. The [Backpropagation](/en/Deep%20Learning/05%20Backpropagation) lesson of the Deep Learning course derives it step by step.* |
| + | *Forward propagation computes and caches the activations, backpropagation sends the loss gradient back through the same edges.* |
| + | |
| + | One practical point completes the recipe: the gradient of step 2 is computed on a mini-batch. [Linear classification](/en/Machine%20Learning/05%20Linear%20classification) offered two extremes, the full batch or a single example per step. Networks train in between, a small batch per step: a gradient accurate enough to make progress, a step cheap enough to take thousands of them, and the vectorized forward propagation of section 6.2.3 processes the whole mini-batch in one matrix product per layer. |
| <details class="proof"> | |
| <summary>Full example: one gradient descent step on the tiny network</summary> | |
| @@ 161,7 163,7 @@ | |
| $$\delta^{[1]}_1 = w^{[2]}_{11}\, \delta^{[2]}\, \sigma'(0) = 1 \cdot 0.80 \cdot 0.25 = 0.20, \qquad \delta^{[1]}_2 = 1 \cdot 0.80 \cdot 0.10 = 0.08$$ | |
| - | (the constant bias neuron takes no error, and note the small slope $0.10$ of unit 2: section 6.5 returns to it). Then the same outer product against $\bar{x} = (1, 1, 2)$: |
| + | (the constant bias neuron takes no error, and note the small slope $0.10$ of unit 2: the [Deep Learning](/en/Deep%20Learning) course makes a story of it). Then the same outer product against $\bar{x} = (1, 1, 2)$: |
| $$\frac{\partial L}{\partial W^{[1]}} = \delta^{[1]}\, \bar{x}^T = \begin{pmatrix} 0.20 & 0.20 & 0.40 \\ 0.08 & 0.08 & 0.16 \end{pmatrix}$$ | |
| @@ 179,88 181,7 @@ | |
| </details> | |
| - | ## 6.5 The vanishing gradient |
| - | |
| - | The backpropagation formula hides a trap. Every layer the error crosses multiplies $\delta^{[l]}$ by the local slope $g'(z^{[l]})$, so the gradient reaching layer 1 contains one such factor per layer. With sigmoid activations those factors are small by construction: |
| - | |
| - | $$\boxed{ \sigma'(z) = \sigma(z)\left(1 - \sigma(z)\right) \le \tfrac{1}{4} }$$ |
| - | |
| - | The result is the vanishing gradient: the layers near the output learn, the layers near the input receive almost nothing and barely move. Deep sigmoid networks stall, and the fix is not a better optimizer, it is a better activation (section 6.6). |
| - | |
| - | <details class="proof"> |
| - | <summary>Proof: the gradient shrinks geometrically with depth</summary> |
| - | |
| - | **Step 1: the sigmoid's slope never exceeds $1/4$.** Differentiate $\sigma(z) = (1 + e^{-z})^{-1}$ with the chain rule: |
| - | |
| - | $$\sigma'(z) = \frac{e^{-z}}{\left(1 + e^{-z}\right)^2} = \frac{1}{1 + e^{-z}} \cdot \frac{e^{-z}}{1 + e^{-z}} = \sigma(z)\left(1 - \sigma(z)\right)$$ |
| - | |
| - | Write $s = \sigma(z) \in (0, 1)$. The product $s(1 - s)$ is a downward parabola in $s$, largest at $s = \tfrac{1}{2}$ where it equals $\tfrac{1}{4}$. So the bound holds, with equality only at $z = 0$, and saturation makes it far worse: in the worked example of section 6.2.4, hidden unit 2 sits at $\sigma(2) \approx 0.88$, where the slope has already dropped to $0.88 \cdot 0.12 \approx 0.10$. |
| - | |
| - | **Step 2: backpropagation multiplies those slopes.** Take the simplest deep network, a chain of $L$ layers with one unit each, so every quantity is a scalar. Applying the chain rule from the output back to layer 1, each layer crossed contributes the factor $\partial z^{[l]} / \partial z^{[l-1]} = w^{[l]}\, \sigma'(z^{[l-1]})$: |
| - | |
| - | $$\frac{\partial L}{\partial z^{[1]}} = \frac{\partial L}{\partial z^{[L]}} \prod_{l=2}^{L} w^{[l]}\, \sigma'(z^{[l-1]})$$ |
| - | |
| - | With weights of typical size $|w^{[l]}| \le 1$, every factor is at most $\tfrac{1}{4}$ in absolute value, so |
| - | |
| - | $$\boxed{ \left|\frac{\partial L}{\partial z^{[1]}}\right| \le \left(\tfrac{1}{4}\right)^{L-1} \left|\frac{\partial L}{\partial z^{[L]}}\right| }$$ |
| - | |
| - | Ten layers already shrink the gradient by about $10^{-6}$. The full matrix case is the recursion of section 6.4, with the same conclusion. $\blacksquare$ |
| - | |
| - | </details> |
| - | |
| - | Weights much larger than $1$ only trade the problem for its mirror image, the exploding gradient. The [Initialization and vanishing gradients](/en/Deep%20Learning/07%20Initialization%20and%20vanishing%20gradients) lesson of the Deep Learning course gives the full treatment. |
| - | |
| - | ## 6.6 Activation functions |
| - | |
| - | So which activation should $g$ be? The candidates, in the order history tried them: |
| - | |
| - | $$\boxed{ \sigma(z) = \frac{1}{1 + e^{-z}}, \qquad \tanh(z), \qquad \mathrm{ReLU}(z) = \max(0, z) }$$ |
| - | |
| - | The sigmoid saturates in both tails, which is exactly what section 6.5 punished, and its outputs are never negative, so a unit's incoming weights all receive gradients of the same sign and the updates zig-zag. The zero-centered $\tanh$ removes that bias but still saturates. ReLU keeps a slope of exactly $1$ on its whole positive side, so the shrinking factors of section 6.5 disappear, and it costs almost nothing to compute. That is why it is the default hidden activation today. |
| - | |
| - |  |
| - | |
| - | *The tanh is zero-centered while the sigmoid is not, and ReLU stays linear for positive inputs.* |
| - | |
| - | ReLU has one blind spot: a unit whose input stays negative outputs $0$, has slope $0$, and stops learning, a dead unit. Variants such as Leaky ReLU, $\max(0.01 z, z)$, and ELU keep a small slope on the negative side to prevent it. In practice: start with ReLU, try its variants if units die, and keep the sigmoid only where section 6.3 needs it, at the output of a binary classifier. The [Activation functions](/en/Deep%20Learning/03%20Activation%20functions) lesson of the Deep Learning course compares them all. |
| - | |
| - | ## 6.7 Good practices |
| - | |
| - | Five habits make the difference between a network that trains and one that stalls. |
| - | |
| - | **Train on mini-batches.** [Linear classification](/en/Machine%20Learning/05%20Linear%20classification) offered two extremes, the full batch or a single example per step. Networks train on mini-batches, a small batch per step: a gradient accurate enough to make progress, a step cheap enough to take thousands of them, and the vectorized forward propagation of section 6.2.3 processes the whole mini-batch in one matrix product per layer. |
| - | |
| - | **Initialize with care.** Equal weights would make every unit of a layer compute the same thing forever, so start small and random to break the symmetry. The scale matters too: too small and the activations shrink toward zero layer after layer, too large and they saturate. Scale the variance by the unit's number of inputs, Xavier for tanh, He for ReLU. |
| - | |
| - | **Center and normalize the inputs.** Standardize each feature (subtract its mean, divide by its standard deviation), so no feature dominates the first dot products and the all-positive-input zig-zag of section 6.6 disappears at the first layer. |
| - | |
| - | **Dropout.** Randomly zero a fraction of units during training so none can lean on its neighbors, a regularizer in the spirit of [General concepts](/en/Machine%20Learning/02%20General%20concepts). At prediction time every unit stays on and outputs are scaled by the keep probability, which approximates averaging the many thinned networks ([Regularization and dropout](/en/Deep%20Learning/09%20Regularization%20and%20dropout)). |
| - | |
| - | **Sanity-check before training long.** A freshly initialized $K$-class classifier should start near the loss $\ln K$ (about $2.3$ for $K = 10$). A tiny training set should be easy to overfit: if the network cannot, the code is broken. Watch the training and validation curves. And since backpropagation is error-prone, check its analytic gradient against a numerical finite-difference estimate: |
| - | |
| - | $$\boxed{ \frac{\partial L}{\partial w} \approx \frac{L(w + \varepsilon) - L(w - \varepsilon)}{2\varepsilon} }$$ |
| - | |
| - | ## 6.8 Gradient descent, improved |
| - | |
| - | Plain gradient descent takes the steepest step and nothing more, and three landscapes defeat it: plateaus, where the slope is nearly zero and progress stalls, saddle points (common in high dimension), where the gradient is exactly zero without being a minimum, and ravines, steep in one direction and shallow in another, where the step oscillates across the walls while crawling along the floor. |
| - | |
| - | **Momentum** treats the update as a velocity with friction: gradients accumulate, persistent directions build up speed, oscillating ones cancel out: |
| - | |
| - | $$\boxed{ v \leftarrow \rho\, v + \nabla_W L, \qquad W \leftarrow W - \alpha\, v }$$ |
| - | |
| - | with the friction $\rho$ typically around $0.9$. |
| - | |
| - | **RMSProp** gives each parameter its own step size, dividing by a running average of the gradient's magnitude, so steep directions are tamed and flat ones sped up: |
| - | |
| - | $$\boxed{ m \leftarrow \beta\, m + (1 - \beta) \left(\nabla_W L\right)^2, \qquad W \leftarrow W - \frac{\alpha}{\sqrt{m} + \varepsilon}\, \nabla_W L }$$ |
| - | |
| - | **Adam** combines the two ideas, a velocity for the direction and a per-parameter scale for the step (the full version also corrects a startup bias in $v$ and $m$), and is the default optimizer in practice: |
| - | |
| - | $$\boxed{ v \leftarrow \beta_1 v + (1 - \beta_1)\, \nabla_W L, \qquad m \leftarrow \beta_2 m + (1 - \beta_2) \left(\nabla_W L\right)^2, \qquad W \leftarrow W - \alpha\, \frac{v}{\sqrt{m} + \varepsilon} }$$ |
| - | |
| - | Two habits complete the picture: decay the learning rate as training advances, and remember that all three methods still consume the mini-batch gradients of section 6.7, they only spend them more wisely. The [Optimization](/en/Deep%20Learning/06%20Optimization) lesson of the Deep Learning course derives each one and adds the learning-rate schedules. |
| - | |
| - | *This module is the doorway to the [Deep Learning](/en/Deep%20Learning) course, which develops architectures, optimizers, initialization, normalization, and regularization in full. The next module returns to linear models from a new angle, the maximum-margin classifier.* |
| + | *This module is the doorway to the [Deep Learning](/en/Deep%20Learning) course, which picks the story up exactly here: the activation functions, the optimizers, initialization, normalization, regularization, and the architectures built on them. The next module returns to linear models from a new angle, the maximum-margin classifier.* |
| --- | |
| Next: [Support Vector Machines](/en/Machine%20Learning/07%20Support%20Vector%20Machines) · [Course overview](/en/Machine%20Learning) | |
| en/Machine Learning/06 Multilayer neural networks/activations.png .. /dev/null | |
| en/Mathematics/01 Mathematical refresher.md .. | |
| @@ 2,12 2,6 @@ | |
| This module gathers the mathematical tools the rest of the course leans on: a little linear algebra, the language of expectation and covariance, the multivariate Gaussian, and the four probability quantities (likelihood, prior, posterior, evidence) that the next module turns into a way of reasoning. It is a reference to return to, not a full treatment. | |
| - | **Objectives** |
| - | - Recall the vector and matrix operations used throughout: dot product, matrix-vector product, transpose, inverse, and norm. |
| - | - Define expectation, variance, and covariance, and assemble the covariance matrix. |
| - | - Write the multivariate Gaussian density and read its shape from the covariance. |
| - | - Name the likelihood, prior, posterior, and evidence, and relate them by Bayes' rule. |
| - | |
| ## 1.1 Linear algebra | |
| A feature vector lives in $\mathbb{R}^n$ and a dataset stacks such vectors into a matrix. The dot product of two vectors sums their elementwise products: | |
| fr/Deep Learning.md .. | |
| @@ 2,25 2,20 @@ | |
| Les réseaux de neurones, du simple perceptron aux transformeurs modernes : comment la profondeur, les bonnes fonctions d'activation et l'entraînement par gradient permettent à un modèle d'apprendre ses propres caractéristiques au lieu de les concevoir à la main. | |
| - | **Prérequis :** le cours [Machine Learning](/fr/Machine%20Learning) (en particulier le perceptron dans [Classification linéaire](/fr/Machine%20Learning/05%20Linear%20classification)), Python de base, calcul différentiel et algèbre linéaire. |
| + | **Prérequis :** le cours [Machine Learning](/fr/Machine%20Learning), en particulier [Réseaux de neurones multi-couches](/fr/Machine%20Learning/06%20Multilayer%20neural%20networks), qui construit le modèle et couvre les pertes et la rétropropagation utilisées tout au long de ce cours. Python de base, calcul différentiel et algèbre linéaire. |
| ## Programme | |
| 1. [Introduction](/fr/Deep%20Learning/01%20Introduction) | |
| - | 2. [Perceptron multicouche](/fr/Deep%20Learning/02%20Multilayer%20perceptron) |
| - | 3. [Fonctions d'activation](/fr/Deep%20Learning/03%20Activation%20functions) |
| - | 4. [Fonctions de perte et couches de sortie](/fr/Deep%20Learning/04%20Loss%20functions%20and%20output%20layers) |
| - | 5. [Rétropropagation](/fr/Deep%20Learning/05%20Backpropagation) |
| - | 6. [Optimisation](/fr/Deep%20Learning/06%20Optimization) |
| - | 7. [Initialisation et disparition du gradient](/fr/Deep%20Learning/07%20Initialization%20and%20vanishing%20gradients) |
| - | 8. [Normalisation](/fr/Deep%20Learning/08%20Normalization) |
| - | 9. [Régularisation et dropout](/fr/Deep%20Learning/09%20Regularization%20and%20dropout) |
| - | 10. [Réseaux convolutifs](/fr/Deep%20Learning/10%20Convolutional%20networks) |
| - | 11. [Plongements et apprentissage de représentations](/fr/Deep%20Learning/11%20Embeddings%20and%20representation%20learning) |
| - | 12. [Réseaux récurrents](/fr/Deep%20Learning/12%20Recurrent%20networks) |
| - | 13. [LSTM et GRU](/fr/Deep%20Learning/13%20LSTM%20and%20GRU) |
| - | 14. [Attention](/fr/Deep%20Learning/14%20Attention) |
| - | 15. [Transformeurs](/fr/Deep%20Learning/15%20Transformers) |
| + | 2. [Fonctions d'activation](/fr/Deep%20Learning/02%20Activation%20functions) |
| + | 3. [Optimisation](/fr/Deep%20Learning/03%20Optimization) |
| + | 4. [Entraîner les réseaux profonds](/fr/Deep%20Learning/04%20Training%20deep%20networks) |
| + | 5. [Réseaux convolutifs](/fr/Deep%20Learning/05%20Convolutional%20networks) |
| + | 6. [Plongements et apprentissage de représentations](/fr/Deep%20Learning/06%20Embeddings%20and%20representation%20learning) |
| + | 7. [Réseaux récurrents](/fr/Deep%20Learning/07%20Recurrent%20networks) |
| + | 8. [LSTM et GRU](/fr/Deep%20Learning/08%20LSTM%20and%20GRU) |
| + | 9. [Attention](/fr/Deep%20Learning/09%20Attention) |
| + | 10. [Transformeurs](/fr/Deep%20Learning/10%20Transformers) |
| --- | |
| [Machine Learning](/fr/Machine%20Learning) · [MLOps](/fr/MLOps) · [Accueil](/fr) | |
| fr/Deep Learning/01 Introduction.md .. | |
| @@ 1,13 1,6 @@ | |
| # 1. Introduction | |
| - | Ce cours prolonge directement le cours de Machine Learning, qui concluait la partie [Classification linéaire](/fr/Machine%20Learning/05%20Linear%20classification) sur une remarque clé : un perceptron est une unité unique, et empilé en couches il devient un réseau de neurones. Cette leçon rend ce pont explicite. Elle rappelle ce qu'une seule unité peut faire, montre la tâche concrète (XOR) où une unité unique échoue, et fixe la notation utilisée dans tout le reste du cours. |
| - | |
| - | **Objectifs** |
| - | - Rappeler le perceptron comme une unité unique avec une activation en marche d'escalier et une frontière linéaire. |
| - | - Comprendre pourquoi une seule unité ne peut pas résoudre XOR, ce qui motive les couches cachées. |
| - | - Comprendre ce que signifie « profond » et pourquoi les couches cachées apprennent des caractéristiques. |
| - | - Adopter la notation à biais explicite, par couche, utilisée tout au long de ce cours. |
| - | - Lire un réseau comme une composition d'applications de couches, de l'entrée à la prédiction. |
| + | Ce cours prolonge directement le cours de Machine Learning, dont le module [Réseaux de neurones multi-couches](/fr/Machine%20Learning/06%20Multilayer%20neural%20networks) a construit le réseau lui-même : le modèle, ses pertes et la rétropropagation. Cette leçon réancre ce pont. Elle rappelle ce qu'une seule unité peut faire, montre la tâche concrète (XOR) où une unité unique échoue, fixe la notation utilisée dans tout le reste du cours, puis réénonce le perceptron multicouche et sa boucle d'entraînement dans cette notation. |
| ## 1.1 Le perceptron, rappel | |
| @@ 48,7 41,7 @@ | |
| Le bénéfice est l'**apprentissage de représentations**. En apprentissage automatique classique, on conçoit les caractéristiques à la main, puis on les fournit à un modèle linéaire. Dans un réseau profond, les couches cachées apprennent leurs propres caractéristiques à partir de l'entrée brute : les premières couches capturent des motifs simples et les couches ultérieures les combinent en motifs plus abstraits. On spécifie l'architecture et l'objectif, et le réseau découvre les représentations intermédiaires par l'entraînement. | |
| - | *Remarque :* empiler des applications linéaires seules reviendrait à une seule application linéaire, donc l'activation non linéaire $g$ entre les couches est essentielle. Sans elle, aucune profondeur n'ajouterait de puissance expressive. Les fonctions d'activation sont traitées dans les leçons suivantes. |
| + | *Remarque :* empiler des applications linéaires seules reviendrait à une seule application linéaire, donc l'activation non linéaire $g$ entre les couches est essentielle. Sans elle, aucune profondeur n'ajouterait de puissance expressive. Les fonctions d'activation sont traitées dans la leçon suivante. |
| ## 1.4 Notation pour ce cours | |
| @@ 97,7 90,25 @@ | |
| Chaque flèche porte un poids issu de $W^{[l]}$, et chaque unité cachée et de sortie ajoute son biais issu de $b^{[l]}$ avant d'appliquer son activation. Cette couche cachée à deux unités est exactement ce qui permet au réseau de résoudre XOR, la tâche qui mettait en échec une unité unique. | |
| - | *La prochaine leçon formalise cette image sous la forme du perceptron multicouche, en écrivant la passe avant complète couche par couche et en choisissant les fonctions d'activation.* |
| + | ## 1.5 Le perceptron multicouche, rappel |
| + | |
| + | Le réseau que décrivent ces symboles est le perceptron multicouche (MLP), construit pas à pas dans [Réseaux de neurones multi-couches](/fr/Machine%20Learning/06%20Multilayer%20neural%20networks) : prendre la régression logistique, insérer des couches cachées faites des mêmes unités à produit scalaire, et lire la composition de la section 1.4.2 de gauche à droite. L'entraînement y a été réglé aussi, et un pas complet se traduit dans la nouvelle notation d'un seul geste. |
| + | |
| + | **En avant.** Propager l'entrée à travers $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$ et $a^{[l]} = g^{[l]}(z^{[l]})$, en mettant en cache chaque $z^{[l]}$ et $a^{[l]}$ au passage. |
| + | |
| + | **La perte.** Comparer $\hat{y} = a^{[L]}$ à l'étiquette avec la perte adaptée à la tâche : erreur quadratique derrière une sortie identité pour la régression, entropie croisée derrière une sigmoïde ou une softmax pour la classification. |
| + | |
| + | **En arrière.** Renvoyer le gradient de la perte par le même câblage avec la règle de dérivation en chaîne, couche par couche : |
| + | |
| + | $$\boxed{ \delta^{[l]} = \left((W^{[l+1]})^T \delta^{[l+1]}\right) \odot g'^{[l]}\!\left(z^{[l]}\right), \qquad \frac{\partial L}{\partial W^{[l]}} = \delta^{[l]} (a^{[l-1]})^T, \qquad \frac{\partial L}{\partial b^{[l]}} = \delta^{[l]} }$$ |
| + | |
| + | La seule nouveauté est la dernière égalité : le gradient du biais reçoit désormais sa propre ligne, car $b^{[l]}$ n'est plus une colonne de $W^{[l]}$ alimentée par un neurone constant. |
| + | |
| + | **La mise à jour.** Faire un pas de gradient sur un mini-lot (la leçon 3 améliore ce pas avec momentum, RMSProp et Adam). |
| + | |
| + | Si une étape reste floue, le module la donne en entier : [le modèle](/fr/Machine%20Learning/06%20Multilayer%20neural%20networks#62-rendre-la-regression-logistique-profonde) avec un exemple détaillé sur le graphe, [les pertes](/fr/Machine%20Learning/06%20Multilayer%20neural%20networks#63-la-fonction-de-perte) et [la rétropropagation](/fr/Machine%20Learning/06%20Multilayer%20neural%20networks#64-comment-optimiser-les-parametres-) avec un pas d'entraînement numérique complet. Ce cours les tient pour acquis et possède tout ce qui suit : les fonctions d'activation (prochaine leçon), les optimiseurs et leurs bonnes pratiques (leçon 3), et la boîte à outils de l'entraînement, initialisation, normalisation et régularisation (leçon 4). |
| + | |
| + | *Le modèle, ses pertes et son entraînement ont été construits dans le cours de Machine Learning. La prochaine leçon reprend l'histoire au choix qui rend la profondeur utile : les fonctions d'activation.* |
| --- | |
| - | Suivant : [Perceptron multicouche](/fr/Deep%20Learning/02%20Multilayer%20perceptron) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| + | Suivant : [Fonctions d'activation](/fr/Deep%20Learning/02%20Activation%20functions) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| fr/Deep Learning/03 Activation functions.md .. fr/Deep Learning/02 Activation functions.md | |
| @@ 1,15 1,8 @@ | |
| - | # 3. Fonctions d'activation |
| + | # 2. Fonctions d'activation |
| Chaque couche calcule une pré-activation $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$ puis une activation $a^{[l]} = g^{[l]}(z^{[l]})$. Le choix de la non-linéarité $g^{[l]}$ est ce qui rend la profondeur utile. Cette leçon explique pourquoi une fonction $g$ non linéaire est nécessaire, passe en revue les familles sigmoïde, tanh et ReLU, présente la softmax utilisée en sortie et donne des conseils pratiques sur l'activation à choisir. | |
| - | **Objectifs** |
| - | - Montrer qu'un empilement de couches purement linéaires se réduit à une seule application linéaire. |
| - | - Définir la sigmoïde et la tanh, dériver leurs dérivées et expliquer la saturation. |
| - | - Passer en revue la famille ReLU (ReLU, leaky ReLU, PReLU, ELU, GELU) et le problème des unités mortes. |
| - | - Définir la softmax et la placer en sortie plutôt que dans les couches cachées. |
| - | - Donner une règle empirique simple pour choisir une activation par couche. |
| - | |
| - | ## 3.1 Pourquoi la non-linéarité est nécessaire |
| + | ## 2.1 Pourquoi la non-linéarité est nécessaire |
| Supposons que chaque activation soit l'identité, $g^{[l]}(z) = z$. Alors chaque couche se réduit à $a^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$, et composer deux d'entre elles donne $W^{[2]}(W^{[1]} x + b^{[1]}) + b^{[2]} = (W^{[2]} W^{[1]}) x + (W^{[2]} b^{[1]} + b^{[2]})$. C'est de nouveau de la forme $W x + b$. Par récurrence, l'ensemble du réseau à $L$ couches se réduit à une seule application affine : | |
| @@ 19,13 12,13 @@ | |
| *Remarque :* le biais est conservé ici de façon explicite sous la forme $b^{[l]}$, contrairement au cours de Machine Learning où l'ordonnée à l'origine était intégrée dans $w^T x$ via l'entrée augmentée $x_0 = 1$. Dans ce cours de Deep Learning, chaque couche possède sa propre matrice de poids $W^{[l]}$ et son propre vecteur de biais $b^{[l]}$. | |
| - | ## 3.2 Sigmoïde et tanh |
| + | ## 2.2 Sigmoïde et tanh |
| - |  |
| + |  |
| *Fonctions d'activation courantes : la sigmoïde et la tanh, bornées, saturent dans leurs queues, tandis que ReLU et ses variantes restent linéaires pour les entrées positives.* | |
| - | ### 3.2.1 Sigmoïde |
| + | ### 2.2.1 Sigmoïde |
| La sigmoïde écrase n'importe quelle pré-activation réelle dans l'intervalle ouvert $(0, 1)$ : | |
| @@ 35,7 28,7 @@ | |
| $$\boxed{ \sigma'(z) = \sigma(z)\left(1 - \sigma(z)\right) }$$ | |
| - | ### 3.2.2 Tanh |
| + | ### 2.2.2 Tanh |
| La tangente hyperbolique est une sigmoïde remise à l'échelle et centrée en zéro, dont la sortie est dans $(-1, 1)$. Sa dérivée s'exprime elle aussi à partir de la valeur avant : | |
| @@ 43,15 36,42 @@ | |
| *Remarque :* $\tanh$ est centrée en zéro alors que $\sigma$ ne l'est pas, si bien que $\tanh$ s'entraîne souvent un peu mieux comme activation cachée. Les deux sont reliées par $\tanh(z) = 2\sigma(2z) - 1$. | |
| - | ### 3.2.3 Saturation |
| + | ### 2.2.3 Saturation |
| - | Les deux courbes s'aplatissent dans leurs queues. Pour de grandes valeurs de $|z|$, la sortie est proche d'une constante ($0$ ou $1$ pour $\sigma$, $\pm 1$ pour $\tanh$), donc la dérivée est proche de zéro : $\sigma'(z) \to 0$ et $\tanh'(z) \to 0$. Une unité située dans cette région plate est dite saturée, et elle ne transmet presque aucun gradient vers l'arrière. Lorsque de nombreux facteurs de ce type se multiplient à travers un empilement profond, le gradient tend vers zéro : c'est le problème de disparition du gradient revu dans [Initialisation et disparition du gradient](/fr/Deep%20Learning/07%20Initialization%20and%20vanishing%20gradients). |
| + | Les deux courbes s'aplatissent dans leurs queues. Pour de grandes valeurs de $|z|$, la sortie est proche d'une constante ($0$ ou $1$ pour $\sigma$, $\pm 1$ pour $\tanh$), donc la dérivée est proche de zéro : $\sigma'(z) \to 0$ et $\tanh'(z) \to 0$. Une unité située dans cette région plate est dite saturée, et elle ne transmet presque aucun gradient vers l'arrière. Lorsque de nombreux facteurs de ce type se multiplient à travers un empilement profond, le gradient tend vers zéro : c'est la disparition du gradient, rendue précise dans la section suivante. |
| - |  |
| + |  |
| *Dérivées des activations : les gradients de la sigmoïde et de la tanh disparaissent dans les queues, tandis que le gradient de ReLU vaut 1 partout où l'unité est active.* | |
| - | ## 3.3 La famille ReLU |
| + | ## 2.3 La disparition du gradient |
| + | |
| + | La saturation n'est pas un défaut cosmétique, elle décide si une pile profonde peut s'entraîner tout court. La rétropropagation multiplie l'erreur par la pente locale $g'(z^{[l]})$ à chaque couche traversée, si bien que le gradient qui atteint la couche 1 contient un tel facteur par couche. Avec des activations sigmoïdes, ces facteurs sont petits par construction : la section 2.2.1 a donné $\sigma'(z) = \sigma(z)(1 - \sigma(z))$, et ce produit ne dépasse jamais |
| + | |
| + | $$\boxed{ \sigma'(z) = \sigma(z)\left(1 - \sigma(z)\right) \le \tfrac{1}{4} }$$ |
| + | |
| + | Le résultat est la disparition du gradient : les couches proches de la sortie apprennent, celles proches de l'entrée ne reçoivent presque rien et bougent à peine. Les réseaux sigmoïdes profonds stagnent, et le remède n'est pas un meilleur optimiseur, c'est une meilleure activation, la ReLU de la section suivante. |
| + | |
| + | <details class="proof"> |
| + | <summary>Preuve : le gradient rétrécit géométriquement avec la profondeur</summary> |
| + | |
| + | **Étape 1 : la pente de la sigmoïde ne dépasse jamais $1/4$.** Posons $s = \sigma(z) \in (0, 1)$ : le produit $s(1 - s)$ est une parabole tournée vers le bas, maximale en $s = \tfrac{1}{2}$ où elle vaut $\tfrac{1}{4}$. La borne tient donc, avec égalité seulement en $z = 0$, et la saturation aggrave tout : une unité située à $\sigma(2) \approx 0{,}88$ a déjà une pente de $0{,}88 \cdot 0{,}12 \approx 0{,}10$. |
| + | |
| + | **Étape 2 : la rétropropagation multiplie ces pentes.** Prenons le réseau profond le plus simple, une chaîne de $L$ couches à une unité chacune, où chaque quantité est un scalaire. Chaque couche traversée apporte le facteur $\partial z^{[l]} / \partial z^{[l-1]} = w^{[l]}\, \sigma'(z^{[l-1]})$ : |
| + | |
| + | $$\frac{\partial L}{\partial z^{[1]}} = \frac{\partial L}{\partial z^{[L]}} \prod_{l=2}^{L} w^{[l]}\, \sigma'(z^{[l-1]})$$ |
| + | |
| + | Avec des poids de taille typique $|w^{[l]}| \le 1$, chaque facteur vaut au plus $\tfrac{1}{4}$ en valeur absolue, donc |
| + | |
| + | $$\boxed{ \left|\frac{\partial L}{\partial z^{[1]}}\right| \le \left(\tfrac{1}{4}\right)^{L-1} \left|\frac{\partial L}{\partial z^{[L]}}\right| }$$ |
| + | |
| + | Dix couches rétrécissent déjà le gradient d'environ $10^{-6}$. $\blacksquare$ |
| + | |
| + | </details> |
| + | |
| + | Des poids bien plus grands que $1$ ne font qu'échanger le problème contre son image miroir, l'explosion du gradient. [Entraîner les réseaux profonds](/fr/Deep%20Learning/04%20Training%20deep%20networks) choisit l'échelle des poids pour que ce produit reste proche de $1$. |
| + | |
| + | ## 2.4 La famille ReLU |
| L'unité linéaire rectifiée conserve la partie positive de son entrée et annule le reste : | |
| @@ 69,7 89,7 @@ | |
| *Remarque :* leaky ReLU et PReLU ajoutent une petite pente $\alpha$ du côté négatif afin qu'une unité ne soit jamais complètement éteinte. GELU pondère l'entrée par la probabilité $\Phi(z)$ qu'une loi normale standard soit inférieure à $z$, ce qui donne une courbe lisse se comportant comme ReLU pour de grandes valeurs de $|z|$. C'est le choix standard à l'intérieur des Transformers. | |
| - | ## 3.4 Softmax pour les sorties multiclasses |
| + | ## 2.5 Softmax pour les sorties multiclasses |
| Pour une classification à $K$ classes, la couche finale produit un vecteur $z \in \mathbb{R}^K$ de scores, et la softmax le transforme en une distribution de probabilité sur les classes : | |
| @@ 77,13 97,13 @@ | |
| Chaque composante appartient à $(0, 1)$ et les composantes somment à $1$, si bien que $\text{softmax}(z)_k$ se lit comme la probabilité prédite de la classe $k$. Le plus grand score devient la classe la plus probable. | |
| - | *Remarque :* la softmax a sa place dans la couche de sortie, pas dans une couche cachée. Elle couple chaque unité par le dénominateur partagé (une normalisation sur tout le vecteur), ce qui est exactement ce dont une sortie probabiliste a besoin mais ne constitue pas une non-linéarité cachée utile par unité. Pour une sortie unique ($K = 1$ contre son complément), la softmax se réduit à la sigmoïde. L'association de la softmax avec sa fonction de perte fait l'objet de [Fonctions de perte et couches de sortie](/fr/Deep%20Learning/04%20Loss%20functions%20and%20output%20layers). |
| + | *Remarque :* la softmax a sa place dans la couche de sortie, pas dans une couche cachée. Elle couple chaque unité par le dénominateur partagé (une normalisation sur tout le vecteur), ce qui est exactement ce dont une sortie probabiliste a besoin mais ne constitue pas une non-linéarité cachée utile par unité. Pour une sortie unique ($K = 1$ contre son complément), la softmax se réduit à la sigmoïde. L'association de la softmax avec sa fonction de perte a été réglée dans la section 6.3 de [Réseaux de neurones multi-couches](/fr/Machine%20Learning/06%20Multilayer%20neural%20networks). |
| - | ## 3.5 Choisir une activation |
| + | ## 2.6 Choisir une activation |
| Un bon choix par défaut : utiliser ReLU ou GELU dans les couches cachées, et choisir l'activation de sortie selon la tâche. Le schéma et le tableau ci-dessous résument la décision. | |
| - |  |
| + |  |
| *Choisir une activation : ReLU ou GELU pour les couches cachées, et une activation de sortie adaptée à la tâche.* | |
| @@ 97,7 117,7 @@ | |
| *Remarque :* la sigmoïde et la tanh sont aujourd'hui rarement utilisées comme activations cachées dans les réseaux profonds à propagation avant, précisément à cause de la saturation vue à la Section 3.2.3. Elles subsistent en sortie (sigmoïde) et à l'intérieur des unités récurrentes à portes, où leur plage bornée est justement recherchée. | |
| - | *Une fois les non-linéarités par couche fixées, la prochaine leçon associe l'activation de sortie à une fonction de perte adaptée afin que le réseau ait quelque chose à minimiser.* |
| + | *Une fois les non-linéarités par couche fixées, la prochaine leçon transforme les gradients en mises à jour : les optimiseurs qui rendent l'entraînement profond praticable.* |
| --- | |
| - | Suivant : [Fonctions de perte et couches de sortie](/fr/Deep%20Learning/04%20Loss%20functions%20and%20output%20layers) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| + | Suivant : [Optimisation](/fr/Deep%20Learning/03%20Optimization) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| fr/Deep Learning/03 Activation functions/activation-choice.svg .. fr/Deep Learning/02 Activation functions/activation-choice.svg | |
| fr/Deep Learning/03 Activation functions/activation-derivatives.png .. fr/Deep Learning/02 Activation functions/activation-derivatives.png | |
| fr/Deep Learning/03 Activation functions/activation-functions.png .. fr/Deep Learning/02 Activation functions/activation-functions.png | |
| fr/Deep Learning/02 Multilayer perceptron.md .. /dev/null | |
| @@ 1,100 0,0 @@ | |
| - | # 2. Perceptron multicouche |
| - | |
| - | Un perceptron est une unité, $a = g(w^T x + b)$. Empilez plusieurs unités qui lisent la même entrée et vous obtenez une couche, empilez des couches et vous obtenez un perceptron multicouche (MLP). Ce module construit le MLP à partir d'unités, écrit la propagation avant pour un exemple et pour un mini-lot, suit les dimensions et le nombre de paramètres, et énonce le théorème d'approximation universelle. |
| - | |
| - | **Objectifs** |
| - | - Construire une couche comme un empilement d'unités de type perceptron lisant une entrée partagée. |
| - | - Écrire la propagation avant pour un exemple avec des poids et un biais explicites par couche. |
| - | - Vectoriser la passe avant sur un mini-lot avec un biais diffusé. |
| - | - Suivre la dimension de chaque $W^{[l]}$ et $b^{[l]}$ et compter les paramètres. |
| - | - Énoncer le théorème d'approximation universelle et opposer la largeur à la profondeur. |
| - | |
| - | ## 2.1 D'une unité à une couche |
| - | |
| - | ### 2.1.1 Une seule unité |
| - | |
| - | Une unité prend un vecteur d'entrée $x \in \mathbb{R}^{n_0}$, forme une somme pondérée avec un vecteur de poids $w$ et un biais scalaire $b$, puis applique une activation non linéaire $g$ : |
| - | |
| - | $$\boxed{ a = g\left(w^T x + b\right) }$$ |
| - | |
| - | C'est le perceptron du cours précédent, sauf que le seuil dur est maintenant une activation lisse comme la sigmoïde ou la ReLU. L'activation est nommée ici et définie complètement dans la [leçon suivante](/fr/Deep%20Learning/03%20Activation%20functions). |
| - | |
| - | ### 2.1.2 Une couche d'unités |
| - | |
| - | Placez maintenant $n_1$ unités côte à côte, lisant toutes la même entrée $x$. L'unité $i$ possède son propre vecteur de poids $w_i$ et son biais $b_i$, produisant $a_i = g(w_i^T x + b_i)$. Rassemblez les vecteurs de poids comme les lignes d'une matrice $W^{[1]}$ et les biais dans un vecteur $b^{[1]}$ : |
| - | |
| - | $$\boxed{ W^{[1]} = \begin{bmatrix} w_1^{T} \\ \vdots \\ w_{n_1}^{T} \end{bmatrix}, \quad b^{[1]} = \begin{bmatrix} b_1 \\ \vdots \\ b_{n_1} \end{bmatrix} }$$ |
| - | |
| - | La couche entière calcule alors un vecteur de pré-activation et un vecteur d'activation en une seule expression matricielle, $z^{[1]} = W^{[1]} x + b^{[1]}$ et $a^{[1]} = g^{[1]}(z^{[1]})$, où $g^{[1]}$ est appliquée élément par élément. |
| - | |
| - | *Remarque :* les lignes de $W^{[1]}$ sont exactement les vecteurs de poids des unités individuelles, donc une couche n'est que de nombreuses unités regroupées dans une seule matrice. Le biais reste explicite ici : contrairement au cours de Machine Learning, qui intégrait l'ordonnée à l'origine dans $w$ via l'entrée augmentée $x_0 = 1$, ce cours conserve $b^{[l]}$ comme son propre vecteur. |
| - | |
| - | ## 2.2 Propagation avant |
| - | |
| - | L'empilement de $L$ telles couches donne le MLP. La couche $l$ lit l'activation de la couche inférieure, $a^{[l-1]}$, et produit $a^{[l]}$. L'entrée est $a^{[0]} = x$ et la prédiction est la sortie de la dernière couche. |
| - | |
| - | ### 2.2.1 Un exemple |
| - | |
| - | Pour $l = 1, \dots, L$ : |
| - | |
| - | $$\boxed{ a^{[0]} = x, \quad z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}, \quad a^{[l]} = g^{[l]}\!\left(z^{[l]}\right), \quad \hat{y} = a^{[L]} }$$ |
| - | |
| - | Chaque couche peut utiliser sa propre activation $g^{[l]}$ : les couches cachées utilisent typiquement la ReLU, tandis que la couche de sortie utilise la sigmoïde ou le softmax pour la classification et l'identité pour la régression. |
| - | |
| - | *Remarque :* la composition $\hat{y} = g^{[L]}(W^{[L]} g^{[L-1]}(\cdots g^{[1]}(W^{[1]} x + b^{[1]}) \cdots) + b^{[L]})$ est ce qui rend le réseau expressif. Sans les $g^{[l]}$ non linéaires, l'empilement entier s'effondrerait en une seule application linéaire $W x + b$. |
| - | |
| - | ### 2.2.2 Vectorisation sur un mini-lot |
| - | |
| - | L'entraînement s'exécute sur des lots, pas sur des exemples isolés. Placez $m$ exemples comme les colonnes d'une matrice, de sorte que $A^{[0]} = X \in \mathbb{R}^{n_0 \times m}$, et la passe avant devient un produit matriciel avec le biais diffusé sur toutes les colonnes : |
| - | |
| - | $$\boxed{ Z^{[l]} = W^{[l]} A^{[l-1]} + b^{[l]}, \quad A^{[l]} = g^{[l]}\!\left(Z^{[l]}\right) }$$ |
| - | |
| - | Ici $Z^{[l]}$ et $A^{[l]}$ ont pour dimension $n_l \times m$, une colonne par exemple. Le biais $b^{[l]} \in \mathbb{R}^{n_l}$ est ajouté à chaque colonne, une opération connue sous le nom de diffusion (broadcasting). |
| - | |
| - | *Remarque :* le seul changement par rapport à la forme à un exemple est que le vecteur $a^{[l-1]}$ devient la matrice $A^{[l-1]}$. Traiter un lot en une seule multiplication matricielle est ce qui permet à un GPU d'exécuter la passe efficacement. |
| - | |
| - | ## 2.3 Dimensions et nombre de paramètres |
| - | |
| - | Les dimensions découlent d'une seule règle : pour calculer $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$, la matrice $W^{[l]}$ doit envoyer un vecteur de dimension $n_{l-1}$ vers un vecteur de dimension $n_l$. |
| - | |
| - | $$\boxed{ W^{[l]} \in \mathbb{R}^{n_l \times n_{l-1}}, \quad b^{[l]} \in \mathbb{R}^{n_l} }$$ |
| - | |
| - | La couche $l$ contient donc $n_l \, n_{l-1}$ poids plus $n_l$ biais. Considérons un petit réseau avec $n_0 = 4$ entrées, deux couches cachées de $5$ et $3$ unités, et une seule unité de sortie. |
| - | |
| - | | Couche $l$ | Dimension de $W^{[l]}$ | Dimension de $b^{[l]}$ | Paramètres | |
| - | | --- | --- | --- | --- | |
| - | | 1 | $5 \times 4$ | $5$ | $25$ | |
| - | | 2 | $3 \times 5$ | $3$ | $18$ | |
| - | | 3 | $1 \times 3$ | $1$ | $4$ | |
| - | | Total | | | $47$ | |
| - | |
| - | *Remarque :* la couche d'entrée ne contient aucun paramètre, elle n'est que les données $a^{[0]} = x$. Quand on compte les couches, on compte celles qui portent des poids, donc ce réseau a $L = 3$. |
| - | |
| - | ## 2.4 Un réseau multicouche |
| - | |
| - | Le diagramme ci-dessous montre le même réseau $4$-$5$-$3$-$1$ sous forme d'un flux d'activations. Chaque groupe de flèches est une matrice de poids complète, et chaque boîte applique son activation à la pré-activation. |
| - | |
| - |  |
| - | |
| - | *Un perceptron multicouche : chaque couche calcule z = W a + b puis a = g(z), composant l'entrée a0 en la prédiction aL.* |
| - | |
| - | L'information circule strictement de gauche à droite pendant la passe avant, c'est pourquoi il s'agit d'un réseau à propagation avant (feedforward). Rien ne boucle en arrière. La direction inverse, utilisée pour calculer les gradients, fait l'objet d'une leçon ultérieure. |
| - | |
| - | ## 2.5 Approximation universelle |
| - | |
| - | Quelle est l'expressivité d'un MLP ? Le théorème d'approximation universelle apporte une réponse forte. Soit $f$ une fonction continue quelconque sur un compact $K \subset \mathbb{R}^{n_0}$, et soit $\varepsilon > 0$. Alors il existe un réseau avec une seule couche cachée de largeur finie, utilisant une activation non linéaire appropriée, dont la sortie $F$ vérifie : |
| - | |
| - | $$\boxed{ \sup_{x \in K} \left| F(x) - f(x) \right| < \varepsilon }$$ |
| - | |
| - | Autrement dit, une seule couche cachée avec suffisamment d'unités peut approcher n'importe quelle fonction continue sur une région bornée avec la précision souhaitée $\varepsilon$. C'est un résultat d'existence, pas une recette : il garantit que de tels poids existent, mais ne dit rien sur le nombre d'unités nécessaires ni sur la manière de les trouver. |
| - | |
| - | *Remarque :* le piège est la largeur. Atteindre une cible avec une précision $\varepsilon$ avec une seule couche cachée peut exiger un nombre énorme d'unités, croissant rapidement à mesure que $\varepsilon$ diminue. La profondeur est généralement bien plus efficace en paramètres : empiler plusieurs couches étroites peut représenter des fonctions qu'une seule couche nécessiterait un nombre exponentiel d'unités pour égaler. Cette efficacité de la profondeur sur la largeur est la raison pratique pour laquelle le domaine s'appelle l'apprentissage profond (deep learning). |
| - | |
| - |  |
| - | |
| - | *Un réseau avec une seule couche cachée approche une fonction cible en sommant de nombreuses unités activées simples.* |
| - | |
| - | *Le réseau n'est défini qu'une fois les activations $g^{[l]}$ fixées. La leçon suivante les définit, sigmoïde, tanh, ReLU et ses variantes, et explique comment chacune façonne l'apprentissage.* |
| - | |
| - | --- |
| - | Suivant : [Fonctions d'activation](/fr/Deep%20Learning/03%20Activation%20functions) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| fr/Deep Learning/02 Multilayer perceptron/mlp-forward.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 760 392" width="760" height="392" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="760" height="392" fill="#ffffff"/><text x="380.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">Forward propagation through a 4-5-3-1 network</text><line x1="136.0" y1="125.0" x2="274.0" y2="100.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="125.0" x2="274.0" y2="150.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="125.0" x2="274.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="125.0" x2="274.0" y2="250.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="125.0" x2="274.0" y2="300.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="175.0" x2="274.0" y2="100.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="175.0" x2="274.0" y2="150.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="175.0" x2="274.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="175.0" x2="274.0" y2="250.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="175.0" x2="274.0" y2="300.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="225.0" x2="274.0" y2="100.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="225.0" x2="274.0" y2="150.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="225.0" x2="274.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="225.0" x2="274.0" y2="250.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="225.0" x2="274.0" y2="300.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="275.0" x2="274.0" y2="100.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="275.0" x2="274.0" y2="150.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="275.0" x2="274.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="275.0" x2="274.0" y2="250.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="136.0" y1="275.0" x2="274.0" y2="300.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="100.0" x2="444.0" y2="150.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="100.0" x2="444.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="100.0" x2="444.0" y2="250.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="150.0" x2="444.0" y2="150.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="150.0" x2="444.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="150.0" x2="444.0" y2="250.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="200.0" x2="444.0" y2="150.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="200.0" x2="444.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="200.0" x2="444.0" y2="250.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="250.0" x2="444.0" y2="150.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="250.0" x2="444.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="250.0" x2="444.0" y2="250.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="300.0" x2="444.0" y2="150.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="300.0" x2="444.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="306.0" y1="300.0" x2="444.0" y2="250.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="476.0" y1="150.0" x2="614.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="476.0" y1="200.0" x2="614.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><line x1="476.0" y1="250.0" x2="614.0" y2="200.0" stroke="#c7d0d9" stroke-width="0.8"/><circle cx="120.0" cy="125.0" r="16.0" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><circle cx="120.0" cy="175.0" r="16.0" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><circle cx="120.0" cy="225.0" r="16.0" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><circle cx="120.0" cy="275.0" r="16.0" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><circle cx="290.0" cy="100.0" r="16.0" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><circle cx="290.0" cy="150.0" r="16.0" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><circle cx="290.0" cy="200.0" r="16.0" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><circle cx="290.0" cy="250.0" r="16.0" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><circle cx="290.0" cy="300.0" r="16.0" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><circle cx="460.0" cy="150.0" r="16.0" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><circle cx="460.0" cy="200.0" r="16.0" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><circle cx="460.0" cy="250.0" r="16.0" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><circle cx="630.0" cy="200.0" r="16.0" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="120.0" y="355.0" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#5b6b7b" text-anchor="middle">a<tspan baseline-shift="sub" font-size="9px">0</tspan> (input)</text><text x="290.0" y="355.0" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#5b6b7b" text-anchor="middle">a<tspan baseline-shift="sub" font-size="9px">1</tspan></text><text x="460.0" y="355.0" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#5b6b7b" text-anchor="middle">a<tspan baseline-shift="sub" font-size="9px">2</tspan></text><text x="630.0" y="355.0" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#5b6b7b" text-anchor="middle">a<tspan baseline-shift="sub" font-size="9px">3</tspan> = ŷ</text><text x="205.0" y="90.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">W a + b</text><text x="380.0" y="372.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">each box computes z = W a + b then a = g(z), information flows left to right</text></svg> |
| \ | No newline at end of file |
| fr/Deep Learning/02 Multilayer perceptron/universal-approximation.png .. /dev/null | |
| fr/Deep Learning/06 Optimization.md .. fr/Deep Learning/03 Optimization.md | |
| @@ 1,16 1,8 @@ | |
| - | # 6. Optimisation |
| + | # 3. Optimisation |
| La rétropropagation renvoie le gradient du coût par rapport à chaque paramètre. Un optimiseur est la règle qui transforme ces gradients en mises à jour. Ce module couvre les variantes de la descente de gradient et les optimiseurs adaptatifs (momentum, RMSProp, Adam) qui rendent les réseaux profonds entraînables, ainsi que les plannings de taux d'apprentissage qui façonnent l'entraînement. | |
| - | **Objectifs** |
| - | - Réutiliser la mise à jour de la descente de gradient vue dans le cours de Machine Learning et nommer ses variantes batch, mini-batch et stochastique. |
| - | - Ajouter le momentum pour amortir les oscillations et accélérer le long des directions cohérentes. |
| - | - Redimensionner chaque coordonnée par la magnitude récente de son gradient avec RMSProp. |
| - | - Combiner les deux idées dans Adam et comprendre sa correction de biais. |
| - | - Choisir un planning de taux d'apprentissage : décroissance par paliers, cosinus ou warmup. |
| - | - Comparer les optimiseurs et savoir quand recourir à chacun. |
| - | |
| - | ## 6.1 Descente de gradient |
| + | ## 3.1 Descente de gradient |
| Soit $w$ l'ensemble de tous les paramètres (chaque $W^{[l]}$ et $b^{[l]}$) et soit $J(w)$ le coût, la moyenne de la perte par exemple $L$. Notons $g = \nabla_w J(w)$ le gradient du coût par rapport aux paramètres, tel que renvoyé par la rétropropagation. La mise à jour de base déplace $w$ dans le sens de la descente : | |
| @@ 20,7 12,7 @@ | |
| *Remarque :* le biais est explicite ici. Le gradient $g$ possède un bloc par $W^{[l]}$ et un par $b^{[l]}$, et la mise à jour s'applique à chaque bloc avec le même $\alpha$. | |
| - | ### 6.1.1 Batch, mini-batch, stochastique |
| + | ### 3.1.1 Batch, mini-batch, stochastique |
| Les variantes ne diffèrent que par le nombre d'exemples qui entrent dans le gradient $g$ à chaque étape. | |
| @@ 32,7 24,7 @@ | |
| *Remarque :* un passage complet sur l'ensemble du jeu de données est une époque. Le mini-batch est le choix standard : des batchs de $32$ à $512$ tiennent dans l'accélérateur, exploitent les produits matriciels vectorisés, et le bruit résiduel dans $g$ aide à échapper aux minima locaux peu profonds. En apprentissage profond, « SGD » est employé de façon souple pour désigner la descente de gradient par mini-batch. | |
| - | ## 6.2 Momentum |
| + | ## 3.2 Momentum |
| La SGD simple zigzague à travers les vallées étroites parce que le gradient pointe davantage en travers de la vallée que le long de celle-ci. Le momentum accumule une moyenne pondérée exponentiellement des gradients passés dans un vecteur de vitesse $v$, puis avance dans cette direction moyennée : | |
| @@ 40,7 32,11 @@ | |
| avec un coefficient de momentum $\beta \in [0, 1)$, typiquement $\beta = 0.9$. Les composantes de $g$ qui gardent le même signe se renforcent mutuellement, si bien que $v$ croît et que le pas accélère le long des directions cohérentes. Les composantes qui changent de signe s'annulent dans la moyenne, si bien que les oscillations en travers de la vallée sont amorties. | |
| - | ### 6.2.1 Momentum de Nesterov |
| + |  |
| + | |
| + | *Les courbes sont les lignes de niveau de la perte, le point est le minimum. Le gradient est perpendiculaire à la ligne de niveau où il se trouve, dans un ravin il pointe donc surtout en travers de la vallée, et la descente de gradient brute rebondit. Le momentum garde une mémoire des pas précédents, les rebonds s'annulent et la direction de la vallée s'accumule.* |
| + | |
| + | ### 3.2.1 Momentum de Nesterov |
| Le gradient accéléré de Nesterov évalue le gradient en un point d'anticipation, après que le pas de momentum a été appliqué à titre provisoire, plutôt qu'au $w$ courant. Cette correction anticipatrice réagit plus tôt lorsque la pente change : | |
| @@ 48,7 44,7 @@ | |
| *Remarque :* voir $\beta \approx 0.9$ comme une moyenne sur environ les $\tfrac{1}{1 - \beta} = 10$ derniers gradients. Nesterov converge en général un peu plus vite que le momentum simple pour les mêmes $\alpha$ et $\beta$. | |
| - | ## 6.3 RMSProp |
| + | ## 3.3 RMSProp |
| Différents paramètres peuvent nécessiter des tailles de pas très différentes, et un unique $\alpha$ global ne peut pas tous les servir. RMSProp maintient une moyenne glissante par coordonnée $s$ des gradients au carré, puis divise le pas par $\sqrt{s}$, de sorte que les coordonnées aux gradients récents importants prennent des pas plus petits et que les coordonnées calmes prennent des pas plus grands : | |
| @@ 58,7 54,7 @@ | |
| *Remarque :* $s$ estime le moment d'ordre deux non centré de chaque coordonnée de $g$, si bien que $\sqrt{s}$ correspond à peu près à sa magnitude quadratique moyenne récente. RMSProp convient aux objectifs non stationnaires, ce qui est exactement le cas d'un gradient de mini-batch mobile. | |
| - | ## 6.4 Adam |
| + | ## 3.4 Adam |
| Adam (adaptive moment estimation) combine le momentum et RMSProp : il maintient une estimation du moment d'ordre un $m$ (la moyenne du gradient) et une estimation du moment d'ordre deux $v$ (la moyenne du gradient au carré). | |
| @@ 76,27 72,27 @@ | |
| *Remarque :* la correction de biais compte surtout dans les premières dizaines d'étapes, quand $t$ est petit et que $\beta_2^t$ est encore proche de $1$. Sans elle, $\hat v$ serait bien trop petit et les premiers pas bien trop grands. AdamW, une variante courante, découple la décroissance des poids (weight decay) de cette mise à jour. | |
| - |  |
| + |  |
| *Adam combine le momentum des gradients moyennés avec la mise à l'échelle par paramètre de RMSProp.* | |
| - | ## 6.5 Plannings de taux d'apprentissage |
| + | ## 3.5 Plannings de taux d'apprentissage |
| Le taux d'apprentissage $\alpha$ est l'hyperparamètre le plus important à lui seul, et le maintenir fixe est rarement optimal. Un grand $\alpha$ accélère les progrès initiaux mais empêche de se stabiliser dans un minimum, si bien que les plannings diminuent généralement $\alpha$ au fil de l'entraînement. Ici $\alpha_0$ est le taux initial et $t$ indexe l'étape ou l'époque. | |
| - | ### 6.5.1 Décroissance par paliers |
| + | ### 3.5.1 Décroissance par paliers |
| Multiplier $\alpha$ par un facteur $\gamma \in (0, 1)$ toutes les $s$ époques, de sorte qu'il chute par étapes discrètes : | |
| $$\boxed{ \alpha_t = \alpha_0\, \gamma^{\lfloor t / s \rfloor} }$$ | |
| - | ### 6.5.2 Décroissance en cosinus |
| + | ### 3.5.2 Décroissance en cosinus |
| Recuire $\alpha$ en douceur depuis $\alpha_0$ vers un plancher nul le long d'un demi-cosinus sur un total de $T$ étapes : | |
| $$\boxed{ \alpha_t = \tfrac{1}{2}\,\alpha_0\left(1 + \cos\frac{\pi t}{T}\right) }$$ | |
| - | ### 6.5.3 Warmup |
| + | ### 3.5.3 Warmup |
| Le warmup fait monter $\alpha$ linéairement depuis une petite valeur au cours des premières centaines à quelques milliers d'étapes, puis passe la main à un planning de décroissance. Il évite les mises à jour importantes et mal conditionnées qu'un démarrage à froid avec un grand $\alpha$ produirait, et il est standard pour les réseaux profonds tels que les transformeurs. | |
| @@ 106,13 102,13 @@ | |
| | Cosinus | recuit en douceur vers zéro | choix par défaut moderne, souvent avec warmup | | |
| | Warmup | montée linéaire, puis décroissance | stabiliser les premières étapes, grands modèles | | |
| - |  |
| + |  |
| *Plannings courants de taux d'apprentissage : décroissance par paliers, décroissance en cosinus, et un warmup suivi de décroissance.* | |
| *Remarque :* le warmup et une décroissance sont généralement combinés, le warmup pour la première phase et le cosinus ou la décroissance par paliers ensuite. | |
| - | ## 6.6 Choisir un optimiseur |
| + | ## 3.6 Choisir un optimiseur |
| | optimiseur | ce qu'il ajoute | suit | usage typique | | |
| | --- | --- | --- | --- | | |
| @@ 123,11 119,21 @@ | |
| *Remarque :* Adam est le choix par défaut sûr et converge vite avec peu de réglage. Une SGD bien réglée avec momentum et un planning atteint souvent une précision de test finale légèrement meilleure sur les grands modèles de vision, raison pour laquelle les deux restent largement utilisés. | |
| - |  |
| + |  |
| *Sur une surface de perte allongée, le momentum et Adam atteignent le minimum bien plus vite que la descente de gradient simple.* | |
| + | ## 3.7 Les bonnes pratiques |
| + | |
| + | Deux des habitudes qui font qu'un entraînement se passe bien, l'initialisation soignée et le dropout, vivent dans la prochaine leçon ([Entraîner les réseaux profonds](/fr/Deep%20Learning/04%20Training%20deep%20networks)). Deux autres ont leur place ici. |
| + | |
| + | **Centrer et normaliser les entrées.** Standardiser chaque caractéristique (soustraire sa moyenne, diviser par son écart-type), pour qu'aucune ne domine les premiers produits scalaires et que les gradients de la première couche démarrent bien mis à l'échelle. |
| + | |
| + | **Vérifier avant d'entraîner longtemps.** Un classifieur à $K$ classes fraîchement initialisé doit démarrer près de la perte $\ln K$ (environ $2{,}3$ pour $K = 10$). Un minuscule jeu d'entraînement doit être facile à surapprendre : si le réseau n'y arrive pas, le code est cassé. Surveiller les courbes d'entraînement et de validation. Et comme la rétropropagation est source d'erreurs, comparer son gradient analytique à une estimation numérique par différences finies : |
| + | |
| + | $$\boxed{ \frac{\partial L}{\partial w} \approx \frac{L(w + \varepsilon) - L(w - \varepsilon)}{2\varepsilon} }$$ |
| + | |
| *Chaque optimiseur présenté ici met à l'échelle le gradient brut, si bien que son comportement dépend de la taille de ces gradients au départ. La partie suivante étudie comment les poids initiaux et la profondeur du réseau fixent cette échelle, et comment de mauvais choix font disparaître ou exploser les gradients.* | |
| --- | |
| - | Suivant : [Initialisation et disparition du gradient](/fr/Deep%20Learning/07%20Initialization%20and%20vanishing%20gradients) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| + | Suivant : [Entraîner les réseaux profonds](/fr/Deep%20Learning/04%20Training%20deep%20networks) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| fr/Deep Learning/06 Optimization/lr-schedules.png .. fr/Deep Learning/03 Optimization/lr-schedules.png | |
| fr/Deep Learning/06 Optimization/optimizer-family.svg .. fr/Deep Learning/03 Optimization/optimizer-family.svg | |
| fr/Deep Learning/06 Optimization/optimizer-paths.png .. fr/Deep Learning/03 Optimization/optimizer-paths.png | |
| /dev/null .. fr/Deep Learning/03 Optimization/ravine-momentum.svg | |
| @@ 0,0 1,51 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 370" width="900" height="370" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs> |
| + | <marker id="arrowred" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#d1495b"/></marker> |
| + | <marker id="arrowblue" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#3b6fb6"/></marker> |
| + | </defs> |
| + | <rect width="900" height="370" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">A ravine in the loss: steep across, shallow along</text> |
| + | |
| + | <!-- Panel 1: plain gradient descent --> |
| + | <text x="225" y="56" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">plain gradient descent</text> |
| + | <ellipse cx="225" cy="195" rx="170" ry="70" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <ellipse cx="225" cy="195" rx="135" ry="55" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <ellipse cx="225" cy="195" rx="100" ry="40" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <ellipse cx="225" cy="195" rx="65" ry="26" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <ellipse cx="225" cy="195" rx="30" ry="12" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <circle cx="225" cy="195" r="3.5" fill="#1f2933"/> |
| + | <polyline points="70,168 86,236 101,163 116,231 130,167 143,226 156,171 168,221 179,175 190,217 200,179 209,212 217,183 224,208 230,187 235,204 239,192" fill="none" stroke="#d1495b" stroke-width="1.6" marker-end="url(#arrowred)"/> |
| + | <circle cx="70" cy="168" r="2.5" fill="#d1495b"/> |
| + | <circle cx="86" cy="236" r="1.8" fill="#d1495b"/> |
| + | <circle cx="101" cy="163" r="1.8" fill="#d1495b"/> |
| + | <circle cx="116" cy="231" r="1.8" fill="#d1495b"/> |
| + | <circle cx="130" cy="167" r="1.8" fill="#d1495b"/> |
| + | <circle cx="143" cy="226" r="1.8" fill="#d1495b"/> |
| + | <circle cx="156" cy="171" r="1.8" fill="#d1495b"/> |
| + | <circle cx="168" cy="221" r="1.8" fill="#d1495b"/> |
| + | <circle cx="179" cy="175" r="1.8" fill="#d1495b"/> |
| + | <circle cx="190" cy="217" r="1.8" fill="#d1495b"/> |
| + | <circle cx="200" cy="179" r="1.8" fill="#d1495b"/> |
| + | <circle cx="209" cy="212" r="1.8" fill="#d1495b"/> |
| + | <circle cx="217" cy="183" r="1.8" fill="#d1495b"/> |
| + | <circle cx="224" cy="208" r="1.8" fill="#d1495b"/> |
| + | <circle cx="230" cy="187" r="1.8" fill="#d1495b"/> |
| + | <circle cx="235" cy="204" r="1.8" fill="#d1495b"/> |
| + | <text x="62" y="156" font-size="10" fill="#5b6b7b" text-anchor="middle">start</text> |
| + | <text x="225" y="328" font-size="11" fill="#5b6b7b" text-anchor="middle">the gradient points mostly across the valley:</text> |
| + | <text x="225" y="344" font-size="11" fill="#5b6b7b" text-anchor="middle">the step bounces between the walls, crawls toward the minimum (dot)</text> |
| + | |
| + | <!-- Panel 2: momentum --> |
| + | <text x="675" y="56" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">with momentum</text> |
| + | <ellipse cx="675" cy="195" rx="170" ry="70" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <ellipse cx="675" cy="195" rx="135" ry="55" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <ellipse cx="675" cy="195" rx="100" ry="40" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <ellipse cx="675" cy="195" rx="65" ry="26" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <ellipse cx="675" cy="195" rx="30" ry="12" fill="none" stroke="#c7d0d9" stroke-width="1.1"/> |
| + | <circle cx="675" cy="195" r="3.5" fill="#1f2933"/> |
| + | <path d="M520,168 C545,210 566,229 592,227 C620,225 646,211 668,203 C688,196 702,193 709,194 C707,198 698,198 690,197" fill="none" stroke="#3b6fb6" stroke-width="1.8" marker-end="url(#arrowblue)"/> |
| + | <circle cx="520" cy="168" r="2.5" fill="#3b6fb6"/> |
| + | <text x="512" y="156" font-size="10" fill="#5b6b7b" text-anchor="middle">start</text> |
| + | <text x="675" y="328" font-size="11" fill="#5b6b7b" text-anchor="middle">the crossings cancel while the along-the-valley component builds speed:</text> |
| + | <text x="675" y="344" font-size="11" fill="#5b6b7b" text-anchor="middle">the path glides to the minimum, with one small overshoot</text> |
| + | </svg> |
| fr/Deep Learning/04 Loss functions and output layers.md .. /dev/null | |
| @@ 1,92 0,0 @@ | |
| - | # 4. Fonctions de perte et couches de sortie |
| - | |
| - | Avant qu'un réseau puisse apprendre, il lui faut une cible vers laquelle descendre. La couche de sortie transforme la dernière activation $a^{[L]}$ en une prédiction, et la perte mesure l'écart entre cette prédiction et le vrai label. Ce module fixe ces deux choix par tâche, car la rétropropagation du prochain module dérive une perte concrète. L'activation de sortie et la perte ne se choisissent pas indépendamment : les accorder à la forme de la tâche est ce qui rend le signal d'entraînement propre. |
| - | |
| - | **Objectifs** |
| - | - Passer d'une perte par exemple $L$ au coût $J$ moyenné sur le lot. |
| - | - Choisir une sortie linéaire avec l'erreur quadratique moyenne pour la régression. |
| - | - Choisir une sortie sigmoïde avec l'entropie croisée binaire pour les problèmes à deux classes. |
| - | - Choisir une sortie softmax avec l'entropie croisée catégorielle pour les problèmes multiclasses. |
| - | - Dériver le gradient propre au niveau des logits de la paire softmax et entropie croisée. |
| - | - Associer toute tâche à son activation de sortie et à sa perte à l'aide d'une simple table de correspondance. |
| - | |
| - | ## 4.1 De la perte par exemple au coût |
| - | |
| - | Le réseau prédit $\hat{y} = a^{[L]}$ à partir de l'entrée $a^{[0]} = x$. Pour un seul exemple, la perte $L(\hat{y}, y)$ évalue cette prédiction par rapport à la cible $y$. L'entraînement minimise le coût $J$, défini comme la moyenne de $L$ sur les $m$ exemples du lot ou du jeu de données : |
| - | |
| - | $$\boxed{ J = \frac{1}{m}\sum_{i=1}^{m} L\!\left(\hat{y}^{(i)}, y^{(i)}\right) }$$ |
| - | |
| - | *Remarque :* la perte $L$ évalue une prédiction, le coût $J$ est ce que l'optimiseur réduit réellement. Le fait de moyenner (plutôt que de sommer) maintient l'échelle du gradient indépendante de la taille du lot, si bien que le taux d'apprentissage n'a pas à être réajusté quand $m$ change. |
| - | |
| - | Les trois tâches ci-dessous réutilisent les pertes présentées dans le cours de Machine Learning. La ligne de l'entropie croisée de la table des pertes des [Concepts généraux](/fr/Machine%20Learning/02%20General%20concepts), intitulée « Réseaux de neurones », est exactement l'objectif que minimise un réseau de classification. La nouveauté ici est d'apparier chaque perte avec l'activation de sortie $g^{[L]}$ qui produit $\hat{y}$. |
| - | |
| - | ## 4.2 Régression : sortie linéaire et erreur quadratique moyenne |
| - | |
| - | Pour une cible continue $y \in \mathbb{R}^{n_L}$, la couche de sortie n'utilise aucune activation, elle est donc linéaire (l'identité) et la prédiction peut prendre n'importe quelle valeur réelle : |
| - | |
| - | $$\boxed{ \hat{y} = a^{[L]} = z^{[L]} = W^{[L]} a^{[L-1]} + b^{[L]} }$$ |
| - | |
| - | La perte par exemple est la distance euclidienne au carré entre la prédiction et la cible, mise à l'échelle par un demi : |
| - | |
| - | $$\boxed{ L(\hat{y}, y) = \tfrac{1}{2}\,\lVert \hat{y} - y \rVert^2 }$$ |
| - | |
| - | *Remarque :* le facteur $\tfrac{1}{2}$ annule le $2$ qui apparaît en dérivant le carré, laissant le gradient résiduel épuré $\partial L / \partial \hat{y} = \hat{y} - y$. C'est le même objectif d'erreur quadratique moyenne utilisé pour la régression linéaire, posé désormais au sommet d'un réseau profond au lieu d'un unique score linéaire. |
| - | |
| - | ## 4.3 Classification binaire : sortie sigmoïde et entropie croisée binaire |
| - | |
| - | Pour un label à deux classes $y \in \{0, 1\}$, la couche de sortie possède une seule unité dont l'activation est la sigmoïde, qui écrase le logit $z^{[L]}$ en une probabilité : |
| - | |
| - | $$\hat{y} = a^{[L]} = \sigma\!\left(z^{[L]}\right) = \frac{1}{1 + e^{-z^{[L]}}} \in (0, 1)$$ |
| - | |
| - | Ici $\hat{y}$ se lit comme $p(y = 1 \mid x)$. La perte associée est l'entropie croisée binaire, la log-vraisemblance négative du label de Bernoulli : |
| - | |
| - | $$\boxed{ L(\hat{y}, y) = -\Big[\, y \log \hat{y} + (1 - y)\log(1 - \hat{y}) \,\Big] }$$ |
| - | |
| - | *Remarque :* un seul des deux termes est actif pour un label donné. Quand $y = 1$, la perte vaut $-\log \hat{y}$, pénalisant une petite probabilité prédite, et quand $y = 0$ elle vaut $-\log(1 - \hat{y})$. L'entropie croisée est préférée ici à l'erreur quadratique parce qu'elle maintient le gradient élevé lorsque la prédiction est confiante et fausse, de sorte que l'apprentissage ne stagne pas. |
| - | |
| - |  |
| - | |
| - | *La perte d'entropie croisée croît sans borne à mesure que la probabilité prédite s'éloigne du vrai label.* |
| - | |
| - | ## 4.4 Classification multiclasse : sortie softmax et entropie croisée catégorielle |
| - | |
| - | Pour un label à $K$ classes, la couche de sortie possède $K$ unités et l'activation softmax transforme le vecteur de logits $z^{[L]} \in \mathbb{R}^{K}$ en une distribution de probabilité sur les classes : |
| - | |
| - | $$\boxed{ \hat{y}_k = \frac{e^{z^{[L]}_k}}{\sum_{j=1}^{K} e^{z^{[L]}_j}} }$$ |
| - | |
| - | Les sorties sont positives et somment à un, donc $\hat{y}$ est une distribution valide et $\hat{y}_k = p(y = k \mid x)$. La cible $y$ est en encodage one-hot : $y_k = 1$ pour la vraie classe et $0$ sinon. La perte associée est l'entropie croisée catégorielle : |
| - | |
| - | $$\boxed{ L(\hat{y}, y) = -\sum_{k=1}^{K} y_k \log \hat{y}_k }$$ |
| - | |
| - | *Remarque :* comme $y$ est en encodage one-hot, la somme se réduit à un seul terme, $-\log \hat{y}_{k^\star}$, où $k^\star$ est la vraie classe. La perte récompense donc le fait de placer la masse de probabilité sur la bonne classe et ignore la façon dont la masse restante est répartie. L'entropie croisée binaire est le cas particulier $K = 2$. |
| - | |
| - | ## 4.5 Le gradient de la softmax et de l'entropie croisée |
| - | |
| - | La sortie softmax et la perte d'entropie croisée catégorielle sont utilisées ensemble parce que leur composition a une dérivée remarquablement propre au niveau des logits $z^{[L]}$. En dérivant $L$ par rapport à un seul logit $z^{[L]}_k$, on obtient : |
| - | |
| - | $$\boxed{ \frac{\partial L}{\partial z^{[L]}_k} = \hat{y}_k - y_k }$$ |
| - | |
| - | Le gradient à la couche de sortie n'est autre que la prédiction moins la cible, un simple résidu sans facteur sigmoïde ou softmax gênant qui subsisterait. La même identité vaut pour la paire sigmoïde et entropie croisée binaire, qui en est le cas $K = 2$. C'est précisément pour cela que chaque activation est couplée à sa perte associée plutôt que mélangée avec, par exemple, l'erreur quadratique. |
| - | |
| - | *Remarque :* la forme élément par élément $\partial L / \partial z^{[L]} = \hat{y} - y$ est ce qui amorce la rétropropagation. Le prochain module démarre la passe arrière depuis ce vecteur, puis applique de façon répétée la règle de dérivation en chaîne et le produit de Hadamard $\odot$ pour le repousser à travers les couches cachées. |
| - | |
| - | ## 4.6 De la tâche à la sortie à la perte |
| - | |
| - | Les trois cas se résument en une seule correspondance. Fixez la tâche, et l'activation de sortie et la perte en découlent. |
| - | |
| - | | Tâche | Activation de sortie $g^{[L]}$ | Perte par exemple $L$ | Gradient au niveau des logits $\partial L / \partial z^{[L]}$ | |
| - | | --- | --- | --- | --- | |
| - | | Régression | linéaire (identité) | erreur quadratique moyenne | $\hat{y} - y$ | |
| - | | Classification binaire | sigmoïde | entropie croisée binaire | $\hat{y} - y$ | |
| - | | Classification multiclasse | softmax | entropie croisée catégorielle | $\hat{y} - y$ | |
| - | |
| - | *Remarque :* la dernière colonne est identique pour les trois lignes. Accorder l'activation de sortie à sa perte naturelle fait que le réseau démarre sa passe arrière depuis le même résidu simple quelle que soit la tâche. |
| - | |
| - |  |
| - | |
| - | *L'activation de sortie et la perte sont choisies ensemble par tâche, et les paires accordées partagent le gradient propre au niveau des logits yhat moins y.* |
| - | |
| - | *Une fois une perte concrète choisie et son gradient à la couche de sortie en main, le prochain module fait remonter la règle de dérivation en chaîne à travers chaque couche : la rétropropagation.* |
| - | |
| - | --- |
| - | Suivant : [Rétropropagation](/fr/Deep%20Learning/05%20Backpropagation) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| fr/Deep Learning/04 Loss functions and output layers/loss-curves.png .. /dev/null | |
| fr/Deep Learning/04 Loss functions and output layers/output-loss-map.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 780 386" width="780" height="386" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="780" height="386" fill="#ffffff"/><text x="390.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">Task, output activation, and loss are matched per task</text><text x="135.0" y="58.0" font-family="Helvetica, Arial, sans-serif" font-size="12" font-weight="600" fill="#5b6b7b" text-anchor="middle">task</text><text x="325.0" y="58.0" font-family="Helvetica, Arial, sans-serif" font-size="12" font-weight="600" fill="#5b6b7b" text-anchor="middle">output activation</text><text x="545.0" y="58.0" font-family="Helvetica, Arial, sans-serif" font-size="12" font-weight="600" fill="#5b6b7b" text-anchor="middle">loss</text><rect x="640.0" y="156.0" width="118.0" height="68.0" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="699.0" y="179.1" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">gradient at</text><text x="699.0" y="194.4" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">logits = ŷ</text><text x="699.0" y="209.8" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">- y</text><rect x="60.0" y="72.0" width="150.0" height="46.0" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="135.0" y="99.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">regression</text><rect x="250.0" y="72.0" width="150.0" height="46.0" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="325.0" y="99.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">linear output</text><rect x="470.0" y="72.0" width="150.0" height="46.0" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="545.0" y="99.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">mean squared error</text><line x1="210.0" y1="95.0" x2="250.0" y2="95.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="400.0" y1="95.0" x2="470.0" y2="95.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="620.0" y1="95.0" x2="640.0" y2="190.0" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><rect x="60.0" y="167.0" width="150.0" height="46.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="135.0" y="194.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">binary</text><rect x="250.0" y="167.0" width="150.0" height="46.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="325.0" y="194.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">sigmoid output</text><rect x="470.0" y="167.0" width="150.0" height="46.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="545.0" y="186.8" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">binary</text><text x="545.0" y="202.1" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">cross-entropy</text><line x1="210.0" y1="190.0" x2="250.0" y2="190.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="400.0" y1="190.0" x2="470.0" y2="190.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="620.0" y1="190.0" x2="640.0" y2="190.0" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><rect x="60.0" y="262.0" width="150.0" height="46.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="135.0" y="289.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">multiclass</text><rect x="250.0" y="262.0" width="150.0" height="46.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="325.0" y="289.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">softmax output</text><rect x="470.0" y="262.0" width="150.0" height="46.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="545.0" y="281.8" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">categorical</text><text x="545.0" y="297.1" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">cross-entropy</text><line x1="210.0" y1="285.0" x2="250.0" y2="285.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="400.0" y1="285.0" x2="470.0" y2="285.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="620.0" y1="285.0" x2="640.0" y2="190.0" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><text x="390.0" y="366.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">the matched pairs all seed the backward pass from the same residual</text></svg> |
| \ | No newline at end of file |
| /dev/null .. fr/Deep Learning/04 Training deep networks.md | |
| @@ 0,0 1,72 @@ | |
| + | # 4. Entraîner les réseaux profonds |
| + | |
| + | La leçon 3 a choisi l'optimiseur. Cette leçon rassemble les trois outils qui font que cette optimisation se comporte bien sur une pile profonde : démarrer les poids à la bonne échelle (l'initialisation), garder les activations bien mises à l'échelle pendant qu'elles bougent (la normalisation), et empêcher le réseau de mémoriser le jeu d'entraînement (la régularisation). Chacun mériterait son propre chapitre. Ici, nous gardons ce qui compte le plus en pratique. |
| + | |
| + | ## 4.1 L'initialisation |
| + | |
| + | **Briser la symétrie.** Poser $W^{[l]} = 0$, ou toute valeur qui rend identiques toutes les unités d'une couche, casse l'apprentissage : deux unités avec les mêmes poids et la même entrée calculent la même activation, reçoivent le même gradient et restent identiques pour toujours. La couche se comporte comme une seule unité, quelle que soit sa largeur. Les poids démarrent donc aléatoires. Les biais peuvent démarrer à zéro : les poids aléatoires diffèrent déjà, et un biais nul garde chaque unité dans la région réactive de son activation. |
| + | |
| + | **Choisir la bonne échelle.** Pour une unité $z = \sum_{j=1}^{n_{\text{in}}} W_j a_j$ à poids et entrées indépendants et centrés, la variance est une somme de $n_{\text{in}}$ termes : |
| + | |
| + | $$\boxed{ \operatorname{Var}(z) = n_{\text{in}} \cdot \operatorname{Var}(W) \cdot \operatorname{Var}(a) }$$ |
| + | |
| + | Si $n_{\text{in}} \operatorname{Var}(W)$ glisse sous $1$, le signal rétrécit couche après couche, et la leçon 2 a prouvé où cela finit : un gradient disparu. Au-dessus de $1$, il explose. Le remède est de tenir $n_{\text{in}} \operatorname{Var}(W) \approx 1$, ce qui fixe la variance des poids à la taille de la couche : |
| + | |
| + | $$\boxed{ \text{Xavier : } \operatorname{Var}(W^{[l]}) = \frac{2}{n_{\text{in}} + n_{\text{out}}}, \qquad \text{He : } \operatorname{Var}(W^{[l]}) = \frac{2}{n_{\text{in}}} }$$ |
| + | |
| + | Xavier (qui équilibre les passes avant et arrière) convient à $\tanh$ et à la sigmoïde. He double la variance parce que ReLU met à zéro la moitié de ses entrées en moyenne, c'est donc la bonne cible pour la famille ReLU. |
| + | |
| + |  |
| + | |
| + | *Amplitude du gradient selon la profondeur : des poids mal mis à l'échelle le font disparaître ou exploser, tandis qu'une initialisation qui préserve la variance le garde proche de un.* |
| + | |
| + | **Écrêter ce qui explose encore.** L'initialisation règle l'échelle une seule fois, au pas zéro. Si les gradients explosent encore pendant l'entraînement (fréquent dans les réseaux récurrents, leçon 7), on remet le gradient à l'échelle pour que sa norme ne dépasse jamais un seuil $\tau$, en gardant sa direction : |
| + | |
| + | $$\boxed{ g \leftarrow g \cdot \min\!\left(1, \frac{\tau}{\lVert g \rVert}\right) }$$ |
| + | |
| + | ## 4.2 La normalisation |
| + | |
| + | L'initialisation ne positionne le réseau qu'au pas zéro : à mesure que l'entraînement déplace les poids, la distribution de l'entrée de chaque couche dérive, et les couches suivantes poursuivent une cible mouvante. Une couche de normalisation fixe les statistiques à la volée : standardiser, puis laisser le réseau réapprendre son échelle. Pour une caractéristique $x$ sur un mini-lot de taille $m$, la normalisation par lot calcule |
| + | |
| + | $$\boxed{ \mu_\mathcal{B} = \frac{1}{m}\sum_{i=1}^{m} x^{(i)}, \qquad \sigma_\mathcal{B}^2 = \frac{1}{m}\sum_{i=1}^{m}\left(x^{(i)} - \mu_\mathcal{B}\right)^2 }$$ |
| + | |
| + | $$\boxed{ \hat{x}^{(i)} = \frac{x^{(i)} - \mu_\mathcal{B}}{\sqrt{\sigma_\mathcal{B}^2 + \epsilon}}, \qquad y^{(i)} = \gamma\, \hat{x}^{(i)} + \beta }$$ |
| + | |
| + | L'échelle $\gamma$ et le décalage $\beta$ s'apprennent comme n'importe quel poids, la normalisation ne retire donc aucune capacité : le réseau peut même apprendre à l'annuler. Le gain : une surface de perte plus lisse, des taux d'apprentissage utilisables plus élevés, moins de sensibilité à l'initialisation. Deux conséquences pratiques : à l'inférence, où il n'y a pas de lot, la BatchNorm bascule sur des moyennes glissantes de $\mu$ et $\sigma^2$ accumulées pendant l'entraînement (oublier cette bascule est le bug BatchNorm classique), et le décalage $\beta$ rend le biais de couche $b^{[l]}$ redondant. |
| + | |
| + | La normalisation par couche garde la même recette standardiser, mettre à l'échelle, décaler, mais moyenne sur les caractéristiques d'un seul exemple au lieu du lot. Ses statistiques ne dépendent plus du lot, elle se comporte donc à l'identique à l'entraînement et à l'inférence et gère les séquences de longueur variable, ce qui en fait le choix des réseaux récurrents et des Transformers (leçon 10). |
| + | |
| + | | Aspect | Normalisation par lot | Normalisation par couche | |
| + | | --- | --- | --- | |
| + | | Axe de normalisation | à travers le lot, par caractéristique | à travers les caractéristiques, par exemple | |
| + | | Entraînement vs inférence | statistiques de lot vs statistiques glissantes | identiques dans les deux | |
| + | | Usage typique | CNN et modèles de vision | RNN et Transformers | |
| + | |
| + |  |
| + | |
| + | *La normalisation par lot calcule ses statistiques le long d'une colonne de caractéristique à travers le lot, la normalisation par couche à travers les caractéristiques d'un seul exemple.* |
| + | |
| + | ## 4.3 La régularisation et le dropout |
| + | |
| + | Un réseau profond a généralement plus de paramètres que d'exemples d'entraînement, il peut donc mémoriser le jeu d'entraînement, bruit compris. La régularisation échange un peu de précision d'entraînement contre de la généralisation, l'histoire biais-variance de [Concepts généraux](/fr/Machine%20Learning/02%20General%20concepts). |
| + | |
| + | **La décroissance des poids ($L_2$).** Ajouter au coût une pénalité sur les poids au carré, de force $\lambda$ : |
| + | |
| + | $$\boxed{ J_{\text{reg}} = J + \frac{\lambda}{2}\sum_{l=1}^{L}\lVert W^{[l]} \rVert_F^2 \quad\Longrightarrow\quad W^{[l]} \leftarrow (1 - \alpha\lambda)\, W^{[l]} - \alpha\,\frac{\partial J}{\partial W^{[l]}} }$$ |
| + | |
| + | Le facteur $(1 - \alpha\lambda)$ rétrécit chaque poids à chaque pas, d'où le nom. Sa cousine $L_1$ pénalise les valeurs absolues et pousse beaucoup de poids exactement à zéro. Les biais restent hors de la pénalité. |
| + | |
| + | **Le dropout.** À chaque passe d'entraînement, garder chaque unité avec probabilité $p$ et la mettre à zéro sinon, puis diviser par $p$ pour que le signal attendu reste inchangé : |
| + | |
| + | $$\boxed{ \tilde{a}^{[l]} = \frac{m \odot a^{[l]}}{p}, \qquad m_i \sim \text{Bernoulli}(p) }$$ |
| + | |
| + | Aucune unité ne peut compter sur ses voisines, la représentation s'étale donc. Chaque pas entraîne l'un des $2^k$ sous-réseaux amincis qui partagent leurs poids, et l'inférence, dropout éteint, approche gratuitement leur prédiction moyenne (c'est ce que le $1/p$ achète). Probabilités de conservation typiques : environ $0{,}8$ en entrée, $0{,}5$ dans les couches cachées. |
| + | |
| + |  |
| + | |
| + | *Le dropout entraîne à chaque pas un sous-réseau aminci différent en retirant des unités au hasard, et les moyenne à l'inférence.* |
| + | |
| + | *Initialisé à la bonne échelle, normalisé en vol et régularisé contre la mémorisation, le réseau est prêt pour l'architecture. La prochaine leçon construit le réseau convolutif, dont le partage de poids est lui-même une forme de régularisation.* |
| + | |
| + | --- |
| + | Suivant : [Réseaux convolutifs](/fr/Deep%20Learning/05%20Convolutional%20networks) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| fr/Deep Learning/08 Normalization/batchnorm-vs-layernorm.svg .. fr/Deep Learning/04 Training deep networks/batchnorm-vs-layernorm.svg | |
| fr/Deep Learning/09 Regularization and dropout/dropout-network.svg .. fr/Deep Learning/04 Training deep networks/dropout-network.svg | |
| fr/Deep Learning/07 Initialization and vanishing gradients/gradient-flow.png .. fr/Deep Learning/04 Training deep networks/gradient-flow.png | |
| fr/Deep Learning/05 Backpropagation.md .. /dev/null | |
| @@ 1,100 0,0 @@ | |
| - | # 5. Rétropropagation |
| - | |
| - | La rétropropagation est l'algorithme qui calcule le gradient du coût par rapport à chaque paramètre d'un réseau. Ce n'est rien de plus que la règle de dérivation en chaîne appliquée avec soin, dans l'ordre inverse, sur le graphe de calcul, en réutilisant les quantités mises en cache lors de la passe avant. Ce module la dérive couche par couche à l'aide du signal d'erreur $\delta^{[l]} = \partial L / \partial z^{[l]}$. |
| - | |
| - | **Objectifs** |
| - | - Lire un réseau comme une composition de fonctions et comprendre pourquoi les gradients circulent en sens inverse par la règle de dérivation en chaîne. |
| - | - Définir l'erreur de couche $\delta^{[l]}$ et calculer l'erreur de la couche de sortie $\delta^{[L]}$. |
| - | - Établir la récurrence arrière qui propage $\delta$ de la couche $L$ jusqu'à la couche $1$. |
| - | - Transformer chaque $\delta^{[l]}$ en gradients des paramètres $W^{[l]}$ et $b^{[l]}$. |
| - | - Assembler l'algorithme complet avant-et-arrière et le relier à la mise à jour des paramètres. |
| - | |
| - | ## 5.1 La règle de dérivation en chaîne sur un graphe de calcul |
| - | |
| - | Un réseau à propagation avant est une composition de fonctions. Chaque couche $l$ prend l'activation précédente $a^{[l-1]}$ et produit une pré-activation et une activation : |
| - | |
| - | $$\boxed{ z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}, \quad a^{[l]} = g^{[l]}\!\left(z^{[l]}\right) }$$ |
| - | |
| - | avec $a^{[0]} = x$ et la prédiction $\hat{y} = a^{[L]}$. La perte scalaire $L$ se trouve à la fin de cette chaîne. Comme le coût est une composition, sa dérivée par rapport à n'importe quelle quantité intermédiaire est un produit de dérivées locales, une par lien du graphe. La règle de dérivation en chaîne nous dit d'accumuler ces produits. |
| - | |
| - | La manière efficace de procéder est de parcourir le graphe en sens inverse. Un seul parcours arrière calcule, pour chaque nœud, la dérivée de la perte finale par rapport à ce nœud, et chaque étape réutilise la dérivée déjà calculée pour le nœud situé juste en aval. C'est cette réutilisation qui fait que la rétropropagation coûte à peu près autant qu'une seule passe avant, plutôt qu'une passe par paramètre. |
| - | |
| - |  |
| - | |
| - | *La rétropropagation parcourt le graphe de calcul en sens inverse : la passe avant (trait plein) met les valeurs en cache, la passe arrière (trait pointillé) propage l'erreur delta.* |
| - | |
| - | *Remarque :* les flèches pleines représentent la passe avant (les données circulant vers la perte) et les flèches pointillées la passe arrière (les gradients circulant depuis la perte). Les deux passes parcourent le même graphe dans des directions opposées. |
| - | |
| - | ## 5.2 L'erreur de couche |
| - | |
| - | L'objet central est l'erreur de la couche $l$, la sensibilité de la perte à la pré-activation $z^{[l]}$ : |
| - | |
| - | $$\boxed{ \delta^{[l]} = \frac{\partial L}{\partial z^{[l]}} \in \mathbb{R}^{n_l} }$$ |
| - | |
| - | Une fois que l'on connaît $\delta^{[l]}$ à chaque couche, tous les gradients des paramètres en découlent immédiatement (section 5.5). L'algorithme tout entier se ramène au calcul de ces vecteurs, d'abord à la couche de sortie, puis récursivement en sens inverse. |
| - | |
| - | *Remarque :* placer $\delta$ à la pré-activation $z^{[l]}$ plutôt qu'à l'activation $a^{[l]}$ est un choix délibéré. Cela fait apparaître la dérivée de l'activation $g'^{[l]}$ exactement une fois par couche et garde la récurrence propre. |
| - | |
| - | ## 5.3 Erreur de la couche de sortie |
| - | |
| - | À la couche de sortie, la règle de dérivation en chaîne comporte deux liens : la perte dépend de $a^{[L]} = \hat{y}$, et $a^{[L]}$ dépend de $z^{[L]}$ à travers l'activation $g^{[L]}$. En multipliant les deux dérivées locales élément par élément, on obtient l'erreur de sortie : |
| - | |
| - | $$\boxed{ \delta^{[L]} = \nabla_{a^{[L]}} L \;\odot\; g'^{[L]}\!\left(z^{[L]}\right) }$$ |
| - | |
| - | Le produit de Hadamard $\odot$ apparaît parce que $g^{[L]}$ agit élément par élément, de sorte que la composante $j$ de $z^{[L]}$ n'influence que la composante $j$ de $a^{[L]}$. |
| - | |
| - | ### 5.3.1 Le raccourci softmax et entropie croisée |
| - | |
| - | Pour la classification multiclasse, l'appariement naturel est une sortie softmax avec la perte d'entropie croisée (introduite dans [Fonctions de perte et couches de sortie](/fr/Deep%20Learning/04%20Loss%20functions%20and%20output%20layers)). Les deux dérivées se combinent et s'annulent, laissant un résultat d'une simplicité frappante : |
| - | |
| - | $$\boxed{ \delta^{[L]} = \hat{y} - y }$$ |
| - | |
| - | *Remarque :* la même forme épurée apparaît pour une sortie sigmoïde avec entropie croisée binaire, et pour une sortie linéaire avec erreur quadratique. Dans chaque cas, l'activation de sortie est la fonction de lien inverse appariée à la perte, si bien que les facteurs encombrants s'annulent et que l'erreur se réduit au résidu $\hat{y} - y$. |
| - | |
| - | ## 5.4 La récurrence arrière |
| - | |
| - | Étant donné l'erreur à la couche $l+1$, on obtient l'erreur à la couche $l$. La perte ne dépend de $z^{[l]}$ qu'à travers $z^{[l+1]} = W^{[l+1]} a^{[l]} + b^{[l]}$, et $a^{[l]} = g^{[l]}(z^{[l]})$. En propageant la sensibilité en arrière à travers la matrice de poids puis à travers l'activation, on obtient : |
| - | |
| - | $$\boxed{ \delta^{[l]} = \left( \left(W^{[l+1]}\right)^{T} \delta^{[l+1]} \right) \odot g'^{[l]}\!\left(z^{[l]}\right) }$$ |
| - | |
| - | Deux opérations ont lieu ici. La transposée $\left(W^{[l+1]}\right)^{T}$ renvoie l'erreur aval à travers l'application linéaire, en répartissant chaque composante aval sur les unités qui l'ont alimentée. Le produit élément par élément avec $g'^{[l]}(z^{[l]})$ la filtre ensuite selon la sensibilité de chaque activation à son point de fonctionnement. |
| - | |
| - | | Symbole | Signification | Forme | |
| - | | --- | --- | --- | |
| - | | $\delta^{[l]}$ | erreur à la couche $l$ | $(n_l)$ | |
| - | | $W^{[l+1]}$ | poids entrant dans la couche $l+1$ | $(n_{l+1} \times n_l)$ | |
| - | | $\left(W^{[l+1]}\right)^{T}\delta^{[l+1]}$ | erreur renvoyée vers la couche $l$ | $(n_l)$ | |
| - | | $g'^{[l]}(z^{[l]})$ | pente locale de l'activation | $(n_l)$ | |
| - | |
| - | *Remarque :* la passe avant utilise $W^{[l+1]}$ et la passe arrière utilise sa transposée. C'est la même application linéaire lue en sens inverse, ce qui explique pourquoi la passe arrière a le même coût que la passe avant. |
| - | |
| - | ## 5.5 Gradients des paramètres |
| - | |
| - | L'erreur $\delta^{[l]}$ est tout ce dont nous avons besoin pour les paramètres de la couche $l$. Puisque $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$ est linéaire en $W^{[l]}$ et $b^{[l]}$, le dernier lien de la règle de dérivation en chaîne est simple. Le gradient des poids est le produit extérieur de l'erreur de couche avec l'activation d'entrée mise en cache : |
| - | |
| - | $$\boxed{ \frac{\partial L}{\partial W^{[l]}} = \delta^{[l]} \left(a^{[l-1]}\right)^{T}, \qquad \frac{\partial L}{\partial b^{[l]}} = \delta^{[l]} }$$ |
| - | |
| - | Le gradient des poids a la forme $(n_l \times n_{l-1})$, correspondant à $W^{[l]}$, et le gradient du biais a la forme $(n_l)$, correspondant à $b^{[l]}$. Le gradient du biais vaut exactement $\delta^{[l]}$ car $\partial z^{[l]} / \partial b^{[l]}$ est l'identité. |
| - | |
| - | *Remarque :* l'activation mise en cache $a^{[l-1]}$ issue de la passe avant est réutilisée telle quelle dans le gradient des poids. C'est le bénéfice concret de la mise en cache : rien de la passe avant n'est recalculé. |
| - | |
| - | ## 5.6 L'algorithme complet |
| - | |
| - | La rétropropagation exécute une passe avant pour remplir un cache, une passe arrière pour propager $\delta$, puis une mise à jour des paramètres. |
| - | |
| - | 1. **Passe avant.** Poser $a^{[0]} = x$. Pour $l = 1, \dots, L$, calculer $z^{[l]}$ et $a^{[l]}$, en mettant chacun en cache. Évaluer la perte $L$ en $\hat{y} = a^{[L]}$. |
| - | 2. **Erreur de sortie.** Calculer $\delta^{[L]}$ d'après la section 5.3. |
| - | 3. **Passe arrière.** Pour $l = L-1, \dots, 1$, appliquer la récurrence de la section 5.4 pour obtenir $\delta^{[l]}$. |
| - | 4. **Gradients.** Pour chaque couche, former $\partial L / \partial W^{[l]}$ et $\partial L / \partial b^{[l]}$ d'après la section 5.5. |
| - | 5. **Mise à jour.** Sur un lot, moyenner les gradients par exemple pour obtenir le gradient du coût $\nabla J$ et effectuer un pas de descente de gradient (détaillé dans [Optimisation](/fr/Deep%20Learning/06%20Optimization)). |
| - | |
| - |  |
| - | |
| - | *L'algorithme de rétropropagation vu comme un pipeline, depuis une passe avant mise en cache jusqu'à la mise à jour des paramètres.* |
| - | |
| - | *Remarque :* la rétropropagation donne le gradient, pas le pas. Elle indique quelle direction abaisse le coût, et de combien par unité de chaque paramètre. Transformer ce gradient en une modification effective des poids est le travail de l'optimiseur. |
| - | |
| - | En résumé, la rétropropagation est une application ordonnée, en une seule passe, de la règle de dérivation en chaîne qui réutilise les quantités avant mises en cache pour calculer chaque gradient au prix d'environ une passe avant supplémentaire. *Le gradient en main, la prochaine leçon étudie comment bien l'utiliser : taux d'apprentissage, momentum, et les méthodes adaptatives qui rendent les réseaux profonds entraînables.* |
| - | |
| - | --- |
| - | Suivant : [Optimisation](/fr/Deep%20Learning/06%20Optimization) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| fr/Deep Learning/05 Backpropagation/backprop-steps.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1120 320" width="1120" height="320" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="1120" height="320" fill="#ffffff"/><text x="560.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">The backpropagation algorithm as a pipeline</text><rect x="25.0" y="130.0" width="150.0" height="92.0" rx="8" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><text x="100.0" y="172.8" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">1. Forward pass:</text><text x="100.0" y="188.1" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">cache z, a</text><rect x="209.0" y="130.0" width="150.0" height="92.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="284.0" y="180.4" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">2. Evaluate loss L</text><rect x="393.0" y="130.0" width="150.0" height="92.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="468.0" y="172.8" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">3. Output error</text><text x="468.0" y="188.1" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">δ<tspan baseline-shift="super" font-size="9px">[L]</tspan></text><rect x="577.0" y="130.0" width="150.0" height="92.0" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="652.0" y="157.4" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">4. Backward</text><text x="652.0" y="172.8" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">recursion</text><text x="652.0" y="188.1" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">δ<tspan baseline-shift="super" font-size="9px">[L]</tspan> to</text><text x="652.0" y="203.4" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">δ<tspan baseline-shift="super" font-size="9px">[1]</tspan></text><rect x="761.0" y="130.0" width="150.0" height="92.0" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="836.0" y="165.1" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">5. Parameter</text><text x="836.0" y="180.4" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">gradients ∇ W,</text><text x="836.0" y="195.8" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">∇ b</text><rect x="945.0" y="130.0" width="150.0" height="92.0" rx="8" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="1020.0" y="165.1" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">6.</text><text x="1020.0" y="180.4" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">Gradient-descent</text><text x="1020.0" y="195.8" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">step</text><line x1="178.0" y1="176.0" x2="206.0" y2="176.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="362.0" y1="176.0" x2="390.0" y2="176.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="546.0" y1="176.0" x2="574.0" y2="176.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="730.0" y1="176.0" x2="758.0" y2="176.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="914.0" y1="176.0" x2="942.0" y2="176.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><text x="100.0" y="256.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">forward</text><text x="652.0" y="256.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">backward</text><text x="1020.0" y="256.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">update</text></svg> |
| \ | No newline at end of file |
| fr/Deep Learning/05 Backpropagation/computational-graph.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1080 380" width="1080" height="380" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="1080" height="380" fill="#ffffff"/><text x="540.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">Forward and backward passes over the computational graph</text><rect x="40.0" y="150.0" width="118.0" height="54.0" rx="8" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><text x="99.0" y="182.1" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">x = a<tspan baseline-shift="super" font-size="11px">[0]</tspan></text><rect x="212.0" y="150.0" width="118.0" height="54.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="271.0" y="182.1" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">z<tspan baseline-shift="super" font-size="11px">[1]</tspan></text><rect x="384.0" y="150.0" width="118.0" height="54.0" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="443.0" y="182.1" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">a<tspan baseline-shift="super" font-size="11px">[1]</tspan></text><rect x="556.0" y="150.0" width="118.0" height="54.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="615.0" y="182.1" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">z<tspan baseline-shift="super" font-size="11px">[2]</tspan></text><rect x="728.0" y="150.0" width="118.0" height="54.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="787.0" y="182.1" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">ŷ</text><rect x="900.0" y="150.0" width="118.0" height="54.0" rx="8" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="959.0" y="182.1" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">loss L</text><text x="99.0" y="124.0" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#5b6b7b" text-anchor="start">forward pass (solid): cache z and a</text><line x1="162.0" y1="171.0" x2="208.0" y2="171.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="334.0" y1="171.0" x2="380.0" y2="171.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="506.0" y1="171.0" x2="552.0" y2="171.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="678.0" y1="171.0" x2="724.0" y2="171.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="850.0" y1="171.0" x2="896.0" y2="171.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="959.0" y1="250.0" x2="787.0" y2="250.0" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)" stroke-dasharray="5 4"/><text x="873.0" y="242.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">δ<tspan baseline-shift="super" font-size="9px">[L]</tspan></text><line x1="787.0" y1="250.0" x2="615.0" y2="250.0" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)" stroke-dasharray="5 4"/><text x="701.0" y="242.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">δ<tspan baseline-shift="super" font-size="9px">[2]</tspan></text><line x1="615.0" y1="250.0" x2="443.0" y2="250.0" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)" stroke-dasharray="5 4"/><text x="529.0" y="242.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">∇ W<tspan baseline-shift="super" font-size="9px">[2]</tspan>, ∇ b<tspan baseline-shift="super" font-size="9px">[2]</tspan></text><line x1="443.0" y1="250.0" x2="271.0" y2="250.0" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)" stroke-dasharray="5 4"/><text x="357.0" y="242.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">δ<tspan baseline-shift="super" font-size="9px">[1]</tspan></text><line x1="271.0" y1="250.0" x2="99.0" y2="250.0" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)" stroke-dasharray="5 4"/><text x="185.0" y="242.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">∇ W<tspan baseline-shift="super" font-size="9px">[1]</tspan>, ∇ b<tspan baseline-shift="super" font-size="9px">[1]</tspan></text><text x="99.0" y="280.0" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#5b6b7b" text-anchor="start">backward pass (dashed): propagate the error δ</text></svg> |
| \ | No newline at end of file |
| fr/Deep Learning/10 Convolutional networks.md .. fr/Deep Learning/05 Convolutional networks.md | |
| @@ 1,17 1,8 @@ | |
| - | # 10. Réseaux convolutifs |
| + | # 5. Réseaux convolutifs |
| Une couche dense traite une image comme un vecteur aplati, elle doit donc apprendre un poids distinct pour chaque pixel et oublie que les pixels voisins vont ensemble. Les réseaux convolutifs remplacent cette connectivité dense par un petit filtre qui glisse sur la grille, en réutilisant les mêmes poids partout. Ce module présente la convolution comme une couche structurée pour les données en grille, puis introduit progressivement le pas, le remplissage, les canaux et le pooling. | |
| - | **Objectifs** |
| - | - Motiver la convolution à partir de la localité, de l'équivariance à la translation et du partage de paramètres. |
| - | - Définir la convolution 2D (corrélation croisée) utilisée en apprentissage profond. |
| - | - Calculer la taille de sortie à partir de la taille d'entrée, du noyau, du remplissage et du pas. |
| - | - Étendre un filtre à plusieurs canaux d'entrée et de sortie (cartes de caractéristiques). |
| - | - Utiliser le max pooling et l'average pooling pour sous-échantillonner et ajouter une petite invariance à la translation. |
| - | - Comparer le nombre de paramètres d'une convolution à celui d'une couche dense équivalente. |
| - | - Reconnaître les architectures marquantes, de LeNet à ResNet, et l'idée que chacune a apportée. |
| - | |
| - | ## 10.1 Pourquoi pas une couche dense |
| + | ## 5.1 Pourquoi pas une couche dense |
| Considérons une image RVB modeste de $224 \times 224$. Aplatie, elle compte $224 \times 224 \times 3 \approx 150{,}000$ entrées, donc une seule couche dense avec ne serait-ce que $1{,}000$ unités porte environ $150$ millions de poids. Trois faits à propos des images rendent presque tous ces poids inutiles. | |
| @@ 21,11 12,24 @@ | |
| Une couche convolutive intègre ces trois principes. Elle utilise un petit filtre (les poids partagés) appliqué à chaque emplacement (localité et équivariance), c'est pourquoi elle a besoin de plusieurs ordres de grandeur de paramètres en moins que la couche dense ci-dessus. | |
| + |  |
| + | |
| + | *La couche dense aplatit l'image et paie un poids privé par pixel et par unité, environ $150$ millions dans cet exemple. Le filtre convolutif porte $27$ poids et un biais, et il est simplement réutilisé à chaque position.* |
| + | |
| + | Le gain en chiffres, cette fois sur une entrée $32 \times 32 \times 3$ plus petite avec une couche de $16$ filtres (les bancs de filtres et les canaux sont précisés dans la section 5.4, seul le décompte importe ici) : |
| + | |
| + | | Couche | Poids | Biais | Total des paramètres | |
| + | | --- | --- | --- | --- | |
| + | | Convolution ($3\times3$, $16$ filtres) | $3 \cdot 3 \cdot 3 \cdot 16 = 432$ | $16$ | $448$ | |
| + | | Couche dense équivalente | $(32\cdot32\cdot3)\cdot(32\cdot32\cdot16) \approx 5.0\times10^{10}$ | $16{,}384$ | $\approx 5.0\times10^{10}$ | |
| + | |
| + | Quelques centaines de paramètres contre environ cinquante milliards, et la convolution généralise mieux, car le même détecteur de caractéristiques est réutilisé partout plutôt que réappris à chaque position. |
| + | |
| *Remarque :* rappelez-vous la notation de l'Introduction. Une couche $l$ calcule $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$ et $a^{[l]} = g^{[l]}(z^{[l]})$, avec un biais explicite $b^{[l]}$. Une convolution n'est qu'un $W^{[l]}$ structuré dont les entrées sont liées entre elles et majoritairement nulles, donc la même équation de couche reste valable. | |
| - | ## 10.2 La convolution 2D |
| + | ## 5.2 La convolution 2D |
| - | ### 10.2.1 Corrélation croisée |
| + | ### 5.2.1 Corrélation croisée |
| Soit $I$ une entrée 2D (un canal d'une image) et $K$ un noyau de taille $k \times k$. L'opération utilisée en apprentissage profond fait glisser $K$ sur $I$ et prend, à chaque position $(i, j)$, la somme des produits terme à terme entre le noyau et la fenêtre qu'il couvre : | |
| @@ 33,13 37,13 @@ | |
| Chaque valeur de sortie est un produit scalaire entre le noyau et une fenêtre locale de l'entrée, donc un petit noyau $3 \times 3$ regarde neuf pixels quelle que soit la taille de l'image. | |
| - |  |
| + |  |
| *Une convolution fait glisser un petit noyau sur l'entrée, et chaque position produit une cellule de la carte de caractéristiques de sortie.* | |
| *Remarque :* il s'agit techniquement d'une corrélation croisée. La convolution mathématique retourne d'abord le noyau, mais les bibliothèques d'apprentissage profond ne le retournent pas et parlent tout de même de convolution, car le noyau appris absorbe simplement le retournement. Nous suivons cette convention tout au long du cours. | |
| - | ### 10.2.2 La sortie de la couche |
| + | ### 5.2.2 La sortie de la couche |
| Une couche convolutive applique cette opération, ajoute le biais explicite $b$, et passe le résultat à travers l'activation $g$ : | |
| @@ 47,7 51,7 @@ | |
| Le biais est un unique scalaire partagé sur chaque position de la sortie, exactement une instance de plus de partage de paramètres. | |
| - | ## 10.3 Pas, remplissage et taille de sortie |
| + | ## 5.3 Pas, remplissage et taille de sortie |
| Deux hyperparamètres contrôlent la manière dont le noyau balaie l'entrée. | |
| @@ 62,7 66,11 @@ | |
| Par exemple, avec $n = 32$, $k = 5$, $p = 0$, $s = 1$ la sortie est $\lfloor (32 - 5)/1 \rfloor + 1 = 28$. Ajouter $p = 2$ ("same") donne $\lfloor (32 + 4 - 5)/1 \rfloor + 1 = 32$. | |
| - | ## 10.4 Canaux et cartes de caractéristiques |
| + |  |
| + | |
| + | *Le même noyau $3 \times 3$ sur une entrée $6 \times 6$, de trois façons. Le pas $1$ visite quatre positions par ligne, le pas $2$ saute une position sur deux, et un anneau de zéros ($p = 1$) laisse le noyau couvrir le bord, la sortie garde donc la taille de l'entrée.* |
| + | |
| + | ## 5.4 Canaux et cartes de caractéristiques |
| Les vraies images ont des canaux (trois pour le RVB), et un noyau s'étend sur tous. Un filtre pour une entrée à $C_\text{in}$ canaux a la forme $k \times k \times C_\text{in}$, et sa convolution somme sur les positions spatiales et les canaux pour produire une seule sortie 2D, appelée **carte de caractéristiques**. | |
| @@ 72,7 80,7 @@ | |
| *Remarque :* le nombre de canaux de sortie $C_\text{out}$ d'une couche devient le nombre de canaux d'entrée $C_\text{in}$ de la suivante, donc la profondeur croît à mesure que la taille spatiale rétrécit. Il y a un biais par canal de sortie, c'est pourquoi $b^{[l]}$ a $C_\text{out}$ entrées. | |
| - | ## 10.5 Pooling |
| + | ## 5.5 Pooling |
| Le pooling sous-échantillonne une carte de caractéristiques en résumant chaque petite fenêtre par un unique nombre, à l'aide d'une règle fixe et sans poids appris. Les deux règles courantes sont le maximum et la moyenne sur chaque fenêtre $k \times k$ : | |
| @@ 80,38 88,27 @@ | |
| Le pooling avec un pas $s = k$ (fenêtres non chevauchantes) réduit chaque dimension spatiale d'un facteur $k$, ce qui diminue le calcul des couches suivantes. Il confère aussi une petite **invariance à la translation** : un max sur une fenêtre renvoie la même valeur si la réponse forte se déplace à l'intérieur de cette fenêtre. | |
| - |  |
| + |  |
| *Le max pooling sous-échantillonne chaque région à sa plus grande valeur, réduisant la carte de caractéristiques et ajoutant une petite invariance à la translation.* | |
| *Remarque :* le pooling n'a aucun paramètre et réduit la résolution, c'est pourquoi les architectures modernes le remplacent souvent par des convolutions à pas. La convolution est équivariante à la translation (la réponse se déplace avec l'entrée), tandis que le pooling ajoute un peu d'invariance (la réponse ignore les petits déplacements). | |
| - | ## 10.6 Le gain en paramètres |
| - | |
| - | L'intérêt du partage de paramètres, c'est la taille. Prenons une entrée de $32 \times 32 \times 3$ et une couche produisant une sortie de $32 \times 32 \times 16$ avec un noyau $3 \times 3$ (remplissage "same"). La convolution partage un petit banc de filtres sur toutes les positions, tandis qu'une couche dense reliant chaque entrée à chaque sortie ne le fait pas. |
| - | |
| - | | Couche | Poids | Biais | Total des paramètres | |
| - | | --- | --- | --- | --- | |
| - | | Convolution ($3\times3$, $16$ filtres) | $3 \cdot 3 \cdot 3 \cdot 16 = 432$ | $16$ | $448$ | |
| - | | Couche dense équivalente | $(32\cdot32\cdot3)\cdot(32\cdot32\cdot16) \approx 5.0\times10^{10}$ | $16{,}384$ | $\approx 5.0\times10^{10}$ | |
| - | |
| - | La convolution utilise quelques centaines de paramètres contre environ cinquante milliards pour la couche dense, et elle généralise mieux car le même détecteur de caractéristiques est réutilisé partout plutôt que réappris à chaque position. |
| - | |
| - | ## 10.7 Un étage convolutif |
| + | ## 5.6 Un étage convolutif |
| Un étage typique enchaîne convolution, activation et pooling, transformant l'image brute en une pile de cartes de caractéristiques que les étages suivants affinent. | |
| - |  |
| + |  |
| *Un étage convolutif : convolution, activation, puis pooling, répété pour construire des cartes de caractéristiques.* | |
| *Remarque :* empiler de tels étages fait croître le champ récepteur (la région d'entrée qui influence une valeur de sortie) avec la profondeur, donc les premières couches voient des contours et les couches profondes voient des objets entiers, le tout construit à partir de la même opération locale. | |
| - | ## 10.8 Des couches aux architectures |
| + | ## 5.7 Des couches aux architectures |
| Les réseaux convolutifs marquants partagent tous la même forme : une pile d'étages de convolution et de pooling qui extrait des caractéristiques, puis une petite tête entièrement connectée qui les classe. | |
| - |  |
| + |  |
| *Un CNN profond réduit progressivement la taille spatiale tout en augmentant la profondeur en canaux, puis aplatit vers des couches entièrement connectées.* | |
| @@ 121,15 118,7 @@ | |
| - **AlexNet** a mis ce squelette à l'échelle des grandes images et des GPU, rendu entraînable par les activations ReLU et le dropout. | |
| - **VGG** a rendu chaque convolution $3 \times 3$ et tire sa profondeur de l'empilement : deux couches $3 \times 3$ voient la même région qu'une $5 \times 5$ avec moins de paramètres ($18c^2$ contre $25c^2$) et une non-linéarité de plus. | |
| - **Inception** lance en parallèle des branches de plusieurs tailles de filtres et les concatène, à coût maîtrisé grâce aux convolutions $1 \times 1$, des applications par position sur les canaux qui compriment une carte épaisse avant les filtres coûteux. | |
| - | - **ResNet** fait apprendre à chaque bloc une correction autour d'un saut identité : |
| - | |
| - | $$\boxed{\ y = F(x, W) + x, \qquad \frac{\partial y}{\partial x} = \frac{\partial F}{\partial x} + I\ }$$ |
| - | |
| - | Le $+I$ donne au gradient une route vers l'arrière qui ne rétrécit jamais, le remède direct au [gradient qui s'évanouit](/fr/Deep%20Learning/07%20Initialization%20and%20vanishing%20gradients) de la leçon 7, et des réseaux de centaines de couches s'entraînent de manière fiable. |
| - | |
| - |  |
| - | |
| - | *Un bloc résiduel ajoute une connexion de saut identité autour du chemin de convolution, de sorte que la couche n'a qu'à apprendre une correction F(x).* |
| + | - **ResNet** fait apprendre à chaque bloc une correction autour d'un saut identité : le saut donne au gradient une route vers l'arrière qui ne rétrécit jamais, le remède direct au [gradient qui s'évanouit](/fr/Deep%20Learning/02%20Activation%20functions) de la leçon 2, et des réseaux de centaines de couches s'entraînent de manière fiable. |
| | Architecture | Profondeur approx. | Idée clé | | |
| | --- | --- | --- | | |
| @@ 144,4 133,4 @@ | |
| *Ces piles profondes apprennent des cartes de caractéristiques dont les activations profondes se comportent comme des représentations réutilisables, la porte d'entrée du module suivant sur les plongements et l'apprentissage de représentations.* | |
| --- | |
| - | Suivant : [Plongements et apprentissage de représentations](/fr/Deep%20Learning/11%20Embeddings%20and%20representation%20learning) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| + | Suivant : [Plongements et apprentissage de représentations](/fr/Deep%20Learning/06%20Embeddings%20and%20representation%20learning) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| fr/Deep Learning/10 Convolutional networks/cnn-stack.svg .. fr/Deep Learning/05 Convolutional networks/cnn-stack.svg | |
| fr/Deep Learning/10 Convolutional networks/conv-pipeline.svg .. fr/Deep Learning/05 Convolutional networks/conv-pipeline.svg | |
| fr/Deep Learning/10 Convolutional networks/convolution.svg .. fr/Deep Learning/05 Convolutional networks/convolution.svg | |
| /dev/null .. fr/Deep Learning/05 Convolutional networks/dense-vs-conv.svg | |
| @@ 0,0 1,107 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 360" width="900" height="360" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs> |
| + | <marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | </defs> |
| + | <rect width="900" height="360" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">The same image, two layers</text> |
| + | <text x="225" y="56" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">dense layer on the flattened image</text> |
| + | <line x1="60" y1="80" x2="60" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="73" y1="80" x2="73" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="86" y1="80" x2="86" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="99" y1="80" x2="99" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="112" y1="80" x2="112" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="125" y1="80" x2="125" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="138" y1="80" x2="138" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="60" y1="80" x2="138" y2="80" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="60" y1="93" x2="138" y2="93" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="60" y1="106" x2="138" y2="106" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="60" y1="119" x2="138" y2="119" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="60" y1="132" x2="138" y2="132" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="60" y1="145" x2="138" y2="145" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="60" y1="158" x2="138" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <text x="99" y="178" font-size="10" fill="#5b6b7b" text-anchor="middle">224 × 224 × 3 image</text> |
| + | <line x1="145" y1="119" x2="178" y2="119" stroke="#1f2933" stroke-width="1.4" marker-end="url(#arrow)"/> |
| + | <text x="161" y="107" font-size="10" fill="#5b6b7b" text-anchor="middle">flatten</text> |
| + | <rect x="190" y="70" width="12" height="12" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1"/> |
| + | <rect x="190" y="84" width="12" height="12" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1"/> |
| + | <rect x="190" y="98" width="12" height="12" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1"/> |
| + | <rect x="190" y="132" width="12" height="12" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1"/> |
| + | <rect x="190" y="146" width="12" height="12" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1"/> |
| + | <rect x="190" y="160" width="12" height="12" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1"/> |
| + | <text x="196" y="124" font-size="11" fill="#5b6b7b" text-anchor="middle">⋮</text> |
| + | <text x="196" y="192" font-size="10" fill="#5b6b7b" text-anchor="middle">150,000</text> |
| + | <text x="196" y="204" font-size="10" fill="#5b6b7b" text-anchor="middle">inputs</text> |
| + | <line x1="204" y1="76" x2="321" y2="92" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="76" x2="321" y2="120" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="76" x2="321" y2="148" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="76" x2="321" y2="176" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="76" x2="321" y2="204" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="90" x2="321" y2="92" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="90" x2="321" y2="120" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="90" x2="321" y2="148" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="90" x2="321" y2="176" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="90" x2="321" y2="204" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="104" x2="321" y2="92" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="104" x2="321" y2="120" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="104" x2="321" y2="148" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="104" x2="321" y2="176" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="104" x2="321" y2="204" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="138" x2="321" y2="92" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="138" x2="321" y2="120" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="138" x2="321" y2="148" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="138" x2="321" y2="176" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="138" x2="321" y2="204" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="152" x2="321" y2="92" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="152" x2="321" y2="120" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="152" x2="321" y2="148" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="152" x2="321" y2="176" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="152" x2="321" y2="204" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="166" x2="321" y2="92" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="166" x2="321" y2="120" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="166" x2="321" y2="148" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="166" x2="321" y2="176" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <line x1="204" y1="166" x2="321" y2="204" stroke="#c7d0d9" stroke-width="0.7"/> |
| + | <circle cx="330" cy="92" r="9" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.4"/> |
| + | <circle cx="330" cy="120" r="9" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.4"/> |
| + | <circle cx="330" cy="148" r="9" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.4"/> |
| + | <circle cx="330" cy="176" r="9" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.4"/> |
| + | <circle cx="330" cy="204" r="9" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.4"/> |
| + | <text x="330" y="228" font-size="10" fill="#5b6b7b" text-anchor="middle">1,000 units</text> |
| + | <text x="225" y="272" font-size="13" font-weight="600" fill="#d1495b" text-anchor="middle">≈ 150,000,000 weights</text> |
| + | <text x="225" y="292" font-size="11" fill="#5b6b7b" text-anchor="middle">one private weight per pixel and per unit, position baked in</text> |
| + | <text x="675" y="56" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">convolutional layer</text> |
| + | <line x1="560" y1="80" x2="560" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="573" y1="80" x2="573" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="586" y1="80" x2="586" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="599" y1="80" x2="599" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="612" y1="80" x2="612" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="625" y1="80" x2="625" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="638" y1="80" x2="638" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="560" y1="80" x2="638" y2="80" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="560" y1="93" x2="638" y2="93" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="560" y1="106" x2="638" y2="106" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="560" y1="119" x2="638" y2="119" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="560" y1="132" x2="638" y2="132" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="560" y1="145" x2="638" y2="145" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="560" y1="158" x2="638" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <rect x="560" y="80" width="39" height="39" fill="#3b6fb6" fill-opacity="0.15" stroke="#3b6fb6" stroke-width="1.8"/> |
| + | <rect x="599" y="119" width="39" height="39" fill="none" stroke="#3b6fb6" stroke-width="1.4" stroke-dasharray="4 3" opacity="0.75"/> |
| + | <text x="599" y="178" font-size="10" fill="#5b6b7b" text-anchor="middle">the same 3 × 3 filter</text> |
| + | <text x="599" y="190" font-size="10" fill="#5b6b7b" text-anchor="middle">slides everywhere</text> |
| + | <line x1="648" y1="119" x2="682" y2="119" stroke="#1f2933" stroke-width="1.4" marker-end="url(#arrow)"/> |
| + | <rect x="692" y="93" width="52" height="52" fill="#e7f5ea"/> |
| + | <line x1="692" y1="93" x2="692" y2="145" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="705" y1="93" x2="705" y2="145" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="718" y1="93" x2="718" y2="145" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="731" y1="93" x2="731" y2="145" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="744" y1="93" x2="744" y2="145" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="692" y1="93" x2="744" y2="93" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="692" y1="106" x2="744" y2="106" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="692" y1="119" x2="744" y2="119" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="692" y1="132" x2="744" y2="132" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="692" y1="145" x2="744" y2="145" stroke="#38a05a" stroke-width="1.0"/> |
| + | <text x="718" y="166" font-size="10" fill="#5b6b7b" text-anchor="middle">feature map</text> |
| + | <text x="675" y="272" font-size="13" font-weight="600" fill="#38a05a" text-anchor="middle">27 weights + 1 bias, shared</text> |
| + | <text x="675" y="292" font-size="11" fill="#5b6b7b" text-anchor="middle">one 3 × 3 × 3 filter reused at every position</text> |
| + | <text x="450" y="334" font-size="11" fill="#5b6b7b" text-anchor="middle">same image, over a million times fewer parameters</text> |
| + | </svg> |
| \ | No newline at end of file |
| fr/Deep Learning/10 Convolutional networks/pooling.svg .. fr/Deep Learning/05 Convolutional networks/pooling.svg | |
| /dev/null .. fr/Deep Learning/05 Convolutional networks/stride-padding.svg | |
| @@ 0,0 1,121 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 360" width="900" height="360" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs> |
| + | <marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | </defs> |
| + | <rect width="900" height="360" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">Stride and padding: a 3 × 3 kernel on a 6 × 6 input</text> |
| + | <text x="165" y="56" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">valid (p = 0), stride 1</text> |
| + | <line x1="126" y1="80" x2="126" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="139" y1="80" x2="139" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="152" y1="80" x2="152" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="165" y1="80" x2="165" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="178" y1="80" x2="178" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="191" y1="80" x2="191" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="204" y1="80" x2="204" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="126" y1="80" x2="204" y2="80" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="126" y1="93" x2="204" y2="93" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="126" y1="106" x2="204" y2="106" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="126" y1="119" x2="204" y2="119" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="126" y1="132" x2="204" y2="132" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="126" y1="145" x2="204" y2="145" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="126" y1="158" x2="204" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <rect x="126" y="80" width="39" height="39" fill="#3b6fb6" fill-opacity="0.15" stroke="#3b6fb6" stroke-width="1.8"/> |
| + | <rect x="139" y="80" width="39" height="39" fill="none" stroke="#3b6fb6" stroke-width="1.3" stroke-dasharray="4 3" opacity="0.7"/> |
| + | <line x1="141" y1="72" x2="153" y2="72" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/> |
| + | <text x="133" y="68" font-size="10" fill="#5b6b7b" text-anchor="middle">+1</text> |
| + | <line x1="165" y1="168" x2="165" y2="196" stroke="#1f2933" stroke-width="1.4" marker-end="url(#arrow)"/> |
| + | <rect x="139" y="204" width="52" height="52" fill="#e7f5ea"/> |
| + | <line x1="139" y1="204" x2="139" y2="256" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="152" y1="204" x2="152" y2="256" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="165" y1="204" x2="165" y2="256" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="178" y1="204" x2="178" y2="256" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="191" y1="204" x2="191" y2="256" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="139" y1="204" x2="191" y2="204" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="139" y1="217" x2="191" y2="217" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="139" y1="230" x2="191" y2="230" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="139" y1="243" x2="191" y2="243" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="139" y1="256" x2="191" y2="256" stroke="#38a05a" stroke-width="1.0"/> |
| + | <text x="165" y="290" font-size="12" fill="#1f2933" text-anchor="middle">o = (6 − 3)/1 + 1 = 4</text> |
| + | <text x="165" y="310" font-size="11" fill="#5b6b7b" text-anchor="middle">the kernel visits 4 positions per row</text> |
| + | <text x="450" y="56" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">stride 2</text> |
| + | <line x1="411" y1="80" x2="411" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="424" y1="80" x2="424" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="437" y1="80" x2="437" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="450" y1="80" x2="450" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="463" y1="80" x2="463" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="476" y1="80" x2="476" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="489" y1="80" x2="489" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="411" y1="80" x2="489" y2="80" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="411" y1="93" x2="489" y2="93" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="411" y1="106" x2="489" y2="106" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="411" y1="119" x2="489" y2="119" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="411" y1="132" x2="489" y2="132" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="411" y1="145" x2="489" y2="145" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="411" y1="158" x2="489" y2="158" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <rect x="411" y="80" width="39" height="39" fill="#3b6fb6" fill-opacity="0.15" stroke="#3b6fb6" stroke-width="1.8"/> |
| + | <rect x="437" y="80" width="39" height="39" fill="none" stroke="#3b6fb6" stroke-width="1.3" stroke-dasharray="4 3" opacity="0.7"/> |
| + | <line x1="426" y1="72" x2="452" y2="72" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/> |
| + | <text x="416" y="68" font-size="10" fill="#5b6b7b" text-anchor="middle">+2</text> |
| + | <line x1="450" y1="168" x2="450" y2="196" stroke="#1f2933" stroke-width="1.4" marker-end="url(#arrow)"/> |
| + | <rect x="437" y="204" width="26" height="26" fill="#e7f5ea"/> |
| + | <line x1="437" y1="204" x2="437" y2="230" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="450" y1="204" x2="450" y2="230" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="463" y1="204" x2="463" y2="230" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="437" y1="204" x2="463" y2="204" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="437" y1="217" x2="463" y2="217" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="437" y1="230" x2="463" y2="230" stroke="#38a05a" stroke-width="1.0"/> |
| + | <text x="450" y="290" font-size="12" fill="#1f2933" text-anchor="middle">o = ⌊(6 − 3)/2⌋ + 1 = 2</text> |
| + | <text x="450" y="310" font-size="11" fill="#5b6b7b" text-anchor="middle">it skips every other position</text> |
| + | <text x="735" y="56" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">“same” padding, p = 1</text> |
| + | <line x1="683" y1="68" x2="683" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="696" y1="68" x2="696" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="709" y1="68" x2="709" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="722" y1="68" x2="722" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="735" y1="68" x2="735" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="748" y1="68" x2="748" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="761" y1="68" x2="761" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="774" y1="68" x2="774" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="787" y1="68" x2="787" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="683" y1="68" x2="787" y2="68" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="683" y1="81" x2="787" y2="81" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="683" y1="94" x2="787" y2="94" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="683" y1="107" x2="787" y2="107" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="683" y1="120" x2="787" y2="120" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="683" y1="133" x2="787" y2="133" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="683" y1="146" x2="787" y2="146" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="683" y1="159" x2="787" y2="159" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="683" y1="172" x2="787" y2="172" stroke="#c7d0d9" stroke-width="0.9" stroke-dasharray="3 2"/> |
| + | <line x1="696" y1="81" x2="696" y2="159" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="709" y1="81" x2="709" y2="159" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="722" y1="81" x2="722" y2="159" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="735" y1="81" x2="735" y2="159" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="748" y1="81" x2="748" y2="159" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="761" y1="81" x2="761" y2="159" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="774" y1="81" x2="774" y2="159" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="696" y1="81" x2="774" y2="81" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="696" y1="94" x2="774" y2="94" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="696" y1="107" x2="774" y2="107" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="696" y1="120" x2="774" y2="120" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="696" y1="133" x2="774" y2="133" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="696" y1="146" x2="774" y2="146" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <line x1="696" y1="159" x2="774" y2="159" stroke="#9aa7b2" stroke-width="1.0"/> |
| + | <rect x="683" y="68" width="39" height="39" fill="#3b6fb6" fill-opacity="0.15" stroke="#3b6fb6" stroke-width="1.8"/> |
| + | <line x1="735" y1="176" x2="735" y2="196" stroke="#1f2933" stroke-width="1.4" marker-end="url(#arrow)"/> |
| + | <rect x="696" y="204" width="78" height="78" fill="#e7f5ea"/> |
| + | <line x1="696" y1="204" x2="696" y2="282" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="709" y1="204" x2="709" y2="282" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="722" y1="204" x2="722" y2="282" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="735" y1="204" x2="735" y2="282" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="748" y1="204" x2="748" y2="282" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="761" y1="204" x2="761" y2="282" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="774" y1="204" x2="774" y2="282" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="696" y1="204" x2="774" y2="204" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="696" y1="217" x2="774" y2="217" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="696" y1="230" x2="774" y2="230" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="696" y1="243" x2="774" y2="243" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="696" y1="256" x2="774" y2="256" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="696" y1="269" x2="774" y2="269" stroke="#38a05a" stroke-width="1.0"/> |
| + | <line x1="696" y1="282" x2="774" y2="282" stroke="#38a05a" stroke-width="1.0"/> |
| + | <text x="735" y="302" font-size="12" fill="#1f2933" text-anchor="middle">o = (6 + 2 − 3)/1 + 1 = 6</text> |
| + | <text x="735" y="322" font-size="11" fill="#5b6b7b" text-anchor="middle">one ring of zeros keeps the output size</text> |
| + | </svg> |
| \ | No newline at end of file |
| fr/Deep Learning/11 Embeddings and representation learning.md .. fr/Deep Learning/06 Embeddings and representation learning.md | |
| @@ 1,18 1,10 @@ | |
| - | # 11. Plongements et apprentissage de représentations |
| + | # 6. Plongements et apprentissage de représentations |
| Les réseaux de neurones transforment des entrées brutes en caractéristiques utiles en les apprenant plutôt qu'en les concevant à la main. Pour les symboles discrets (mots, identifiants de produits, identifiants d'utilisateurs, catégories), la représentation naturelle est un vecteur dense appris appelé plongement (embedding). Cette leçon montre pourquoi les codes one-hot sont une mauvaise entrée, comment une matrice de plongement associe chaque symbole à un vecteur compact, comment word2vec apprend de tels vecteurs à partir de la cooccurrence, et pourquoi les plongements sont l'entrée standard des modèles de séquences et des Transformers qui suivent. | |
| - | **Objectifs** |
| - | - Expliquer pourquoi les encodages one-hot sont volumineux, creux et aveugles à la similarité. |
| - | - Définir un plongement comme une recherche dans une matrice apprise $E$ et traiter ses lignes comme des paramètres. |
| - | - Énoncer l'objectif skip-gram de word2vec et le rôle de l'échantillonnage négatif. |
| - | - Mesurer la proximité sémantique avec la similarité cosinus. |
| - | - Voir comment la même idée couvre les articles, les utilisateurs et les caractéristiques catégorielles. |
| - | - Relier les plongements aux réseaux récurrents et aux Transformers en tant que couche d'entrée. |
| + | ## 6.1 Du one-hot aux vecteurs denses |
| - | ## 11.1 Du one-hot aux vecteurs denses |
| - | |
| - | ### 11.1.1 La représentation one-hot |
| + | ### 6.1.1 La représentation one-hot |
| Supposons que le vocabulaire comporte $V$ symboles distincts. La façon classique de fournir le symbole $i$ à un réseau est le vecteur one-hot $x_{\text{onehot}} \in \{0, 1\}^V$, qui ne contient que des zéros à l'exception d'un unique $1$ à la position $i$. Il ne porte aucune structure : chaque paire de symboles distincts est exactement aussi éloignée que toutes les autres paires, si bien que le code ne contient aucune notion de similarité. Il est aussi énorme, un vocabulaire moderne a un $V$ de l'ordre de dizaines ou de centaines de milliers, et il est presque entièrement composé de zéros. | |
| @@ 26,7 18,7 @@ | |
| *Remarque :* fournir un vecteur one-hot à une couche linéaire $W x_{\text{onehot}}$ revient simplement à sélectionner une colonne de $W$. La recherche de plongement ci-dessous rend cette sélection explicite et peu coûteuse. | |
| - | ### 11.1.2 La recherche de plongement |
| + | ### 6.1.2 La recherche de plongement |
| Une matrice de plongement $E \in \mathbb{R}^{V \times d}$ stocke une ligne de dimension $d$ par symbole. Le plongement d'une entrée one-hot est le produit matrice-vecteur | |
| @@ 36,11 28,11 @@ | |
| *Remarque :* les lignes de $E$ sont des paramètres ordinaires. Elles démarrent aléatoires et sont mises à jour par rétropropagation en même temps que le reste du réseau, de sorte que la géométrie de l'espace est façonnée par la tâche sur laquelle le réseau est entraîné. | |
| - | ## 11.2 Apprendre des plongements de mots avec word2vec |
| + | ## 6.2 Apprendre des plongements de mots avec word2vec |
| Les plongements peuvent être appris de bout en bout à l'intérieur de n'importe quelle tâche, mais ils peuvent aussi être appris seuls à partir de texte non étiqueté. Le modèle skip-gram de word2vec fait exactement cela : il apprend un vecteur par mot en prédisant les mots de contexte environnants à partir d'un mot central. | |
| - | ### 11.2.1 Objectif skip-gram |
| + | ### 6.2.1 Objectif skip-gram |
| Chaque mot $w$ possède un vecteur d'entrée $v_w$ (sa ligne dans la matrice de plongement). Étant donné un mot central $w_I$, le modèle attribue un score à chaque mot de sortie candidat $w_O$ par un produit scalaire et normalise sur l'ensemble du vocabulaire avec un softmax : | |
| @@ 48,17 40,17 @@ | |
| L'entraînement maximise cette probabilité pour les paires (central, contexte) qui cooccurrent réellement dans une fenêtre glissante le long du texte. Les mots qui apparaissent dans des contextes similaires sont poussés à avoir de grands produits scalaires, de sorte que leurs vecteurs finissent proches les uns des autres. | |
| - | ### 11.2.2 Échantillonnage négatif |
| + | ### 6.2.2 Échantillonnage négatif |
| Le dénominateur somme sur tous les $V$ mots, ce qui est bien trop coûteux à calculer pour chaque paire d'entraînement. L'échantillonnage négatif remplace le softmax complet par un problème binaire peu coûteux : pour chaque paire réelle (central, contexte), on tire quelques mots aléatoires comme négatifs et on entraîne le modèle à distinguer le vrai mot de contexte des faux. Cela transforme une normalisation à $V$ voies en une poignée de mises à jour logistiques par étape, ce qui rend word2vec assez rapide pour s'entraîner sur des milliards de mots. | |
| - |  |
| + |  |
| *Le modèle skip-gram apprend des plongements en prédisant le contexte d'un mot à partir d'un mot central.* | |
| *Remarque :* l'espace appris présente une structure linéaire frappante. Les directions qu'il contient encodent des relations cohérentes, de sorte que les analogies apparaissent sous forme d'arithmétique vectorielle, l'exemple classique étant que le vecteur de « roi » moins « homme » plus « femme » tombe près de « reine ». | |
| - | ## 11.3 Mesurer la similarité |
| + | ## 6.3 Mesurer la similarité |
| Une fois les symboles devenus des vecteurs denses, « à quel point deux symboles sont-ils liés » devient une question géométrique. La réponse standard est la similarité cosinus, le cosinus de l'angle entre deux vecteurs $u$ et $v$ : | |
| @@ 66,13 58,13 @@ | |
| Elle se situe dans $[-1, 1]$ : une valeur proche de $1$ signifie que les vecteurs pointent dans la même direction (très similaires), proche de $0$ signifie qu'ils sont sans rapport, et proche de $-1$ signifie qu'ils sont opposés. Le cosinus ignore la longueur des vecteurs et ne considère que la direction, ce qui est généralement ce que nous voulons, puisque le sens d'un mot ne devrait pas dépendre de sa fréquence d'apparition. | |
| - |  |
| + |  |
| *Les plongements appris placent les mots liés les uns près des autres, et des directions cohérentes dans l'espace capturent les analogies.* | |
| *Remarque :* la recherche des plus proches voisins sous la similarité cosinus est la façon dont les plongements alimentent la recherche d'information et la recommandation. Trouvez les vecteurs stockés dont la direction est la plus proche d'un vecteur de requête et vous obtenez les articles les plus pertinents. | |
| - | ## 11.4 Les plongements au-delà des mots |
| + | ## 6.4 Les plongements au-delà des mots |
| Rien dans cette construction n'est spécifique au langage. Tout ensemble de symboles discrets peut être plongé en lui attribuant une matrice $E$ et en apprenant ses lignes. | |
| @@ 91,11 83,11 @@ | |
| *Remarque :* les plongements sont aussi une forme de réduction de dimension. Ils compressent un symbole à $V$ voies en $d$ nombres tout en conservant l'information dont une tâche en aval a besoin, ce qui est l'essence même de l'apprentissage de représentations. | |
| - | ## 11.5 Les plongements comme entrée des modèles de séquences |
| + | ## 6.5 Les plongements comme entrée des modèles de séquences |
| - | Une séquence de symboles devient une séquence de vecteurs en recherchant chacun d'eux dans $E$. Cette matrice de plongements est exactement l'entrée qu'un réseau récurrent lit étape par étape (leçon [Réseaux récurrents](/fr/Deep%20Learning/12%20Recurrent%20networks)) et l'entrée sur laquelle un Transformer porte son attention (leçon [Transformers](/fr/Deep%20Learning/15%20Transformers)). Dans les deux cas, la table de plongement est apprise conjointement avec le reste du modèle, de sorte que les représentations sont ajustées à la tâche finale plutôt que fixées à l'avance. |
| + | Une séquence de symboles devient une séquence de vecteurs en recherchant chacun d'eux dans $E$. Cette matrice de plongements est exactement l'entrée qu'un réseau récurrent lit étape par étape (leçon [Réseaux récurrents](/fr/Deep%20Learning/07%20Recurrent%20networks)) et l'entrée sur laquelle un Transformer porte son attention (leçon [Transformers](/fr/Deep%20Learning/10%20Transformers)). Dans les deux cas, la table de plongement est apprise conjointement avec le reste du modèle, de sorte que les représentations sont ajustées à la tâche finale plutôt que fixées à l'avance. |
| - |  |
| + |  |
| *Une recherche de plongement sélectionne une ligne de la matrice E, faisant correspondre un token one-hot creux à un vecteur dense appris.* | |
| @@ 104,4 96,4 @@ | |
| *Les vecteurs denses nous donnent une entrée compacte et sensible à la similarité. La prochaine leçon fournit une telle séquence de vecteurs, une étape à la fois, à un réseau récurrent qui transporte un état caché à travers le temps.* | |
| --- | |
| - | Suivant : [Réseaux récurrents](/fr/Deep%20Learning/12%20Recurrent%20networks) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| \ | No newline at end of file |
| + | Suivant : [Réseaux récurrents](/fr/Deep%20Learning/07%20Recurrent%20networks) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| \ | No newline at end of file |
| fr/Deep Learning/11 Embeddings and representation learning/embedding-lookup.svg .. fr/Deep Learning/06 Embeddings and representation learning/embedding-lookup.svg | |
| fr/Deep Learning/11 Embeddings and representation learning/embedding-space.png .. fr/Deep Learning/06 Embeddings and representation learning/embedding-space.png | |
| fr/Deep Learning/11 Embeddings and representation learning/skipgram.svg .. fr/Deep Learning/06 Embeddings and representation learning/skipgram.svg | |
| fr/Deep Learning/07 Initialization and vanishing gradients.md .. /dev/null | |
| @@ 1,117 0,0 @@ | |
| - | # 7. Initialisation et disparition du gradient |
| - | |
| - | Les réseaux profonds sont difficiles à entraîner parce que la rétropropagation multiplie une jacobienne par couche, si bien qu'un signal peut se contracter ou exploser géométriquement avec la profondeur. Ce module explique d'où vient cette instabilité, pourquoi une initialisation naïve des poids l'aggrave, et les deux remèdes qui rendent l'entraînement profond routinier : l'initialisation préservant la variance (Xavier et He) et l'écrêtage du gradient. |
| - | |
| - | **Objectifs** |
| - | - Écrire la rétropropagation comme un produit de jacobiennes par couche et voir quand elle disparaît ou explose. |
| - | - Relier l'effet à la saturation des activations vue au chapitre 3. |
| - | - Expliquer pourquoi les initialisations à zéro et mal mises à l'échelle échouent. |
| - | - Dériver la cible de variance que satisfont les initialisations Xavier et He. |
| - | - Appliquer l'écrêtage du gradient pour maîtriser les gradients explosifs. |
| - | - Choisir un initialiseur à partir de la fonction d'activation. |
| - | |
| - | ## 7.1 Pourquoi la profondeur est instable |
| - | |
| - | ### 7.1.1 Le produit des jacobiennes |
| - | |
| - | Rappelons la passe avant du chapitre 6 : la couche $l$ calcule $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$ et $a^{[l]} = g^{[l]}(z^{[l]})$, avec $a^{[0]} = x$ et $\hat{y} = a^{[L]}$. La rétropropagation renvoie le gradient de la perte de la sortie jusqu'à la couche $l$ par la règle de dérivation en chaîne. Le signal d'erreur $\delta^{[l]} = \partial L / \partial z^{[l]}$ obéit à la récurrence $\delta^{[l]} = (W^{[l+1]})^T \delta^{[l+1]} \odot g'^{[l]}(z^{[l]})$, donc en la déroulant depuis la couche supérieure $L$ jusqu'à la couche $l$ on obtient un produit : |
| - | |
| - | $$\boxed{ \frac{\partial L}{\partial z^{[l]}} = \left( \prod_{k=l+1}^{L} \operatorname{diag}\!\left(g'^{[k-1]}(z^{[k-1]})\right) (W^{[k]})^T \right) \frac{\partial L}{\partial z^{[L]}} }$$ |
| - | |
| - | Chaque facteur est une jacobienne de couche : une matrice de poids $W^{[k]}$ combinée à la matrice diagonale $\operatorname{diag}(g'(z))$ des pentes d'activation. Le gradient qui atteint la couche $l$ est ce produit entier agissant sur l'erreur de la couche supérieure. |
| - | |
| - | ### 7.1.2 Disparition et explosion |
| - | |
| - | Un produit de nombreux facteurs est régi par leur amplitude typique. Notons $\rho$ la taille typique (une norme spectrale) d'un facteur $W^{[k]} \odot \operatorname{diag}(g')$. À travers $L - l$ couches, le signal se met à l'échelle à peu près comme $\rho^{\,L-l}$ : |
| - | |
| - | $$\boxed{ \left\| \frac{\partial L}{\partial z^{[l]}} \right\| \;\approx\; \rho^{\,L-l} \left\| \frac{\partial L}{\partial z^{[L]}} \right\| }$$ |
| - | |
| - | Si $\rho < 1$ de façon constante, le gradient se contracte vers zéro à mesure qu'il remonte (la **disparition du gradient**), si bien que les premières couches se mettent à peine à jour et cessent en pratique d'apprendre. Si $\rho > 1$, il croît sans limite (l'**explosion du gradient**), si bien que les mises à jour dépassent leur cible et que la perte diverge vers `NaN`. Seul $\rho \approx 1$ maintient le signal vivant à travers la profondeur. |
| - | |
| - |  |
| - | |
| - | *Amplitude du gradient selon la profondeur : des poids mal mis à l'échelle la font disparaître ou exploser, tandis qu'une initialisation préservant la variance la maintient proche de un.* |
| - | |
| - | *Remarque :* le même produit s'applique en avant pour les activations elles-mêmes. Si les sorties des couches se contractent ou croissent géométriquement, le réseau ne peut représenter rien d'utile avant même qu'un gradient ne soit calculé, on veut donc que le signal avant et le gradient arrière soient tous deux proches de l'échelle unité. |
| - | |
| - | ### 7.1.3 Le lien avec la saturation |
| - | |
| - | Le facteur $g'(z)$ relie directement ce phénomène à la saturation vue au chapitre 3. La sigmoïde et $\tanh$ s'aplatissent pour de grands $|z|$, donc leurs dérivées y tombent proche de zéro. |
| - | |
| - | | Activation | $g'(z)$ | pente maximale | pente en saturation | |
| - | | --- | --- | --- | --- | |
| - | | sigmoïde | $g(z)(1-g(z))$ | $0.25$ | $\to 0$ | |
| - | | $\tanh$ | $1 - \tanh^2(z)$ | $1$ | $\to 0$ | |
| - | | ReLU | $1$ pour $z>0$, sinon $0$ | $1$ | $0$ du côté mort | |
| - | |
| - | *Remarque :* la pente de la sigmoïde ne dépasse jamais $0.25$, donc chaque couche multiplie le signal arrière par au plus un quart. Empilez dix couches sigmoïdes et le gradient est mis à l'échelle par au plus $0.25^{10} \approx 10^{-6}$ avant même de considérer un poids. C'est pourquoi les empilements profonds d'unités saturantes s'entraînent mal, et pourquoi ReLU (pente $1$ du côté actif) est devenue le choix par défaut. |
| - | |
| - | ## 7.2 Mauvaises initialisations |
| - | |
| - | ### 7.2.1 Tout à zéro |
| - | |
| - | Poser $W^{[l]} = 0$ (ou toute valeur qui rend identiques toutes les unités d'une couche) casse l'apprentissage par **symétrie**. Si deux unités d'une couche démarrent avec les mêmes poids et voient la même entrée, elles calculent la même activation et reçoivent le même gradient, donc elles se mettent à jour de façon identique et restent identiques pour toujours. La couche se comporte alors comme une seule unité, quelle que soit sa largeur. L'initialisation aléatoire existe précisément pour briser cette symétrie afin que les unités puissent se spécialiser. |
| - | |
| - | ### 7.2.2 Mauvaise échelle |
| - | |
| - | Même avec des poids aléatoires qui brisent la symétrie, la **variance** importe. Considérons une unité linéaire $z = \sum_{j=1}^{n_{\text{in}}} W_j a_j$ avec des poids et des entrées indépendants et de moyenne nulle. Sa variance est une somme de $n_{\text{in}}$ termes indépendants : |
| - | |
| - | $$\boxed{ \operatorname{Var}(z) = n_{\text{in}} \cdot \operatorname{Var}(W) \cdot \operatorname{Var}(a) }$$ |
| - | |
| - | Si $n_{\text{in}} \cdot \operatorname{Var}(W) > 1$, la variance du signal croît couche après couche et explose, et si elle est $< 1$, la variance décroît et disparaît. Pour maintenir $\operatorname{Var}(z) \approx \operatorname{Var}(a)$ d'une couche à l'autre, il faut $n_{\text{in}} \cdot \operatorname{Var}(W) \approx 1$, ce qui fixe la variance des poids à environ $1 / n_{\text{in}}$. Cette seule condition est le germe des deux initialiseurs ci-dessous. |
| - | |
| - | ## 7.3 Initialisation préservant la variance |
| - | |
| - | ### 7.3.1 Xavier / Glorot |
| - | |
| - | Glorot et Bengio équilibrent la passe avant ($\operatorname{Var}(W) = 1/n_{\text{in}}$) contre la passe arrière ($\operatorname{Var}(W) = 1/n_{\text{out}}$) en faisant la moyenne des deux, ce qui donne l'initialisation de Xavier : |
| - | |
| - | $$\boxed{ \operatorname{Var}(W^{[l]}) = \frac{2}{n_{\text{in}} + n_{\text{out}}} }$$ |
| - | |
| - | Ici $n_{\text{in}} = n_{l-1}$ est le fan-in et $n_{\text{out}} = n_l$ est le fan-out de la couche. Xavier est dérivée en supposant que l'activation est à peu près linéaire près de l'origine, elle convient donc aux activations **symétriques, de pente unité** comme $\tanh$ et la sigmoïde. |
| - | |
| - | ### 7.3.2 He |
| - | |
| - | ReLU annule en moyenne la moitié de ses entrées, donc elle divise par deux la variance de ce qui passe. L'initialisation de He compense par un facteur deux : |
| - | |
| - | $$\boxed{ \operatorname{Var}(W^{[l]}) = \frac{2}{n_{\text{in}}} }$$ |
| - | |
| - | C'est la bonne cible pour **ReLU et ses variantes** (leaky ReLU, ELU, GELU). En pratique, on tire les poids d'une gaussienne de cette variance, ou d'une distribution uniforme d'étendue correspondante, et on fixe le biais $b^{[l]}$ à zéro. |
| - | |
| - | *Remarque :* c'est le biais qui démarre à zéro, pas les poids. Un biais nul ne crée pas de symétrie (les poids diffèrent déjà), et il maintient la pré-activation initiale centrée de sorte que l'activation démarre dans sa région réactive plutôt qu'en saturation. |
| - | |
| - | ### 7.3.3 Le mécanisme |
| - | |
| - |  |
| - | |
| - | *L'initialisation préservant la variance maintient le signal et le gradient proches de l'échelle unité à travers la profondeur.* |
| - | |
| - | Choisir la variance est un correctif unique au début de l'entraînement. Il positionne le réseau de sorte que le produit de jacobiennes de la section 7.1 ait des facteurs proches de $1$, mais rien ne l'y maintient à mesure que les poids bougent pendant l'entraînement. C'est ce que traite le module suivant. |
| - | |
| - | ## 7.4 Gradients explosifs et écrêtage |
| - | |
| - | Une bonne initialisation maîtrise la disparition des gradients et réduit fortement les explosions, mais des explosions peuvent tout de même apparaître pendant l'entraînement, en particulier dans les réseaux récurrents où la même matrice de poids est réutilisée à chaque pas de temps. Le remède standard est l'**écrêtage du gradient** : remettre à l'échelle le vecteur gradient entier $g$ pour que sa norme ne dépasse jamais un seuil $\tau$. |
| - | |
| - | $$\boxed{ g \leftarrow g \cdot \min\!\left(1, \frac{\tau}{\lVert g \rVert}\right) }$$ |
| - | |
| - | Lorsque $\lVert g \rVert \le \tau$, le facteur vaut $1$ et le gradient n'est pas touché. Lorsque $\lVert g \rVert > \tau$, le gradient est ramené à une norme exactement égale à $\tau$ tout en conservant sa direction, si bien qu'un unique pas énorme ne peut pas faire exploser les poids. |
| - | |
| - | *Remarque :* l'écrêtage par la norme globale (remettant à l'échelle le vecteur entier ensemble) préserve la direction de la mise à jour, alors qu'écrêter chaque coordonnée indépendamment vers $[-\tau, \tau]$ peut fausser la direction. L'écrêtage par norme globale est le choix par défaut habituel. |
| - | |
| - | ## 7.5 Choisir un initialiseur |
| - | |
| - | Accordez l'initialiseur à l'activation de la couche qu'il alimente. |
| - | |
| - | | Activation | Initialiseur recommandé | Variance des poids | |
| - | | --- | --- | --- | |
| - | | ReLU, leaky ReLU, ELU, GELU | He | $2 / n_{\text{in}}$ | |
| - | | $\tanh$ | Xavier / Glorot | $2 / (n_{\text{in}} + n_{\text{out}})$ | |
| - | | sigmoïde | Xavier / Glorot | $2 / (n_{\text{in}} + n_{\text{out}})$ | |
| - | | softmax / sortie linéaire | Xavier / Glorot | $2 / (n_{\text{in}} + n_{\text{out}})$ | |
| - | |
| - | *Remarque :* l'initialisation et l'écrêtage ne gèrent le signal qu'aux extrémités de l'entraînement et lors des grands pas. Deux remèdes structurels le gardent maîtrisé tout du long : la **normalisation** recentre et remet à l'échelle les activations à chaque couche, et les **connexions résiduelles** ajoutent un raccourci qui laisse le gradient contourner entièrement le produit de jacobiennes. |
| - | |
| - | *Une bonne initialisation maintient le signal bien mis à l'échelle au pas zéro, mais les statistiques dérivent au fil de l'entraînement. Le module suivant les garde sous contrôle à chaque pas grâce à la normalisation.* |
| - | |
| - | --- |
| - | Suivant : [Normalisation](/fr/Deep%20Learning/08%20Normalization) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| fr/Deep Learning/07 Initialization and vanishing gradients/init-reasoning.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1080 190" width="1080" height="190" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="1080" height="190" fill="#ffffff"/><text x="540.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">Why variance-preserving initialization stabilizes depth</text><rect x="24.0" y="78.0" width="176.0" height="66.0" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="112.0" y="107.8" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">random W break</text><text x="112.0" y="123.1" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">symmetry</text><rect x="224.0" y="78.0" width="176.0" height="66.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="312.0" y="107.8" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">set Var(W) near</text><text x="312.0" y="123.1" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">1/n<tspan baseline-shift="sub" font-size="9px">in</tspan></text><rect x="424.0" y="78.0" width="176.0" height="66.0" rx="8" fill="#e4f5f4" stroke="#2a9d9a" stroke-width="1.6"/><text x="512.0" y="107.8" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">signal Var(z) near</text><text x="512.0" y="123.1" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">Var(a)</text><rect x="624.0" y="78.0" width="176.0" height="66.0" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="712.0" y="107.8" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">gradient factor ρ</text><text x="712.0" y="123.1" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">near 1</text><rect x="824.0" y="78.0" width="176.0" height="66.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="912.0" y="107.8" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">deep network trains</text><text x="912.0" y="123.1" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">stably</text><line x1="200.0" y1="111.0" x2="224.0" y2="111.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="400.0" y1="111.0" x2="424.0" y2="111.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="600.0" y1="111.0" x2="624.0" y2="111.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="800.0" y1="111.0" x2="824.0" y2="111.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/></svg> |
| \ | No newline at end of file |
| fr/Deep Learning/12 Recurrent networks.md .. fr/Deep Learning/07 Recurrent networks.md | |
| @@ 1,21 1,23 @@ | |
| - | # 12. Réseaux récurrents |
| + | # 7. Réseaux récurrents |
| Les réseaux à propagation avant et les réseaux convolutifs transforment une entrée de taille fixe en une sortie en une seule passe, mais de nombreux problèmes se présentent sous forme de séquences dont la longueur varie et dont l'ordre importe (texte, audio, séries temporelles). Un réseau de neurones récurrent (RNN) traite une séquence pas à pas et propage un état caché vers l'avant, de sorte que les entrées passées influencent la sortie courante. Ce module introduit la récurrence, la cellule RNN de base, la façon dont elle est entraînée par rétropropagation à travers le temps, et pourquoi les gradients à longue portée ont tendance à s'évanouir ou à exploser. | |
| - | **Objectifs** |
| - | - Expliquer pourquoi les données séquentielles ont besoin de mémoire et de partage de poids entre les pas de temps. |
| - | - Écrire la récurrence du RNN de base pour l'état caché et la sortie. |
| - | - Dérouler une cellule récurrente dans le temps et en lire les paramètres partagés. |
| - | - Dériver comment la rétropropagation à travers le temps (BPTT) accumule le gradient sur tous les pas. |
| - | - Diagnostiquer l'évanouissement et l'explosion des gradients à partir du produit des jacobiennes au fil du temps. |
| + | ## 7.1 Pourquoi pas un MLP ou un CNN ? |
| - | ## 12.1 Données séquentielles et mémoire |
| + | Une séquence est une liste ordonnée d'entrées $x_1, x_2, \dots, x_T$ : les mots d'une critique, les échantillons d'un extrait audio, les valeurs quotidiennes d'une série temporelle. Deux choses la rendent inconfortable pour les réseaux que nous avons déjà. La longueur $T$ change d'un exemple à l'autre, et l'ordre porte le sens : « pas bon, en fait excellent » et « pas excellent, en fait bon » contiennent exactement les mêmes mots. |
| - | Une séquence est une liste ordonnée d'entrées $x_1, x_2, \dots, x_T$, où $T$ peut différer d'un exemple à l'autre. Un réseau à propagation avant du type vu dans les leçons précédentes attend un unique vecteur de taille fixe $a^{[0]} = x$, de sorte qu'il n'a aucun moyen naturel de consommer une entrée de longueur variable ni de se souvenir de ce qui précédait l'élément courant. |
| + | Essayons les outils déjà sur la table : |
| - | Deux idées corrigent cela. Premièrement, le réseau conserve un **état caché** (ou mémoire) $h_t$ qui résume tout ce qui est pertinent jusqu'au pas $t$. Deuxièmement, le réseau **partage** un unique jeu de paramètres à chaque pas, de sorte que la même transformation s'applique que la séquence ait 5 éléments ou 500. Le partage garde le nombre de paramètres indépendant de $T$ et permet à un motif appris à une position de généraliser à n'importe quelle autre. |
| + | - **Un MLP** attend un unique vecteur de taille fixe $a^{[0]} = x$, une séquence plus longue n'entre donc tout simplement pas dans sa couche d'entrée. Pire, chaque position reçoit ses propres poids privés : un motif appris sur le 2e mot n'apprend rien sur le même motif au 9e, le problème de la couche dense de la leçon 5 à nouveau, dans le temps plutôt que dans l'espace. |
| + | - **Un CNN 1D** corrige le partage : un noyau glisse le long de la séquence, les mêmes poids à chaque position. Mais sa fenêtre fait $k$ pas de large, deux entrées ne se rencontrent donc que si elles tombent dans le même champ réceptif. La fin d'une longue séquence ne peut pas voir son début sans empiler beaucoup de couches, et la portée reste fixée à la construction. |
| - | *Remarque :* le partage de poids dans le temps est l'analogue séquentiel du partage de poids dans l'espace dans un réseau convolutif. Tous deux encodent l'a priori qu'une même caractéristique peut apparaître n'importe où. |
| + | Ce que les séquences demandent, ce sont trois choses à la fois : accepter n'importe quelle longueur, appliquer les mêmes poids à chaque pas, et porter une mémoire de ce qui a été vu jusque-là, aussi loin que ce soit. |
| + | |
| + |  |
| + | |
| + | *La même séquence de six pas, deux fois. Le MLP a des cases fixes et des poids privés par position. La convolution 1D partage ses poids, mais ne relie que les entrées qui tombent dans sa fenêtre.* |
| + | |
| + | Ce qui satisfera les trois exigences à la fois sera le sujet de cette leçon. Avant de le construire, notons les formes que prennent les tâches séquentielles : |
| | Configuration | Entrée | Sortie | Exemple | | |
| | --- | --- | --- | --- | | |
| @@ 24,9 26,13 @@ | |
| | Plusieurs vers plusieurs (seq2seq) | séquence | séquence, autre longueur | traduction automatique | | |
| | Un vers plusieurs | vecteur unique | séquence | légendage d'image | | |
| - | ## 12.2 La cellule RNN de base |
| + | ## 7.2 La cellule RNN de base |
| + | |
| + | Le réseau récurrent satisfait les trois exigences d'un seul geste. Il conserve un **état caché** (ou mémoire) $h_t$ qui résume tout ce qui est pertinent jusqu'au pas $t$, et il **partage** un unique jeu de paramètres à chaque pas, de sorte que la même transformation s'applique que la séquence ait 5 éléments ou 500. Le partage garde le nombre de paramètres indépendant de $T$ et permet à un motif appris à une position de généraliser à n'importe quelle autre. |
| + | |
| + | *Remarque :* le partage de poids dans le temps est l'analogue séquentiel du partage de poids dans l'espace dans un réseau convolutif. Tous deux encodent l'a priori qu'une même caractéristique peut apparaître n'importe où. |
| - | ### 12.2.1 Récurrence |
| + | ### 7.2.1 Récurrence |
| Au pas $t$ la cellule lit l'entrée courante $x_t$ et l'état caché précédent $h_{t-1}$, puis produit un nouvel état caché via une activation $g$ (habituellement $\tanh$) : | |
| @@ 40,7 46,7 @@ | |
| *Remarque :* cela conserve la convention de biais explicite de tout le cours de Deep Learning. Le biais $b_h$ est un terme additif séparé, jamais incorporé dans les matrices de poids comme le cours de Machine Learning incorporait l'ordonnée à l'origine dans $w^T x$ avec $x_0 = 1$. | |
| - | ### 12.2.2 Poids partagés |
| + | ### 7.2.2 Poids partagés |
| Le point crucial est que $W_{hh}$, $W_{xh}$, $W_{hy}$, $b_h$ et $b_y$ ne dépendent **pas** de $t$. Les cinq mêmes paramètres sont réutilisés à chaque pas : | |
| @@ 48,17 54,17 @@ | |
| Un RNN n'est donc pas un réseau très profond avec des couches distinctes, c'est une petite cellule appliquée de façon répétée, réinjectant sa propre sortie en entrée. | |
| - | ## 12.3 Déroulement dans le temps |
| + | ## 7.3 Déroulement dans le temps |
| Comme la même cellule est réutilisée, on peut **dérouler** la récurrence en une chaîne : on dessine une copie de la cellule par pas de temps et on connecte l'état caché de chaque copie à la suivante. La vue déroulée est un graphe à propagation avant ordinaire (à poids liés), ce qui est exactement ce qui rend possible le calcul du gradient. | |
| - |  |
| + |  |
| *Déroulé dans le temps, un réseau récurrent réutilise les mêmes poids à chaque pas et propage l'état caché vers l'avant.* | |
| *Remarque :* les flèches horizontales entre états cachés sont le seul chemin par lequel l'information du passé atteint le présent. Chacune d'elles multiplie par la même matrice $W_{hh}$, ce qui est à la fois la source de la puissance du modèle et de sa difficulté d'entraînement. | |
| - | ## 12.4 Rétropropagation à travers le temps |
| + | ## 7.4 Rétropropagation à travers le temps |
| L'entraînement minimise un coût total qui somme la perte par pas sur la séquence. Avec une perte par pas $L_t$ comparant $\hat{y}_t$ à la cible $y_t$, le coût pour une séquence est : | |
| @@ 74,7 80,7 @@ | |
| *Remarque :* en pratique la somme sur $k$ est tronquée après une fenêtre fixe, ce qu'on appelle la BPTT tronquée. Elle borne la mémoire et le calcul par mise à jour au prix d'ignorer les dépendances plus longues que la fenêtre. | |
| - | ## 12.5 Évanouissement et explosion des gradients |
| + | ## 7.5 Évanouissement et explosion des gradients |
| Le produit interne $\prod_{i=k+1}^{t} \frac{\partial h_i}{\partial h_{i-1}}$ est ce qui transporte l'information de gradient du pas $t$ jusqu'au pas $k$. À partir de la récurrence $h_i = g(W_{hh} h_{i-1} + W_{xh} x_i + b_h)$, chaque facteur vaut : | |
| @@ 86,7 92,7 @@ | |
| Ce produit de $t - k$ facteurs quasi identiques se comporte à peu près comme une matrice élevée à la puissance $t - k$. Si la magnitude pertinente (informellement, la plus grande valeur singulière de $\operatorname{diag}(g'(z_i)) W_{hh}$) est inférieure à $1$, le produit rétrécit géométriquement vers zéro à mesure que l'écart grandit, de sorte que les gradients lointains **s'évanouissent**. Si elle est supérieure à $1$, le produit explose et les gradients **explosent**. | |
| - |  |
| + |  |
| *Au fil de nombreux pas de temps, le gradient rétrécit ou croît géométriquement, de sorte que les dépendances à longue portée sont difficiles à apprendre pour un RNN simple.* | |
| @@ 103,4 109,4 @@ | |
| *La prochaine leçon introduit le LSTM et le GRU, des architectures à portes qui transportent un état de cellule à travers des mises à jour additives afin que les gradients puissent traverser de longues portées sans s'évanouir.* | |
| --- | |
| - | Suivant : [LSTM et GRU](/fr/Deep%20Learning/13%20LSTM%20and%20GRU) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| + | Suivant : [LSTM et GRU](/fr/Deep%20Learning/08%20LSTM%20and%20GRU) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| fr/Deep Learning/12 Recurrent networks/bptt-decay.png .. fr/Deep Learning/07 Recurrent networks/bptt-decay.png | |
| /dev/null .. fr/Deep Learning/07 Recurrent networks/rnn-motivation.svg | |
| @@ 0,0 1,37 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 280" width="900" height="280" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs> |
| + | <marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | </defs> |
| + | <rect width="900" height="280" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">One sequence, two networks that fall short</text> |
| + | <text x="40" y="62" font-size="13" font-weight="600" fill="#1f2933">an MLP: flatten into fixed slots</text> |
| + | <rect x="40" y="80" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="50" y="94" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">1</tspan></text> |
| + | <rect x="66" y="80" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="76" y="94" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">2</tspan></text> |
| + | <rect x="92" y="80" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="102" y="94" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">3</tspan></text> |
| + | <rect x="118" y="80" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="128" y="94" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">4</tspan></text> |
| + | <rect x="144" y="80" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="154" y="94" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">5</tspan></text> |
| + | <rect x="170" y="80" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="180" y="94" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">6</tspan></text> |
| + | <rect x="202" y="80" width="20" height="20" fill="#eef1f4" stroke="#d1495b" stroke-width="1.3" stroke-dasharray="4 3"/><text x="212" y="94" font-size="10" fill="#d1495b" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">7</tspan></text> |
| + | <text x="212" y="118" font-size="9" fill="#d1495b" text-anchor="middle">does not fit</text> |
| + | <text x="115" y="118" font-size="10" fill="#5b6b7b" text-anchor="middle">fixed input width</text> |
| + | <line x1="242" y1="90" x2="272" y2="90" stroke="#1f2933" stroke-width="1.4" marker-end="url(#arrow)"/> |
| + | <rect x="282" y="74" width="92" height="32" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/> |
| + | <text x="328" y="94" font-size="10" fill="#1f2933" text-anchor="middle">dense layer</text> |
| + | <text x="328" y="122" font-size="10" fill="#5b6b7b" text-anchor="middle">one private weight per slot</text> |
| + | <text x="560" y="84" font-size="12" font-weight="600" fill="#d1495b">✗</text><text x="578" y="84" font-size="11" fill="#1f2933">fixed length only</text> |
| + | <text x="560" y="104" font-size="12" font-weight="600" fill="#d1495b">✗</text><text x="578" y="104" font-size="11" fill="#1f2933">nothing transfers across positions</text> |
| + | <text x="40" y="168" font-size="13" font-weight="600" fill="#1f2933">a 1D CNN: slide a shared window</text> |
| + | <rect x="40" y="186" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="50" y="200" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">1</tspan></text> |
| + | <rect x="66" y="186" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="76" y="200" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">2</tspan></text> |
| + | <rect x="92" y="186" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="102" y="200" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">3</tspan></text> |
| + | <rect x="118" y="186" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="128" y="200" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">4</tspan></text> |
| + | <rect x="144" y="186" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="154" y="200" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">5</tspan></text> |
| + | <rect x="170" y="186" width="20" height="20" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.3"/><text x="180" y="200" font-size="10" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="7px">6</tspan></text> |
| + | <rect x="38" y="184" width="76" height="24" fill="#3b6fb6" fill-opacity="0.15" stroke="#3b6fb6" stroke-width="1.8"/> |
| + | <rect x="116" y="184" width="76" height="24" fill="none" stroke="#3b6fb6" stroke-width="1.2" stroke-dasharray="4 3" opacity="0.7"/> |
| + | <path d="M180,210 C158,240 72,240 50,210" fill="none" stroke="#d1495b" stroke-width="1.2" stroke-dasharray="4 3"/> |
| + | <text x="115" y="238" font-size="11" font-weight="600" fill="#d1495b" text-anchor="middle">?</text> |
| + | <text x="118" y="258" font-size="10" fill="#5b6b7b" text-anchor="middle">window of 3: distant steps never meet</text> |
| + | <text x="560" y="190" font-size="12" font-weight="600" fill="#38a05a">✓</text><text x="578" y="190" font-size="11" fill="#1f2933">weights shared across positions</text> |
| + | <text x="560" y="210" font-size="12" font-weight="600" fill="#d1495b">✗</text><text x="578" y="210" font-size="11" fill="#1f2933">reach limited to the window</text> |
| + | </svg> |
| \ | No newline at end of file |
| fr/Deep Learning/12 Recurrent networks/rnn-unrolled.svg .. fr/Deep Learning/07 Recurrent networks/rnn-unrolled.svg | |
| fr/Deep Learning/13 LSTM and GRU.md .. fr/Deep Learning/08 LSTM and GRU.md | |
| @@ 1,15 1,8 @@ | |
| - | # 13. LSTM et GRU |
| + | # 8. LSTM et GRU |
| Un réseau récurrent simple peine à transporter l'information sur de nombreux pas de temps, car la multiplication répétée par la même matrice de poids fait s'évanouir ou exploser les gradients. Les cellules récurrentes à portes corrigent cela en ajoutant un état qui circule dans le temps par des mises à jour essentiellement additives, contrôlées par des portes apprises. Ce module construit la cellule à mémoire à long et court terme (LSTM) et l'unité récurrente à porte (GRU), plus légère, et compare les situations où privilégier l'une ou l'autre. | |
| - | **Objectifs** |
| - | - Expliquer pourquoi un état de cellule à portes préserve le flux de gradient à longue portée (le carrousel d'erreur constant). |
| - | - Écrire les trois portes du LSTM comme des sigmoïdes d'une application affine de l'entrée concaténée. |
| - | - Dériver le candidat, la mise à jour de la cellule et l'état caché du LSTM. |
| - | - Écrire les portes de réinitialisation et de mise à jour du GRU ainsi que son état caché interpolé. |
| - | - Comparer LSTM et GRU sur le nombre de portes, l'état de cellule, le nombre de paramètres et l'usage typique. |
| - | |
| - | ## 13.1 L'idée des portes |
| + | ## 8.1 L'idée des portes |
| Une couche récurrente classique met à jour son état caché par $h_t = g(W_h h_{t-1} + W_x x_t + b)$. La rétropropagation de la perte à travers $T$ pas multiplie de nombreuses jacobiennes de cette application entre elles, de sorte que l'amplitude du gradient croît à peu près comme la puissance $T$-ième du rayon spectral du poids récurrent. En dessous de un elle s'évanouit, au-dessus de un elle explose, et dans les deux cas le réseau ne peut pas apprendre des dépendances qui s'étalent sur de nombreux pas. | |
| @@ 17,11 10,11 @@ | |
| *Remarque :* le mot clé est additif. La récurrence multiplicative compose un facteur à chaque pas, tandis qu'un chemin additif laisse l'état persister par défaut et ne changer que lorsqu'une porte s'ouvre. | |
| - | ## 13.2 La cellule LSTM |
| + | ## 8.2 La cellule LSTM |
| Tout au long, $[h_{t-1}, x_t]$ désigne la concaténation de l'état caché précédent et de l'entrée courante en un seul vecteur. Chaque porte est un vecteur dans $(0, 1)$ produit par une sigmoïde $\sigma$ appliquée à une application affine de cette concaténation, de sorte qu'une valeur de porte proche de $1$ laisse passer l'information et une valeur proche de $0$ la bloque. | |
| - | ### 13.2.1 Les trois portes |
| + | ### 8.2.1 Les trois portes |
| La porte d'**oubli** $f_t$ décide de la part de l'ancien état de cellule à conserver, la porte d'**entrée** $i_t$ décide de la part du nouveau candidat à écrire, et la porte de **sortie** $o_t$ décide de la part de l'état de cellule à exposer comme état caché : | |
| @@ 29,7 22,11 @@ | |
| *Remarque :* les portes partagent la même forme fonctionnelle et ne diffèrent que par leurs paramètres appris. Le biais est explicite ici, exactement comme pour les couches à propagation avant des modules précédents, et n'est jamais intégré à la matrice de poids. | |
| - | ### 13.2.2 Candidat et mise à jour de la cellule |
| + |  |
| + | |
| + | *La cellule jusqu'ici : trois portes sigmoïdes lisant $[h_{t-1}, x_t]$ sur le rail d'entrée. Les chemins qu'elles contrôleront sont encore grisés.* |
| + | |
| + | ### 8.2.2 Candidat et mise à jour de la cellule |
| Une couche $\tanh$ propose une mise à jour **candidate** $\tilde{c}_t$, le nouveau contenu que la cellule pourrait stocker : | |
| @@ 41,7 38,11 @@ | |
| Lorsque $f_t \approx 1$ et $i_t \approx 0$, la cellule copie simplement $c_{t-1}$, ce qui constitue le carrousel d'erreur constant : $\partial c_t / \partial c_{t-1} \approx \mathrm{diag}(f_t)$, de sorte que les gradients passent presque sans atténuation. | |
| - | ### 13.2.3 État caché |
| + |  |
| + | |
| + | *L'étape 2 allume le carrousel : le candidat propose un contenu, la multiplication d'oubli et l'addition d'écriture modifient l'état de cellule pendant qu'il traverse le haut. Seul le côté sortie reste grisé.* |
| + | |
| + | ### 8.2.3 État caché |
| L'état caché est l'état de cellule écrasé, contrôlé par la porte de sortie : | |
| @@ 49,21 50,27 @@ | |
| *Remarque :* l'état de cellule $c_t$ est la mémoire à long terme qui circule le long du carrousel, tandis que l'état caché $h_t$ est la vue filtrée exposée à la couche suivante et à la sortie de ce pas. C'est le fait de les garder distincts qui différencie le LSTM du GRU ci-dessous. | |
| - | ## 13.3 Le GRU |
| + | La cellule entière, assemblée : |
| - | Le GRU fusionne l'état de cellule et l'état caché en un seul $h_t$ et n'utilise que deux portes, ce qui lui donne moins de paramètres tout en conservant l'avantage de la mise à jour additive. |
| + |  |
| + | |
| + | *Un pas de LSTM. Le long du haut court le carrousel : l'état de cellule traverse la cellule touché seulement par la multiplication d'oubli et l'addition d'écriture, jamais par une multiplication matricielle. En dessous, les quatre blocs lisent $[h_{t-1}, x_t]$ et décident quoi oublier ($f_t$), quoi écrire ($i_t \odot \tilde{c}_t$) et quoi exposer ($h_t = o_t \odot \tanh(c_t)$).* |
| - |  |
| + | ## 8.3 Le GRU |
| - | *Le GRU fusionne l'état de cellule et l'état caché et n'utilise qu'une porte de réinitialisation et une porte de mise à jour.* |
| + | Le GRU fusionne l'état de cellule et l'état caché en un seul $h_t$ et n'utilise que deux portes, ce qui lui donne moins de paramètres tout en conservant l'avantage de la mise à jour additive. |
| - | ### 13.3.1 Portes de réinitialisation et de mise à jour |
| + | ### 8.3.1 Portes de réinitialisation et de mise à jour |
| La porte de **réinitialisation** $r_t$ contrôle la part de l'état passé qui alimente le candidat, et la porte de **mise à jour** $z_t$ contrôle la part de l'état à rafraîchir : | |
| $$\boxed{ r_t = \sigma\!\left(W_r\,[h_{t-1}, x_t] + b_r\right), \quad z_t = \sigma\!\left(W_z\,[h_{t-1}, x_t] + b_z\right) }$$ | |
| - | ### 13.3.2 Candidat et état interpolé |
| + |  |
| + | |
| + | *La cellule jusqu'ici : deux portes seulement sur le rail d'entrée, le chemin d'état encore grisé.* |
| + | |
| + | ### 8.3.2 Candidat et état interpolé |
| Le candidat utilise une version de l'état caché précédent contrôlée par la porte de réinitialisation, et le nouvel état est une interpolation contrôlée par une porte entre l'ancien état et le candidat : | |
| @@ 71,7 78,11 @@ | |
| *Remarque :* la forme interpolée lie les fractions de conservation et d'écriture par une seule porte : quel que soit le poids $z_t$ attribué au candidat, $1 - z_t$ est laissé au passé. Le LSTM fixe indépendamment sa fraction de conservation $f_t$ et sa fraction d'écriture $i_t$, ce qui représente une porte de plus et une matrice de plus. | |
| - | ## 13.4 LSTM contre GRU |
| + |  |
| + | |
| + | *Un pas de GRU. La porte de réinitialisation $r_t$ filtre la part du passé qui alimente le candidat, puis la porte de mise à jour partage l'état entre conservation ($1 - z_t$) et rafraîchissement ($z_t$). Une seule ligne d'état, deux portes, le même chemin additif.* |
| + | |
| + | ## 8.4 LSTM contre GRU |
| Les deux cellules résolvent le problème de l'évanouissement du gradient grâce à un chemin d'état additif. Elles diffèrent par le nombre de portes qui portent ce chemin et par le fait que la mémoire à long terme soit ou non maintenue distincte de l'état exposé. | |
| @@ 85,17 96,7 @@ | |
| *Remarque :* en pratique, les deux atteignent souvent une précision comparable. Le GRU s'entraîne plus vite et généralise bien sur des ensembles de données plus petits, tandis que la capacité supplémentaire du LSTM peut aider sur des séquences très longues. Traitez ce choix comme un hyperparamètre à régler plutôt que comme une règle établie. | |
| - | ## 13.5 Anatomie d'une cellule à portes |
| - | |
| - | Le diagramme retrace un pas de LSTM : l'état de cellule précédent entre sur le chemin additif, les portes modulent ce qui est oublié, écrit et exposé, et les sorties alimentent le pas suivant. |
| - | |
| - |  |
| - | |
| - | *La cellule LSTM transporte un état de cellule le long du haut, modifié par une multiplication d'oubli et une addition d'entrée, avec des portes sigmoïdes contrôlant le flux.* |
| - | |
| - | *Remarque :* le chemin horizontal de l'état de cellule précédent vers le nouvel état de cellule est le carrousel, et il ne porte aucune multiplication matricielle complète, seulement les produits de portes élément par élément. |
| - | |
| *Les portes permettent à un état récurrent de persister sur de longues portées, mais elles lisent toujours un pas à la fois. La prochaine partie laisse chaque position s'intéresser directement à toutes les autres, supprimant le goulot d'étranglement séquentiel.* | |
| --- | |
| - | Suivant : [Attention](/fr/Deep%20Learning/14%20Attention) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| + | Suivant : [Attention](/fr/Deep%20Learning/09%20Attention) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| /dev/null .. fr/Deep Learning/08 LSTM and GRU/gru-cell.svg | |
| @@ 0,0 1,24 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 400" width="900" height="400" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker></defs> |
| + | <rect width="900" height="400" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">Inside the GRU cell</text> |
| + | <rect x="50" y="46" width="720" height="290" rx="14" fill="#f9fafb" stroke="#c7d0d9" stroke-width="1.2"/> |
| + | <line x1="20" y1="95" x2="247" y2="95" stroke="#1f2933" stroke-width="2.2"/> |
| + | <line x1="273" y1="95" x2="547" y2="95" stroke="#1f2933" stroke-width="2.2"/> |
| + | <line x1="573" y1="95" x2="795" y2="95" stroke="#1f2933" stroke-width="2.2"/> |
| + | <line x1="780" y1="95" x2="800" y2="95" stroke="#1f2933" stroke-width="1.8" marker-end="url(#arrow)"/> |
| + | <text x="28" y="80" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text><text x="822" y="99" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <circle cx="260" cy="95" r="13" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="260" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><text x="260" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">keep</text> |
| + | <circle cx="560" cy="95" r="13" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="560" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">+</text><text x="560" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">update</text> |
| + | <line x1="120" y1="95" x2="120" y2="165" stroke="#1f2933" stroke-width="1.5"/><line x1="120" y1="165" x2="155" y2="165" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><circle cx="170" cy="165" r="12" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="170" y="170" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text> |
| + | <line x1="183" y1="165" x2="300" y2="165" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="242" y="156" font-size="9" fill="#5b6b7b" text-anchor="middle">r<tspan baseline-shift="sub" font-size="9px">t</tspan> ⊙ h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text> |
| + | <rect x="300" y="148" width="96" height="34" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="348.0" y="163" font-size="12" fill="#1f2933" text-anchor="middle">tanh</text><text x="348.0" y="176" font-size="9" fill="#5b6b7b" text-anchor="middle">candidate</text><line x1="396" y1="165" x2="465" y2="165" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="430" y="156" font-size="11" fill="#1f2933" text-anchor="middle">h̃<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <circle cx="480" cy="165" r="12" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="480" y="170" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><line x1="489" y1="155" x2="551" y2="105" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <path d="M440,231 L260,231 L260,110" fill="none" stroke="#1f2933" stroke-width="1.5"/><line x1="260" y1="120" x2="260" y2="110" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="272" y="195" font-size="11" fill="#1f2933" text-anchor="start">1 − z<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="130" y="215" width="80" height="32" rx="8" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="170.0" y="230" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">r</tspan></text><text x="170.0" y="243" font-size="9" fill="#5b6b7b" text-anchor="middle">reset gate</text><line x1="170" y1="215" x2="170" y2="180" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="182" y="200" font-size="11" fill="#1f2933" text-anchor="start">r<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="440" y="215" width="80" height="32" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="480.0" y="230" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">z</tspan></text><text x="480.0" y="243" font-size="9" fill="#5b6b7b" text-anchor="middle">update gate</text><line x1="480" y1="215" x2="480" y2="180" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="492" y="202" font-size="11" fill="#1f2933" text-anchor="start">z<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <line x1="20" y1="288" x2="620" y2="288" stroke="#5b6b7b" stroke-width="1.6"/><line x1="170" y1="288" x2="170" y2="251" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="480" y1="288" x2="480" y2="251" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><text x="90" y="279" font-size="11" fill="#1f2933" text-anchor="middle">[h<tspan baseline-shift="sub" font-size="9px">t-1</tspan>, x<tspan baseline-shift="sub" font-size="9px">t</tspan>]</text> |
| + | <line x1="348" y1="288" x2="348" y2="186" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/> |
| + | <line x1="560" y1="356" x2="560" y2="292" stroke="#1f2933" stroke-width="1.6" marker-end="url(#arrow)"/><text x="560" y="372" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">x<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <text x="450" y="392" font-size="11" fill="#5b6b7b" text-anchor="middle">σ: gate in (0, 1) · ⊙: elementwise product · +: elementwise add</text> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. fr/Deep Learning/08 LSTM and GRU/gru-step1.svg | |
| @@ 0,0 1,24 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 400" width="900" height="400" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker></defs> |
| + | <rect width="900" height="400" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">The GRU cell, step 1: the gates</text> |
| + | <rect x="50" y="46" width="720" height="290" rx="14" fill="#f9fafb" stroke="#c7d0d9" stroke-width="1.2"/> |
| + | <g opacity="0.22"><line x1="20" y1="95" x2="247" y2="95" stroke="#1f2933" stroke-width="2.2"/></g> |
| + | <g opacity="0.22"><line x1="273" y1="95" x2="547" y2="95" stroke="#1f2933" stroke-width="2.2"/></g> |
| + | <g opacity="0.22"><line x1="573" y1="95" x2="795" y2="95" stroke="#1f2933" stroke-width="2.2"/></g> |
| + | <g opacity="0.22"><line x1="780" y1="95" x2="800" y2="95" stroke="#1f2933" stroke-width="1.8" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><text x="28" y="80" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text><text x="822" y="99" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t</tspan></text></g> |
| + | <g opacity="0.22"><circle cx="260" cy="95" r="13" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="260" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><text x="260" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">keep</text></g> |
| + | <g opacity="0.22"><circle cx="560" cy="95" r="13" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="560" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">+</text><text x="560" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">update</text></g> |
| + | <g opacity="0.22"><line x1="120" y1="95" x2="120" y2="165" stroke="#1f2933" stroke-width="1.5"/><line x1="120" y1="165" x2="155" y2="165" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><circle cx="170" cy="165" r="12" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="170" y="170" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text></g> |
| + | <g opacity="0.22"><line x1="183" y1="165" x2="300" y2="165" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="242" y="156" font-size="9" fill="#5b6b7b" text-anchor="middle">r<tspan baseline-shift="sub" font-size="9px">t</tspan> ⊙ h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text></g> |
| + | <g opacity="0.22"><rect x="300" y="148" width="96" height="34" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="348.0" y="163" font-size="12" fill="#1f2933" text-anchor="middle">tanh</text><text x="348.0" y="176" font-size="9" fill="#5b6b7b" text-anchor="middle">candidate</text><line x1="396" y1="165" x2="465" y2="165" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="430" y="156" font-size="11" fill="#1f2933" text-anchor="middle">h̃<tspan baseline-shift="sub" font-size="9px">t</tspan></text></g> |
| + | <g opacity="0.22"><circle cx="480" cy="165" r="12" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="480" y="170" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><line x1="489" y1="155" x2="551" y2="105" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><path d="M440,231 L260,231 L260,110" fill="none" stroke="#1f2933" stroke-width="1.5"/><line x1="260" y1="120" x2="260" y2="110" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="272" y="195" font-size="11" fill="#1f2933" text-anchor="start">1 − z<tspan baseline-shift="sub" font-size="9px">t</tspan></text></g> |
| + | <rect x="130" y="215" width="80" height="32" rx="8" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="170.0" y="230" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">r</tspan></text><text x="170.0" y="243" font-size="9" fill="#5b6b7b" text-anchor="middle">reset gate</text><line x1="170" y1="215" x2="170" y2="180" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="182" y="200" font-size="11" fill="#1f2933" text-anchor="start">r<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="440" y="215" width="80" height="32" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="480.0" y="230" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">z</tspan></text><text x="480.0" y="243" font-size="9" fill="#5b6b7b" text-anchor="middle">update gate</text><line x1="480" y1="215" x2="480" y2="180" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="492" y="202" font-size="11" fill="#1f2933" text-anchor="start">z<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <line x1="20" y1="288" x2="620" y2="288" stroke="#5b6b7b" stroke-width="1.6"/><line x1="170" y1="288" x2="170" y2="251" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="480" y1="288" x2="480" y2="251" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><text x="90" y="279" font-size="11" fill="#1f2933" text-anchor="middle">[h<tspan baseline-shift="sub" font-size="9px">t-1</tspan>, x<tspan baseline-shift="sub" font-size="9px">t</tspan>]</text> |
| + | <g opacity="0.22"><line x1="348" y1="288" x2="348" y2="186" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/></g> |
| + | <line x1="560" y1="356" x2="560" y2="292" stroke="#1f2933" stroke-width="1.6" marker-end="url(#arrow)"/><text x="560" y="372" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">x<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <text x="450" y="392" font-size="11" fill="#5b6b7b" text-anchor="middle">σ: gate in (0, 1) · ⊙: elementwise product · +: elementwise add</text> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. fr/Deep Learning/08 LSTM and GRU/lstm-cell.svg | |
| @@ 0,0 1,26 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 410" width="900" height="410" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker></defs> |
| + | <rect width="900" height="410" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">Inside the LSTM cell</text> |
| + | <rect x="50" y="46" width="720" height="296" rx="14" fill="#f9fafb" stroke="#c7d0d9" stroke-width="1.2"/> |
| + | <line x1="20" y1="95" x2="237" y2="95" stroke="#1f2933" stroke-width="2.2"/> |
| + | <line x1="263" y1="95" x2="477" y2="95" stroke="#1f2933" stroke-width="2.2"/> |
| + | <line x1="503" y1="95" x2="795" y2="95" stroke="#1f2933" stroke-width="2.2"/> |
| + | <line x1="780" y1="95" x2="800" y2="95" stroke="#1f2933" stroke-width="1.8" marker-end="url(#arrow)"/> |
| + | <text x="28" y="80" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">c<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text> |
| + | <text x="822" y="99" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">c<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <circle cx="250" cy="95" r="13" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="250" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><text x="250" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">forget</text> |
| + | <circle cx="490" cy="95" r="13" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="490" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">+</text><text x="490" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">write</text> |
| + | <circle cx="490" cy="160" r="12" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="490" y="165" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><line x1="490" y1="148" x2="490" y2="110" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <rect x="470" y="200" width="90" height="34" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="515.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">tanh</text><text x="515.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">candidate</text><line x1="515" y1="200" x2="497" y2="174" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="526" y="190" font-size="11" fill="#1f2933" text-anchor="start">c̃<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <line x1="600" y1="95" x2="600" y2="138" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><rect x="558" y="140" width="84" height="34" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="600.0" y="155" font-size="12" fill="#1f2933" text-anchor="middle">tanh</text><text x="600.0" y="168" font-size="9" fill="#5b6b7b" text-anchor="middle"></text> |
| + | <line x1="642" y1="157" x2="685" y2="157" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><circle cx="700" cy="157" r="13" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="700" y="162" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><line x1="713" y1="157" x2="795" y2="157" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="822" y="161" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="210" y="200" width="80" height="34" rx="8" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="250.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">f</tspan></text><text x="250.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">forget gate</text><line x1="250" y1="200" x2="250" y2="110" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="240" y="152" font-size="11" fill="#1f2933" text-anchor="end">f<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="348" y="200" width="80" height="34" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="388.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">i</tspan></text><text x="388.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">input gate</text><line x1="388" y1="198" x2="477" y2="167" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="420" y="175" font-size="11" fill="#1f2933" text-anchor="end">i<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="660" y="200" width="80" height="34" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="700.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">o</tspan></text><text x="700.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">output gate</text><line x1="700" y1="200" x2="700" y2="172" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="712" y="190" font-size="11" fill="#1f2933" text-anchor="start">o<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <line x1="70" y1="292" x2="700" y2="292" stroke="#5b6b7b" stroke-width="1.6"/><line x1="250" y1="292" x2="250" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="388" y1="292" x2="388" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="700" y1="292" x2="700" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><text x="150" y="283" font-size="11" fill="#1f2933" text-anchor="middle">[h<tspan baseline-shift="sub" font-size="9px">t-1</tspan>, x<tspan baseline-shift="sub" font-size="9px">t</tspan>]</text> |
| + | <line x1="515" y1="292" x2="515" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/> |
| + | <line x1="20" y1="292" x2="70" y2="292" stroke="#5b6b7b" stroke-width="1.6"/><text x="28" y="277" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text> |
| + | <line x1="130" y1="360" x2="130" y2="296" stroke="#1f2933" stroke-width="1.6" marker-end="url(#arrow)"/><text x="130" y="376" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">x<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <text x="450" y="398" font-size="11" fill="#5b6b7b" text-anchor="middle">σ: gate in (0, 1) · ⊙: elementwise product · +: elementwise add</text> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. fr/Deep Learning/08 LSTM and GRU/lstm-step1.svg | |
| @@ 0,0 1,26 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 410" width="900" height="410" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker></defs> |
| + | <rect width="900" height="410" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">The LSTM cell, step 1: the gates</text> |
| + | <rect x="50" y="46" width="720" height="296" rx="14" fill="#f9fafb" stroke="#c7d0d9" stroke-width="1.2"/> |
| + | <g opacity="0.22"><line x1="20" y1="95" x2="237" y2="95" stroke="#1f2933" stroke-width="2.2"/></g> |
| + | <g opacity="0.22"><line x1="263" y1="95" x2="477" y2="95" stroke="#1f2933" stroke-width="2.2"/></g> |
| + | <g opacity="0.22"><line x1="503" y1="95" x2="795" y2="95" stroke="#1f2933" stroke-width="2.2"/></g> |
| + | <g opacity="0.22"><line x1="780" y1="95" x2="800" y2="95" stroke="#1f2933" stroke-width="1.8" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><text x="28" y="80" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">c<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text></g> |
| + | <g opacity="0.22"><text x="822" y="99" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">c<tspan baseline-shift="sub" font-size="9px">t</tspan></text></g> |
| + | <g opacity="0.22"><circle cx="250" cy="95" r="13" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="250" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><text x="250" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">forget</text></g> |
| + | <g opacity="0.22"><circle cx="490" cy="95" r="13" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="490" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">+</text><text x="490" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">write</text></g> |
| + | <g opacity="0.22"><circle cx="490" cy="160" r="12" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="490" y="165" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><line x1="490" y1="148" x2="490" y2="110" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><rect x="470" y="200" width="90" height="34" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="515.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">tanh</text><text x="515.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">candidate</text><line x1="515" y1="200" x2="497" y2="174" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="526" y="190" font-size="11" fill="#1f2933" text-anchor="start">c̃<tspan baseline-shift="sub" font-size="9px">t</tspan></text></g> |
| + | <g opacity="0.22"><line x1="600" y1="95" x2="600" y2="138" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><rect x="558" y="140" width="84" height="34" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="600.0" y="155" font-size="12" fill="#1f2933" text-anchor="middle">tanh</text><text x="600.0" y="168" font-size="9" fill="#5b6b7b" text-anchor="middle"></text></g> |
| + | <g opacity="0.22"><line x1="642" y1="157" x2="685" y2="157" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><circle cx="700" cy="157" r="13" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="700" y="162" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><line x1="713" y1="157" x2="795" y2="157" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="822" y="161" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t</tspan></text></g> |
| + | <rect x="210" y="200" width="80" height="34" rx="8" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="250.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">f</tspan></text><text x="250.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">forget gate</text><line x1="250" y1="200" x2="250" y2="110" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="240" y="152" font-size="11" fill="#1f2933" text-anchor="end">f<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="348" y="200" width="80" height="34" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="388.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">i</tspan></text><text x="388.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">input gate</text><line x1="388" y1="198" x2="477" y2="167" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="420" y="175" font-size="11" fill="#1f2933" text-anchor="end">i<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="660" y="200" width="80" height="34" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="700.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">o</tspan></text><text x="700.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">output gate</text><line x1="700" y1="200" x2="700" y2="172" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="712" y="190" font-size="11" fill="#1f2933" text-anchor="start">o<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <line x1="70" y1="292" x2="700" y2="292" stroke="#5b6b7b" stroke-width="1.6"/><line x1="250" y1="292" x2="250" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="388" y1="292" x2="388" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="700" y1="292" x2="700" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><text x="150" y="283" font-size="11" fill="#1f2933" text-anchor="middle">[h<tspan baseline-shift="sub" font-size="9px">t-1</tspan>, x<tspan baseline-shift="sub" font-size="9px">t</tspan>]</text> |
| + | <g opacity="0.22"><line x1="515" y1="292" x2="515" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/></g> |
| + | <line x1="20" y1="292" x2="70" y2="292" stroke="#5b6b7b" stroke-width="1.6"/><text x="28" y="277" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text> |
| + | <line x1="130" y1="360" x2="130" y2="296" stroke="#1f2933" stroke-width="1.6" marker-end="url(#arrow)"/><text x="130" y="376" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">x<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <text x="450" y="398" font-size="11" fill="#5b6b7b" text-anchor="middle">σ: gate in (0, 1) · ⊙: elementwise product · +: elementwise add</text> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. fr/Deep Learning/08 LSTM and GRU/lstm-step2.svg | |
| @@ 0,0 1,26 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 410" width="900" height="410" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker></defs> |
| + | <rect width="900" height="410" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">The LSTM cell, step 2: editing the carousel</text> |
| + | <rect x="50" y="46" width="720" height="296" rx="14" fill="#f9fafb" stroke="#c7d0d9" stroke-width="1.2"/> |
| + | <line x1="20" y1="95" x2="237" y2="95" stroke="#1f2933" stroke-width="2.2"/> |
| + | <line x1="263" y1="95" x2="477" y2="95" stroke="#1f2933" stroke-width="2.2"/> |
| + | <line x1="503" y1="95" x2="795" y2="95" stroke="#1f2933" stroke-width="2.2"/> |
| + | <line x1="780" y1="95" x2="800" y2="95" stroke="#1f2933" stroke-width="1.8" marker-end="url(#arrow)"/> |
| + | <text x="28" y="80" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">c<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text> |
| + | <text x="822" y="99" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">c<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <circle cx="250" cy="95" r="13" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="250" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><text x="250" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">forget</text> |
| + | <circle cx="490" cy="95" r="13" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="490" y="100" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">+</text><text x="490" y="70" font-size="11" fill="#5b6b7b" text-anchor="middle">write</text> |
| + | <circle cx="490" cy="160" r="12" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="490" y="165" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><line x1="490" y1="148" x2="490" y2="110" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <rect x="470" y="200" width="90" height="34" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="515.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">tanh</text><text x="515.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">candidate</text><line x1="515" y1="200" x2="497" y2="174" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="526" y="190" font-size="11" fill="#1f2933" text-anchor="start">c̃<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <g opacity="0.22"><line x1="600" y1="95" x2="600" y2="138" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><rect x="558" y="140" width="84" height="34" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="600.0" y="155" font-size="12" fill="#1f2933" text-anchor="middle">tanh</text><text x="600.0" y="168" font-size="9" fill="#5b6b7b" text-anchor="middle"></text></g> |
| + | <g opacity="0.22"><line x1="642" y1="157" x2="685" y2="157" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><circle cx="700" cy="157" r="13" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="700" y="162" font-size="14" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><line x1="713" y1="157" x2="795" y2="157" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="822" y="161" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t</tspan></text></g> |
| + | <rect x="210" y="200" width="80" height="34" rx="8" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="250.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">f</tspan></text><text x="250.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">forget gate</text><line x1="250" y1="200" x2="250" y2="110" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="240" y="152" font-size="11" fill="#1f2933" text-anchor="end">f<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="348" y="200" width="80" height="34" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="388.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">i</tspan></text><text x="388.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">input gate</text><line x1="388" y1="198" x2="477" y2="167" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="420" y="175" font-size="11" fill="#1f2933" text-anchor="end">i<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <rect x="660" y="200" width="80" height="34" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="700.0" y="215" font-size="12" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">o</tspan></text><text x="700.0" y="228" font-size="9" fill="#5b6b7b" text-anchor="middle">output gate</text><line x1="700" y1="200" x2="700" y2="172" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="712" y="190" font-size="11" fill="#1f2933" text-anchor="start">o<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <line x1="70" y1="292" x2="700" y2="292" stroke="#5b6b7b" stroke-width="1.6"/><line x1="250" y1="292" x2="250" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="388" y1="292" x2="388" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="700" y1="292" x2="700" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><text x="150" y="283" font-size="11" fill="#1f2933" text-anchor="middle">[h<tspan baseline-shift="sub" font-size="9px">t-1</tspan>, x<tspan baseline-shift="sub" font-size="9px">t</tspan>]</text> |
| + | <line x1="515" y1="292" x2="515" y2="238" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/> |
| + | <line x1="20" y1="292" x2="70" y2="292" stroke="#5b6b7b" stroke-width="1.6"/><text x="28" y="277" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text> |
| + | <line x1="130" y1="360" x2="130" y2="296" stroke="#1f2933" stroke-width="1.6" marker-end="url(#arrow)"/><text x="130" y="376" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">x<tspan baseline-shift="sub" font-size="9px">t</tspan></text> |
| + | <text x="450" y="398" font-size="11" fill="#5b6b7b" text-anchor="middle">σ: gate in (0, 1) · ⊙: elementwise product · +: elementwise add</text> |
| + | </svg> |
| \ | No newline at end of file |
| fr/Deep Learning/08 Normalization.md .. /dev/null | |
| @@ 1,122 0,0 @@ | |
| - | # 8. Normalisation |
| - | |
| - | Les réseaux profonds s'entraînent plus vite et de façon plus fiable lorsque les activations qui circulent entre les couches restent bien mises à l'échelle. Cette leçon introduit les couches de normalisation, qui standardisent à la volée les entrées d'une couche, puis apprennent à les remettre à l'échelle. Nous couvrons la normalisation par lot (batch normalization) et la normalisation par couche (layer normalization), où chacune calcule ses statistiques, comment elles se comportent à l'inférence, et où les placer. |
| - | |
| - | **Objectifs** |
| - | - Expliquer pourquoi normaliser les activations à l'intérieur du réseau stabilise et accélère l'entraînement. |
| - | - Dériver la transformation de la batch normalization : normaliser, puis mettre à l'échelle et décaler avec des paramètres appris $\gamma, \beta$. |
| - | - Comprendre pourquoi des statistiques courantes (moyenne mobile) remplacent les statistiques du lot à l'inférence. |
| - | - Définir la layer normalization et voir pourquoi elle convient aux réseaux récurrents et aux Transformers. |
| - | - Décider où placer une couche de normalisation par rapport à l'activation $g^{[l]}$. |
| - | - Comparer la batch normalization et la layer normalization selon leur axe de normalisation et leurs cas d'usage. |
| - | |
| - | ## 8.1 Pourquoi normaliser à l'intérieur du réseau |
| - | |
| - | Rappelons qu'une couche calcule $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$ et $a^{[l]} = g^{[l]}(z^{[l]})$. À mesure que l'entraînement met à jour chaque $W^{[l]}$, la distribution de l'entrée $a^{[l-1]}$ de chaque couche ne cesse de se déplacer. Cette cible mouvante, parfois appelée décalage de covariance interne (internal covariate shift), force les couches suivantes à se ré-adapter en permanence et ralentit l'ensemble du réseau. |
| - | |
| - | Normaliser les activations à chaque couche maintient leur moyenne et leur variance stables au fil des mises à jour. Les bénéfices immédiats : |
| - | |
| - | - La surface de la fonction de coût devient plus lisse, ce qui permet d'utiliser un taux d'apprentissage plus élevé sans diverger. |
| - | - L'entraînement converge en moins d'époques et est moins sensible à l'initialisation des poids. |
| - | - L'échelle et le décalage appris redonnent au réseau la liberté d'annuler la normalisation si cela aide. |
| - | |
| - | *Remarque :* la normalisation est appliquée à la pré-activation $z^{[l]}$ ou à l'activation $a^{[l]}$, pas aux paramètres. C'est une couche insérée dans la passe avant, avec ses propres paramètres apprenables. |
| - | |
| - | ## 8.2 Normalisation par lot (batch normalization) |
| - | |
| - | La normalisation par lot (BatchNorm) standardise chaque caractéristique à travers les exemples d'un mini-lot, puis applique une transformation affine apprise. Elle opère par caractéristique, de sorte que chaque caractéristique conserve ses propres statistiques. |
| - | |
| - |  |
| - | |
| - | *La normalisation recentre et remet à l'échelle une entrée de couche à moyenne nulle et variance unitaire avant l'échelle et le décalage appris.* |
| - | |
| - | ### 8.2.1 Statistiques du lot |
| - | |
| - | Pour une caractéristique $x$ sur un mini-lot $\mathcal{B} = \{x^{(1)}, \dots, x^{(m)}\}$ de taille $m$, on calcule la moyenne et la variance du lot : |
| - | |
| - | $$\boxed{ \mu_\mathcal{B} = \frac{1}{m}\sum_{i=1}^{m} x^{(i)}, \qquad \sigma_\mathcal{B}^2 = \frac{1}{m}\sum_{i=1}^{m}\left(x^{(i)} - \mu_\mathcal{B}\right)^2 }$$ |
| - | |
| - | ### 8.2.2 Normaliser, mettre à l'échelle et décaler |
| - | |
| - | On standardise chaque valeur à moyenne nulle et variance unitaire, en utilisant une petite constante $\epsilon > 0$ pour la stabilité numérique : |
| - | |
| - | $$\boxed{ \hat{x}^{(i)} = \frac{x^{(i)} - \mu_\mathcal{B}}{\sqrt{\sigma_\mathcal{B}^2 + \epsilon}} }$$ |
| - | |
| - | Puis on remet à l'échelle avec deux paramètres appris par caractéristique, une échelle $\gamma$ et un décalage $\beta$ : |
| - | |
| - | $$\boxed{ y^{(i)} = \gamma\, \hat{x}^{(i)} + \beta }$$ |
| - | |
| - | *Remarque :* $\gamma$ et $\beta$ sont appris par descente de gradient comme n'importe quel poids. Si le comportement optimal correspond à l'entrée brute, le réseau peut le retrouver en apprenant $\gamma = \sqrt{\sigma_\mathcal{B}^2 + \epsilon}$ et $\beta = \mu_\mathcal{B}$. La normalisation ne retire jamais de capacité, elle ne fait que la reparamétrer. |
| - | |
| - | ### 8.2.3 Inférence avec des statistiques courantes |
| - | |
| - | À l'inférence, on évalue souvent un seul exemple, de sorte qu'une moyenne et une variance de lot sont indéfinies ou dénuées de sens. À la place, BatchNorm utilise des estimations de population accumulées pendant l'entraînement sous forme de moyennes mobiles exponentielles, avec un momentum $\alpha \in [0, 1)$ : |
| - | |
| - | $$\boxed{ \mu \leftarrow \alpha\, \mu + (1 - \alpha)\, \mu_\mathcal{B}, \qquad \sigma^2 \leftarrow \alpha\, \sigma^2 + (1 - \alpha)\, \sigma_\mathcal{B}^2 }$$ |
| - | |
| - | Au moment du test, la transformation est fixe et déterministe, en utilisant ces statistiques courantes à la place de celles du lot : |
| - | |
| - | $$\boxed{ y = \gamma\, \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta }$$ |
| - | |
| - | *Remarque :* cette séparation entraînement/inférence est la source de la plupart des bugs de BatchNorm. Oublier de basculer la couche en mode évaluation la laisse calculer les statistiques du lot au moment du test, ce qui corrompt les prédictions. |
| - | |
| - | ## 8.3 Normalisation par couche (layer normalization) |
| - | |
| - | La normalisation par couche (LayerNorm) conserve la même recette normaliser-mettre à l'échelle-décaler mais change l'axe sur lequel elle moyenne. Plutôt que d'agréger à travers le lot, elle calcule les statistiques sur les caractéristiques d'un seul exemple. Chaque exemple est donc normalisé de manière autonome, indépendamment des autres dans le lot. |
| - | |
| - | ### 8.3.1 Statistiques par exemple |
| - | |
| - | Pour un exemple avec un vecteur de caractéristiques $a \in \mathbb{R}^{H}$ (ses $H$ activations dans une couche), on moyenne sur les caractéristiques : |
| - | |
| - | $$\boxed{ \mu = \frac{1}{H}\sum_{k=1}^{H} a_k, \qquad \sigma^2 = \frac{1}{H}\sum_{k=1}^{H}\left(a_k - \mu\right)^2 }$$ |
| - | |
| - | La normalisation, l'échelle et le décalage ont une forme identique à BatchNorm, appliqués par exemple : |
| - | |
| - | $$\boxed{ \hat{a}_k = \frac{a_k - \mu}{\sqrt{\sigma^2 + \epsilon}}, \qquad y_k = \gamma_k\, \hat{a}_k + \beta_k }$$ |
| - | |
| - | ### 8.3.2 Pourquoi LayerNorm pour les séquences |
| - | |
| - | Parce que les statistiques proviennent d'un seul exemple, LayerNorm se comporte de la même manière à l'entraînement et à l'inférence, et ne dépend pas de la taille du lot. Cela compte lorsque le lot est minuscule ou lorsque les exemples ont une longueur variable, comme dans le texte. LayerNorm est la normalisation de choix pour les réseaux récurrents et les Transformers, où la longueur de la séquence varie et où une moyenne de lot par pas de temps serait mal définie. |
| - | |
| - | *Remarque :* LayerNorm n'a besoin d'aucune statistique courante, il n'y a donc aucune divergence entraînement/inférence à gérer. Cela seul la rend plus simple à déployer que BatchNorm. |
| - | |
| - | ## 8.4 Placement et effets pratiques |
| - | |
| - | Une couche de normalisation se situe entre l'étape linéaire $W^{[l]} a^{[l-1]} + b^{[l]}$ et la non-linéarité $g^{[l]}$. Deux ordres sont courants. |
| - | |
| - |  |
| - | |
| - | *La normalisation est insérée entre la transformation linéaire et l'activation à l'intérieur de chaque couche.* |
| - | |
| - | - **Avant l'activation** (normaliser $z^{[l]}$, puis appliquer $g^{[l]}$) : le placement d'origine et le plus courant. Il maintient l'entrée de la non-linéarité centrée, là où la saturation nuit le plus. |
| - | - **Après l'activation** (normaliser $a^{[l]}$) : parfois utilisé et occasionnellement meilleur en pratique, bien qu'il soit moins standard. |
| - | |
| - | Deux autres points pratiques : |
| - | |
| - | - **Le biais devient redondant.** Le décalage $\beta$ remplace le biais de la couche, puisque la normalisation soustrait la moyenne et annulerait de toute façon $b^{[l]}$. Les couches suivies d'une normalisation sont souvent écrites sans leur propre biais. |
| - | - **BatchNorm dépend de la taille du lot.** Ses statistiques sont plus bruitées avec de petits lots, ce qui agit comme un régulariseur léger mais se dégrade fortement lorsque le lot est très petit. LayerNorm y est insensible, ce qui est une autre raison pour laquelle les modèles de séquence la préfèrent. |
| - | |
| - | *Remarque :* le bruit dépendant du lot dans BatchNorm peut se substituer en partie à d'autres régulariseurs, de sorte que les réseaux qui l'utilisent ont parfois besoin de moins de dropout. |
| - | |
| - | ## 8.5 BatchNorm contre LayerNorm |
| - | |
| - | Les deux couches partagent la transformation normaliser-mettre à l'échelle-décaler et ne diffèrent que par l'axe des statistiques et les conséquences qui en découlent. |
| - | |
| - | | Aspect | Normalisation par lot | Normalisation par couche | |
| - | | --- | --- | --- | |
| - | | Axe de normalisation | à travers le lot, par caractéristique | à travers les caractéristiques, par exemple | |
| - | | Dépend de la taille du lot | oui | non | |
| - | | Entraînement vs inférence | statistiques du lot vs statistiques courantes | identique dans les deux | |
| - | | Statistiques courantes nécessaires | oui | non | |
| - | | Usage typique | CNN et modèles de vision feedforward | RNN et Transformers | |
| - | |
| - |  |
| - | |
| - | *La normalisation par lot calcule les statistiques le long d'une colonne de caractéristique à travers le lot, la normalisation par couche à travers les caractéristiques d'un seul exemple.* |
| - | |
| - | *Remarque :* le bloc Transformer de la leçon 16 place une LayerNorm avant ou après chaque sous-couche, précisément parce qu'elle supprime la dépendance au lot qui, sinon, lierait entre eux des exemples de longueurs différentes. |
| - | |
| - | *Les activations restant bien mises à l'échelle, le réseau s'entraîne de façon stable à des taux d'apprentissage plus élevés. La prochaine leçon aborde le contrôle du surapprentissage par la régularisation et le dropout.* |
| - | |
| - | --- |
| - | Suivant : [Régularisation et dropout](/fr/Deep%20Learning/09%20Regularization%20and%20dropout) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| fr/Deep Learning/08 Normalization/norm-placement.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 760 200" width="760" height="200" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="760" height="200" fill="#ffffff"/><text x="380.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">Normalization inside a layer</text><rect x="40.0" y="90.0" width="150.0" height="60.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="115.0" y="124.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">linear W a + b</text><rect x="210.0" y="90.0" width="150.0" height="60.0" rx="8" fill="#e4f5f4" stroke="#2a9d9a" stroke-width="1.6"/><text x="285.0" y="124.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">normalization</text><rect x="380.0" y="90.0" width="150.0" height="60.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="455.0" y="124.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">activation g</text><rect x="550.0" y="90.0" width="150.0" height="60.0" rx="8" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><text x="625.0" y="124.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">next layer</text><line x1="190.0" y1="120.0" x2="210.0" y2="120.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="360.0" y1="120.0" x2="380.0" y2="120.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="530.0" y1="120.0" x2="550.0" y2="120.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><text x="285.0" y="176.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">recenter and rescale z</text></svg> |
| \ | No newline at end of file |
| fr/Deep Learning/08 Normalization/normalization-effect.png .. /dev/null | |
| fr/Deep Learning/14 Attention.md .. fr/Deep Learning/09 Attention.md | |
| @@ 1,17 1,10 @@ | |
| - | # 14. Attention |
| + | # 9. Attention |
| Les modèles encodeur-décodeur récurrents font passer toute une séquence d'entrée par un unique vecteur de contexte de taille fixe, ce qui limite ce qu'ils peuvent mémoriser pour de longues entrées. L'attention supprime ce goulot d'étranglement en laissant le décodeur lire directement chaque état de l'encodeur, en pondérant chacun selon sa pertinence pour l'étape de sortie courante. Cette leçon construit le mécanisme depuis les scores d'alignement jusqu'à la vision requête-clé-valeur, qui est le socle que le Transformeur va généraliser. | |
| - | **Objectifs** |
| - | - Expliquer pourquoi le vecteur de contexte de taille fixe est un goulot d'étranglement dans les modèles séquence-à-séquence. |
| - | - Définir les scores d'alignement, les poids d'attention et le vecteur de contexte. |
| - | - Opposer les fonctions de score additive (Bahdanau) et multiplicative (Luong). |
| - | - Reformuler l'attention comme une requête qui porte son attention sur des clés et des valeurs. |
| - | - Relier ce cadre à l'auto-attention et au Transformeur. |
| + | ## 9.1 Le goulot d'étranglement du seq2seq |
| - | ## 14.1 Le goulot d'étranglement du seq2seq |
| - | |
| - | Un modèle séquence-à-séquence utilise un réseau récurrent encodeur pour lire les jetons d'entrée $x_1, \dots, x_T$ vers des états cachés $h_1, \dots, h_T$, puis un réseau récurrent décodeur pour émettre les jetons de sortie. Dans la conception de base, le décodeur est initialisé à partir d'un unique vecteur de contexte, le dernier état caché de l'encodeur : |
| + | Prenons un exemple fil rouge : traduire « nous mangeons du pain » en l'anglais « we are eating bread ». Un modèle séquence-à-séquence utilise un réseau récurrent encodeur pour lire les jetons d'entrée $x_1, \dots, x_T$ vers des états cachés $h_1, \dots, h_T$, puis un réseau récurrent décodeur pour émettre les jetons de sortie, un par pas. Dans la conception de base, le décodeur est initialisé à partir d'un unique vecteur de contexte, le dernier état caché de l'encodeur : |
| $$\boxed{ c = h_T }$$ | |
| @@ 19,21 12,17 @@ | |
| *Remarque :* il s'agit d'un véritable goulot d'étranglement de l'information. Pour une phrase courte, $h_T$ peut en contenir assez, mais à mesure que $T$ grandit les premiers jetons sont écrasés et la qualité de la traduction ou du résumé chute fortement sur les longues entrées. | |
| - |  |
| + |  |
| *Le séquence-à-séquence simple comprime toute l'entrée dans un unique vecteur de contexte de taille fixe, un goulot d'étranglement pour les longues séquences.* | |
| La solution consiste à garder disponibles tous les états de l'encodeur $h_1, \dots, h_T$ et à laisser le décodeur décider, à chaque étape, lesquels lire. | |
| - | ## 14.2 Le mécanisme d'attention |
| + | ## 9.2 Le mécanisme d'attention |
| Au lieu d'un unique vecteur de contexte partagé entre toutes les étapes, l'attention construit un nouveau vecteur de contexte $c_i$ pour chaque étape $i$ du décodeur. Elle procède en trois temps : scorer, normaliser, combiner. | |
| - |  |
| - | |
| - | *L'attention score chaque état de l'encodeur par rapport à la requête du décodeur, puis forme le contexte comme une somme pondérée de tous les états.* |
| - | |
| - | ### 14.2.1 Scores d'alignement |
| + | ### 9.2.1 Scores d'alignement |
| Pour l'étape $i$ du décodeur d'état $s_i$, une fonction de score mesure à quel point cet état s'aligne avec chaque état de l'encodeur $h_j$ : | |
| @@ 43,7 32,11 @@ | |
| *Remarque :* $s_i$ est généralement l'état du décodeur juste avant d'émettre le jeton $i$, de sorte que le modèle choisit ce qu'il regarde en fonction de ce qu'il a produit jusqu'ici. | |
| - | ### 14.2.2 Poids d'attention |
| + |  |
| + | |
| + | *Étape 1 : l'état du décodeur $s_i$ est confronté à chaque état de l'encodeur $h_j$, un score par position d'entrée. Tout l'aval est encore grisé.* |
| + | |
| + | ### 9.2.2 Poids d'attention |
| Les scores sont transformés en une distribution de probabilité sur les positions d'entrée par une softmax sur $j$ : | |
| @@ 51,7 44,11 @@ | |
| Chaque $\alpha_{i,j} \in (0,1)$ et $\sum_j \alpha_{i,j} = 1$, donc les poids indiquent quelle part de l'attention du décodeur à l'étape $i$ va à la position d'entrée $j$. | |
| - | ### 14.2.3 Vecteur de contexte |
| + |  |
| + | |
| + | *Étape 2 : la softmax transforme les $T$ scores en poids $\alpha_{i,j}$ de somme un, une barre par position d'entrée.* |
| + | |
| + | ### 9.2.3 Vecteur de contexte |
| Le vecteur de contexte pour l'étape $i$ est la moyenne pondérée des états de l'encodeur, en utilisant les poids d'attention : | |
| @@ 59,13 56,17 @@ | |
| Ce $c_i$ est recalculé à chaque étape du décodeur, de sorte que le modèle lit un mélange différent de l'entrée pour chaque jeton de sortie. Le décodeur combine ensuite $c_i$ avec son état $s_i$ pour prédire le jeton, et les poids d'alignement $\alpha_{i,j}$ peuvent se visualiser comme une matrice douce qui montre quels mots d'entrée chaque mot de sortie regarde. | |
| - | *Remarque :* comme chaque étape effectue une moyenne sur tous les $h_j$, aucun vecteur fixe unique n'a à porter toute l'entrée. Le goulot d'étranglement de la section 15.1 a disparu, et les longues entrées ne se dégradent plus aussi vite. |
| + | *Remarque :* comme chaque étape effectue une moyenne sur tous les $h_j$, aucun vecteur fixe unique n'a à porter toute l'entrée. Le goulot d'étranglement de la section 9.1 a disparu, et les longues entrées ne se dégradent plus aussi vite. |
| + | |
| + |  |
| + | |
| + | *Le mécanisme complet : scorer, normaliser, combiner. Le contexte $c_i$ est la moyenne des états de l'encodeur pondérée par $\alpha$, reconstruite à chaque pas du décodeur.* |
| - | ## 14.3 Fonctions de score |
| + | ## 9.3 Fonctions de score |
| - | La fonction de score de la section 15.2.1 est un choix de conception. Deux formes dominent la littérature initiale sur l'attention. |
| + | La fonction de score de la section 9.2.1 est un choix de conception. Deux formes dominent la littérature initiale sur l'attention. |
| - | ### 14.3.1 Score additif (Bahdanau) |
| + | ### 9.3.1 Score additif (Bahdanau) |
| Le score additif, dû à Bahdanau et ses co-auteurs, fait passer les deux états dans un petit réseau à une couche cachée avec des matrices apprises $W_1$ et $W_2$ et un vecteur appris $v$ : | |
| @@ 73,7 74,7 @@ | |
| Il fonctionne même lorsque $s_i$ et $h_j$ ont des dimensions différentes, puisque $W_1$ et $W_2$ projettent les deux dans un espace commun avant le $\tanh$. | |
| - | ### 14.3.2 Score multiplicatif (Luong) |
| + | ### 9.3.2 Score multiplicatif (Luong) |
| Le score multiplicatif, dû à Luong et ses co-auteurs, est un simple produit scalaire entre les deux états : | |
| @@ 81,7 82,7 @@ | |
| Il n'a aucun paramètre supplémentaire dans sa forme la plus simple et est bien moins coûteux à calculer, puisqu'une matrice entière de scores est un unique produit matriciel. Une variante générale insère une matrice apprise $W$ sous la forme $s_i^{\top} W h_j$ pour gérer les dimensions non concordantes. | |
| - | ### 14.3.3 Lequel utiliser |
| + | ### 9.3.3 Lequel utiliser |
| | Aspect | Additif (Bahdanau) | Multiplicatif (Luong) | | |
| | --- | --- | --- | | |
| @@ 93,11 94,11 @@ | |
| *Remarque :* le produit scalaire croît avec la dimension des états, donc à grande largeur sa variance devient grande et pousse la softmax vers des régions plates. Mettre le score à l'échelle par $1/\sqrt{d}$ corrige cela, et ce produit scalaire mis à l'échelle est exactement ce que le Transformeur va adopter. | |
| - | ## 14.4 Requête, clé, valeur |
| + | ## 9.4 Requête, clé, valeur |
| L'attention admet une lecture plus nette qui abandonne le cadre encodeur-décodeur. Renommons les pièces : l'état qui regarde est une requête, et chaque chose qui peut être regardée fournit une clé (utilisée pour le scoring) et une valeur (utilisée dans la somme). | |
| - |  |
| + |  |
| *Une matrice de poids d'attention : chaque jeton de sortie puise surtout dans quelques jetons d'entrée.* | |
| @@ 114,4 115,4 @@ | |
| *Construire la requête, la clé et la valeur à partir d'une seule séquence avec des projections apprises transforme l'attention en une couche de séquence générale, ce qui est exactement là où commence la prochaine leçon sur les Transformeurs.* | |
| --- | |
| - | Suivant : [Transformeurs](/fr/Deep%20Learning/15%20Transformers) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| + | Suivant : [Transformeurs](/fr/Deep%20Learning/10%20Transformers) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| fr/Deep Learning/14 Attention/attention-heatmap.png .. fr/Deep Learning/09 Attention/attention-heatmap.png | |
| /dev/null .. fr/Deep Learning/09 Attention/attention-step1.svg | |
| @@ 0,0 1,12 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 400" width="900" height="400" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | <marker id="arrowmut" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs> |
| + | <rect width="900" height="400" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">Attention, step 1: score the query against every state</text> |
| + | <text x="255" y="56" font-size="11" fill="#5b6b7b" text-anchor="middle">encoder states h<tspan baseline-shift="sub" font-size="8px">j</tspan></text><rect x="80" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="108" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">1</tspan></text><rect x="180" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="208" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">2</tspan></text><rect x="280" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="308" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">3</tspan></text><rect x="380" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="408" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">4</tspan></text> |
| + | <rect x="690" y="72" width="64" height="28" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="722" y="91" font-size="12" fill="#1f2933" text-anchor="middle">s<tspan baseline-shift="sub" font-size="8px">i</tspan></text><text x="722" y="56" font-size="11" fill="#5b6b7b" text-anchor="middle">decoder state (the query)</text> |
| + | <line x1="108" y1="100" x2="108" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="208" y1="100" x2="208" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="308" y1="100" x2="308" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="408" y1="100" x2="408" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="122" y2="170" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="221" y2="168" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="319" y2="166" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="417" y2="164" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><circle cx="108" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="108" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><circle cx="208" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="208" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><circle cx="308" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="308" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><circle cx="408" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="408" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><text x="470" y="152" font-size="10" fill="#5b6b7b" text-anchor="middle">score(s<tspan baseline-shift="sub" font-size="8px">i</tspan>, h<tspan baseline-shift="sub" font-size="8px">j</tspan>)</text> |
| + | <g opacity="0.22"><rect x="72" y="214" width="372" height="30" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="258" y="233" font-size="11" fill="#1f2933" text-anchor="middle">softmax over positions j</text><line x1="108" y1="192" x2="108" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="208" y1="192" x2="208" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="308" y1="192" x2="308" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="408" y1="192" x2="408" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="108" y1="244" x2="108" y2="311" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="96" y="315" width="24" height="7" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="208" y1="244" x2="208" y2="262" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="196" y="266" width="24" height="56" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="308" y1="244" x2="308" y2="298" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="296" y="302" width="24" height="20" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="408" y1="244" x2="408" y2="311" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="396" y="315" width="24" height="7" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="80" y1="322" x2="436" y2="322" stroke="#5b6b7b" stroke-width="1"/><text x="108" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,1</tspan></text><text x="208" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,2</tspan></text><text x="308" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,3</tspan></text><text x="408" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,4</tspan></text><text x="258" y="356" font-size="10" fill="#5b6b7b" text-anchor="middle">weights, sum to 1</text></g> |
| + | <g opacity="0.22"><line x1="108" y1="346" x2="108" y2="356" stroke="none"/><line x1="208" y1="346" x2="208" y2="356" stroke="none"/><line x1="308" y1="346" x2="308" y2="356" stroke="none"/><line x1="408" y1="346" x2="408" y2="356" stroke="none"/><line x1="118" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="218" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="318" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="418" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><path d="M436,86 C580,90 550,230 522,288" fill="none" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><text x="612" y="182" font-size="10" fill="#5b6b7b" text-anchor="middle">values h<tspan baseline-shift="sub" font-size="8px">j</tspan></text><circle cx="516" cy="300" r="14" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="516" y="305" font-size="14" fill="#1f2933" text-anchor="middle" font-weight="600">+</text><line x1="530" y1="300" x2="558" y2="300" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="560" y="286" width="64" height="28" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="592" y="305" font-size="12" fill="#1f2933" text-anchor="middle">c<tspan baseline-shift="sub" font-size="8px">i</tspan></text><text x="592" y="330" font-size="9" fill="#5b6b7b" text-anchor="middle">context for step i</text><line x1="624" y1="300" x2="676" y2="300" stroke="#5b6b7b" stroke-width="1.4" marker-end="url(#arrowmut)"/><text x="745" y="304" font-size="11" fill="#5b6b7b" text-anchor="middle">to the decoder step</text></g> |
| + | <text x="450" y="388" font-size="11" fill="#5b6b7b" text-anchor="middle">c<tspan baseline-shift="sub" font-size="8px">i</tspan> = Σ α<tspan baseline-shift="sub" font-size="8px">i,j</tspan> h<tspan baseline-shift="sub" font-size="8px">j</tspan> · score, normalise, combine</text> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. fr/Deep Learning/09 Attention/attention-step2.svg | |
| @@ 0,0 1,12 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 400" width="900" height="400" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | <marker id="arrowmut" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs> |
| + | <rect width="900" height="400" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">Attention, step 2: normalise the scores into weights</text> |
| + | <text x="255" y="56" font-size="11" fill="#5b6b7b" text-anchor="middle">encoder states h<tspan baseline-shift="sub" font-size="8px">j</tspan></text><rect x="80" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="108" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">1</tspan></text><rect x="180" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="208" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">2</tspan></text><rect x="280" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="308" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">3</tspan></text><rect x="380" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="408" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">4</tspan></text> |
| + | <rect x="690" y="72" width="64" height="28" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="722" y="91" font-size="12" fill="#1f2933" text-anchor="middle">s<tspan baseline-shift="sub" font-size="8px">i</tspan></text><text x="722" y="56" font-size="11" fill="#5b6b7b" text-anchor="middle">decoder state (the query)</text> |
| + | <line x1="108" y1="100" x2="108" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="208" y1="100" x2="208" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="308" y1="100" x2="308" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="408" y1="100" x2="408" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="122" y2="170" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="221" y2="168" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="319" y2="166" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="417" y2="164" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><circle cx="108" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="108" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><circle cx="208" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="208" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><circle cx="308" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="308" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><circle cx="408" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="408" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><text x="470" y="152" font-size="10" fill="#5b6b7b" text-anchor="middle">score(s<tspan baseline-shift="sub" font-size="8px">i</tspan>, h<tspan baseline-shift="sub" font-size="8px">j</tspan>)</text> |
| + | <rect x="72" y="214" width="372" height="30" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="258" y="233" font-size="11" fill="#1f2933" text-anchor="middle">softmax over positions j</text><line x1="108" y1="192" x2="108" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="208" y1="192" x2="208" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="308" y1="192" x2="308" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="408" y1="192" x2="408" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="108" y1="244" x2="108" y2="311" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="96" y="315" width="24" height="7" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="208" y1="244" x2="208" y2="262" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="196" y="266" width="24" height="56" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="308" y1="244" x2="308" y2="298" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="296" y="302" width="24" height="20" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="408" y1="244" x2="408" y2="311" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="396" y="315" width="24" height="7" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="80" y1="322" x2="436" y2="322" stroke="#5b6b7b" stroke-width="1"/><text x="108" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,1</tspan></text><text x="208" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,2</tspan></text><text x="308" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,3</tspan></text><text x="408" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,4</tspan></text><text x="258" y="356" font-size="10" fill="#5b6b7b" text-anchor="middle">weights, sum to 1</text> |
| + | <g opacity="0.22"><line x1="108" y1="346" x2="108" y2="356" stroke="none"/><line x1="208" y1="346" x2="208" y2="356" stroke="none"/><line x1="308" y1="346" x2="308" y2="356" stroke="none"/><line x1="408" y1="346" x2="408" y2="356" stroke="none"/><line x1="118" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="218" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="318" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="418" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><path d="M436,86 C580,90 550,230 522,288" fill="none" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><text x="612" y="182" font-size="10" fill="#5b6b7b" text-anchor="middle">values h<tspan baseline-shift="sub" font-size="8px">j</tspan></text><circle cx="516" cy="300" r="14" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="516" y="305" font-size="14" fill="#1f2933" text-anchor="middle" font-weight="600">+</text><line x1="530" y1="300" x2="558" y2="300" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="560" y="286" width="64" height="28" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="592" y="305" font-size="12" fill="#1f2933" text-anchor="middle">c<tspan baseline-shift="sub" font-size="8px">i</tspan></text><text x="592" y="330" font-size="9" fill="#5b6b7b" text-anchor="middle">context for step i</text><line x1="624" y1="300" x2="676" y2="300" stroke="#5b6b7b" stroke-width="1.4" marker-end="url(#arrowmut)"/><text x="745" y="304" font-size="11" fill="#5b6b7b" text-anchor="middle">to the decoder step</text></g> |
| + | <text x="450" y="388" font-size="11" fill="#5b6b7b" text-anchor="middle">c<tspan baseline-shift="sub" font-size="8px">i</tspan> = Σ α<tspan baseline-shift="sub" font-size="8px">i,j</tspan> h<tspan baseline-shift="sub" font-size="8px">j</tspan> · score, normalise, combine</text> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. fr/Deep Learning/09 Attention/attention-weights.svg | |
| @@ 0,0 1,12 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 400" width="900" height="400" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | <marker id="arrowmut" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs> |
| + | <rect width="900" height="400" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">Attention: score, normalise, combine</text> |
| + | <text x="255" y="56" font-size="11" fill="#5b6b7b" text-anchor="middle">encoder states h<tspan baseline-shift="sub" font-size="8px">j</tspan></text><rect x="80" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="108" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">1</tspan></text><rect x="180" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="208" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">2</tspan></text><rect x="280" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="308" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">3</tspan></text><rect x="380" y="72" width="56" height="28" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="408" y="91" font-size="12" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="8px">4</tspan></text> |
| + | <rect x="690" y="72" width="64" height="28" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="722" y="91" font-size="12" fill="#1f2933" text-anchor="middle">s<tspan baseline-shift="sub" font-size="8px">i</tspan></text><text x="722" y="56" font-size="11" fill="#5b6b7b" text-anchor="middle">decoder state (the query)</text> |
| + | <line x1="108" y1="100" x2="108" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="208" y1="100" x2="208" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="308" y1="100" x2="308" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="408" y1="100" x2="408" y2="162" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="122" y2="170" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="221" y2="168" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="319" y2="166" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="700" y1="102" x2="417" y2="164" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><circle cx="108" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="108" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><circle cx="208" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="208" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><circle cx="308" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="308" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><circle cx="408" cy="178" r="14" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="408" y="182" font-size="10" fill="#1f2933" text-anchor="middle">e<tspan baseline-shift="sub" font-size="8px">i,j</tspan></text><text x="470" y="152" font-size="10" fill="#5b6b7b" text-anchor="middle">score(s<tspan baseline-shift="sub" font-size="8px">i</tspan>, h<tspan baseline-shift="sub" font-size="8px">j</tspan>)</text> |
| + | <rect x="72" y="214" width="372" height="30" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="258" y="233" font-size="11" fill="#1f2933" text-anchor="middle">softmax over positions j</text><line x1="108" y1="192" x2="108" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="208" y1="192" x2="208" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="308" y1="192" x2="308" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="408" y1="192" x2="408" y2="212" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="108" y1="244" x2="108" y2="311" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="96" y="315" width="24" height="7" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="208" y1="244" x2="208" y2="262" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="196" y="266" width="24" height="56" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="308" y1="244" x2="308" y2="298" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="296" y="302" width="24" height="20" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="408" y1="244" x2="408" y2="311" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="396" y="315" width="24" height="7" fill="#fff1e0" stroke="#e0872e" stroke-width="1.4"/><line x1="80" y1="322" x2="436" y2="322" stroke="#5b6b7b" stroke-width="1"/><text x="108" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,1</tspan></text><text x="208" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,2</tspan></text><text x="308" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,3</tspan></text><text x="408" y="338" font-size="10" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="8px">i,4</tspan></text><text x="258" y="356" font-size="10" fill="#5b6b7b" text-anchor="middle">weights, sum to 1</text> |
| + | <line x1="108" y1="346" x2="108" y2="356" stroke="none"/><line x1="208" y1="346" x2="208" y2="356" stroke="none"/><line x1="308" y1="346" x2="308" y2="356" stroke="none"/><line x1="408" y1="346" x2="408" y2="356" stroke="none"/><line x1="118" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="218" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="318" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><line x1="418" y1="326" x2="502" y2="300" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><path d="M436,86 C580,90 550,230 522,288" fill="none" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><text x="612" y="182" font-size="10" fill="#5b6b7b" text-anchor="middle">values h<tspan baseline-shift="sub" font-size="8px">j</tspan></text><circle cx="516" cy="300" r="14" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="516" y="305" font-size="14" fill="#1f2933" text-anchor="middle" font-weight="600">+</text><line x1="530" y1="300" x2="558" y2="300" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="560" y="286" width="64" height="28" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="592" y="305" font-size="12" fill="#1f2933" text-anchor="middle">c<tspan baseline-shift="sub" font-size="8px">i</tspan></text><text x="592" y="330" font-size="9" fill="#5b6b7b" text-anchor="middle">context for step i</text><line x1="624" y1="300" x2="676" y2="300" stroke="#5b6b7b" stroke-width="1.4" marker-end="url(#arrowmut)"/><text x="745" y="304" font-size="11" fill="#5b6b7b" text-anchor="middle">to the decoder step</text> |
| + | <text x="450" y="388" font-size="11" fill="#5b6b7b" text-anchor="middle">c<tspan baseline-shift="sub" font-size="8px">i</tspan> = Σ α<tspan baseline-shift="sub" font-size="8px">i,j</tspan> h<tspan baseline-shift="sub" font-size="8px">j</tspan> · score, normalise, combine</text> |
| + | </svg> |
| \ | No newline at end of file |
| fr/Deep Learning/14 Attention/seq2seq-bottleneck.svg .. fr/Deep Learning/09 Attention/seq2seq-bottleneck.svg | |
| fr/Deep Learning/09 Regularization and dropout.md .. /dev/null | |
| @@ 1,117 0,0 @@ | |
| - | # 9. Régularisation et dropout |
| - | |
| - | Un réseau profond possède une capacité suffisante pour ajuster presque n'importe quel jeu d'entraînement, y compris son bruit. La régularisation regroupe les techniques qui échangent un peu de précision sur l'entraînement contre une meilleure généralisation. Ce module couvre la décroissance des poids ($L_2$), son pendant $L_1$, le dropout avec le rééchelonnement de l'inverted dropout, ainsi que les régularisateurs plus légers que sont l'arrêt précoce et l'augmentation de données. |
| - | |
| - | **Objectifs** |
| - | - Rappeler ce qu'est le surapprentissage et pourquoi les réseaux à forte capacité y sont sujets. |
| - | - Ajouter une pénalité $L_2$ au coût et en lire l'effet sur le gradient (décroissance des poids). |
| - | - Opposer $L_2$ et $L_1$ et leurs pressions différentes sur les poids. |
| - | - Appliquer l'inverted dropout comme un masque de Bernoulli avec rééchelonnement en $1/p$. |
| - | - Expliquer la vision « ensemble » du dropout et pourquoi le rééchelonnement laisse les activations non biaisées. |
| - | - Placer l'arrêt précoce et l'augmentation de données dans la même boîte à outils de généralisation. |
| - | |
| - | ## 9.1 Rappel sur le surapprentissage |
| - | |
| - | Un modèle surapprend lorsqu'il fait tendre son coût d'entraînement $J$ vers zéro en mémorisant les exemples d'entraînement, y compris leur bruit, si bien qu'il généralise mal à des données non vues. L'écart entre la performance sur l'entraînement et celle sur le test est le signe révélateur. Les réseaux profonds y sont particulièrement exposés car leur nombre de paramètres $\sum_l n_l\, n_{l-1}$ dépasse généralement le nombre d'exemples d'entraînement, ils ont donc la capacité de mémoriser. |
| - | |
| - | *Remarque :* le compromis biais-variance sous-jacent a été introduit dans le cours de Machine Learning, voir [Concepts généraux](/fr/Machine%20Learning/02%20General%20concepts). La régularisation ramène un modèle à forte variance vers le point idéal. |
| - | |
| - | Le remède consiste à contraindre la capacité effective pour que le réseau préfère des fonctions plus simples. Chaque technique ci-dessous est une telle contrainte. |
| - | |
| - | ## 9.2 Régularisation L2 (décroissance des poids) |
| - | |
| - | ### 9.2.1 La pénalité |
| - | |
| - | La régularisation $L_2$ ajoute au coût une pénalité proportionnelle à la magnitude au carré de chaque matrice de poids. Avec $\lambda \ge 0$ l'intensité de régularisation, le coût régularisé est : |
| - | |
| - | $$\boxed{ J_{\text{reg}} = J + \frac{\lambda}{2}\sum_{l=1}^{L}\lVert W^{[l]} \rVert_F^2 }$$ |
| - | |
| - | où $\lVert W^{[l]} \rVert_F^2 = \sum_{i,j}\big(W^{[l]}_{ij}\big)^2$ est la norme de Frobenius au carré. Les biais $b^{[l]}$ sont normalement exclus de la pénalité, car ils ajoutent une capacité négligeable et les pénaliser tend à provoquer du sous-apprentissage. |
| - | |
| - | ### 9.2.2 Effet sur le gradient |
| - | |
| - | C'est la dérivation de la pénalité qui donne à la technique son second nom. Le terme supplémentaire contribue à hauteur de $\lambda W^{[l]}$ au gradient par rapport à $W^{[l]}$ : |
| - | |
| - | $$\boxed{ \frac{\partial J_{\text{reg}}}{\partial W^{[l]}} = \frac{\partial J}{\partial W^{[l]}} + \lambda\, W^{[l]} }$$ |
| - | |
| - | En injectant ceci dans une étape de descente de gradient avec taux d'apprentissage $\alpha$, on rétrécit le poids avant d'appliquer la mise à jour guidée par les données : |
| - | |
| - | $$\boxed{ W^{[l]} \leftarrow (1 - \alpha\lambda)\, W^{[l]} - \alpha\,\frac{\partial J}{\partial W^{[l]}} }$$ |
| - | |
| - | *Remarque :* le facteur $(1 - \alpha\lambda) < 1$ multiplie chaque poids à chaque étape, ce qui est littéralement une décroissance vers zéro. C'est pourquoi la régularisation $L_2$ est appelée décroissance des poids. Des poids plus petits signifient une fonction plus lisse et de plus faible variance. |
| - | |
| - | ### 9.2.3 Comparaison avec L1 |
| - | |
| - | Remplacer la norme au carré par la norme en valeur absolue donne la régularisation $L_1$, qui pénalise à hauteur de $\lambda\sum_l \lVert W^{[l]} \rVert_1 = \lambda\sum_{l,i,j}\lvert W^{[l]}_{ij}\rvert$. Sa contribution au gradient est $\lambda\,\operatorname{sign}(W^{[l]})$, une attraction constante vers zéro indépendante de la magnitude. |
| - | |
| - | | Pénalité | Ajout au coût | Terme de gradient | Pression sur les poids | |
| - | | --- | --- | --- | --- | |
| - | | $L_2$ | $\tfrac{\lambda}{2}\lVert W \rVert_F^2$ | $\lambda W$ | rétrécit tous les poids proportionnellement, rarement exactement nuls | |
| - | | $L_1$ | $\lambda\lVert W \rVert_1$ | $\lambda\,\operatorname{sign}(W)$ | pousse de nombreux poids exactement à zéro (parcimonie) | |
| - | |
| - | *Remarque :* $L_1$ produit des matrices de poids parcimonieuses et fait donc aussi office de sélection de variables. $L_2$ est le choix par défaut en apprentissage profond car elle est lisse partout et s'associe proprement à la descente de gradient. |
| - | |
| - | ## 9.3 Dropout |
| - | |
| - | ### 9.3.1 L'idée |
| - | |
| - | Le dropout régularise en injectant du bruit dans les activations. À chaque passe avant d'entraînement, chaque unité est conservée avec probabilité $p$ et mise à zéro avec probabilité $1 - p$, de façon indépendante. Le réseau ne peut donc s'appuyer sur aucune unité isolée, il répartit alors la représentation sur de nombreuses unités et cesse de les co-adapter. |
| - | |
| - | ### 9.3.2 Inverted dropout |
| - | |
| - | Soit $m$ un masque de Bernoulli$(p)$ de même forme que l'activation $a^{[l]}$, tiré à neuf à chaque étape. L'inverted dropout applique le masque puis divise immédiatement par $p$ : |
| - | |
| - | $$\boxed{ \tilde{a}^{[l]} = \frac{m \odot a^{[l]}}{p}, \qquad m_i \sim \text{Bernoulli}(p) }$$ |
| - | |
| - | L'activation masquée et rééchelonnée $\tilde{a}^{[l]}$ circule alors vers la couche $l+1$ à la place de $a^{[l]}$. Au moment de l'inférence, le dropout est désactivé et se comporte comme l'identité, $\tilde{a}^{[l]} = a^{[l]}$, sans masque ni rééchelonnement. |
| - | |
| - | *Remarque :* conserver le rééchelonnement en $1/p$ au moment de l'entraînement (d'où « inverted ») est ce qui permet à l'inférence de rester une simple passe avant. La forme ancienne, non inversée, multipliait au contraire les poids par $p$ au moment du test, ce qui est facile à oublier. |
| - | |
| - | ### 9.3.3 Pourquoi le rééchelonnement |
| - | |
| - | Comme $\mathbb{E}[m_i] = p$, l'espérance d'une unité conservée et rééchelonnée est égale à l'activation d'origine : |
| - | |
| - | $$\boxed{ \mathbb{E}\!\left[\tilde{a}^{[l]}_i\right] = \frac{p\cdot a^{[l]}_i + (1-p)\cdot 0}{p} = a^{[l]}_i }$$ |
| - | |
| - | L'entrée attendue de la couche suivante est donc inchangée, et le réseau voit le même signal moyen que le dropout soit activé ou non. C'est précisément pourquoi aucune correction n'est nécessaire à l'inférence. |
| - | |
| - | ### 9.3.4 La vision « ensemble » |
| - | |
| - | Un réseau comportant $k$ unités susceptibles d'être supprimées définit $2^k$ sous-réseaux amincis possibles, un par masque. Chaque étape d'entraînement échantillonne un sous-réseau et effectue une étape de gradient dessus, et tous les sous-réseaux partagent leurs poids. Au moment du test, le réseau complet avec ses activations rééchelonnées approxime la prédiction moyenne de cet ensemble exponentiellement grand, ce qui explique pourquoi le dropout se comporte comme une moyenne de modèles à bas coût. |
| - | |
| - |  |
| - | |
| - | *Le dropout entraîne à chaque étape un sous-réseau aminci différent en supprimant des unités au hasard, puis les moyenne à l'inférence.* |
| - | |
| - | *Remarque :* les probabilités de conservation typiques sont $p$ autour de $0{,}8$ pour les couches d'entrée et $0{,}5$ pour les couches cachées. Un $p$ plus petit signifie une régularisation plus forte. |
| - | |
| - | ## 9.4 Autres régularisateurs |
| - | |
| - | ### 9.4.1 Arrêt précoce |
| - | |
| - | On suit le coût de validation pendant l'entraînement et on s'arrête à l'époque où il commence à remonter, alors même que le coût d'entraînement continue de baisser. S'arrêter tôt maintient les poids près de leurs faibles valeurs initiales, ce qui agit comme une pénalité $L_2$ implicite sans ajouter de terme au coût. |
| - | |
| - |  |
| - | |
| - | *La perte d'entraînement continue de baisser tandis que la perte de validation repart à la hausse, l'écart est le surapprentissage et son minimum est l'endroit où l'arrêt précoce interrompt l'entraînement.* |
| - | |
| - | ### 9.4.2 Augmentation de données |
| - | |
| - | On élargit le jeu d'entraînement avec des transformations des entrées qui préservent l'étiquette (recadrages aléatoires, retournements, petites rotations, variation de couleur pour les images, bruit pour l'audio). Une variété plus effective dans les données réduit directement la variance, c'est de la régularisation appliquée au jeu de données plutôt qu'aux poids. |
| - | |
| - | ### 9.4.3 Récapitulatif |
| - | |
| - | | Technique | Où elle agit | Effet | |
| - | | --- | --- | --- | |
| - | | $L_2$ (décroissance des poids) | coût via $\lambda W$ | rétrécit les poids, fonction plus lisse | |
| - | | $L_1$ | coût via $\lambda\,\operatorname{sign}(W)$ | poids parcimonieux, sélection de variables | |
| - | | Dropout | activations à l'entraînement | ensemble de sous-réseaux amincis | |
| - | | Arrêt précoce | boucle d'entraînement | maintient les poids près de l'initialisation | |
| - | | Augmentation de données | données d'entraînement | plus de variété, variance plus faible | |
| - | |
| - | *Remarque :* ces techniques se composent. Un réseau convolutif utilise couramment ensemble la décroissance des poids, le dropout et une forte augmentation de données. |
| - | |
| - | *Le surapprentissage étant maîtrisé, le module suivant construit une architecture dont le partage des poids est lui-même une forme de régularisation : le réseau convolutif.* |
| - | |
| - | --- |
| - | Suivant : [Réseaux convolutifs](/fr/Deep%20Learning/10%20Convolutional%20networks) · [Vue d'ensemble du cours](/fr/Deep%20Learning) |
| fr/Deep Learning/09 Regularization and dropout/overfitting-curves.png .. /dev/null | |
| fr/Deep Learning/10 Convolutional networks/residual-block.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 760 321" width="760" height="321" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="760" height="321" fill="#ffffff"/><text x="380.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">A residual block</text><rect x="40.0" y="144.0" width="90.0" height="52.0" rx="8" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><text x="85.0" y="174.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">input x</text><rect x="210.0" y="144.0" width="150.0" height="52.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="285.0" y="174.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">conv path F(x)</text><circle cx="470.0" cy="170.0" r="20.0" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="470.0" y="176.1" font-family="Helvetica, Arial, sans-serif" font-size="18" fill="#1f2933" text-anchor="middle">+</text><rect x="540.0" y="144.0" width="90.0" height="52.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="585.0" y="174.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">relu</text><rect x="660.0" y="144.0" width="80.0" height="52.0" rx="8" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><text x="700.0" y="174.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">output y</text><line x1="130.0" y1="170.0" x2="210.0" y2="170.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="360.0" y1="170.0" x2="450.0" y2="170.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="490.0" y1="170.0" x2="540.0" y2="170.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="630.0" y1="170.0" x2="660.0" y2="170.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><circle cx="160.0" cy="170.0" r="4.0" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><path d="M160.0 170.0 Q268.5 70.0 470.0 150.0" fill="none" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><text x="315.0" y="155.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">identity skip x</text><text x="285.0" y="222.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">learns a correction F(x)</text><text x="380.0" y="300.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">y = relu( F(x) + x )</text></svg> |
| \ | No newline at end of file |
| fr/Deep Learning/15 Transformers.md .. fr/Deep Learning/10 Transformers.md | |
| @@ 1,16 1,8 @@ | |
| - | # 15. Transformeurs |
| + | # 10. Transformeurs |
| - | Le Transformeur remplace la récurrence par la seule attention. Il traite en parallèle une séquence entière de plongements de tokens, ce qui permet à chaque token de porter attention à tous les autres au moyen de requêtes, de clés et de valeurs apprises. Cette leçon construit l'architecture à partir de l'auto-attention, en supposant connus les plongements (leçon 12) et le mécanisme d'attention (leçon 15), et elle réutilise la normalisation (leçon 8) et les connexions résiduelles (leçon 11). |
| + | Le Transformeur remplace la récurrence par la seule attention. Il traite en parallèle une séquence entière de plongements de tokens, ce qui permet à chaque token de porter attention à tous les autres au moyen de requêtes, de clés et de valeurs apprises. Cette leçon construit l'architecture à partir de l'auto-attention, en supposant connus les plongements (leçon 6) et le mécanisme d'attention (leçon 9), et elle réutilise la normalisation (leçon 4) et les connexions résiduelles (leçon 5). |
| - | **Objectifs** |
| - | - Projeter les plongements de tokens en requêtes $Q$, clés $K$ et valeurs $V$ à l'aide de matrices apprises. |
| - | - Définir l'attention par produit scalaire mis à l'échelle et expliquer le facteur d'échelle $1/\sqrt{d_k}$. |
| - | - Exécuter plusieurs têtes d'attention en parallèle et les combiner par l'attention multi-têtes. |
| - | - Injecter l'ordre dans une opération ensembliste au moyen d'encodages positionnels. |
| - | - Assembler un bloc de Transformeur à partir de connexions résiduelles et de la normalisation par couche. |
| - | - Placer le bloc dans la pile encodeur-décodeur et nommer ses variantes encodeur seul et décodeur seul. |
| - | |
| - | ## 15.1 Auto-attention et Q, K, V |
| + | ## 10.1 Auto-attention et Q, K, V |
| Une séquence de $n$ tokens est représentée par une matrice de plongements $X \in \mathbb{R}^{n \times d}$, une ligne par token. L'auto-attention permet à chaque token de recueillir de l'information auprès des autres en posant une question (une requête), en la comparant à l'étiquette de chaque token (une clé) et en en lisant le contenu (une valeur). | |
| @@ 20,7 12,11 @@ | |
| *Remarque :* les projections sont ici les seuls paramètres appris, et les trois mêmes matrices sont partagées entre toutes les positions. Comme un token est comparé à tous les autres, l'opération capture les dépendances à longue portée en une seule étape, contrairement à une récurrence qui doit propager l'information une position à la fois. | |
| - | ## 15.2 Attention par produit scalaire mis à l'échelle |
| + |  |
| + | |
| + | *D'une seule matrice d'entrée (ici quatre tokens, « nous mangeons du pain »), trois projections apprises donnent à chaque token sa requête, sa clé et sa valeur. Ce qu'elles alimentent est encore grisé.* |
| + | |
| + | ## 10.2 Attention par produit scalaire mis à l'échelle |
| Chaque requête est comparée à toutes les clés par un produit scalaire, ce qui donne une matrice $n \times n$ de scores bruts. Les scores sont mis à l'échelle, transformés en poids par un softmax appliqué ligne par ligne, puis utilisés pour moyenner les valeurs : | |
| @@ 28,7 24,11 @@ | |
| La ligne $i$ du softmax est une distribution de probabilité sur tous les tokens, si bien que la ligne $i$ de la sortie est une moyenne pondérée des vecteurs de valeurs, pondérée par la pertinence de chaque token vis-à-vis du token $i$. | |
| - | ### 15.2.1 Pourquoi diviser par $\sqrt{d_k}$ |
| + |  |
| + | |
| + | *Toute la couche en produits matriciels, formes comprises : $QK^T/\sqrt{d_k}$ compare chaque token à tous les autres ($n \times n$), la softmax par ligne transforme les scores en poids, et la multiplication par $V$ rend une ligne de sortie par token.* |
| + | |
| + | ### 10.2.1 Pourquoi diviser par $\sqrt{d_k}$ |
| Si les composantes de $q$ et $k$ sont indépendantes, de moyenne nulle et de variance unité, le produit scalaire $q^{T} k = \sum_{j=1}^{d_k} q_j k_j$ a une variance $d_k$, de sorte que sa magnitude typique croît comme $\sqrt{d_k}$. | |
| @@ 36,7 36,11 @@ | |
| Des scores élevés poussent le softmax dans un régime saturé où un poids est proche de $1$ et les autres proches de $0$, régime dans lequel le gradient du softmax est minuscule. Diviser par $\sqrt{d_k}$ maintient les logits à une échelle modérée, ce qui garde les gradients du softmax en bonne santé et stabilise l'entraînement. | |
| - | ## 15.3 Attention multi-têtes |
| + |  |
| + | |
| + | *L'unité jusqu'ici : une seule tête d'attention par produit scalaire normé, déjà à sa place. Le reste du bloc est encore grisé.* |
| + | |
| + | ## 10.3 Attention multi-têtes |
| Un unique calcul d'attention contraint chaque relation à être lue à travers un seul sous-espace de dimension $d_k$. L'attention multi-têtes exécute $h$ opérations d'attention en parallèle, chacune avec ses propres projections, de sorte que différentes têtes peuvent se spécialiser (l'une sur la syntaxe, une autre sur la coréférence, et ainsi de suite). | |
| @@ 50,7 54,11 @@ | |
| *Remarque :* la largeur par tête est habituellement fixée à $d_k = d_v = d / h$, si bien que la concaténation revient à la largeur $d$ et que le coût total égale celui d'une seule tête de pleine largeur. Les têtes sont indépendantes et calculées en parallèle, ce qui explique en partie pourquoi les Transformeurs s'entraînent efficacement sur le matériel moderne. | |
| - | ## 15.4 Encodage positionnel |
| + |  |
| + | |
| + | *Étape 2 : plusieurs têtes tournent en parallèle sur leurs propres projections, et leurs sorties sont concaténées puis mélangées par $W^O$. La sous-couche multi-têtes est complète.* |
| + | |
| + | ## 10.4 Encodage positionnel |
| L'attention traite son entrée comme un ensemble : permuter les lignes de $X$ permute la sortie de la même manière, l'opération est donc indifférente à l'ordre. Le langage ne l'est pas, la position doit donc être fournie explicitement. Le Transformeur original ajoute aux plongements un encodage sinusoïdal fixe, en utilisant une fréquence différente par dimension de caractéristique : | |
| @@ 58,29 66,33 @@ | |
| Ici, $pos$ est la position du token et $i$ indexe la dimension de caractéristique. Les dimensions basses varient rapidement avec la position et les dimensions hautes varient lentement, de sorte que le vecteur encode la position à travers de nombreuses échelles. L'encodage est ajouté au plongement du token avant le premier bloc. | |
| - |  |
| + |  |
| *Les encodages positionnels sinusoïdaux varient rapidement dans les dimensions basses et lentement dans les dimensions hautes, donnant à chaque position une signature multi-échelle unique.* | |
| *Remarque :* les sinusoïdes permettent d'écrire un décalage relatif $PE_{pos+k}$ comme une fonction linéaire de $PE_{pos}$, de sorte que le modèle peut apprendre à porter attention par décalage relatif. Les encodages sont fixes (non appris) et s'étendent à des longueurs de séquence non vues pendant l'entraînement. De nombreux modèles ultérieurs les remplacent par des schémas de position appris ou relatifs. | |
| - | ## 15.5 Le bloc de Transformeur |
| + |  |
| + | |
| + | *Étape 3 : le côté entrée. Les plongements de tokens entrent par une addition avec l'encodage positionnel, ce qui donne à l'attention son sens de l'ordre.* |
| + | |
| + | ## 10.5 Le bloc de Transformeur |
| Chaque sous-couche est enveloppée dans une connexion résiduelle suivie d'une normalisation par couche, ce qui maintient la circulation des gradients à travers des piles profondes et stabilise l'échelle des activations : | |
| $$\boxed{ x \leftarrow \mathrm{LayerNorm}\!\left(x + \mathrm{Sublayer}(x)\right) }$$ | |
| - |  |
| + |  |
| - | *Un bloc de Transformeur enveloppe une attention multi-têtes et un réseau à propagation avant, chacun dans une connexion résiduelle suivie d'une normalisation par couche.* |
| + | *Le bloc assemblé : l'auto-attention multi-têtes, puis le réseau à propagation avant par position, chacun enveloppé dans une connexion résiduelle et une normalisation par couche. Empilé $N$ fois, c'est le Transformeur.* |
| - | Un bloc enchaîne deux sous-couches selon ce motif. La première est l'auto-attention multi-têtes (les tokens échangent de l'information). La seconde est un réseau à propagation avant appliqué par position, un MLP à deux couches appliqué indépendamment à chaque position, avec la notation utilisée depuis la leçon 12 : |
| + | Un bloc enchaîne deux sous-couches selon ce motif. La première est l'auto-attention multi-têtes (les tokens échangent de l'information). La seconde est un réseau à propagation avant appliqué par position, un MLP à deux couches appliqué indépendamment à chaque position, avec la notation de la leçon 1 : |
| $$\boxed{ \mathrm{FFN}(x) = g\!\left(x W_1 + b_1\right) W_2 + b_2 }$$ | |
| avec une non-linéarité $g$ (ReLU ou GELU) et une largeur interne plusieurs fois plus grande que $d$. | |
| - | *Remarque :* la connexion résiduelle réutilise le raccourci identité de la leçon 11, de sorte que la sous-couche n'a qu'à apprendre une correction de son entrée. La normalisation par couche (leçon 8) normalise selon la dimension des caractéristiques pour chaque token, ce qui convient mieux aux séquences de longueur variable que la normalisation par lot. La forme ci-dessus correspond au placement post-norm original. De nombreuses implémentations modernes utilisent le pré-norm, $x \leftarrow x + \mathrm{Sublayer}(\mathrm{LayerNorm}(x))$, qui s'entraîne de façon plus stable à grande profondeur. |
| + | *Remarque :* la connexion résiduelle réutilise le raccourci identité de la leçon 5, de sorte que la sous-couche n'a qu'à apprendre une correction de son entrée. La normalisation par couche (leçon 4) normalise selon la dimension des caractéristiques pour chaque token, ce qui convient mieux aux séquences de longueur variable que la normalisation par lot. La forme ci-dessus correspond au placement post-norm original. De nombreuses implémentations modernes utilisent le pré-norm, $x \leftarrow x + \mathrm{Sublayer}(\mathrm{LayerNorm}(x))$, qui s'entraîne de façon plus stable à grande profondeur. |
| | Composant | Rôle | Agit selon | | |
| | --- | --- | --- | | |
| @@ 89,15 101,15 @@ | |
| | Connexion résiduelle | préserver un chemin de gradient | la profondeur | | |
| | Normalisation par couche | stabiliser l'échelle des activations | les caractéristiques par token | | |
| - | ## 15.6 L'architecture encodeur-décodeur |
| + | ## 10.6 L'architecture encodeur-décodeur |
| Le Transformeur complet empile $N$ blocs identiques dans un encodeur et $N$ dans un décodeur. L'encodeur associe la séquence d'entrée à un ensemble de vecteurs de contexte. Chaque bloc décodeur comporte trois sous-couches : une auto-attention masquée sur les tokens générés jusqu'ici (le masque bloque l'attention vers les positions futures), une attention croisée dont les requêtes proviennent du décodeur et dont les clés et les valeurs proviennent de la sortie de l'encodeur, et un réseau à propagation avant. Une dernière couche linéaire suivie d'un softmax transforme les états du sommet du décodeur en une distribution sur le vocabulaire. | |
| - |  |
| + |  |
| *Le Transformeur complet : une pile de blocs encodeurs et une pile de blocs décodeurs reliées par l'attention croisée.* | |
| - | ### 15.6.1 Variantes |
| + | ### 10.6.1 Variantes |
| Toutes les tâches n'ont pas besoin des deux moitiés. Deux familles dominent la pratique : | |
| fr/Deep Learning/15 Transformers/positional-encoding.png .. fr/Deep Learning/10 Transformers/positional-encoding.png | |
| /dev/null .. fr/Deep Learning/10 Transformers/selfattention-matrices.svg | |
| @@ 0,0 1,10 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 330" width="900" height="330" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker></defs> |
| + | <rect width="900" height="330" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">Self-attention as matrix products</text> |
| + | <rect x="88" y="110" width="72" height="88" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="88" y1="132.0" x2="160" y2="132.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="88" y1="154.0" x2="160" y2="154.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="88" y1="176.0" x2="160" y2="176.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="100.0" y1="110" x2="100.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="112.0" y1="110" x2="112.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="124.0" y1="110" x2="124.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="136.0" y1="110" x2="136.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="148.0" y1="110" x2="148.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="124.0" y="102" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">X</text><text x="124.0" y="212" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d</text><text x="82" y="125" font-size="9" fill="#5b6b7b" text-anchor="end">nous</text><text x="82" y="147" font-size="9" fill="#5b6b7b" text-anchor="end">mangeons</text><text x="82" y="169" font-size="9" fill="#5b6b7b" text-anchor="end">du</text><text x="82" y="191" font-size="9" fill="#5b6b7b" text-anchor="end">pain</text> |
| + | <rect x="208" y="77" width="50" height="26" rx="7" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.5"/><text x="233" y="94" font-size="11" fill="#1f2933" text-anchor="middle">W<tspan baseline-shift="super" font-size="7px">Q</tspan></text><line x1="160" y1="154" x2="206" y2="90" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="208" y="153" width="50" height="26" rx="7" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.5"/><text x="233" y="170" font-size="11" fill="#1f2933" text-anchor="middle">W<tspan baseline-shift="super" font-size="7px">K</tspan></text><line x1="160" y1="154" x2="206" y2="166" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="208" y="229" width="50" height="26" rx="7" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.5"/><text x="233" y="246" font-size="11" fill="#1f2933" text-anchor="middle">W<tspan baseline-shift="super" font-size="7px">V</tspan></text><line x1="160" y1="154" x2="206" y2="242" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/> |
| + | <line x1="258" y1="90" x2="292" y2="90" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="294" y="70" width="44" height="40" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="294" y1="80.0" x2="338" y2="80.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="90.0" x2="338" y2="90.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="100.0" x2="338" y2="100.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="308.6666666666667" y1="70" x2="308.6666666666667" y2="110" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="323.3333333333333" y1="70" x2="323.3333333333333" y2="110" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="316.0" y="62" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">Q</text><text x="316.0" y="124" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d<tspan baseline-shift="sub" font-size="7px">k</tspan></text><line x1="258" y1="166" x2="292" y2="166" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="294" y="146" width="44" height="40" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="294" y1="156.0" x2="338" y2="156.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="166.0" x2="338" y2="166.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="176.0" x2="338" y2="176.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="308.6666666666667" y1="146" x2="308.6666666666667" y2="186" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="323.3333333333333" y1="146" x2="323.3333333333333" y2="186" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="316.0" y="138" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">K</text><text x="316.0" y="200" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d<tspan baseline-shift="sub" font-size="7px">k</tspan></text><line x1="258" y1="242" x2="292" y2="242" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="294" y="222" width="44" height="40" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="294" y1="232.0" x2="338" y2="232.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="242.0" x2="338" y2="242.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="252.0" x2="338" y2="252.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="308.6666666666667" y1="222" x2="308.6666666666667" y2="262" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="323.3333333333333" y1="222" x2="323.3333333333333" y2="262" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="316.0" y="214" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">V</text><text x="316.0" y="276" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d<tspan baseline-shift="sub" font-size="7px">v</tspan></text> |
| + | <line x1="338" y1="90" x2="396" y2="116" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="338" y1="166" x2="396" y2="134" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="398" y="96" width="56" height="56" fill="#fff1e0" stroke="#e0872e" stroke-width="1.5"/><line x1="398" y1="110.0" x2="454" y2="110.0" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="398" y1="124.0" x2="454" y2="124.0" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="398" y1="138.0" x2="454" y2="138.0" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="412.0" y1="96" x2="412.0" y2="152" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="426.0" y1="96" x2="426.0" y2="152" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="440.0" y1="96" x2="440.0" y2="152" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><text x="426.0" y="88" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">QK​<tspan baseline-shift="super" font-size="7px">T</tspan>/√dₖ</text><text x="426.0" y="166" font-size="9" fill="#5b6b7b" text-anchor="middle">scores · n × n</text><line x1="454" y1="124" x2="490" y2="124" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="492" y="108" width="88" height="32" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.5"/><text x="536" y="122" font-size="11" fill="#1f2933" text-anchor="middle">softmax</text><text x="536" y="134" font-size="8" fill="#5b6b7b" text-anchor="middle">per row</text><line x1="580" y1="124" x2="616" y2="124" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="618" y="96" width="56" height="56" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.5"/><line x1="618" y1="110.0" x2="674" y2="110.0" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="618" y1="124.0" x2="674" y2="124.0" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="618" y1="138.0" x2="674" y2="138.0" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="632.0" y1="96" x2="632.0" y2="152" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="646.0" y1="96" x2="646.0" y2="152" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="660.0" y1="96" x2="660.0" y2="152" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><text x="646.0" y="88" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">A</text><text x="646.0" y="166" font-size="9" fill="#5b6b7b" text-anchor="middle">weights · rows sum to 1</text><line x1="674" y1="124" x2="712" y2="124" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><circle cx="726" cy="124" r="12" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="726" y="129" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">×</text><line x1="338" y1="242" x2="726" y2="242" stroke="#1f2933" stroke-width="1.3"/><line x1="726" y1="242" x2="726" y2="138" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="766" y="100" width="48" height="48" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="766" y1="112.0" x2="814" y2="112.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="766" y1="124.0" x2="814" y2="124.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="766" y1="136.0" x2="814" y2="136.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="782.0" y1="100" x2="782.0" y2="148" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="798.0" y1="100" x2="798.0" y2="148" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="790.0" y="92" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">Y</text><text x="790.0" y="162" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d<tspan baseline-shift="sub" font-size="7px">v</tspan></text><line x1="738" y1="124" x2="764" y2="124" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/> |
| + | <text x="450" y="316" font-size="11" fill="#5b6b7b" text-anchor="middle">n tokens as rows · one matrix product per step, the whole sequence at once</text> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. fr/Deep Learning/10 Transformers/selfattention-step1.svg | |
| @@ 0,0 1,10 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 330" width="900" height="330" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker></defs> |
| + | <rect width="900" height="330" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">Self-attention, step 1: three projections of the same input</text> |
| + | <rect x="88" y="110" width="72" height="88" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="88" y1="132.0" x2="160" y2="132.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="88" y1="154.0" x2="160" y2="154.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="88" y1="176.0" x2="160" y2="176.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="100.0" y1="110" x2="100.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="112.0" y1="110" x2="112.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="124.0" y1="110" x2="124.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="136.0" y1="110" x2="136.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="148.0" y1="110" x2="148.0" y2="198" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="124.0" y="102" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">X</text><text x="124.0" y="212" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d</text><text x="82" y="125" font-size="9" fill="#5b6b7b" text-anchor="end">nous</text><text x="82" y="147" font-size="9" fill="#5b6b7b" text-anchor="end">mangeons</text><text x="82" y="169" font-size="9" fill="#5b6b7b" text-anchor="end">du</text><text x="82" y="191" font-size="9" fill="#5b6b7b" text-anchor="end">pain</text> |
| + | <rect x="208" y="77" width="50" height="26" rx="7" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.5"/><text x="233" y="94" font-size="11" fill="#1f2933" text-anchor="middle">W<tspan baseline-shift="super" font-size="7px">Q</tspan></text><line x1="160" y1="154" x2="206" y2="90" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="208" y="153" width="50" height="26" rx="7" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.5"/><text x="233" y="170" font-size="11" fill="#1f2933" text-anchor="middle">W<tspan baseline-shift="super" font-size="7px">K</tspan></text><line x1="160" y1="154" x2="206" y2="166" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="208" y="229" width="50" height="26" rx="7" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.5"/><text x="233" y="246" font-size="11" fill="#1f2933" text-anchor="middle">W<tspan baseline-shift="super" font-size="7px">V</tspan></text><line x1="160" y1="154" x2="206" y2="242" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/> |
| + | <line x1="258" y1="90" x2="292" y2="90" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="294" y="70" width="44" height="40" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="294" y1="80.0" x2="338" y2="80.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="90.0" x2="338" y2="90.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="100.0" x2="338" y2="100.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="308.6666666666667" y1="70" x2="308.6666666666667" y2="110" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="323.3333333333333" y1="70" x2="323.3333333333333" y2="110" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="316.0" y="62" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">Q</text><text x="316.0" y="124" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d<tspan baseline-shift="sub" font-size="7px">k</tspan></text><line x1="258" y1="166" x2="292" y2="166" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="294" y="146" width="44" height="40" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="294" y1="156.0" x2="338" y2="156.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="166.0" x2="338" y2="166.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="176.0" x2="338" y2="176.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="308.6666666666667" y1="146" x2="308.6666666666667" y2="186" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="323.3333333333333" y1="146" x2="323.3333333333333" y2="186" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="316.0" y="138" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">K</text><text x="316.0" y="200" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d<tspan baseline-shift="sub" font-size="7px">k</tspan></text><line x1="258" y1="242" x2="292" y2="242" stroke="#1f2933" stroke-width="1.2" marker-end="url(#arrow)"/><rect x="294" y="222" width="44" height="40" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="294" y1="232.0" x2="338" y2="232.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="242.0" x2="338" y2="242.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="294" y1="252.0" x2="338" y2="252.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="308.6666666666667" y1="222" x2="308.6666666666667" y2="262" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="323.3333333333333" y1="222" x2="323.3333333333333" y2="262" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="316.0" y="214" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">V</text><text x="316.0" y="276" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d<tspan baseline-shift="sub" font-size="7px">v</tspan></text> |
| + | <g opacity="0.22"><line x1="338" y1="90" x2="396" y2="116" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><line x1="338" y1="166" x2="396" y2="134" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="398" y="96" width="56" height="56" fill="#fff1e0" stroke="#e0872e" stroke-width="1.5"/><line x1="398" y1="110.0" x2="454" y2="110.0" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="398" y1="124.0" x2="454" y2="124.0" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="398" y1="138.0" x2="454" y2="138.0" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="412.0" y1="96" x2="412.0" y2="152" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="426.0" y1="96" x2="426.0" y2="152" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><line x1="440.0" y1="96" x2="440.0" y2="152" stroke="#e0872e" stroke-width="0.6" opacity="0.55"/><text x="426.0" y="88" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">QK​<tspan baseline-shift="super" font-size="7px">T</tspan>/√dₖ</text><text x="426.0" y="166" font-size="9" fill="#5b6b7b" text-anchor="middle">scores · n × n</text><line x1="454" y1="124" x2="490" y2="124" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="492" y="108" width="88" height="32" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.5"/><text x="536" y="122" font-size="11" fill="#1f2933" text-anchor="middle">softmax</text><text x="536" y="134" font-size="8" fill="#5b6b7b" text-anchor="middle">per row</text><line x1="580" y1="124" x2="616" y2="124" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="618" y="96" width="56" height="56" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.5"/><line x1="618" y1="110.0" x2="674" y2="110.0" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="618" y1="124.0" x2="674" y2="124.0" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="618" y1="138.0" x2="674" y2="138.0" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="632.0" y1="96" x2="632.0" y2="152" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="646.0" y1="96" x2="646.0" y2="152" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><line x1="660.0" y1="96" x2="660.0" y2="152" stroke="#38a05a" stroke-width="0.6" opacity="0.55"/><text x="646.0" y="88" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">A</text><text x="646.0" y="166" font-size="9" fill="#5b6b7b" text-anchor="middle">weights · rows sum to 1</text><line x1="674" y1="124" x2="712" y2="124" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><circle cx="726" cy="124" r="12" fill="#ffffff" stroke="#1f2933" stroke-width="1.5"/><text x="726" y="129" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">×</text><line x1="338" y1="242" x2="726" y2="242" stroke="#1f2933" stroke-width="1.3"/><line x1="726" y1="242" x2="726" y2="138" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/><rect x="766" y="100" width="48" height="48" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.5"/><line x1="766" y1="112.0" x2="814" y2="112.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="766" y1="124.0" x2="814" y2="124.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="766" y1="136.0" x2="814" y2="136.0" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="782.0" y1="100" x2="782.0" y2="148" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><line x1="798.0" y1="100" x2="798.0" y2="148" stroke="#3b6fb6" stroke-width="0.6" opacity="0.55"/><text x="790.0" y="92" font-size="12" fill="#1f2933" text-anchor="middle" font-weight="600">Y</text><text x="790.0" y="162" font-size="9" fill="#5b6b7b" text-anchor="middle">n × d<tspan baseline-shift="sub" font-size="7px">v</tspan></text><line x1="738" y1="124" x2="764" y2="124" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/></g> |
| + | <text x="450" y="316" font-size="11" fill="#5b6b7b" text-anchor="middle">n tokens as rows · one matrix product per step, the whole sequence at once</text> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. fr/Deep Learning/10 Transformers/transformer-block.svg | |
| @@ 0,0 1,30 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 545" width="900" height="545" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | <marker id="arrowmut" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs> |
| + | <rect width="900" height="545" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">The Transformer block, assembled</text> |
| + | <rect x="280" y="110" width="340" height="330" rx="14" fill="#f9fafb" stroke="#c7d0d9" stroke-width="1.2"/> |
| + | <rect x="340" y="330" width="220" height="86" rx="10" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="450" y="345" font-size="10" fill="#1f2933" text-anchor="middle">multi-head self-attention</text> |
| + | <rect x="352" y="352" width="196" height="20" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.2"/><text x="450" y="366" font-size="9" fill="#1f2933" text-anchor="middle">concat, then W<tspan baseline-shift="super" font-size="7px">O</tspan></text> |
| + | <rect x="352" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="380" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="380" y1="382" x2="380" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <rect x="422" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="450" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="450" y1="382" x2="450" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <rect x="492" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="520" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="520" y1="382" x2="520" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <line x1="628" y1="392" x2="552" y2="392" stroke="#5b6b7b" stroke-width="1" stroke-dasharray="3 3"/><text x="634" y="389" font-size="10" fill="#5b6b7b" text-anchor="start">one head = scaled dot-product attention:</text><text x="634" y="403" font-size="10" fill="#5b6b7b" text-anchor="start">softmax(QK<tspan baseline-shift="super" font-size="7px">T</tspan>/√dₖ) V</text> |
| + | <rect x="365" y="492" width="170" height="30" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="450.0" y="511.0" font-size="11" fill="#1f2933" text-anchor="middle">token embeddings X</text> |
| + | <line x1="450" y1="492" x2="450" y2="470" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <circle cx="450" cy="455" r="12" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="450" y="460" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">+</text> |
| + | <rect x="560" y="441" width="170" height="28" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="645.0" y="459.0" font-size="10" fill="#1f2933" text-anchor="middle">positional encoding</text> |
| + | <line x1="560" y1="455" x2="465" y2="455" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <line x1="450" y1="443" x2="450" y2="418" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <rect x="350" y="286" width="200" height="26" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="450.0" y="303.0" font-size="10" fill="#1f2933" text-anchor="middle">add & norm</text> |
| + | <rect x="350" y="205" width="200" height="52" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="450.0" y="229.0" font-size="11" fill="#1f2933" text-anchor="middle">feed-forward network</text><text x="450.0" y="242.0" font-size="9" fill="#5b6b7b" text-anchor="middle">two-layer MLP, per position</text> |
| + | <rect x="350" y="150" width="200" height="26" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="450.0" y="167.0" font-size="10" fill="#1f2933" text-anchor="middle">add & norm</text> |
| + | <line x1="450" y1="330" x2="450" y2="314" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <line x1="450" y1="286" x2="450" y2="259" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <line x1="450" y1="205" x2="450" y2="178" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <line x1="450" y1="150" x2="450" y2="80" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="450" y="66" font-size="11" fill="#5b6b7b" text-anchor="middle">to the next block</text> |
| + | <path d="M452,430 C606,430 606,299 554,299" fill="none" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/> |
| + | <path d="M452,268 C606,268 606,163 554,163" fill="none" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/> |
| + | <text x="648" y="240" font-size="10" fill="#5b6b7b" text-anchor="start">residual shortcuts</text> |
| + | <text x="648" y="130" font-size="11" fill="#5b6b7b" text-anchor="start">× N identical blocks</text> |
| + | </svg> |
| \ | No newline at end of file |
| fr/Deep Learning/15 Transformers/transformer-stack.svg .. fr/Deep Learning/10 Transformers/transformer-stack.svg | |
| /dev/null .. fr/Deep Learning/10 Transformers/transformer-step1.svg | |
| @@ 0,0 1,30 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 545" width="900" height="545" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | <marker id="arrowmut" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs> |
| + | <rect width="900" height="545" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">The Transformer block, step 1: one attention head</text> |
| + | <g opacity="0.22"><rect x="280" y="110" width="340" height="330" rx="14" fill="#f9fafb" stroke="#c7d0d9" stroke-width="1.2"/></g> |
| + | <g opacity="0.22"><rect x="340" y="330" width="220" height="86" rx="10" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="450" y="345" font-size="10" fill="#1f2933" text-anchor="middle">multi-head self-attention</text></g> |
| + | <g opacity="0.22"><rect x="352" y="352" width="196" height="20" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.2"/><text x="450" y="366" font-size="9" fill="#1f2933" text-anchor="middle">concat, then W<tspan baseline-shift="super" font-size="7px">O</tspan></text></g> |
| + | <g opacity="0.22"><rect x="352" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="380" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="380" y1="382" x2="380" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/></g> |
| + | <rect x="422" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="450" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="450" y1="382" x2="450" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <g opacity="0.22"><rect x="492" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="520" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="520" y1="382" x2="520" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/></g> |
| + | <line x1="628" y1="392" x2="552" y2="392" stroke="#5b6b7b" stroke-width="1" stroke-dasharray="3 3"/><text x="634" y="389" font-size="10" fill="#5b6b7b" text-anchor="start">one head = scaled dot-product attention:</text><text x="634" y="403" font-size="10" fill="#5b6b7b" text-anchor="start">softmax(QK<tspan baseline-shift="super" font-size="7px">T</tspan>/√dₖ) V</text> |
| + | <g opacity="0.22"><rect x="365" y="492" width="170" height="30" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="450.0" y="511.0" font-size="11" fill="#1f2933" text-anchor="middle">token embeddings X</text></g> |
| + | <g opacity="0.22"><line x1="450" y1="492" x2="450" y2="470" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><circle cx="450" cy="455" r="12" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="450" y="460" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">+</text></g> |
| + | <g opacity="0.22"><rect x="560" y="441" width="170" height="28" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="645.0" y="459.0" font-size="10" fill="#1f2933" text-anchor="middle">positional encoding</text></g> |
| + | <g opacity="0.22"><line x1="560" y1="455" x2="465" y2="455" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="443" x2="450" y2="418" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><rect x="350" y="286" width="200" height="26" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="450.0" y="303.0" font-size="10" fill="#1f2933" text-anchor="middle">add & norm</text></g> |
| + | <g opacity="0.22"><rect x="350" y="205" width="200" height="52" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="450.0" y="229.0" font-size="11" fill="#1f2933" text-anchor="middle">feed-forward network</text><text x="450.0" y="242.0" font-size="9" fill="#5b6b7b" text-anchor="middle">two-layer MLP, per position</text></g> |
| + | <g opacity="0.22"><rect x="350" y="150" width="200" height="26" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="450.0" y="167.0" font-size="10" fill="#1f2933" text-anchor="middle">add & norm</text></g> |
| + | <g opacity="0.22"><line x1="450" y1="330" x2="450" y2="314" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="286" x2="450" y2="259" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="205" x2="450" y2="178" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="150" x2="450" y2="80" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="450" y="66" font-size="11" fill="#5b6b7b" text-anchor="middle">to the next block</text></g> |
| + | <g opacity="0.22"><path d="M452,430 C606,430 606,299 554,299" fill="none" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><path d="M452,268 C606,268 606,163 554,163" fill="none" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><text x="648" y="240" font-size="10" fill="#5b6b7b" text-anchor="start">residual shortcuts</text></g> |
| + | <g opacity="0.22"><text x="648" y="130" font-size="11" fill="#5b6b7b" text-anchor="start">× N identical blocks</text></g> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. fr/Deep Learning/10 Transformers/transformer-step2.svg | |
| @@ 0,0 1,30 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 545" width="900" height="545" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | <marker id="arrowmut" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs> |
| + | <rect width="900" height="545" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">The Transformer block, step 2: multi-head attention</text> |
| + | <g opacity="0.22"><rect x="280" y="110" width="340" height="330" rx="14" fill="#f9fafb" stroke="#c7d0d9" stroke-width="1.2"/></g> |
| + | <rect x="340" y="330" width="220" height="86" rx="10" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="450" y="345" font-size="10" fill="#1f2933" text-anchor="middle">multi-head self-attention</text> |
| + | <rect x="352" y="352" width="196" height="20" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.2"/><text x="450" y="366" font-size="9" fill="#1f2933" text-anchor="middle">concat, then W<tspan baseline-shift="super" font-size="7px">O</tspan></text> |
| + | <rect x="352" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="380" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="380" y1="382" x2="380" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <rect x="422" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="450" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="450" y1="382" x2="450" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <rect x="492" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="520" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="520" y1="382" x2="520" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <line x1="628" y1="392" x2="552" y2="392" stroke="#5b6b7b" stroke-width="1" stroke-dasharray="3 3"/><text x="634" y="389" font-size="10" fill="#5b6b7b" text-anchor="start">one head = scaled dot-product attention:</text><text x="634" y="403" font-size="10" fill="#5b6b7b" text-anchor="start">softmax(QK<tspan baseline-shift="super" font-size="7px">T</tspan>/√dₖ) V</text> |
| + | <g opacity="0.22"><rect x="365" y="492" width="170" height="30" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="450.0" y="511.0" font-size="11" fill="#1f2933" text-anchor="middle">token embeddings X</text></g> |
| + | <g opacity="0.22"><line x1="450" y1="492" x2="450" y2="470" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><circle cx="450" cy="455" r="12" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="450" y="460" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">+</text></g> |
| + | <g opacity="0.22"><rect x="560" y="441" width="170" height="28" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="645.0" y="459.0" font-size="10" fill="#1f2933" text-anchor="middle">positional encoding</text></g> |
| + | <g opacity="0.22"><line x1="560" y1="455" x2="465" y2="455" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="443" x2="450" y2="418" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><rect x="350" y="286" width="200" height="26" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="450.0" y="303.0" font-size="10" fill="#1f2933" text-anchor="middle">add & norm</text></g> |
| + | <g opacity="0.22"><rect x="350" y="205" width="200" height="52" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="450.0" y="229.0" font-size="11" fill="#1f2933" text-anchor="middle">feed-forward network</text><text x="450.0" y="242.0" font-size="9" fill="#5b6b7b" text-anchor="middle">two-layer MLP, per position</text></g> |
| + | <g opacity="0.22"><rect x="350" y="150" width="200" height="26" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="450.0" y="167.0" font-size="10" fill="#1f2933" text-anchor="middle">add & norm</text></g> |
| + | <g opacity="0.22"><line x1="450" y1="330" x2="450" y2="314" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="286" x2="450" y2="259" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="205" x2="450" y2="178" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="150" x2="450" y2="80" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="450" y="66" font-size="11" fill="#5b6b7b" text-anchor="middle">to the next block</text></g> |
| + | <g opacity="0.22"><path d="M452,430 C606,430 606,299 554,299" fill="none" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><path d="M452,268 C606,268 606,163 554,163" fill="none" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><text x="648" y="240" font-size="10" fill="#5b6b7b" text-anchor="start">residual shortcuts</text></g> |
| + | <g opacity="0.22"><text x="648" y="130" font-size="11" fill="#5b6b7b" text-anchor="start">× N identical blocks</text></g> |
| + | </svg> |
| \ | No newline at end of file |
| /dev/null .. fr/Deep Learning/10 Transformers/transformer-step3.svg | |
| @@ 0,0 1,30 @@ | |
| + | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 545" width="900" height="545" font-family="Helvetica, Arial, sans-serif"> |
| + | <defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker> |
| + | <marker id="arrowmut" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs> |
| + | <rect width="900" height="545" fill="#ffffff"/> |
| + | <text x="450" y="24" font-size="15" fill="#1f2933" text-anchor="middle" font-weight="600">The Transformer block, step 3: embeddings and positions enter</text> |
| + | <g opacity="0.22"><rect x="280" y="110" width="340" height="330" rx="14" fill="#f9fafb" stroke="#c7d0d9" stroke-width="1.2"/></g> |
| + | <rect x="340" y="330" width="220" height="86" rx="10" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="450" y="345" font-size="10" fill="#1f2933" text-anchor="middle">multi-head self-attention</text> |
| + | <rect x="352" y="352" width="196" height="20" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.2"/><text x="450" y="366" font-size="9" fill="#1f2933" text-anchor="middle">concat, then W<tspan baseline-shift="super" font-size="7px">O</tspan></text> |
| + | <rect x="352" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="380" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="380" y1="382" x2="380" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <rect x="422" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="450" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="450" y1="382" x2="450" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <rect x="492" y="382" width="56" height="24" rx="6" fill="#ffffff" stroke="#8257d6" stroke-width="1.3"/><text x="520" y="398" font-size="10" fill="#1f2933" text-anchor="middle">head</text><line x1="520" y1="382" x2="520" y2="374" stroke="#1f2933" stroke-width="1.1" marker-end="url(#arrow)"/> |
| + | <line x1="628" y1="392" x2="552" y2="392" stroke="#5b6b7b" stroke-width="1" stroke-dasharray="3 3"/><text x="634" y="389" font-size="10" fill="#5b6b7b" text-anchor="start">one head = scaled dot-product attention:</text><text x="634" y="403" font-size="10" fill="#5b6b7b" text-anchor="start">softmax(QK<tspan baseline-shift="super" font-size="7px">T</tspan>/√dₖ) V</text> |
| + | <rect x="365" y="492" width="170" height="30" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="450.0" y="511.0" font-size="11" fill="#1f2933" text-anchor="middle">token embeddings X</text> |
| + | <line x1="450" y1="492" x2="450" y2="470" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <circle cx="450" cy="455" r="12" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="450" y="460" font-size="13" fill="#1f2933" text-anchor="middle" font-weight="600">+</text> |
| + | <rect x="560" y="441" width="170" height="28" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="645.0" y="459.0" font-size="10" fill="#1f2933" text-anchor="middle">positional encoding</text> |
| + | <line x1="560" y1="455" x2="465" y2="455" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <line x1="450" y1="443" x2="450" y2="418" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/> |
| + | <g opacity="0.22"><rect x="350" y="286" width="200" height="26" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="450.0" y="303.0" font-size="10" fill="#1f2933" text-anchor="middle">add & norm</text></g> |
| + | <g opacity="0.22"><rect x="350" y="205" width="200" height="52" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="450.0" y="229.0" font-size="11" fill="#1f2933" text-anchor="middle">feed-forward network</text><text x="450.0" y="242.0" font-size="9" fill="#5b6b7b" text-anchor="middle">two-layer MLP, per position</text></g> |
| + | <g opacity="0.22"><rect x="350" y="150" width="200" height="26" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="450.0" y="167.0" font-size="10" fill="#1f2933" text-anchor="middle">add & norm</text></g> |
| + | <g opacity="0.22"><line x1="450" y1="330" x2="450" y2="314" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="286" x2="450" y2="259" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="205" x2="450" y2="178" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><line x1="450" y1="150" x2="450" y2="80" stroke="#1f2933" stroke-width="1.5" marker-end="url(#arrow)"/><text x="450" y="66" font-size="11" fill="#5b6b7b" text-anchor="middle">to the next block</text></g> |
| + | <g opacity="0.22"><path d="M452,430 C606,430 606,299 554,299" fill="none" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><path d="M452,268 C606,268 606,163 554,163" fill="none" stroke="#1f2933" stroke-width="1.3" marker-end="url(#arrow)"/></g> |
| + | <g opacity="0.22"><text x="648" y="240" font-size="10" fill="#5b6b7b" text-anchor="start">residual shortcuts</text></g> |
| + | <g opacity="0.22"><text x="648" y="130" font-size="11" fill="#5b6b7b" text-anchor="start">× N identical blocks</text></g> |
| + | </svg> |
| \ | No newline at end of file |
| fr/Deep Learning/13 LSTM and GRU/gru-cell.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 760 446" width="760" height="446" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="760" height="446" fill="#ffffff"/><text x="380.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">Inside a GRU cell</text><text x="50.0" y="84.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text><line x1="30.0" y1="100.0" x2="690.0" y2="100.0" stroke="#1f2933" stroke-width="2.2"/><line x1="670.0" y1="100.0" x2="690.0" y2="100.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><text x="716.0" y="104.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="9px">t</tspan></text><circle cx="300.0" cy="100.0" r="15.0" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="300.0" y="105.0" font-family="Helvetica, Arial, sans-serif" font-size="16" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><text x="300.0" y="72.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">keep 1-z</text><circle cx="500.0" cy="100.0" r="15.0" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="500.0" y="105.0" font-family="Helvetica, Arial, sans-serif" font-size="16" font-weight="600" fill="#1f2933" text-anchor="middle">+</text><text x="500.0" y="72.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">combine</text><circle cx="500.0" cy="200.0" r="15.0" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="500.0" y="205.0" font-family="Helvetica, Arial, sans-serif" font-size="16" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><text x="500.0" y="176.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">write z</text><circle cx="200.0" cy="250.0" r="15.0" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="200.0" y="255.0" font-family="Helvetica, Arial, sans-serif" font-size="16" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><text x="158.0" y="254.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="end">reset r</text><rect x="60.0" y="320.0" width="122.0" height="52.0" rx="8" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="121.0" y="350.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">r</tspan></text><rect x="300.0" y="320.0" width="122.0" height="52.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="361.0" y="350.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">z</tspan></text><rect x="560.0" y="320.0" width="122.0" height="52.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="621.0" y="350.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">tanh cand</text><text x="55.0" y="425.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text><text x="135.0" y="425.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="9px">t</tspan></text><line x1="55.0" y1="413.0" x2="100.0" y2="403.0" stroke="#5b6b7b" stroke-width="1.4"/><line x1="135.0" y1="413.0" x2="100.0" y2="403.0" stroke="#5b6b7b" stroke-width="1.4"/><path d="M100.0 399.0 Q110.5 356.0 121.0 372.0" fill="none" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><path d="M100.0 399.0 Q230.5 356.0 361.0 372.0" fill="none" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><path d="M100.0 399.0 Q360.5 356.0 621.0 372.0" fill="none" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><line x1="121.0" y1="320.0" x2="200.0" y2="265.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><path d="M300.0 100.0 Q240.0 175.0 210.0 236.0" fill="none" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><path d="M200.0 250.0 Q360.0 300.0 601.0 320.0" fill="none" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><path d="M621.0 320.0 Q560.0 260.0 512.0 214.0" fill="none" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="361.0" y1="320.0" x2="490.0" y2="214.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><path d="M361.0 320.0 Q320.0 240.0 300.0 115.0" fill="none" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="500.0" y1="185.0" x2="500.0" y2="115.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/></svg> |
| \ | No newline at end of file |
| fr/Deep Learning/13 LSTM and GRU/lstm-cell.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 840 473" width="840" height="473" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="840" height="473" fill="#ffffff"/><text x="420.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">Inside an LSTM cell</text><text x="50.0" y="79.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">c<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text><line x1="30.0" y1="95.0" x2="770.0" y2="95.0" stroke="#1f2933" stroke-width="2.2"/><line x1="750.0" y1="95.0" x2="770.0" y2="95.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><text x="796.0" y="99.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">c<tspan baseline-shift="sub" font-size="9px">t</tspan></text><circle cx="250.0" cy="95.0" r="15.0" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="250.0" y="100.0" font-family="Helvetica, Arial, sans-serif" font-size="16" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><text x="250.0" y="67.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">forget</text><circle cx="470.0" cy="95.0" r="15.0" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="470.0" y="100.0" font-family="Helvetica, Arial, sans-serif" font-size="16" font-weight="600" fill="#1f2933" text-anchor="middle">+</text><text x="470.0" y="67.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">input add</text><circle cx="470.0" cy="195.0" r="15.0" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="470.0" y="200.0" font-family="Helvetica, Arial, sans-serif" font-size="16" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><rect x="190.0" y="315.0" width="122.0" height="54.0" rx="8" fill="#fdecec" stroke="#d1495b" stroke-width="1.6"/><text x="251.0" y="346.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">f</tspan></text><rect x="342.0" y="315.0" width="122.0" height="54.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="403.0" y="346.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">i</tspan></text><rect x="590.0" y="315.0" width="122.0" height="54.0" rx="8" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="651.0" y="346.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">σ<tspan baseline-shift="sub" font-size="9px">o</tspan></text><rect x="342.0" y="398.0" width="122.0" height="48.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="403.0" y="426.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">tanh cand</text><text x="70.0" y="452.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="9px">t-1</tspan></text><text x="150.0" y="452.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">x<tspan baseline-shift="sub" font-size="9px">t</tspan></text><line x1="70.0" y1="440.0" x2="118.0" y2="430.0" stroke="#5b6b7b" stroke-width="1.4"/><line x1="150.0" y1="440.0" x2="118.0" y2="430.0" stroke="#5b6b7b" stroke-width="1.4"/><path d="M118.0 426.0 Q184.5 351.0 251.0 369.0" fill="none" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><path d="M118.0 426.0 Q260.5 351.0 403.0 369.0" fill="none" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><path d="M118.0 426.0 Q384.5 351.0 651.0 369.0" fill="none" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><path d="M118.0 426.0 Q260.5 412.0 403.0 446.0" fill="none" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><line x1="251.0" y1="315.0" x2="250.0" y2="110.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="403.0" y1="315.0" x2="458.0" y2="210.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="403.0" y1="398.0" x2="482.0" y2="210.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="470.0" y1="180.0" x2="470.0" y2="110.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><circle cx="720.0" cy="153.0" r="15.0" fill="#f2e9fd" stroke="#8257d6" stroke-width="1.6"/><text x="720.0" y="158.0" font-family="Helvetica, Arial, sans-serif" font-size="16" font-weight="600" fill="#1f2933" text-anchor="middle">⊙</text><rect x="590.0" y="200.0" width="122.0" height="46.0" rx="8" fill="#e4f5f4" stroke="#2a9d9a" stroke-width="1.6"/><text x="651.0" y="227.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">tanh</text><line x1="651.0" y1="95.0" x2="651.0" y2="200.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><path d="M651.0 200.0 Q675.0 187.0 707.0 159.0" fill="none" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="651.0" y1="315.0" x2="720.0" y2="168.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="720.0" y1="153.0" x2="770.0" y2="153.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><text x="796.0" y="157.0" font-family="Helvetica, Arial, sans-serif" font-size="13" font-weight="600" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="9px">t</tspan></text></svg> |
| \ | No newline at end of file |
| fr/Deep Learning/14 Attention/attention-weights.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 760 404" width="760" height="404" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="760" height="404" fill="#ffffff"/><text x="380.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">Attention: context is a weighted sum of encoder states</text><rect x="60.0" y="300.0" width="110.0" height="44.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="115.0" y="326.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="9px">1</tspan></text><rect x="230.0" y="300.0" width="110.0" height="44.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="285.0" y="326.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="9px">2</tspan></text><rect x="400.0" y="300.0" width="110.0" height="44.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="455.0" y="326.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="9px">3</tspan></text><rect x="570.0" y="300.0" width="110.0" height="44.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="625.0" y="326.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">h<tspan baseline-shift="sub" font-size="9px">4</tspan></text><text x="370.0" y="362.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">encoder states</text><text x="370.0" y="384.0" font-family="Helvetica, Arial, sans-serif" font-size="11" font-style="italic" fill="#5b6b7b" text-anchor="middle">thicker line = larger weight</text><circle cx="380.0" cy="175.0" r="34.0" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="380.0" y="180.1" font-family="Helvetica, Arial, sans-serif" font-size="15" fill="#1f2933" text-anchor="middle">c</text><text x="454.0" y="175.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">context</text><circle cx="380.0" cy="60.0" r="30.0" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="380.0" y="65.1" font-family="Helvetica, Arial, sans-serif" font-size="15" fill="#1f2933" text-anchor="middle">s</text><text x="465.0" y="60.0" font-family="Helvetica, Arial, sans-serif" font-size="12" fill="#5b6b7b" text-anchor="middle">decoder query</text><line x1="380.0" y1="90.0" x2="380.0" y2="141.0" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><line x1="115.0" y1="300.0" x2="380.0" y2="209.0" stroke="#38a05a" stroke-width="2.2800000000000002"/><text x="173.3" y="272.0" font-family="Helvetica, Arial, sans-serif" font-size="12" font-weight="600" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="9px">1</tspan></text><line x1="285.0" y1="300.0" x2="380.0" y2="209.0" stroke="#38a05a" stroke-width="6.15"/><text x="305.9" y="272.0" font-family="Helvetica, Arial, sans-serif" font-size="12" font-weight="600" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="9px">2</tspan></text><line x1="455.0" y1="300.0" x2="380.0" y2="209.0" stroke="#38a05a" stroke-width="3.45"/><text x="438.5" y="272.0" font-family="Helvetica, Arial, sans-serif" font-size="12" font-weight="600" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="9px">3</tspan></text><line x1="625.0" y1="300.0" x2="380.0" y2="209.0" stroke="#38a05a" stroke-width="1.92"/><text x="571.1" y="272.0" font-family="Helvetica, Arial, sans-serif" font-size="12" font-weight="600" fill="#1f2933" text-anchor="middle">α<tspan baseline-shift="sub" font-size="9px">4</tspan></text></svg> |
| \ | No newline at end of file |
| fr/Deep Learning/15 Transformers/transformer-block.svg .. /dev/null | |
| @@ 1,1 0,0 @@ | |
| - | <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 560 600" width="560" height="600" font-family="Helvetica, Arial, sans-serif"><defs><marker id="arrow" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#1f2933"/></marker><marker id="arrowmuted" viewBox="0 0 10 10" refX="8.5" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0 0L10 5L0 10z" fill="#5b6b7b"/></marker></defs><rect width="560" height="600" fill="#ffffff"/><text x="280.0" y="22.0" font-family="Helvetica, Arial, sans-serif" font-size="15" font-weight="600" fill="#1f2933" text-anchor="middle">A Transformer block</text><rect x="130.0" y="50.0" width="240.0" height="46.0" rx="8" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><text x="250.0" y="77.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">output</text><rect x="130.0" y="140.0" width="240.0" height="46.0" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="250.0" y="167.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">Add and Norm</text><rect x="130.0" y="240.0" width="240.0" height="46.0" rx="8" fill="#fff1e0" stroke="#e0872e" stroke-width="1.6"/><text x="250.0" y="267.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">Feed Forward</text><rect x="130.0" y="340.0" width="240.0" height="46.0" rx="8" fill="#e7f5ea" stroke="#38a05a" stroke-width="1.6"/><text x="250.0" y="367.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">Add and Norm</text><rect x="130.0" y="440.0" width="240.0" height="46.0" rx="8" fill="#e8f0fe" stroke="#3b6fb6" stroke-width="1.6"/><text x="250.0" y="467.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">Multi-Head Attention</text><rect x="130.0" y="530.0" width="240.0" height="46.0" rx="8" fill="#eef1f4" stroke="#9aa7b2" stroke-width="1.6"/><text x="250.0" y="557.4" font-family="Helvetica, Arial, sans-serif" font-size="13" fill="#1f2933" text-anchor="middle">input</text><line x1="250.0" y1="530.0" x2="250.0" y2="486.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="250.0" y1="440.0" x2="250.0" y2="386.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="250.0" y1="340.0" x2="250.0" y2="286.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="250.0" y1="240.0" x2="250.0" y2="186.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><line x1="250.0" y1="140.0" x2="250.0" y2="96.0" stroke="#1f2933" stroke-width="1.7" marker-end="url(#arrow)"/><path d="M370.0 494.0 Q450.0 428.5 370.0 363.0" fill="none" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><text x="468.0" y="413.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">residual</text><path d="M370.0 294.0 Q450.0 228.5 370.0 163.0" fill="none" stroke="#5b6b7b" stroke-width="1.7" marker-end="url(#arrowmuted)"/><text x="468.0" y="213.0" font-family="Helvetica, Arial, sans-serif" font-size="11" fill="#5b6b7b" text-anchor="middle">residual</text></svg> |
| \ | No newline at end of file |
| fr/MLOps/01 Introduction.md .. | |
| @@ 2,11 2,6 @@ | |
| > Leçon 01 · MLOps | |
| - | **Objectifs** |
| - | - Comprendre ce qu'est le MLOps et pourquoi un bon score ne suffit pas. |
| - | - Voir en quoi les systèmes de ML diffèrent du logiciel classique. |
| - | - Découvrir le cycle de vie sur lequel s'appuie le reste du cours. |
| - | |
| ## Qu'est-ce que le MLOps ? | |
| Le MLOps applique les pratiques d'ingénierie et d'exploitation au cycle de vie du machine | |
| fr/Machine Learning/05 Linear classification.md .. | |
| @@ 60,7 60,7 @@ | |
| La variante par lots calcule le gradient sur tout l'ensemble d'entraînement avant chaque pas, une descente lisse qui relit chaque exemple à chaque fois. La variante stochastique (SGD) avance sur un exemple à la fois, peu coûteuse et bruitée, et c'est le choix par défaut sur les grands jeux de données. Si $\alpha$ est trop grand les itérés peuvent diverger, s'il est trop petit la convergence se traîne. | |
| - | *Remarque :* des optimiseurs plus élaborés existent, momentum, Adam et leurs cousins, des raffinements de cette même règle qui comptent pour les réseaux profonds ([Optimisation](/fr/Deep%20Learning/06%20Optimization) dans le cours de Deep Learning). Tout ce module se contente de la version simple. |
| + | *Remarque :* des optimiseurs plus élaborés existent, momentum, Adam et leurs cousins, des raffinements de cette même règle qui comptent pour les réseaux profonds ([Optimisation](/fr/Deep%20Learning/03%20Optimization) dans le cours de Deep Learning). Tout ce module se contente de la version simple. |
| Sur un seul exemple mal classé le gradient du critère vaut $-y^{(i)} x^{(i)}$, le pas stochastique est donc la mise à jour du perceptron : sur une erreur, | |
| fr/Machine Learning/06 Multilayer neural networks.md .. | |
| @@ 1,6 1,6 @@ | |
| # 6. Réseaux de neurones multi-couches | |
| - | Une seule unité linéaire ne trace qu'une frontière droite. Empiler de nombreuses unités simples avec une non-linéarité entre elles donne un réseau de neurones multi-couches, qui ajuste des frontières courbes et apprend ses propres caractéristiques. Ce module construit ce modèle en douceur : prendre la régression logistique du module précédent, la dessiner comme un graphe, et la rendre profonde, une étape à la fois. La recette est celle de chaque module : un modèle (des couches, exécutées par la propagation avant), une fonction de perte adaptée à la tâche, et la descente de gradient, désormais propulsée par la rétropropagation. L'histoire continue ensuite comme la pratique l'a imposé : les gradients disparaissent dans les piles profondes, de meilleures activations les raniment, les bonnes pratiques stabilisent l'entraînement, et la descente de gradient elle-même reçoit une amélioration. Ce module est la porte d'entrée du cours de [Deep Learning](/fr/Deep%20Learning), qui développe en profondeur chaque sujet abordé ici. |
| + | Une seule unité linéaire ne trace qu'une frontière droite. Empiler de nombreuses unités simples avec une non-linéarité entre elles donne un réseau de neurones multi-couches, qui ajuste des frontières courbes et apprend ses propres caractéristiques. Ce module construit ce modèle en douceur : prendre la régression logistique du module précédent, la dessiner comme un graphe, et la rendre profonde, une étape à la fois. La recette est celle de chaque module : un modèle (des couches, exécutées par la propagation avant), une fonction de perte adaptée à la tâche, et la descente de gradient, désormais propulsée par la rétropropagation. L'histoire continue ensuite dans le cours de [Deep Learning](/fr/Deep%20Learning), qui reprend exactement ici : pourquoi les gradients disparaissent dans les piles profondes, les activations qui les raniment, les bonnes pratiques qui stabilisent l'entraînement, et les améliorations de la descente de gradient. |
| ## 6.1 Linéaire contre non linéaire | |
| @@ 138,7 138,9 @@ | |
|  | |
| - | *La propagation avant calcule et met en cache les activations, la rétropropagation renvoie le gradient de la perte par les mêmes arêtes. La leçon [Rétropropagation](/fr/Deep%20Learning/05%20Backpropagation) du cours de Deep Learning la dérive pas à pas.* |
| + | *La propagation avant calcule et met en cache les activations, la rétropropagation renvoie le gradient de la perte par les mêmes arêtes.* |
| + | |
| + | Un point pratique complète la recette : le gradient de l'étape 2 se calcule sur un mini-lot. [Classification linéaire](/fr/Machine%20Learning/05%20Linear%20classification) offrait deux extrêmes, le lot complet ou un seul exemple par pas. Les réseaux s'entraînent entre les deux, un petit lot par pas : un gradient assez précis pour progresser, un pas assez bon marché pour en faire des milliers, et la propagation avant vectorisée de la section 6.2.3 traite tout le mini-lot en un produit matriciel par couche. |
| <details class="proof"> | |
| <summary>Exemple complet : un pas de descente de gradient sur le petit réseau</summary> | |
| @@ 161,7 163,7 @@ | |
| $$\delta^{[1]}_1 = w^{[2]}_{11}\, \delta^{[2]}\, \sigma'(0) = 1 \cdot 0{,}80 \cdot 0{,}25 = 0{,}20, \qquad \delta^{[1]}_2 = 1 \cdot 0{,}80 \cdot 0{,}10 = 0{,}08$$ | |
| - | (le neurone de biais constant ne prend aucune erreur, et notez la petite pente $0{,}10$ de l'unité 2 : la section 6.5 y reviendra). Puis le même produit extérieur contre $\bar{x} = (1, 1, 2)$ : |
| + | (le neurone de biais constant ne prend aucune erreur, et notez la petite pente $0{,}10$ de l'unité 2 : le cours de [Deep Learning](/fr/Deep%20Learning) en fera toute une histoire). Puis le même produit extérieur contre $\bar{x} = (1, 1, 2)$ : |
| $$\frac{\partial L}{\partial W^{[1]}} = \delta^{[1]}\, \bar{x}^T = \begin{pmatrix} 0{,}20 & 0{,}20 & 0{,}40 \\ 0{,}08 & 0{,}08 & 0{,}16 \end{pmatrix}$$ | |
| @@ 179,88 181,7 @@ | |
| </details> | |
| - | ## 6.5 La disparition du gradient |
| - | |
| - | La formule de rétropropagation cache un piège. À chaque couche traversée, l'erreur $\delta^{[l]}$ est multipliée par la pente locale $g'(z^{[l]})$, si bien que le gradient qui atteint la couche 1 contient un tel facteur par couche. Avec des activations sigmoïdes, ces facteurs sont petits par construction : |
| - | |
| - | $$\boxed{ \sigma'(z) = \sigma(z)\left(1 - \sigma(z)\right) \le \tfrac{1}{4} }$$ |
| - | |
| - | Le résultat est la disparition du gradient : les couches proches de la sortie apprennent, celles proches de l'entrée ne reçoivent presque rien et bougent à peine. Les réseaux sigmoïdes profonds stagnent, et le remède n'est pas un meilleur optimiseur, c'est une meilleure activation (section 6.6). |
| - | |
| - | <details class="proof"> |
| - | <summary>Preuve : le gradient rétrécit géométriquement avec la profondeur</summary> |
| - | |
| - | **Étape 1 : la pente de la sigmoïde ne dépasse jamais $1/4$.** Dérivons $\sigma(z) = (1 + e^{-z})^{-1}$ avec la règle de dérivation en chaîne : |
| - | |
| - | $$\sigma'(z) = \frac{e^{-z}}{\left(1 + e^{-z}\right)^2} = \frac{1}{1 + e^{-z}} \cdot \frac{e^{-z}}{1 + e^{-z}} = \sigma(z)\left(1 - \sigma(z)\right)$$ |
| - | |
| - | Posons $s = \sigma(z) \in (0, 1)$. Le produit $s(1 - s)$ est une parabole tournée vers le bas, maximale en $s = \tfrac{1}{2}$ où elle vaut $\tfrac{1}{4}$. La borne tient donc, avec égalité seulement en $z = 0$, et la saturation aggrave tout : dans l'exemple détaillé de la section 6.2.4, l'unité cachée 2 se trouve à $\sigma(2) \approx 0{,}88$, où la pente est déjà tombée à $0{,}88 \cdot 0{,}12 \approx 0{,}10$. |
| - | |
| - | **Étape 2 : la rétropropagation multiplie ces pentes.** Prenons le réseau profond le plus simple, une chaîne de $L$ couches à une unité chacune, où chaque quantité est un scalaire. En appliquant la règle de dérivation en chaîne de la sortie vers la couche 1, chaque couche traversée apporte le facteur $\partial z^{[l]} / \partial z^{[l-1]} = w^{[l]}\, \sigma'(z^{[l-1]})$ : |
| - | |
| - | $$\frac{\partial L}{\partial z^{[1]}} = \frac{\partial L}{\partial z^{[L]}} \prod_{l=2}^{L} w^{[l]}\, \sigma'(z^{[l-1]})$$ |
| - | |
| - | Avec des poids de taille typique $|w^{[l]}| \le 1$, chaque facteur vaut au plus $\tfrac{1}{4}$ en valeur absolue, donc |
| - | |
| - | $$\boxed{ \left|\frac{\partial L}{\partial z^{[1]}}\right| \le \left(\tfrac{1}{4}\right)^{L-1} \left|\frac{\partial L}{\partial z^{[L]}}\right| }$$ |
| - | |
| - | Dix couches rétrécissent déjà le gradient d'environ $10^{-6}$. Le cas matriciel complet est la récursion de la section 6.4, avec la même conclusion. $\blacksquare$ |
| - | |
| - | </details> |
| - | |
| - | Des poids bien plus grands que $1$ ne font qu'échanger le problème contre son image miroir, l'explosion du gradient. La leçon [Initialisation et disparition du gradient](/fr/Deep%20Learning/07%20Initialization%20and%20vanishing%20gradients) du cours de Deep Learning en donne le traitement complet. |
| - | |
| - | ## 6.6 Les fonctions d'activation |
| - | |
| - | Alors, quelle activation choisir pour $g$ ? Les candidates, dans l'ordre où l'histoire les a essayées : |
| - | |
| - | $$\boxed{ \sigma(z) = \frac{1}{1 + e^{-z}}, \qquad \tanh(z), \qquad \mathrm{ReLU}(z) = \max(0, z) }$$ |
| - | |
| - | La sigmoïde sature dans ses deux queues, exactement ce que la section 6.5 vient de punir, et ses sorties ne sont jamais négatives, donc les poids entrants d'une unité reçoivent des gradients de même signe et les mises à jour zigzaguent. La $\tanh$, centrée en zéro, supprime ce biais mais sature encore. ReLU garde une pente d'exactement $1$ sur tout son côté positif, les facteurs rétrécissants de la section 6.5 disparaissent donc, et elle ne coûte presque rien à calculer. C'est pourquoi elle est aujourd'hui l'activation cachée par défaut. |
| - | |
| - |  |
| - | |
| - | *La tanh est centrée en zéro alors que la sigmoïde ne l'est pas, et ReLU reste linéaire pour les entrées positives.* |
| - | |
| - | ReLU a un angle mort : une unité dont l'entrée reste négative sort $0$, a une pente de $0$ et cesse d'apprendre, une unité morte. Des variantes comme Leaky ReLU, $\max(0{,}01\, z, z)$, et ELU gardent une petite pente du côté négatif pour l'éviter. En pratique : commencer avec ReLU, essayer ses variantes si des unités meurent, et réserver la sigmoïde là où la section 6.3 en a besoin, à la sortie d'un classifieur binaire. La leçon [Fonctions d'activation](/fr/Deep%20Learning/03%20Activation%20functions) du cours de Deep Learning les compare toutes. |
| - | |
| - | ## 6.7 Les bonnes pratiques |
| - | |
| - | Cinq habitudes font la différence entre un réseau qui s'entraîne et un réseau qui stagne. |
| - | |
| - | **Entraîner par mini-lots.** [Classification linéaire](/fr/Machine%20Learning/05%20Linear%20classification) offrait deux extrêmes, le lot complet ou un seul exemple par pas. Les réseaux s'entraînent par mini-lots, un petit lot par pas : un gradient assez précis pour progresser, un pas assez bon marché pour en faire des milliers, et la propagation avant vectorisée de la section 6.2.3 traite tout le mini-lot en un produit matriciel par couche. |
| - | |
| - | **Initialiser avec soin.** Des poids tous égaux feraient calculer la même chose à chaque unité d'une couche pour toujours, on démarre donc petit et aléatoire pour briser la symétrie. L'échelle compte aussi : trop petit et les activations rétrécissent vers zéro couche après couche, trop grand et elles saturent. Mettre la variance à l'échelle du nombre d'entrées de l'unité, Xavier pour tanh, He pour ReLU. |
| - | |
| - | **Centrer et normaliser les entrées.** Standardiser chaque caractéristique (soustraire sa moyenne, diviser par son écart-type), pour qu'aucune ne domine les premiers produits scalaires et que le zigzag des entrées toutes positives de la section 6.6 disparaisse dès la première couche. |
| - | |
| - | **Dropout.** Mettre à zéro au hasard une fraction des unités pendant l'entraînement pour qu'aucune ne puisse s'appuyer sur ses voisines, un régulariseur dans l'esprit de [Concepts généraux](/fr/Machine%20Learning/02%20General%20concepts). En prédiction, toutes les unités restent actives et les sorties sont mises à l'échelle par la probabilité de conservation, ce qui approche la moyenne des nombreux réseaux amincis ([Régularisation et dropout](/fr/Deep%20Learning/09%20Regularization%20and%20dropout)). |
| - | |
| - | **Vérifier avant d'entraîner longtemps.** Un classifieur à $K$ classes fraîchement initialisé doit démarrer près de la perte $\ln K$ (environ $2{,}3$ pour $K = 10$). Un minuscule jeu d'entraînement doit être facile à surapprendre : si le réseau n'y arrive pas, le code est cassé. Surveiller les courbes d'entraînement et de validation. Et comme la rétropropagation est source d'erreurs, comparer son gradient analytique à une estimation numérique par différences finies : |
| - | |
| - | $$\boxed{ \frac{\partial L}{\partial w} \approx \frac{L(w + \varepsilon) - L(w - \varepsilon)}{2\varepsilon} }$$ |
| - | |
| - | ## 6.8 La descente de gradient améliorée |
| - | |
| - | La descente de gradient brute prend le pas le plus pentu et rien de plus, et trois paysages la mettent en échec : les plateaux, où la pente est presque nulle et le progrès s'arrête, les points de selle (fréquents en haute dimension), où le gradient est exactement nul sans être un minimum, et les ravins, pentus dans une direction et doux dans l'autre, où le pas oscille entre les parois en rampant le long du fond. |
| - | |
| - | **Le momentum** traite la mise à jour comme une vitesse avec frottement : les gradients s'accumulent, les directions persistantes prennent de la vitesse, les directions oscillantes s'annulent : |
| - | |
| - | $$\boxed{ v \leftarrow \rho\, v + \nabla_W L, \qquad W \leftarrow W - \alpha\, v }$$ |
| - | |
| - | avec un frottement $\rho$ typiquement autour de $0{,}9$. |
| - | |
| - | **RMSProp** donne à chaque paramètre son propre pas, en divisant par une moyenne glissante de l'amplitude du gradient, ce qui tempère les directions pentues et accélère les plates : |
| - | |
| - | $$\boxed{ m \leftarrow \beta\, m + (1 - \beta) \left(\nabla_W L\right)^2, \qquad W \leftarrow W - \frac{\alpha}{\sqrt{m} + \varepsilon}\, \nabla_W L }$$ |
| - | |
| - | **Adam** combine les deux idées, une vitesse pour la direction et une échelle par paramètre pour le pas (la version complète corrige aussi un biais de démarrage dans $v$ et $m$), et c'est l'optimiseur par défaut en pratique : |
| - | |
| - | $$\boxed{ v \leftarrow \beta_1 v + (1 - \beta_1)\, \nabla_W L, \qquad m \leftarrow \beta_2 m + (1 - \beta_2) \left(\nabla_W L\right)^2, \qquad W \leftarrow W - \alpha\, \frac{v}{\sqrt{m} + \varepsilon} }$$ |
| - | |
| - | Deux habitudes complètent le tableau : faire décroître le taux d'apprentissage au fil de l'entraînement, et se rappeler que ces trois méthodes consomment toujours les gradients par mini-lots de la section 6.7, elles ne font que mieux les dépenser. La leçon [Optimisation](/fr/Deep%20Learning/06%20Optimization) du cours de Deep Learning les dérive une à une et ajoute les calendriers de taux d'apprentissage. |
| - | |
| - | *Ce module est la porte d'entrée du cours de [Deep Learning](/fr/Deep%20Learning), qui développe pleinement les architectures, les optimiseurs, l'initialisation, la normalisation et la régularisation. Le module suivant revient aux modèles linéaires sous un nouvel angle, le classifieur à marge maximale.* |
| + | *Ce module est la porte d'entrée du cours de [Deep Learning](/fr/Deep%20Learning), qui reprend l'histoire exactement ici : les fonctions d'activation, les optimiseurs, l'initialisation, la normalisation, la régularisation et les architectures bâties dessus. Le module suivant revient aux modèles linéaires sous un nouvel angle, le classifieur à marge maximale.* |
| --- | |
| Suivant : [Machines à vecteurs de support](/fr/Machine%20Learning/07%20Support%20Vector%20Machines) · [Vue d'ensemble du cours](/fr/Machine%20Learning) | |
| fr/Machine Learning/06 Multilayer neural networks/activations.png .. /dev/null | |
| fr/Mathematics/01 Mathematical refresher.md .. | |
| @@ 2,12 2,6 @@ | |
| Ce module rassemble les outils mathématiques sur lesquels s'appuie le reste du cours : un peu d'algèbre linéaire, le langage de l'espérance et de la covariance, la gaussienne multivariée, et les quatre quantités probabilistes (vraisemblance, a priori, a posteriori, évidence) que le module suivant transforme en une manière de raisonner. C'est une référence à consulter, pas un traitement complet. | |
| - | **Objectifs** |
| - | - Rappeler les opérations sur vecteurs et matrices utilisées partout : produit scalaire, produit matrice-vecteur, transposée, inverse et norme. |
| - | - Définir l'espérance, la variance et la covariance, et assembler la matrice de covariance. |
| - | - Écrire la densité de la gaussienne multivariée et lire sa forme à partir de la covariance. |
| - | - Nommer la vraisemblance, l'a priori, l'a posteriori et l'évidence, et les relier par la règle de Bayes. |
| - | |
| ## 1.1 Algèbre linéaire | |
| Un vecteur de caractéristiques vit dans $\mathbb{R}^n$ et un jeu de données empile de tels vecteurs dans une matrice. Le produit scalaire de deux vecteurs somme leurs produits terme à terme : | |
