The layers and operations of the previous module compose into full networks, and a handful of landmark architectures shaped how those pieces are assembled. This module surveys LeNet, AlexNet, VGG, Inception, and ResNet, extracting the one idea each contributed. The through-line is a search for depth: how to stack more layers without the training signal decaying, which ties directly back to the vanishing-gradient problem of lesson 7.

- Trace the progression from the early convolutional stacks of LeNet and AlexNet.

- Explain why VGG replaced large filters with deep stacks of small $3 \times 3$ convolutions.

- Read an Inception module as parallel branches and understand the $1 \times 1$ convolution as a channel bottleneck.

- Write the residual block $y = F(x, W) + x$ and connect the skip connection to gradient flow.

- Compare the five architectures by depth, key idea, and contribution.

All of these architectures share the same overall shape: a stack of convolution and pooling layers that extract features, followed by a small fully connected head that classifies them.


LeNet is the original convolutional network, built for handwritten-digit recognition. It alternates convolution and pooling layers to extract features, then finishes with fully connected layers for classification. A layer $l$ still computes $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$ followed by $a^{[l]} = g^{[l]}(z^{[l]})$, but $W^{[l]}$ is now a bank of small shared filters rather than a dense matrix. The activation $g^{[l]}$ was a saturating sigmoid or $\tanh$, and the whole network was only a handful of layers deep.

AlexNet kept the convolution-then-pool skeleton but scaled it to large natural images and trained it on GPUs. Two ideas from this course made deep training practical at that scale. First, the ReLU activation

$$\boxed{\ g(z) = \max(0, z)\ }$$

replaced the saturating sigmoid, so the gradient is $1$ wherever $z > 0$ and does not vanish for large positive inputs. Second, dropout randomly zeroes a fraction $p$ of activations during training, which regularizes the large fully connected layers:

$$\boxed{\ a^{[l]} \leftarrow \frac{1}{1-p}\, m \odot a^{[l]}, \quad m_j \sim \text{Bernoulli}(1-p)\ }$$

*Remark:* the mask $m$ is applied elementwise with the Hadamard product $\odot$, and the $1/(1-p)$ factor keeps the expected activation unchanged so that no rescaling is needed at test time.

VGG made one design choice and pushed it hard: every convolution is $3 \times 3$, and depth comes from stacking many of them. Two stacked $3 \times 3$ convolutions see the same input region as one $5 \times 5$ convolution, and three stacked see the same region as one $7 \times 7$. The stack is cheaper and more expressive, because it inserts a nonlinearity between each layer while using fewer parameters.

For a filter of side $k$ mapping $c_{\text{in}}$ input channels to $c_{\text{out}}$ output channels, the weight count is

$$\boxed{\ \#\text{params} = k^2 \cdot c_{\text{in}} \cdot c_{\text{out}} \ }$$

so with $c_{\text{in}} = c_{\text{out}} = c$ a single $5 \times 5$ layer costs $25 c^2$ weights, while two $3 \times 3$ layers cost $2 \cdot 9 c^2 = 18 c^2$. The deeper stack is both smaller and adds an extra ReLU.

*Remark:* the regular structure is what made VGG a favourite backbone. The trade is cost, because its wide fully connected head holds most of the parameters.

Instead of choosing a single filter size, an Inception module (GoogLeNet) runs several in parallel and concatenates their outputs along the channel axis. One branch is $1 \times 1$, one is $3 \times 3$, one is $5 \times 5$, and one is a pooling branch, so the network learns which scale matters at each stage rather than fixing it by hand.

The key trick is the $1 \times 1$ convolution. It has no spatial extent, so it does not mix neighbouring pixels. Instead it acts as a per-position linear map across channels, computing at each spatial location $(i, j)$

$$\boxed{\ y_{ij} = W\, a_{ij} + b, \quad W \in \mathbb{R}^{c_{\text{out}} \times c_{\text{in}}}\ }$$

Choosing $c_{\text{out}} < c_{\text{in}}$ makes it a channel bottleneck: it projects a thick feature map down to fewer channels before an expensive $3 \times 3$ or $5 \times 5$ convolution, cutting the cost of that convolution sharply. This is why Inception can be both wide and affordable.

*Remark:* a $1 \times 1$ convolution followed by a ReLU is exactly a small fully connected network applied identically at every spatial position, sharing one weight matrix $W$ across the whole feature map.

Very deep plain stacks train worse than shallow ones, not because they overfit but because the signal degrades. ResNet fixes this by having each block learn a residual and adding the input back through a skip connection:

$$\boxed{\ y = F(x, W) + x\ }$$

Here $F$ is a short stack of convolutions with weights $W$, and the term $+x$ is the identity skip. If the optimal map for a block is close to the identity, the network only has to drive $F$ toward zero, which is far easier than learning the identity from scratch through several nonlinear layers.


Differentiating the block, the skip contributes an identity term to the Jacobian:

$$\boxed{\ \frac{\partial y}{\partial x} = \frac{\partial F}{\partial x} + I\ }$$

During backpropagation the upstream gradient is multiplied by this factor at every block. The $+I$ term gives the gradient a direct route backward that never shrinks, so even when the $\partial F / \partial x$ contributions are small the product across many blocks does not collapse toward zero. This is the direct remedy to the vanishing-gradient problem from lesson 7, where repeated multiplication by small Jacobians in a plain deep stack drives early-layer gradients to nothing. With skip connections, networks of hundreds of layers train reliably.

*Remark:* when $F$ changes the number of channels or the spatial size, the skip uses a $1 \times 1$ convolution to match shapes so the sum $F(x, W) + x$ is well defined.

| Architecture | Approx. depth | Key idea | Contribution |

| --- | --- | --- | --- |

| LeNet | 5 to 7 layers | conv and pool stack | first working CNN for digits |

| AlexNet | 8 layers | ReLU and dropout at scale | deep CNNs on large images and GPUs |

| VGG | 16 to 19 layers | stacks of $3 \times 3$ convolutions | depth from uniform small filters |

| Inception | 22 layers | multi-branch modules, $1 \times 1$ bottleneck | width and efficiency together |

| ResNet | 50 to 152 layers | residual block $y = F(x, W) + x$ | trains very deep networks |

*Remark:* the trend is monotonic in depth, and each jump was unlocked by a specific fix, better activations, smaller filters, channel bottlenecks, and finally skip connections.

---

Next: [Embeddings and representation learning](/en/Deep%20Learning/12%20Embeddings%20and%20representation%20learning) · [Course overview](/en/Deep%20Learning)