Every lesson so far derived the mechanics of neural networks by hand: forward pass, loss, backpropagation, and the optimizer. In practice you write almost none of that. Modern frameworks store data as tensors, record the operations you perform, and differentiate them automatically, so the training loop you code is short and the gradients come for free. This capstone connects the theory to the tools, the hardware, and the habits that make a model actually train.

- Explain what a tensor and automatic differentiation give you, and how autograd implements backpropagation.

- Write a framework-agnostic training loop from memory.

- Reason about batch size, accelerators, and mixed precision as practical trade-offs.

- Apply transfer learning: reuse a pretrained backbone, freeze early layers, fine-tune the rest.

- Recognize and fix the common failure modes that quietly wreck a run.

- Place the models from this course on a single map and hand them off to production.

The two dominant stacks are **PyTorch** and **TensorFlow**, with **JAX** a fast-growing third that pairs a NumPy-like API with function transformations. All three share two ideas.

A **tensor** is an n-dimensional array that lives on a device (CPU or accelerator) and carries a data type. A scalar is a 0-D tensor, a vector 1-D, a matrix 2-D, and a batch of RGB images is typically a 4-D tensor of shape (batch, channels, height, width). Every activation $a^{[l]}$, weight $W^{[l]}$, and bias $b^{[l]}$ from the earlier lessons is a tensor.

**Automatic differentiation** (autograd) is what saves you from coding backprop. As the forward pass runs, the framework records each primitive operation into a computation graph. Calling `backward()` walks that graph in reverse and applies the chain rule, giving $\partial J / \partial W^{[l]}$ and $\partial J / \partial b^{[l]}$ for every parameter. This is exactly the backpropagation you derived earlier, executed for you:

$$\boxed{ \frac{\partial J}{\partial z^{[l]}} = \left( W^{[l+1]} \right)^{T} \frac{\partial J}{\partial z^{[l+1]}} \odot g'^{[l]}\!\left(z^{[l]}\right) }$$

*Remark:* PyTorch builds the graph dynamically on each forward pass (define-by-run), which makes debugging feel like ordinary Python. TensorFlow and JAX can trace and compile the graph ahead of time for speed. You rarely call the gradient math yourself, but knowing the formula above is why you can diagnose a vanishing or exploding gradient when a deep network refuses to learn.

Underneath every framework the loop is the same. You iterate over epochs, and within each epoch over mini-batches, running four steps per batch: forward pass, loss, backward pass, optimizer step. One detail trips up newcomers: gradients accumulate by default, so you must clear them each iteration.

for epoch in range(num_epochs):

for x_batch, y_batch in dataloader: # mini-batches, shuffled

optimizer.zero_grad() # clear accumulated gradients

yhat = model(x_batch) # forward pass a[L] = model(x)

loss = loss_fn(yhat, y_batch) # per-batch cost J

loss.backward() # autograd: backpropagation

optimizer.step() # update W[l], b[l]

validate(model, val_loader) # track generalization

*Remark:* the order matters. Zero the gradients before `backward()`, and never call `optimizer.step()` before the backward pass has populated the gradients. In TensorFlow the same four steps live inside a `GradientTape` context, but the structure is identical.

Neural networks are dense linear algebra, which maps perfectly onto **GPUs** and other accelerators (TPUs). A GPU runs thousands of matrix multiplications in parallel, so moving both the model and the data to the device is usually the single largest speedup you will get.

The batch size is a core trade-off, not a detail.

| Batch size | Gradient quality | Hardware use | Generalization |

| --- | --- | --- | --- |

| Small (8 to 32) | noisy estimate | underuses the GPU | noise can help escape sharp minima |

| Large (256+) | smooth, accurate estimate | saturates the GPU | may converge to sharp minima, needs a warmup |

*Remark:* a common rule of thumb is to pick the largest batch that fits in memory, then tune the learning rate to match, since a larger batch usually needs a larger (or warmed-up) learning rate.

Storing activations and weights in 16-bit floats (`float16` or `bfloat16`) instead of 32-bit halves the memory and speeds up the matrix multiplies, while a master copy of the weights and the loss stay in 32-bit for numerical stability. This is **mixed precision**, and on modern accelerators it is close to free performance.

Training a large network from scratch needs a lot of data and compute. **Transfer learning** sidesteps that by reusing a model already trained on a large corpus. You keep its **backbone** (the feature-extracting layers), replace the final task-specific head, and train on your smaller dataset.

The usual recipe:

1. **Freeze** the early layers, whose features (edges, textures, generic token patterns) transfer across tasks.

2. **Replace the head** with one sized for your classes or outputs.

3. **Fine-tune** the later layers, and optionally unfreeze the rest at a small learning rate once the head has settled.


*Remark:* this is where **self-supervised pretraining** pays off. A model pretrained BERT-style or GPT-style on huge unlabelled text already encodes rich language structure, so fine-tuning it on a small labelled set beats training a fresh model many times over. The same holds for vision backbones pretrained on large image collections.

Most failed runs are not exotic. They come from a short list of mistakes, and each has a direct fix.

| Pitfall | Symptom | Fix |

| --- | --- | --- |

| Overfitting | train loss drops, validation loss rises | regularize, add dropout, augment, or stop early |

| Bad learning rate | loss diverges or is flat | sweep the rate, use a scheduler or warmup |

| Data leakage | great validation score, poor in production | split before preprocessing, keep test data unseen |

| Forgetting to shuffle | loss plateaus or cycles | shuffle the training set every epoch |

| Not normalizing inputs | slow or unstable training | standardize features to zero mean, unit variance |

*Remark:* data leakage is the most dangerous because it hides as success. If you fit a scaler or select features using the whole dataset before splitting, information about the test set bleeds into training, and the reported score is a mirage.

The models across this course form a lineage. Fully connected multilayer perceptrons gave the core mechanics. Convolutions added spatial structure for images. Recurrent networks and LSTMs handled sequences. Attention removed the sequential bottleneck, transformers scaled it, and pretraining transformers at scale produced the foundation models that now anchor most applications.


A trained model is only half the job. Serving it reliably, monitoring for drift, versioning data, and automating retraining are their own discipline.

---

Next: [Course overview](/en/Deep%20Learning)