17. Deep learning in practice
Every lesson so far derived the mechanics of neural networks by hand: forward pass, loss, backpropagation, and the optimizer. In practice you write almost none of that. Modern frameworks store data as tensors, record the operations you perform, and differentiate them automatically, so the training loop you code is short and the gradients come for free. This capstone connects the theory to the tools, the hardware, and the habits that make a model actually train.
Objectives
- Explain what a tensor and automatic differentiation give you, and how autograd implements backpropagation.
- Write a framework-agnostic training loop from memory.
- Reason about batch size, accelerators, and mixed precision as practical trade-offs.
- Apply transfer learning: reuse a pretrained backbone, freeze early layers, fine-tune the rest.
- Recognize and fix the common failure modes that quietly wreck a run.
- Place the models from this course on a single map and hand them off to production.
17.1 Frameworks, tensors, and autograd
The two dominant stacks are PyTorch and TensorFlow, with JAX a fast-growing third that pairs a NumPy-like API with function transformations. All three share two ideas.
A tensor is an n-dimensional array that lives on a device (CPU or accelerator) and carries a data type. A scalar is a 0-D tensor, a vector 1-D, a matrix 2-D, and a batch of RGB images is typically a 4-D tensor of shape (batch, channels, height, width). Every activation \(a^{[l]}\), weight \(W^{[l]}\), and bias \(b^{[l]}\) from the earlier lessons is a tensor.
Automatic differentiation (autograd) is what saves you from coding backprop. As the forward pass runs, the framework records each primitive operation into a computation graph. Calling backward() walks that graph in reverse and applies the chain rule, giving \(\partial J / \partial W^{[l]}\) and \(\partial J / \partial b^{[l]}\) for every parameter. This is exactly the backpropagation you derived earlier, executed for you:
Remark: PyTorch builds the graph dynamically on each forward pass (define-by-run), which makes debugging feel like ordinary Python. TensorFlow and JAX can trace and compile the graph ahead of time for speed. You rarely call the gradient math yourself, but knowing the formula above is why you can diagnose a vanishing or exploding gradient when a deep network refuses to learn.
17.2 The training loop
Underneath every framework the loop is the same. You iterate over epochs, and within each epoch over mini-batches, running four steps per batch: forward pass, loss, backward pass, optimizer step. One detail trips up newcomers: gradients accumulate by default, so you must clear them each iteration.
for epoch in range(num_epochs): for x_batch, y_batch in dataloader: # mini-batches, shuffled optimizer.zero_grad() # clear accumulated gradients yhat = model(x_batch) # forward pass a[L] = model(x) loss = loss_fn(yhat, y_batch) # per-batch cost J loss.backward() # autograd: backpropagation optimizer.step() # update W[l], b[l] validate(model, val_loader) # track generalization
Remark: the order matters. Zero the gradients before backward(), and never call optimizer.step() before the backward pass has populated the gradients. In TensorFlow the same four steps live inside a GradientTape context, but the structure is identical.
17.3 Hardware and batching
Neural networks are dense linear algebra, which maps perfectly onto GPUs and other accelerators (TPUs). A GPU runs thousands of matrix multiplications in parallel, so moving both the model and the data to the device is usually the single largest speedup you will get.
17.3.1 Mini-batch size
The batch size is a core trade-off, not a detail.
| Batch size | Gradient quality | Hardware use | Generalization |
|---|---|---|---|
| Small (8 to 32) | noisy estimate | underuses the GPU | noise can help escape sharp minima |
| Large (256+) | smooth, accurate estimate | saturates the GPU | may converge to sharp minima, needs a warmup |
Remark: a common rule of thumb is to pick the largest batch that fits in memory, then tune the learning rate to match, since a larger batch usually needs a larger (or warmed-up) learning rate.
17.3.2 Mixed precision
Storing activations and weights in 16-bit floats (float16 or bfloat16) instead of 32-bit halves the memory and speeds up the matrix multiplies, while a master copy of the weights and the loss stay in 32-bit for numerical stability. This is mixed precision, and on modern accelerators it is close to free performance.
17.4 Transfer learning and fine-tuning
Training a large network from scratch needs a lot of data and compute. Transfer learning sidesteps that by reusing a model already trained on a large corpus. You keep its backbone (the feature-extracting layers), replace the final task-specific head, and train on your smaller dataset.
The usual recipe:
- Freeze the early layers, whose features (edges, textures, generic token patterns) transfer across tasks.
- Replace the head with one sized for your classes or outputs.
- Fine-tune the later layers, and optionally unfreeze the rest at a small learning rate once the head has settled.
Transfer learning reuses a pretrained backbone, replaces the head, and fine-tunes the later layers on the new task.
Remark: this is where self-supervised pretraining pays off. A model pretrained BERT-style or GPT-style on huge unlabelled text already encodes rich language structure, so fine-tuning it on a small labelled set beats training a fresh model many times over. The same holds for vision backbones pretrained on large image collections.
17.5 Common pitfalls
Most failed runs are not exotic. They come from a short list of mistakes, and each has a direct fix.
| Pitfall | Symptom | Fix |
|---|---|---|
| Overfitting | train loss drops, validation loss rises | regularize, add dropout, augment, or stop early |
| Bad learning rate | loss diverges or is flat | sweep the rate, use a scheduler or warmup |
| Data leakage | great validation score, poor in production | split before preprocessing, keep test data unseen |
| Forgetting to shuffle | loss plateaus or cycles | shuffle the training set every epoch |
| Not normalizing inputs | slow or unstable training | standardize features to zero mean, unit variance |
Remark: data leakage is the most dangerous because it hides as success. If you fit a scaler or select features using the whole dataset before splitting, information about the test set bleeds into training, and the reported score is a mirage.
17.6 A map of the field
The models across this course form a lineage. Fully connected multilayer perceptrons gave the core mechanics. Convolutions added spatial structure for images. Recurrent networks and LSTMs handled sequences. Attention removed the sequential bottleneck, transformers scaled it, and pretraining transformers at scale produced the foundation models that now anchor most applications.
A map of the course: from the multilayer perceptron through convolutional and recurrent networks to attention, Transformers, and foundation models.
A trained model is only half the job. Serving it reliably, monitoring for drift, versioning data, and automating retraining are their own discipline.
To take any of these models from a notebook to a reliable production service, continue with the MLOps course.
Next: Course overview