Feedforward and convolutional networks map a fixed-size input to an output in one pass, but many problems come as sequences whose length varies and whose order matters (text, audio, time series). A recurrent neural network (RNN) processes a sequence one step at a time and carries a hidden state forward, so past inputs influence the current output. This module introduces recurrence, the vanilla RNN cell, how it is trained by backpropagation through time, and why long-range gradients tend to vanish or explode.

- Explain why sequence data needs memory and weight sharing across time steps.

- Write the vanilla RNN recurrence for the hidden state and the output.

- Unroll a recurrent cell across time and read off the shared parameters.

- Derive how backpropagation through time (BPTT) accumulates the gradient over all steps.

- Diagnose vanishing and exploding gradients from the product of Jacobians over time.

A sequence is an ordered list of inputs $x_1, x_2, \dots, x_T$, where $T$ can differ from one example to the next. A feedforward network of the kind seen in earlier lessons expects a single fixed-size vector $a^{[0]} = x$, so it has no natural way to consume a variable-length input or to remember what came before the current element.

Two ideas fix this. First, the network keeps a **hidden state** (or memory) $h_t$ that summarizes everything relevant seen up to step $t$. Second, the network **shares** one set of parameters across every step, so the same transformation applies whether the sequence has 5 elements or 500. Sharing keeps the parameter count independent of $T$ and lets a pattern learned at one position generalize to any other.

*Remark:* weight sharing across time is the sequential analogue of weight sharing across space in a convolutional network. Both encode a prior that the same feature can appear anywhere.

| Setup | Input | Output | Example |

| --- | --- | --- | --- |

| Many to one | sequence | single vector | sentiment of a sentence |

| Many to many (aligned) | sequence | sequence, same length | part-of-speech tagging |

| Many to many (seq2seq) | sequence | sequence, other length | machine translation |

| One to many | single vector | sequence | image captioning |

At step $t$ the cell reads the current input $x_t$ and the previous hidden state $h_{t-1}$, then produces a new hidden state through an activation $g$ (usually $\tanh$):

$$\boxed{ h_t = g\left(W_{hh}\, h_{t-1} + W_{xh}\, x_t + b_h\right) }$$

The hidden state is initialized to $h_0 = \mathbf{0}$ (or a learned vector). The per-step output is a linear readout of the hidden state:

$$\boxed{ \hat{y}_t = W_{hy}\, h_t + b_y }$$

Here $W_{hh}$ maps state to state, $W_{xh}$ maps input to state, and $W_{hy}$ maps state to output. If the hidden size is $n_h$ and the input size is $n_x$, then $W_{hh}$ is $(n_h \times n_h)$, $W_{xh}$ is $(n_h \times n_x)$, and $b_h$ has shape $n_h$.

The crucial point is that $W_{hh}$, $W_{xh}$, $W_{hy}$, $b_h$, and $b_y$ do **not** depend on $t$. The same five parameters are reused at every step:

So an RNN is not a very deep network with distinct layers, it is one small cell applied repeatedly, feeding its own output back as input.

Because the same cell is reused, we can **unroll** the recurrence into a chain: draw one copy of the cell per time step and connect the hidden state of each copy to the next. The unrolled view is an ordinary feedforward graph (with tied weights), which is exactly what makes gradient computation possible.

*Remark:* the horizontal arrows between hidden states are the only path along which information from the past reaches the present. Every one of them multiplies by the same matrix $W_{hh}$, which is the source of both the model's power and its training difficulty.

Training minimizes a total cost that sums the per-step loss over the sequence. With per-step loss $L_t$ comparing $\hat{y}_t$ to the target $y_t$, the cost for one sequence is:

$$\boxed{ J = \sum_{t=1}^{T} L_t\left(\hat{y}_t, y_t\right) }$$

Backpropagation through time (BPTT) is ordinary backpropagation run on the unrolled graph. Because $W_{hh}$ is reused at every step, its gradient is the **sum** of the contributions from all steps:

$$\boxed{ \frac{\partial J}{\partial W_{hh}} = \sum_{t=1}^{T} \frac{\partial L_t}{\partial W_{hh}} }$$

For a single step $t$, the loss depends on $W_{hh}$ both directly (through $h_t$) and indirectly through every earlier hidden state $h_k$ with $k \le t$, since each of those was itself produced with $W_{hh}$. Applying the chain rule through the state chain gives:

$$\boxed{ \frac{\partial L_t}{\partial W_{hh}} = \sum_{k=1}^{t} \frac{\partial L_t}{\partial h_t}\left(\prod_{i=k+1}^{t} \frac{\partial h_i}{\partial h_{i-1}}\right)\frac{\partial h_k}{\partial W_{hh}} }$$

*Remark:* in practice the sum over $k$ is cut off after a fixed window, which is called truncated BPTT. It bounds memory and compute per update at the cost of ignoring dependencies longer than the window.

The inner product $\prod_{i=k+1}^{t} \frac{\partial h_i}{\partial h_{i-1}}$ is what carries gradient information from step $t$ back to step $k$. From the recurrence $h_i = g(W_{hh} h_{i-1} + W_{xh} x_i + b_h)$, each factor is:

$$\boxed{ \frac{\partial h_i}{\partial h_{i-1}} = \operatorname{diag}\!\left(g'(z_i)\right) W_{hh} }$$

where $z_i = W_{hh} h_{i-1} + W_{xh} x_i + b_h$ is the pre-activation at step $i$. Composing over the whole gap from $k$ to $t$ gives a product of $t - k$ such matrices:

$$\boxed{ \prod_{i=k+1}^{t} \frac{\partial h_i}{\partial h_{i-1}} = \prod_{i=k+1}^{t} \operatorname{diag}\!\left(g'(z_i)\right) W_{hh} }$$

This product of $t - k$ near-identical factors behaves roughly like a matrix raised to the power $t - k$. If the relevant magnitude (informally, the largest singular value of $\operatorname{diag}(g'(z_i)) W_{hh}$) is below $1$, the product shrinks geometrically toward zero as the gap grows, so distant gradients **vanish**. If it is above $1$, the product blows up and gradients **explode**.

| Regime | Product over time | Effect on training |

| --- | --- | --- |

| Factor magnitude $< 1$ | decays toward $0$ | long-range gradients vanish, no long memory learned |

| Factor magnitude $\approx 1$ | stays bounded | stable, the ideal case |

| Factor magnitude $> 1$ | grows without bound | gradients explode, updates diverge |

*Remark:* exploding gradients are usually tamed with **gradient clipping** (rescale the gradient when its norm exceeds a threshold). Vanishing gradients are harder, because the signal is lost rather than merely large, and no simple rescaling recovers it.

Since a saturating activation such as $\tanh$ has $g' \le 1$ everywhere, the diagonal factor tends to pull the product toward vanishing, which makes it hard for a vanilla RNN to learn dependencies more than a few dozen steps apart. This limitation is precisely what motivates gated cells, which add a near-linear path for the state to flow along without repeated squashing.

---