Recurrent encoder-decoder models push a whole input sequence through a single fixed context vector, which caps how much they can remember for long inputs. Attention removes that bottleneck by letting the decoder read every encoder state directly, weighting each one by how relevant it is to the current output step. This lesson builds the mechanism from alignment scores to the query-key-value view, which is the foundation the Transformer will generalize.

$$\boxed{ c = h_T }$$

Every decoder step $i$ produces its state $s_i$ and its output from this one vector $c$ plus the previous output. The whole meaning of the input, however long, has to be squeezed into a single fixed-size $h_T$.

*Remark:* this is a genuine information bottleneck. For a short sentence $h_T$ can hold enough, but as $T$ grows the early tokens are overwritten and translation or summarisation quality drops sharply on long inputs.

The fix is to keep all encoder states $h_1, \dots, h_T$ available and let the decoder decide, at each step, which of them to read.

Instead of one context vector shared across all steps, attention builds a fresh context vector $c_i$ for each decoder step $i$. It does this in three stages: score, normalise, combine.

For decoder step $i$ with state $s_i$, a score function measures how well that state aligns with each encoder state $h_j$:

$$e_{i,j} = \operatorname{score}(s_i, h_j)$$

A high $e_{i,j}$ means encoder position $j$ is relevant to producing output $i$. The scores form a vector over the $T$ input positions.

*Remark:* $s_i$ is usually the decoder state just before emitting token $i$, so the model chooses what to look at using what it has produced so far.

The scores are turned into a probability distribution over input positions with a softmax across $j$:

$$\boxed{ \alpha_{i,j} = \frac{\exp(e_{i,j})}{\sum_{k=1}^{T} \exp(e_{i,k})} }$$

Each $\alpha_{i,j} \in (0,1)$ and $\sum_j \alpha_{i,j} = 1$, so the weights say how much of the decoder's attention at step $i$ goes to input position $j$.

The context vector for step $i$ is the weighted average of the encoder states, using the attention weights:

$$\boxed{ c_i = \sum_{j=1}^{T} \alpha_{i,j}\, h_j }$$

This $c_i$ is recomputed at every decoder step, so the model reads a different mixture of the input for each output token. The decoder then combines $c_i$ with its state $s_i$ to predict the token, and the alignment weights $\alpha_{i,j}$ can be visualised as a soft matrix that shows which input words each output word attends to.

The additive score, from Bahdanau and co-authors, feeds the two states through a small one-hidden-layer network with learned matrices $W_1$ and $W_2$ and a learned vector $v$:

$$\boxed{ e_{i,j} = v^{\top} \tanh\!\left( W_1 s_i + W_2 h_j \right) }$$

It works even when $s_i$ and $h_j$ have different dimensions, since $W_1$ and $W_2$ project both into a shared space before the $\tanh$.

The multiplicative score, from Luong and co-authors, is a plain dot product between the two states:

$$\boxed{ e_{i,j} = s_i^{\top} h_j }$$

It has no extra parameters in its simplest form and is far cheaper to compute, since a whole matrix of scores is a single matrix multiplication. A general variant inserts a learned matrix $W$ as $s_i^{\top} W h_j$ to handle mismatched dimensions.

| Aspect | Additive (Bahdanau) | Multiplicative (Luong) |

| --- | --- | --- |

| Formula | $v^{\top}\tanh(W_1 s_i + W_2 h_j)$ | $s_i^{\top} h_j$ |

| Extra parameters | $W_1$, $W_2$, $v$ | none (or one matrix $W$) |

| Different dims | handled by projection | needs the $W$ variant |

| Cost | slower, small network per pair | fast, one matrix product |

| Best when | small models, mixed dimensions | large models, matched dimensions |

*Remark:* the dot product grows with the dimension of the states, so at large widths its variance gets big and pushes the softmax into flat regions. Scaling the score by $1/\sqrt{d}$ fixes this, and that scaled dot product is exactly what the Transformer will adopt.

Attention has a cleaner reading that drops the encoder-decoder framing. Rename the pieces: the state that does the looking is a query, and each thing that can be looked at contributes a key (used for scoring) and a value (used in the sum).

$$\boxed{ q = s_i, \quad k_j = h_j, \quad v_j = h_j }$$

With this naming the score compares the query against each key, the softmax turns the scores into weights, and the output is the weighted sum of the values:

$$\boxed{ \operatorname{Attention}(q, K, V) = \sum_{j} \operatorname{softmax}_j\!\left(\operatorname{score}(q, k_j)\right) v_j }$$

In classic seq2seq attention the key and the value are the same encoder state $h_j$, but nothing forces that. Separating the three roles is what unlocks the next step.

*Remark:* in this lesson the query comes from the decoder while the keys and values come from the encoder, so the query attends over a different sequence. When the query, keys, and values all come from the same sequence, each token attends over its own neighbours. That is self-attention, and stacking it is the entire idea behind the Transformer.

---