Neural networks turn raw inputs into useful features by learning them rather than hand-crafting them. For discrete symbols (words, product IDs, user IDs, categories) the natural representation is a learned dense vector called an embedding. This lesson shows why one-hot codes are a poor input, how an embedding matrix maps each symbol to a compact vector, how word2vec learns such vectors from co-occurrence, and why embeddings are the standard input to the sequence models and Transformers that follow.

- Explain why one-hot encodings are large, sparse, and blind to similarity.

- Define an embedding as a lookup into a learned matrix $E$ and treat its rows as parameters.

- State the word2vec skip-gram objective and the role of negative sampling.

- Measure semantic closeness with cosine similarity.

- See how the same idea covers items, users, and categorical features.

- Connect embeddings to recurrent networks and Transformers as the input layer.

Suppose the vocabulary has $V$ distinct symbols. The classic way to feed symbol $i$ to a network is the one-hot vector $x_{\text{onehot}} \in \{0, 1\}^V$, which is all zeros except for a single $1$ at position $i$. It carries no structure: every pair of distinct symbols is exactly as far apart as every other pair, so the code holds no notion of similarity. It is also enormous, a modern vocabulary has $V$ in the tens or hundreds of thousands, and it is almost entirely zeros.

| property | one-hot | learned embedding |

| --- | --- | --- |

| dimension | $V$ (tens of thousands) | $d$ (tens to hundreds) |

| sparsity | one nonzero entry | dense, all entries used |

| similarity | all pairs equidistant | close vectors mean related symbols |

| parameters | none, fixed | learned from data |

| downstream size | huge weight matrices | compact, reusable features |

*Remark:* feeding a one-hot vector into a linear layer $W x_{\text{onehot}}$ simply selects one column of $W$. The embedding lookup below makes that selection explicit and cheap.

An embedding matrix $E \in \mathbb{R}^{V \times d}$ stores one $d$-dimensional row per symbol. The embedding of a one-hot input is the matrix-vector product

$$\boxed{\; e = E^{T} x_{\text{onehot}} \in \mathbb{R}^{d} \;}$$

Because $x_{\text{onehot}}$ has a single $1$ at position $i$, this product just returns row $i$ of $E$, so in practice it is implemented as a table lookup $e = E_{i,:}$ and never as a real multiplication. The vector $e$ is short (dimension $d \ll V$) and dense.

*Remark:* the rows of $E$ are ordinary parameters. They start random and are updated by backpropagation together with the rest of the network, so the geometry of the space is shaped by whatever task the network is trained on.

Embeddings can be learned end to end inside any task, but they can also be learned on their own from unlabelled text. The word2vec skip-gram model does exactly this: it learns a vector per word by predicting the surrounding context words from a centre word.

Each word $w$ has an input vector $v_w$ (its row in the embedding matrix). Given a centre word $w_I$, the model scores each candidate output word $w_O$ by a dot product and normalizes over the whole vocabulary with a softmax:

$$\boxed{\; p(w_O \mid w_I) = \frac{\exp\!\left(v_{w_O}^{T} v_{w_I}\right)}{\sum_{w=1}^{V} \exp\!\left(v_{w}^{T} v_{w_I}\right)} \;}$$

Training maximizes this probability for the (centre, context) pairs that actually co-occur in a sliding window over the text. Words that appear in similar contexts are pushed to have large dot products, so their vectors end up close together.

The denominator sums over all $V$ words, which is far too expensive to compute for every training pair. Negative sampling replaces the full softmax with a cheap binary problem: for each real (centre, context) pair, draw a few random words as negatives and train the model to tell the true context word from the fakes. This turns one $V$-way normalization into a handful of logistic updates per step and is what makes word2vec fast enough to train on billions of words.

*Remark:* the learned space has a striking linear structure. Directions in it encode consistent relations, so analogies show up as vector arithmetic, the classic example being that the vector for "king" minus "man" plus "woman" lands near "queen".

Once symbols are dense vectors, "how related are two symbols" becomes a geometric question. The standard answer is cosine similarity, the cosine of the angle between two vectors $u$ and $v$:

$$\boxed{\; \cos(u, v) = \frac{u^{T} v}{\lVert u \rVert \, \lVert v \rVert} \;}$$

It lies in $[-1, 1]$: a value near $1$ means the vectors point the same way (very similar), near $0$ means unrelated, and near $-1$ means opposite. Cosine ignores vector length and looks only at direction, which is usually what we want, since a word's meaning should not depend on how often it appears.

*Remark:* nearest-neighbour search under cosine similarity is how embeddings power retrieval and recommendation. Find the stored vectors whose direction is closest to a query vector and you have the most relevant items.

Nothing in the construction is specific to language. Any set of discrete symbols can be embedded by giving it a matrix $E$ and learning its rows.

| domain | symbol | what the embedding captures |

| --- | --- | --- |

| language | word or token | meaning and usage |

| recommendation | item ID | products bought or viewed together |

| recommendation | user ID | a user's taste profile |

| tabular data | category level | behaviour of that category |

In a recommender, a predicted affinity between a user and an item is read off as the dot product of their embeddings, the same operation that scored words above:

$$\boxed{\; \text{score}(\text{user}, \text{item}) = v_{\text{user}}^{T} \, v_{\text{item}} \;}$$

In tabular models, replacing a high-cardinality categorical column with a learned embedding often beats one-hot encoding, because the model can place similar categories near each other instead of treating them as unrelated.

*Remark:* embeddings are also a form of dimensionality reduction. They compress a $V$-way symbol into $d$ numbers while keeping the information a downstream task needs, which is the essence of representation learning.

*Remark:* pretrained embeddings can be loaded as a starting point and then fine-tuned, so a model does not have to relearn basic semantics from scratch. This transfer of learned representations is one of the reasons deep models generalize so well on limited data.

---