How Transformers Actually Work: The Math, With Real Numbers

A modern language system runs on one equation: softmax of Q times K transpose over the square root of d, times V. This video builds the whole pipeline from the bottom and works every step by hand, with small numbers you can check with a pen. Byte pair encoding (merging e+s, es+t, lo+w on a real toy corpus); one-hot times the embedding matrix as a lookup; words as points and king - man + woman; the dot product (3x2 + 4x5 = 26) and cosine similarity (0.97); sinusoidal positional encoding; query, key and value; scores 6, 5, 1.5, 0.4 scaled by the square root of 2; softmax to 64% / 32% / 3% / 1%; the weighted sum that moves "bank" to (1.69, 0.37); QK transpose and the causal mask; multi-head attention, residual connections, layer norm and the feed-forward network; logits, softmax and temperature on a five-word vocabulary; cross-entropy loss, the p - y gradient, and gradient descent. ⏱ CHAPTERS 00:00 One equation behind every language model 00:42 Tokens: byte pair encoding by hand 01:41 One-hot times the embedding matrix 02:14 Words as points: king - man + woman 02:46 The dot product and cosine similarity 03:42 Positional encoding: sines and cosines 04:31 One word, two meanings 04:57 Query, key, value 05:29 Scores and the square root of d 06:12 Softmax, worked out 06:52 The weighted sum: bank moves 07:24 All words at once: QK transpose and the mask 08:10 The transformer block: heads, residual, norm, FFN 09:26 Logits, softmax and temperature 10:20 Loss, gradient, gradient descent 11:21 The whole chain 📚 SOURCES & NOTES • Vaswani et al., "Attention Is All You Need", 2017 (arXiv 1706.03762): scaled dot-product attention, sinusoidal positional encoding, multi-head attention, residual connections with layer norm, and the position-wise feed-forward layer. • Sennrich, Haddow & Birch, "Neural Machine Translation of Rare Words with Subword Units", 2016 (arXiv 1508.07909): byte pair encoding; the low/lower/newest/widest corpus is their worked example. • Mikolov et al., 2013 (arXiv 1301.3781): word2vec and the king - man + woman analogy. • Radford et al., 2019: GPT-2 (small model: 12 layers, 12 heads, 768 dimensions; vocabulary 50,257 tokens). • Every number on screen was recomputed in Python for this video; the 2-D embeddings and weights are toy values chosen so the arithmetic fits on screen, and the method is the same at full size. Animated with Manim; maths typeset with LaTeX. #machinelearning #transformers