Foundations track
Start here: how a transformer runs, and how it is trained
Two sites, one model. The Decoder follows your own text through every operation of a small decoder; Training trains that same model, in your browser, and then scales the ideas up to thousands of GPUs. Read them in order: every later site in the family (serving, hardware, agents) assumes these two.
Reading order for the Foundations track
Each chapter builds on the one before it. Start at the Decoder's chapter 1; the Training site picks up where the Decoder ends.
What you need first
- Vectors and matrices. A dot product, a matrix times a vector, and which shapes can be multiplied. Every operation in the Decoder is one of these. A linear algebra refresher (slides) covers it.
- Exponentials and logarithms. What ex and ln x do, and that a log turns a product into a sum. The softmax and the loss are built from them.
- A little probability. A distribution’s probabilities add up to 1, and sampling draws from it.
- Derivatives and the chain rule, for Training chapters 4 and 5 only. The Concept layer explains them in words; the Maths layer gives the formulas.
- Reading code helps but is optional. Each chapter’s Code layer shows the TypeScript that runs; hide it with the layer switch at the top of the chapter.
You don’t need PyTorch, a GPU or an account: everything runs in the browser or on this site’s server, and signing in only keeps your comments and progress.
How a chapter works
A chapter opens with an animation of its mechanism, computed by the site’s tested model, then explains it in three layers you can switch on and off: Concept (words and pictures), Maths (the formulas) and Code (the code that runs). It ends with Check yourself, three questions to commit to before you reveal the answers (the numbers come from the same model), and What to remember, the chapter in a few lines.
Glossary
Each term links to the section that defines it. The Transformer Training slide series’ glossary has more.
- Decoder, logits
- a stack of blocks that turns tokens into a score (logit) for every possible next token
- Token, embedding
- a token's id picks one learned row of the embedding table
- Positional encoding
- a vector added to each row that says where the token sits
- Attention
- each position takes a weighted average of earlier positions' values
- Causal mask
- sets future scores to −∞, so a position never sees what comes next
- Multi-head attention
- several attention heads on slices of the model width, joined
- Feed-forward network (FFN)
- a two-layer MLP applied to each position on its own
- GELU
- the smooth non-linearity inside the FFN
- Residual stream
- the running vector every sub-layer reads and adds to
- LayerNorm, pre-norm
- normalise each position's vector before each sub-layer
- Logit lens
- reading the stream early with the final LayerNorm and head
- Temperature, top-k, top-p
- the rules that turn logits into a sampled next token
- Training step
- one batch, one forward and backward pass, one update
- Validation loss
- the loss on held-out text the model never trains on
- Cross-entropy, perplexity
- minus the log of the right token's probability; e to the loss
- Gradient, backpropagation
- the chain rule, one operation at a time, from the loss back
- SGD, momentum
- step downhill; keep a running sum of past gradients
- Nesterov momentum
- momentum with the gradient taken at the look-ahead point
- Adam, AdamW
- per-weight step sizes; decoupled weight decay
- Learning-rate sweep, warmup
- how the most important number is found and started
- Critical batch size
- beyond it, a bigger batch barely saves steps
- Pre-LN, post-LN, µP
- where the norm sits; one learning rate for every width
- QK-norm, z-loss
- cures for growing attention scores and drifting logits
- BF16, mixed precision
- 16-bit compute with an FP32 master copy of the weights
- 16 bytes per parameter
- weights, gradients, Adam's two moments and the master copy
- Activation checkpointing
- recompute activations in the backward pass to save memory
- Data parallelism, all-reduce
- every worker a full copy; the gradients summed by a ring
- ZeRO, FSDP
- shard the optimiser states, gradients and weights
- MFU
- the model's useful FLOPs as a share of the hardware's peak
- Pipeline bubble
- time a pipeline stage waits, (p − 1)/m of the work
- Tensor parallelism
- each layer's matrices split across the GPUs of a server
- Top-k, DeMo, DiLoCo
- sending less, or less often, over slow links
- Scaling law, compute 6ND
- the best model size for a compute budget
- Checkpoint, goodput
- saving a run to resume it; the share of time spent training
- SFT, DPO
- fine-tuning on responses; optimising preferences directly