transformer-explainer

Transformer Decoder Explainer

Watch a decoder think, one matrix at a time.

Type in your own text and follow it through every operation of a GPT-style decoder: embeddings, masked self-attention, the feed-forward network, LayerNorm and residuals, stacked blocks, and next-token sampling. The numbers you see are computed on the server by a small, readable TypeScript implementation.

Start the lessons →Open the playground
Attention weights for head 0 on the text "the cat sat". Each row is a query position; it spreads its attention over itself and earlier positions only, and the cells above the diagonal are masked.tthhee␣␣ccaatt␣␣ssaatt"t" → "t": 1.000"t" → "h": masked (a later token)"t" → "e": masked (a later token)"t" → "␣": masked (a later token)"t" → "c": masked (a later token)"t" → "a": masked (a later token)"t" → "t": masked (a later token)"t" → "␣": masked (a later token)"t" → "s": masked (a later token)"t" → "a": masked (a later token)"t" → "t": masked (a later token)"h" → "t": 0.500"h" → "h": 0.500"h" → "e": masked (a later token)"h" → "␣": masked (a later token)"h" → "c": masked (a later token)"h" → "a": masked (a later token)"h" → "t": masked (a later token)"h" → "␣": masked (a later token)"h" → "s": masked (a later token)"h" → "a": masked (a later token)"h" → "t": masked (a later token)"e" → "t": 0.333"e" → "h": 0.333"e" → "e": 0.335"e" → "␣": masked (a later token)"e" → "c": masked (a later token)"e" → "a": masked (a later token)"e" → "t": masked (a later token)"e" → "␣": masked (a later token)"e" → "s": masked (a later token)"e" → "a": masked (a later token)"e" → "t": masked (a later token)"␣" → "t": 0.249"␣" → "h": 0.249"␣" → "e": 0.250"␣" → "␣": 0.252"␣" → "c": masked (a later token)"␣" → "a": masked (a later token)"␣" → "t": masked (a later token)"␣" → "␣": masked (a later token)"␣" → "s": masked (a later token)"␣" → "a": masked (a later token)"␣" → "t": masked (a later token)"c" → "t": 0.199"c" → "h": 0.199"c" → "e": 0.200"c" → "␣": 0.201"c" → "c": 0.201"c" → "a": masked (a later token)"c" → "t": masked (a later token)"c" → "␣": masked (a later token)"c" → "s": masked (a later token)"c" → "a": masked (a later token)"c" → "t": masked (a later token)"a" → "t": 0.166"a" → "h": 0.166"a" → "e": 0.166"a" → "␣": 0.167"a" → "c": 0.167"a" → "a": 0.168"a" → "t": masked (a later token)"a" → "␣": masked (a later token)"a" → "s": masked (a later token)"a" → "a": masked (a later token)"a" → "t": masked (a later token)"t" → "t": 0.142"t" → "h": 0.142"t" → "e": 0.142"t" → "␣": 0.143"t" → "c": 0.143"t" → "a": 0.143"t" → "t": 0.143"t" → "␣": masked (a later token)"t" → "s": masked (a later token)"t" → "a": masked (a later token)"t" → "t": masked (a later token)"␣" → "t": 0.124"␣" → "h": 0.124"␣" → "e": 0.125"␣" → "␣": 0.125"␣" → "c": 0.125"␣" → "a": 0.125"␣" → "t": 0.125"␣" → "␣": 0.126"␣" → "s": masked (a later token)"␣" → "a": masked (a later token)"␣" → "t": masked (a later token)"s" → "t": 0.111"s" → "h": 0.110"s" → "e": 0.111"s" → "␣": 0.111"s" → "c": 0.111"s" → "a": 0.111"s" → "t": 0.111"s" → "␣": 0.111"s" → "s": 0.112"s" → "a": masked (a later token)"s" → "t": masked (a later token)"a" → "t": 0.100"a" → "h": 0.099"a" → "e": 0.100"a" → "␣": 0.100"a" → "c": 0.100"a" → "a": 0.100"a" → "t": 0.100"a" → "␣": 0.100"a" → "s": 0.100"a" → "a": 0.101"a" → "t": masked (a later token)"t" → "t": 0.091"t" → "h": 0.091"t" → "e": 0.091"t" → "␣": 0.091"t" → "c": 0.091"t" → "a": 0.091"t" → "t": 0.091"t" → "␣": 0.091"t" → "s": 0.091"t" → "a": 0.091"t" → "t": 0.091
Real attention weights for “the cat sat” (head 0, seed 42). Each row is a letter looking back at itself and earlier letters; grey cells are masked. The model’s weights are random, not trained, so each row spreads its attention almost evenly: the last row gives each of its 11 letters about 1/11. Try your own text

The code is a teaching artefact too: read the source on GitHub for Next.js full-stack patterns alongside the maths.