An induction head, from scratch
This is a real transformer: 2 pre-norm blocks, 2 attention heads, width 32, written from scratch with hand-written backpropagation through attention, layer norm and GELU. No libraries. Every gradient was verified against finite differences before training, and the model reaches 100.0% accuracy against a10% chance baseline.
It was trained on one job: copy from context. The sequence repeats, so to predict the next token the model has to find the earlier copy of the current token and read off whatever followed it. That circuit is called an induction head, and it is thought to be the mechanism behind in-context learning in large language models. You can see it below as a single bright arc reaching back to the match.
Attention decides which token you read from. These are the units that then compute with it, for the query position only. Every dot is a real unit of this trained model, sized and lit by its measured activation, and the faint lines are the actual weights (top few hundred by magnitude, or the model would be a solid block of ink). Hover a unit to see its value and where its weight goes.
Training curve
Loss and accuracy over training. The task is solved abruptly rather than gradually, which is characteristic of induction heads forming.
18048 parameters · 1500 iterations · accuracy 100.0% (chance 10%) · loss 0.0036 · gradients checked to 1e-6 · 2026-07-28