Vibe codingGenerative motionFall 2026
Studies of what happens inside AI models,drawn in code from the math they run on.
Motion studies of the mechanisms inside today’s models: how text becomes tokens, what each word attends to, how the next word gets chosen and how an image model paints. Each drawing is computed by a small working version of its mechanism. A tokenizer and a language model train in your browser when this page opens, and every study can be played with.
Research
Before drawing anything, I looked at how AI gets pictured: the stock images people reach for, the art labs commission, and the graphics a few designers make from the systems themselves.
The nonprofit Better Images of AI, started by We and AI with BBC R&D, found stock libraries dominated by science-fiction robots, glowing brains and blue backgrounds. It argues these images mislead people about what AI is and hide the people who build it.
Better Images of AI (opens in a new tab)DeepMind commissions artists to interpret research themes and releases the work free on Unsplash and Pexels. The pieces are beautiful and varied. They express ideas around AI more than the way a model computes.
Visualising AI on Unsplash (opens in a new tab)Justin Jay Wang’s graphics start from what a system does. For OpenAI, gradients generated fresh for each visitor. For Cursor, line drawings where reinforcement learning becomes meshing gears and codebase indexing becomes stacked grids. He writes each one as rules in code, directing an agent.
Designing with code (opens in a new tab)Tokens
This tokenizer learned its merges from a page of sentences about light, water and painting. Type anything and it splits into the pieces it knows; slide back to see it with fewer merges learned.
10 tokens: the, painter, re, paint, e, d, the, mo, on, light
Byte-pair encoding starts from single characters, counts every neighbouring pair in the training text and merges the most frequent pair into a new token, again and again. Encoding replays those merges in the order they were learned.
Common words close into one token. A word the training text never had splits into pieces it did have, which is how a model can read any string at all.
Each arch is one merge, joining the centres of the two pieces it combined, one level above the merges inside it. Ink arches finish a token; gray ones are the steps inside it.
The readout names the merge and its rank in the learned list. The bar under each token carries its ID, the number the model actually receives.
Limits and prices are counted in tokens, so a product should count in tokens where people write, not in characters.
Spelling tasks, like counting the letters in a word, are hard for a model that never saw the letters; a product can hand them to code. Tokenizers trained mostly on English cut other languages into more pieces, so the same message can cost more in Chinese than in English.
Sources: Sennrich, Haddow and Birch, 2016, Petrov et al., 2023
Attention
Point at a word to see where it looks. The sentence repeats on purpose: the second time through, the induction head finds the first copy and predicts the name that followed it. Write your own sentence with a repeated phrase.
Next word from the induction head: Vela, copied from word 4.
In every transformer layer, each token scores the tokens before it and takes a weighted mix of what it finds. Interpretability research has found heads with recognizable jobs.
A previous-word head links each word to the one before it. An induction head builds on that link: it finds an earlier copy of the current word and attends to whatever came next. That is how a model continues a pattern it has seen only once in the prompt.
Curves below the line are one word’s attention, as thick as their weight. The arches above are the previous-word head. The matrix is the whole head at once, a row per word, darker where it looks.
The weights apply each head’s published rule to your sentence, with some weight resting on the first word as large models do. They are not read out of a trained network.
This is why examples in a prompt work, and why a name or format mentioned once keeps coming back.
It also explains a failure: a model can copy a pattern nobody meant to set. Letting people see, pin, edit or remove earlier context gives them control over what gets copied.
Sources: Vaswani et al., 2017, Elhage et al., 2021, Olsson et al., 2022, Wang et al., 2022, Xiao et al., 2023
Sampling
Every sentence this small model could write after “the light” is drawn at once. Change the temperature, draw a sentence, and point at any fork to see its next-word odds.
Next word after the light
A language model gives every word in its vocabulary a probability. Dividing the scores by a temperature before the softmax sharpens the odds below 1 and flattens them above 1. At zero the model always takes the top word, so the same prompt always gives the same answer.
The model here is a trigram model, the kind Shannon sampled by hand in 1948, trained in your browser on the same page of sentences as the tokenizer.
Each fork is a possible next word. Its band is as tall as its probability, with the likeliest word in the middle, and the river’s ink fades to gray as paths get less likely.
The cobalt pen is a real sample. Its random numbers are seeded, so the cover draws the same sentences on every visit.
Regenerate exists because answers are samples. A product can set temperature per task: low for extraction and code, higher for brainstorming and names.
When a question has several good answers, showing a few side by side tells people more than presenting one draw as the answer.
Sources: Shannon, 1948, Holtzman et al., 2020
Diffusion
This is one of my oil paintings, buried in noise and recovered step by step. Drag through the steps or let it play.
A diffusion model learns to undo a process that mixes an image with Gaussian noise over many steps. Generating means starting from pure noise and removing a little at every step.
Natural images keep most of their energy in large shapes, while noise spreads evenly across every scale. So as noise comes off, large shapes clear first and fine detail last, and generation tends to run from coarse to fine.
On the left is the noisy input at each step, on the cosine schedule. On the right is the best linear guess of the painting from that input: a Wiener filter applied band by band, which a trained model’s guesses roughly follow.
The bar splits the picture into frequency bands, largest shapes first, and fills each band once its signal is stronger than its noise.
Composition is decided early, so a generation can be previewed while it forms. Variations can be offered by how many steps to redo: redo only the late steps and the layout holds while details change; start earlier and the layout moves too.
When I paint, I block in the large masses first and leave the figure’s edges for last. The model ends up working in the same order.
Sources: Ho, Jain and Abbeel, 2020, Nichol and Dhariwal, 2021, Meng et al., 2022, Dieleman, 2024
Drawing rules
Merges arch above the line, attention hangs below it, and the height says how deep or how far. Forks split probability; bands split a picture by scale.
Ink is what the model has settled. Cobalt is what it is doing at this moment. Gray is what it passed over. Things fade by moving to a lighter solid color, so a still frame reads like the motion.
Each study plays once, from blank paper to its result and back to paper, so the next one cuts in cleanly. With reduced motion, each shows its finished frame instead.
Sources
