UnfinishedI left the shirt as bare paper. A model read a shirt into it. Code draws it in the model’s numbers.

A reflective project: one of my portraits, and what two open AI models did with it on my laptop. Every word, number and frame below comes from those runs.

Role: Artist
Creative Technologist
Type: Reflective project
Models: BLIP, SDXL Inpainting
Tools: Claude Code, Python, PyTorch, React
Compute: On-device
Rendering: Real-time Canvas 2D

Each digit is its spot’s value, 0 darkest to 9 lightest. The shirt is the model’s token IDs, in cobalt. Point at the picture, or press it, to read them.

Read as

“”

a 90%man 1%the 1%

Looked for the shirt at

the collar, and the bare paper below it

“white” at 62%, then “shirt” at 88%. I never painted one.

Woven from

320308653932078653032015792523131253651815061

The 13 token IDs the image model ran on, in cobalt.

Overview

I paint, and I design AI products.
Here a model helps finish my own painting.

Design question: Where should a model’s help stop in my own painting, and how should its part show?
Models: BLIP, a captioning model, reads the painting, and Stable Diffusion XL inpainting drafts the shirt. The painting stayed on my laptop.
Constraint: The model’s part shows as its own material: numbers in cobalt over my untouched paint.
Research: Justin Jay Wang’s graphics made in code, where each drawing is a rule set built from what a system does; Better Images of AI, on showing the machine as it is; Grad-CAM, for where a model looks; and Model Studies, my first study, where I drew the same mechanisms with toy models.

Paint

Before any model: strokes, values and one palette.

This is how I read the same picture as a painter. Each stroke is redrawn as a line of its own values, 0 for the darkest to 9 for the lightest, in the eight colours the painting is made of, drawn darkest first, the way I block in a portrait.

Value, 0 darkest to 9 lightest

My palette, eight colours sampled from the paint

Cobalt, the machine's colour: outside my palette, and opposite its ochres on the colour wheel.

Reading

It reads the painting one word at a time.

BLIP starts from “a painting of” and chooses each next word from a ranked list. It took “in” at 21% over “with” at 23%, and “looking” at 9%. Then it gave a painting a camera, at 82%. I took its own second choice, “viewer” at 5%, before anything was drawn.

a painting ofGiven to it
a
- a90%
- man1%
- the1%
- an1%
- young<1%
man
- man60%
- young13%
- painting4%
- portrait3%
- close2%
in
- with23%
- in21%
- is8%
- wearing7%
- looking5%
a
- a67%
- white18%
- an1%
- front1%
- yellow1%
white
- white62%
- shirt10%
- yellow3%
- polo2%
- baseball1%
shirt
- shirt88%
- sweater1%
- polo1%
- collar1%
- t1%
looking
- is13%
- and9%
- looking9%
- with8%
- looks7%
at
- at21%
- to10%
- off7%
- into7%
- away5%
the
- the68%
- something16%
- a5%
- camera2%
- his1%
viewerMy edit
- camera82%
- viewer5%
- side1%
- right1%
- left<1%

Readings it could have written

a young man wearing a white shirt and a brown hat
a man with a baseball bat in his hands
a man in a white shirt stares off into the distance

Temperature 0.6. Cooler than its own odds: it stays near its first choices, and the readings stay plausible. Three of the 8 readings it drew; I left out the ones that happened to name people.

Looking

When it wrote “shirt”, it was looking at bare paper.

Its vision side cuts the painting into 576 patches. For each word, the squares mark the patches whose attention raised that word's chance. “Man” finds the face, and its “camera” was his eyes.

My portrait, with the patches the captioner looked at for “shirt”

For “shirt” it looked at the collar and the bare paper below it. I never painted a shirt there.

576 patches, 24 by 24. A square's size is how much that patch's attention raised the chance of “shirt”, which it gave 88%.

Why a white shirt

It fills bare paper with what usually goes there, in the paper’s colour.

To find out why, I changed one thing at a time and asked the same model again, from the same first words. Each test shows the picture it was given, what it read, and how likely each word was after “a man in a”.

As I painted it“a man in a white shirt looking at the camera”white 62%
The paper filled black“a man in a black shirt and tie”black 53%white 1%
The paper filled blue“a man in a blue shirt looking at the camera”blue 85%white <1%
The paper filled yellow“a man in a yellow shirt looking at the camera”yellow 83%white 3%
The head alone, above the paper“a man in a suit and tie in front of a painting”suit 11%white 9%
No picture, flat gray“a man riding a surfboard on top of a wave”suit 11%white 5%

The colour is the paper’s.: Fill the paper black, blue or yellow and the shirt turns black, blue or yellow: 53%, 85% and 83%. My paper photographs a warm cream, but as the palest thing in the picture it reads as white, 62% to yellow’s 3%.
Below a face, it expects clothes.: Whatever the paper’s colour, the word after it is “shirt”, 51% to 88%. Crop the paper away and it still dresses him, in a suit and tie.
It isn’t the words alone.: With no picture, “white” follows “a man in a” only 5% of the time. The 62% comes from looking at the paper.
It has no word for unfinished.: Started from “an unfinished painting of”, it still writes “a young man in a white shirt”. It can’t read bare paper as bare.

That is why the tool shows the reading, and the words it nearly chose, before anything is drawn.

Numbers

The image model gets numbers, not words.

Its own tokenizer breaks each word into pieces it learned from its training text and swaps them for IDs. These 13 numbers are all of the reading it ever sees, and the digits the shirt will be woven from.

Lettersshirt</w>
41st mergeshirt</w>
71st mergeshirt</w>
1,445th mergeshirt</w>
2,012th mergeshirt</w>
Token ID2523

CLIP's tokenizer starts from letters and keeps joining the pair it learned earliest, until “shirt” is one piece it knows. The rank is where the pair sits among the 48,894 merges it learned from its training text; </w> marks the end of a word. The captioner spelled the same word as 3797 in its own vocabulary: the two models do not share numbers.

Draft

It drafts the shirt out of noise, large shapes first.

It takes the 13 IDs and the painting with the bare paper masked. Over 29 steps it guesses the finished picture, coarse first and detail last. I keep its draft as a draft: it decides where the folds and shadows go, and none of its paint reaches the canvas.

the image model drafting the shirt, step 1 of 29 — Step 1 of 29

Prompt, from its own reading: a man in a white shirt looking at the viewer. Each frame is its guess of the finished picture at that step. Outside the outline the painting is kept as I left it.

Every draft it made, one per seed

Code

Code weaves the shirt from those numbers.

Switch between my painting, the model’s draft and the code, and scrub through the steps.

Diffusion step29 of 29

The shirt is woven from the 13 token IDs the image model received. Darker in the draft means a heavier digit, and each digit lands at the step the model settled its spot. My sketched lines stay on top.