An image looks nothing like a sentence. But once you slice it into small pieces and number those pieces, it becomes a sequence — and a transformer can read it the same way it reads words.
The obvious way to feed an image into a model is to flatten it into a long list of pixel values. A 224×224 image has 150,528 pixels. Each pixel has three values (red, green, blue), giving you a sequence of about 450,000 numbers.
The problem is attention. As we covered in the Deep Dive, attention costs scale quadratically with sequence length. A 450,000-token sequence would make the attention computation astronomically expensive — completely impractical. You'd need attention scores between every pair of pixels across the entire image, every forward pass.
The insight that unlocked vision transformers was a simple spatial one: individual pixels aren't meaningful in isolation anyway. A single red pixel tells you almost nothing. A 16×16 patch of pixels — showing texture, edge, color gradient, shape fragment — tells you quite a lot.
The Vision Transformer (ViT), introduced by Google in 2020, solves the sequence length problem by splitting the image into non-overlapping patches — typically 16×16 pixels each. A 224×224 image divided into 16×16 patches gives you 196 patches. That's 196 tokens instead of 450,000. Now attention is manageable.
Each patch is flattened into a vector of numbers and projected into the model's working dimension — exactly like how a word gets converted to an embedding vector in a language model. The 196 patch embeddings become the sequence that gets fed into the transformer.
In a language model: sentence → words → embedding vectors → transformer.
In a vision model: image → patches → embedding vectors → transformer.
The transformer itself is unchanged. Only the tokenization strategy differs.
Trained on millions of labeled images, a vision transformer learns to recognize which patch patterns co-occur, which patches attend to which other patches when identifying a given object, and how to combine patch-level features into image-level understanding.
The attention mechanism is particularly striking here. When researchers visualize which patches a vision transformer attends to when identifying a dog, the attention maps often highlight the face, the fur texture, and the body outline — the same features a human would look at. The model discovered what to look for without being told.
Once images are just sequences of patch tokens, combining them with text tokens becomes straightforward: you concatenate the two sequences. A text prompt becomes tokens. An image becomes tokens. Stack them together and feed them into a transformer that was trained on both.
This is exactly how GPT-4 Vision, Claude's vision capability, and Google's Gemini work. When you upload an image and ask a question about it, the model receives a sequence that begins with image patch tokens and ends with your text tokens. The attention mechanism can relate any text token to any image patch — letting the model answer questions about specific regions of the image, describe visual content, or connect what it sees to what it knows from text training.
DALL-E, Stable Diffusion, and Midjourney run this in reverse — they generate image patches from text tokens. The same fundamental relationship between text embeddings and image patch embeddings applies; the generation direction is flipped. You're still running a transformer over a combined sequence. The output is patch values instead of word probabilities.
Here's what's worth pausing on: the transformer architecture that was designed for predicting the next word in a sentence — with attention heads, feed-forward layers, residual connections, and layer normalization — was applied to images by changing nothing but how the input is tokenized. No new architecture was invented. No new training paradigm was required.
The pre-2020 assumption was that images required convolutional neural networks — a completely different architecture designed specifically for spatial data. Vision transformers showed that if you tokenize correctly, the general-purpose sequence learner is at least as good, often better, and dramatically more flexible.
A 16×16 pixel patch is the "word" in a vision model. What's lost by treating patches as independent tokens rather than raw pixels? What's gained? And what does it tell you that the same attention mechanism that relates "the" to "cat" in a sentence can also relate a fur-texture patch to an ear-shape patch in an image?