In 2017, a Google paper titled "Attention Is All You Need" introduced the transformer architecture. Every major language model since — GPT, Claude, Llama, Gemini — is built on the same core idea. Here's what it actually does.
Before transformers, the dominant approach to language modeling used recurrent neural networks (RNNs). An RNN processed text one token at a time, left to right, maintaining a hidden state that carried context forward. To understand the word at position 100, it had to have correctly propagated relevant information through 99 previous steps.
This created two serious problems:
The vanishing gradient problem: During training, gradients had to travel backwards through all those sequential steps. Over long sequences, they would become extremely small — effectively meaning the model couldn't learn dependencies between words that were far apart.
No parallelism: Because each step depended on the previous step, RNNs couldn't process a sequence in parallel. You had to process token 1, then token 2, then token 3. On a GPU designed for massive parallelism, this was a severe bottleneck.
Attention solves both problems at once.
With attention, every token looks directly at every other token in the sequence — simultaneously. There's no chain of steps, no information bottleneck. A word at position 1 and a word at position 500 can directly influence each other in a single operation.
Here's the intuition behind attention: when processing a word, instead of using a fixed context window or a compressed hidden state, let the model decide which other words in the sequence are relevant to understanding the current word — and weight its attention accordingly.
Consider the sentence: "The animal didn't cross the street because it was too tired."
What does "it" refer to? The animal, not the street. A human reader resolves this instantly. For a model, this requires connecting "it" back to "animal" despite several words in between. Attention allows the model to learn to give high weight to "animal" when processing "it," regardless of distance.
The attention mechanism introduces three concepts for each token: a Query, a Key, and a Value. The names come from information retrieval — think of it as a database lookup.
Imagine a library where each book has a key (its catalog entry — what it's about) and a value (its actual content). When you come in with a query (what you're looking for), the librarian matches your query against all the keys, gives you a weighted stack of books based on how well each key matches, and you read a blend of the most relevant ones.
In attention, every token generates all three:
For each token, the model computes how well its Query matches the Key of every other token. Those match scores become weights. The output is a weighted blend of all the Values — heavily weighted toward tokens whose Keys matched well.
# Attention in pseudo-code — the actual core is this simple # For each token, compute Q, K, V vectors via learned weight matrices Q = token_embeddings @ W_query K = token_embeddings @ W_key V = token_embeddings @ W_value # Compute match scores: how much should each token attend to each other? scores = Q @ K.transpose() # every Q dot-producted with every K # Normalize into probabilities (softmax = all scores sum to 1.0) attention_weights = softmax(scores) # Weighted sum of values: blend information from attended tokens output = attention_weights @ V
The matrices W_query, W_key, and W_value are learned during training. The model learns what kind of queries to ask and what kind of keys to expose based on the task — the attention pattern itself is not hand-designed.
A single attention operation is called one "head." In practice, transformers run multiple attention heads in parallel — typically 8, 16, or 32 heads depending on the model size.
Each head has its own Q, K, V weight matrices and can learn to attend to different kinds of relationships. One head might learn to track grammatical dependencies (verbs attending to their subjects). Another might track coreference (pronouns attending to the nouns they refer to). Another might capture semantic similarity.
The outputs of all the heads are concatenated and projected back to the model's working dimension. This is why the operation is called multi-head attention — parallel perspectives combined into one representation.
Nobody tells the heads what to learn. Training discovers useful attention patterns automatically. When researchers visualize attention weights in trained models, they find that different heads have spontaneously learned interpretable roles — but the model found those patterns by optimizing for prediction, not because they were specified.
Recall that attention computes match scores between every token and every other token. For a sequence of N tokens, that's N × N comparisons — the computation grows quadratically with sequence length.
Double the context window, and you quadruple the attention computation. This is the fundamental reason context windows are expensive: it's not storage, it's the quadratic cost of attending across the entire sequence.
A 128,000-token context window requires attention computation across 128,000 × 128,000 = 16.4 billion comparisons per layer per forward pass. Managing this efficiently is one of the hardest active research problems in the field.
When API providers charge more for longer context, the quadratic cost of attention is largely why. It's not just storage — it's compute. A 100k-token input is not 10× more expensive than a 10k-token input; it's closer to 100× more expensive at the attention layer.
The original transformer paper discarded recurrence entirely. No more sequential processing. No more hidden states carrying context token-by-token. Just attention — applied in parallel across the full sequence — plus a feed-forward network per layer.
The result was a model that:
GPT-1 used this architecture in 2018. By 2020, GPT-3 had scaled it to 175 billion parameters. The architecture hasn't changed fundamentally since 2017 — the gains have come almost entirely from scale and data.