Before we look at a single line of code, we need to agree on what problem we're solving. It turns out to be much simpler — and much stranger — than most people expect.
When most people hear the word "intelligence," they think of reasoning, creativity, understanding. When a computer scientist hears it, they think of something far more concrete: pattern matching at scale.
That's not a reduction. It's actually the key insight. A language model doesn't "understand" text the way you do. It has been exposed to an enormous amount of text and has learned the statistical patterns of how words follow other words, how ideas connect, how sentences are structured across billions of documents.
When you ask it a question, it's not retrieving an answer from a database or reasoning from first principles. It's producing the sequence of words that — based on everything it has seen — is most likely to follow the words you gave it.
Intelligence, in this context, is sophisticated pattern completion. That turns out to be enough to do things that look remarkably like thinking.
Here's the central idea of the entire course: almost everything a language model does reduces to next-token prediction.
A "token" is roughly a word or a word-fragment. Given a sequence of tokens, the model's job is to predict the next one. That's it. That's the whole task at training time.
This sounds trivial. It isn't. To reliably predict the next word in a sentence, you have to implicitly understand grammar, context, facts about the world, conversational conventions, and much more. The model doesn't learn any of this explicitly — it learns it as a side effect of getting really good at prediction.
When you ask a model to summarize a document, translate a sentence, write code, or answer a question — all of that is still, under the hood, next-token prediction. The framing of the task changes. The fundamental operation doesn't.
Think about autocomplete on your phone. It predicts the next word. A language model is doing the same thing — just with a much larger model, much more training data, and the ability to maintain coherence across thousands of words rather than just one. What do you think changes when you scale that up?
A model takes in tokens (text, broken into pieces) and outputs a probability distribution over the entire vocabulary — every word it knows, each assigned a likelihood of being the right next token.
So when you type "The sky is," the model doesn't return "blue." It returns something like: blue: 38%, clear: 12%, dark: 9%, grey: 8%, filled: 5%... for every word in its vocabulary. We then pick from that distribution — usually taking the most likely word, but sometimes introducing some randomness to make the output feel more natural. That randomness is what's called temperature.
This is important: the model doesn't have opinions. It has probabilities. What feels like a confident, opinionated response is actually the output of a carefully calibrated probability distribution learned from human-generated text.
When you set "temperature = 0" in an API call, you're saying: always pick the single most probable next token. Higher temperature = more randomness = more "creative" but also more unpredictable output.
Every language model goes through three completely distinct phases, and it's important not to confuse them:
Before the model exists, there's just text — an enormous amount of it. Books, websites, code repositories, scientific papers. This is assembled, cleaned, and formatted into a training corpus. The model hasn't seen any of it yet. The dataset is static by the time training begins.
The model (initially just random numbers) is shown the training data over and over. For each piece of text, it tries to predict the next token. It's wrong most of the time at first. Each wrong prediction triggers an update to the model's weights — tiny adjustments that make it slightly less wrong the next time. This process runs for days or weeks on thousands of GPUs. By the end, the model's weights have encoded an enormous amount of structure from human language.
The model's weights are now frozen. No more learning. You give it a prompt, it predicts the next token, appends it, predicts the next one, and so on. This is what happens every time you chat with an AI. The model is not updating based on your conversation — it's just running the forward pass on static weights.
The model you talk to today learned everything it knows before you arrived. Your conversation doesn't teach it anything new. Fine-tuning and RAG exist specifically to work around this limitation — but they're not the base model learning in real time.
The code we'll walk through in this course is a minimal implementation of a transformer — the architecture that underlies every major language model: GPT, Claude, Llama, Gemini. All of them.
It contains, in roughly this distribution:
By the time we finish Section 06, every one of those lines will map to a concept you understand. Nothing will be a black box. And from that foundation, you'll be able to reason about why GPT-4 is different from GPT-3, what a 70B model actually means, and why context windows are expensive.