The 200 lines aren't the destination. They're the lens. Now we use that understanding to decode real AI systems, make sense of marketing claims, and reason about decisions you'll actually face.
You now know that model size (the B in 7B, 13B, 70B) refers to the number of parameters — the individual weight values across all the matrices in the model. More parameters means:
Bigger is not always better for a given use case. A 7B model fine-tuned on medical literature may outperform a 70B general-purpose model on medical Q&A — because the fine-tuning has concentrated the model's capacity on a specific domain, and the 70B model is spending its parameters on knowledge you don't need.
When evaluating a model for a specific task, benchmark performance on that task matters more than raw parameter count.
The context window is block_size in the code — the maximum number of tokens the model can see at once during a single forward pass. Everything outside the context window is invisible to the model.
This is not a storage limitation — it's a computational one. Attention is O(N²) in sequence length. A 128k context window requires attention matrices of 128,000 × 128,000 per layer. At 32 layers, per forward pass, per token generated, this is why long-context inference is expensive.
Practical implications:
Fine-tuning is taking a pretrained model's weights and continuing the training process on a smaller, specific dataset — adjusting those weights toward a target behavior.
You're not teaching the model a new language or new reasoning skills. You're adjusting the probability distributions it outputs toward patterns in your fine-tuning data. A model fine-tuned on customer service transcripts will generate responses that sound more like customer service representatives — because those patterns now have higher weight in the model's outputs.
This is powerful but comes with risks. Fine-tuning on a biased or low-quality dataset will steer the model toward those patterns — including bad ones. Fine-tuning is not a magic alignment tool; it's weight adjustment, and it follows the same logic as all training.
RLHF — Reinforcement Learning from Human Feedback — is a form of fine-tuning. Human raters score model outputs, and those scores are used to train a "reward model" that then guides further fine-tuning. It's how raw next-token prediction gets shaped into a helpful assistant. It's still weight adjustment.
You now have the vocabulary to understand why hallucination happens. The model's job is to produce the most probable next token given the context. It has no built-in mechanism to distinguish "things I know from training data" from "things I'm generating based on statistical patterns."
When a model confidently names a fake paper, invents a citation, or states a false fact, it's not "lying" in any intentional sense. It's producing tokens that are statistically consistent with the kind of text that appears in contexts like the current one. Academic text contains citations; given the context, citation-shaped text has high probability — regardless of whether the cited paper exists.
The model has no truth-checking mechanism at the token generation level. It has pattern matching. Hallucination is pattern matching producing plausible-sounding output that isn't grounded in reality. Every AI system you deploy will do this to some degree — understanding why helps you design around it.
You've now seen the complete architecture. The concepts you have are sufficient to understand:
None of these require new foundational concepts. They're all extensions of what you already know.
Andrej Karpathy's "Let's build GPT from scratch" video on YouTube walks through code very similar to what we read in Section 06, live, with narration. It's the best next step if you want to see the code in motion. His "Neural Networks: Zero to Hero" series goes deeper on every concept in this course.
There's a version of AI education that tries to maintain a sense of mystery — that frames these systems as fundamentally inscrutable, that leaves you dependent on expert interpretation. That framing serves people who want to sell you certainty.
The truth is more interesting. These systems are complex, but they're not mysterious. The architecture is public. The math is accessible. The code fits in 200 lines. The concepts are learnable by anyone willing to spend time with them.
Understanding what's actually happening inside these systems doesn't make them less impressive. It makes you better equipped to use them well, to reason about their failures, and to make decisions about where they belong in the systems you build and maintain.
That was always the point.