Section 03 of 07

Networks — Layers, Matrix Math, and Why GPUs Exist

A single neuron draws a line. A layer of neurons draws many lines at once. Stack enough layers and you can approximate any function — including the function that maps text to more text.

From One Neuron to a Layer

A layer is just multiple neurons operating in parallel on the same input. If you have 512 neurons in a layer, each one receives the same inputs and produces its own output — giving you 512 different "views" of the input data, each weighted differently.

The outputs of that layer become the inputs to the next layer. Each subsequent layer can detect increasingly abstract patterns — the early layers might detect simple features, while deeper layers combine those into higher-level concepts. This is the core intuition behind "deep" learning: the depth refers to the number of layers.

In practice, a modern language model might have 32, 64, or even 96 layers stacked on top of each other. The information flows from the input (your prompt) through each layer in sequence until the final layer produces a probability distribution over the vocabulary.

Key Insight

You don't design what each layer learns. The training process figures that out automatically. You just decide the architecture — how many layers, how wide each layer is — and training fills in the weights.

The Forward Pass: Data as a Pipeline

When you run inference — when you give a model a prompt and it generates a response — the data flows through the network in one direction: forward. Input goes in one end, output comes out the other. This is called the forward pass.

Think of it as a pipeline:

  1. Your prompt is tokenized and converted to numbers (embeddings)
  2. Those numbers enter Layer 1, which transforms them according to its weights
  3. The transformed values enter Layer 2, and so on through all layers
  4. The final layer produces probabilities over every token in the vocabulary
  5. The highest-probability token is selected as the next word
  6. That token is appended to the input and the whole process repeats

Notice that step 6: to generate the second token, the model runs a complete forward pass through all layers again — this time with one more token appended. For a response that's 500 tokens long, the model runs 500 separate forward passes. This is why inference isn't instantaneous, and why longer contexts are more expensive to process.

Matrix Multiplication: Why This Operation Is the Entire Job

Here's the mathematical reality underneath everything we've described: the forward pass is almost entirely matrix multiplication.

A matrix is just a 2D grid of numbers. When you multiply two matrices together, you compute dot products between rows and columns in a very specific pattern. It sounds abstract until you realize: running a layer of neurons on an input is exactly a matrix multiplication.

The weight matrix for a layer has one row per output neuron and one column per input. Multiply the input vector by that matrix and you get the output of every neuron in the layer simultaneously — in one operation.

# A whole layer of neurons — same as a matrix multiply
# inputs: vector of length N
# weights: matrix of shape (N, M) — N inputs, M output neurons
# bias: vector of length M

output = relu(inputs @ weights + bias)

# The @ operator is matrix multiplication in Python
# This one line replaces a loop over M neurons

The entire network, every layer, is a chain of these operations. The forward pass for a full 7B parameter model on a single token is thousands of matrix multiplications.

Why This Matters

Matrix multiplication is the core operation. Everything else in the architecture — attention, normalization, embeddings — either is matrix multiplication or is structured specifically to work well with it.

Why GPUs Exist: The Embarrassingly Parallel Problem

A CPU is designed to execute instructions sequentially and quickly, with sophisticated logic for handling conditional branches, memory hierarchies, and complex instruction sets. It's a general-purpose problem solver with a small number of very powerful cores — typically 8 to 64.

A GPU was originally designed for graphics — specifically for transforming millions of pixels simultaneously. To do that, it sacrifices the complex per-core logic of a CPU in favor of having thousands of simple cores that all execute the same operation at the same time.

Matrix multiplication is embarrassingly parallel — a term in computer science meaning the computation can be broken into independent pieces with no dependencies between them. Each element of the output matrix depends only on a single row and column of the input matrices — no element needs to wait for another to be computed first.

This is a perfect match for GPU architecture. An A100 GPU has 6,912 CUDA cores. An H100 has nearly 17,000. They can all multiply simultaneously. What would take a CPU minutes can take a GPU seconds — or milliseconds.

Key Insight

The AI hardware boom isn't about AI being mysterious. It's about one operation — matrix multiply — being central to the entire field, and GPUs being extremely good at it. NVIDIA didn't predict the AI boom; AI happened to need exactly what NVIDIA was already building.

VRAM: Where the Weights Live

When a model runs inference, its weights must be loaded into GPU memory (VRAM). Not RAM. Not disk. VRAM — the fast memory physically on the GPU card. The CPU can't run the matrix multiplications; only the GPU can. So the weights have to live where the GPU can reach them instantly.

This is the hard constraint that determines what hardware you need:

A consumer GPU like an RTX 4090 has 24GB VRAM. That fits a 7B model in FP16 with some headroom for inference — but nothing larger. Running a 70B model requires either multiple high-end GPUs, quantization (reducing precision from 16-bit to 4-bit, cutting VRAM requirements by 4×), or a server-grade card like an A100 (80GB) or H100 (80GB).

Practical Note

Quantization is why you'll see model filenames like llama-3-70b-Q4_K_M.gguf. Q4 means 4-bit quantization — approximately 4× reduction in VRAM at the cost of some precision. Q5 and Q6 are better quality but larger. Q4 is usually the sweet spot for consumer hardware.

Parameters and Model Size

When someone says a model has "7 billion parameters," they mean: the model's weight matrices, combined, contain 7 billion individual numbers.

Bigger models have more weights, which means more capacity to represent complex patterns in language. But bigger also means more VRAM required, slower inference, and vastly more compute to train. The art of modern AI research is getting maximum capability out of minimum parameters — which is why small models are often more interesting than their headline numbers suggest.

Discussion
  • You manage GPU infrastructure. How does knowing that inference is "mostly matrix multiply" change how you'd think about capacity planning for an AI workload?
  • Why might a quantized 70B model (Q4) perform better than a full-precision 7B model, even on the same hardware?
  • If the forward pass runs separately for every output token, what does that imply about latency for a 2,000-token response vs. a 10-token response?
← Section 02 Course Home Section 04 →