A single neuron draws a line. A layer of neurons draws many lines at once. Stack enough layers and you can approximate any function — including the function that maps text to more text.
A layer is just multiple neurons operating in parallel on the same input. If you have 512 neurons in a layer, each one receives the same inputs and produces its own output — giving you 512 different "views" of the input data, each weighted differently.
The outputs of that layer become the inputs to the next layer. Each subsequent layer can detect increasingly abstract patterns — the early layers might detect simple features, while deeper layers combine those into higher-level concepts. This is the core intuition behind "deep" learning: the depth refers to the number of layers.
In practice, a modern language model might have 32, 64, or even 96 layers stacked on top of each other. The information flows from the input (your prompt) through each layer in sequence until the final layer produces a probability distribution over the vocabulary.
You don't design what each layer learns. The training process figures that out automatically. You just decide the architecture — how many layers, how wide each layer is — and training fills in the weights.
When you run inference — when you give a model a prompt and it generates a response — the data flows through the network in one direction: forward. Input goes in one end, output comes out the other. This is called the forward pass.
Think of it as a pipeline:
Notice that step 6: to generate the second token, the model runs a complete forward pass through all layers again — this time with one more token appended. For a response that's 500 tokens long, the model runs 500 separate forward passes. This is why inference isn't instantaneous, and why longer contexts are more expensive to process.
Here's the mathematical reality underneath everything we've described: the forward pass is almost entirely matrix multiplication.
A matrix is just a 2D grid of numbers. When you multiply two matrices together, you compute dot products between rows and columns in a very specific pattern. It sounds abstract until you realize: running a layer of neurons on an input is exactly a matrix multiplication.
The weight matrix for a layer has one row per output neuron and one column per input. Multiply the input vector by that matrix and you get the output of every neuron in the layer simultaneously — in one operation.
# A whole layer of neurons — same as a matrix multiply # inputs: vector of length N # weights: matrix of shape (N, M) — N inputs, M output neurons # bias: vector of length M output = relu(inputs @ weights + bias) # The @ operator is matrix multiplication in Python # This one line replaces a loop over M neurons
The entire network, every layer, is a chain of these operations. The forward pass for a full 7B parameter model on a single token is thousands of matrix multiplications.
Matrix multiplication is the core operation. Everything else in the architecture — attention, normalization, embeddings — either is matrix multiplication or is structured specifically to work well with it.
A CPU is designed to execute instructions sequentially and quickly, with sophisticated logic for handling conditional branches, memory hierarchies, and complex instruction sets. It's a general-purpose problem solver with a small number of very powerful cores — typically 8 to 64.
A GPU was originally designed for graphics — specifically for transforming millions of pixels simultaneously. To do that, it sacrifices the complex per-core logic of a CPU in favor of having thousands of simple cores that all execute the same operation at the same time.
Matrix multiplication is embarrassingly parallel — a term in computer science meaning the computation can be broken into independent pieces with no dependencies between them. Each element of the output matrix depends only on a single row and column of the input matrices — no element needs to wait for another to be computed first.
This is a perfect match for GPU architecture. An A100 GPU has 6,912 CUDA cores. An H100 has nearly 17,000. They can all multiply simultaneously. What would take a CPU minutes can take a GPU seconds — or milliseconds.
The AI hardware boom isn't about AI being mysterious. It's about one operation — matrix multiply — being central to the entire field, and GPUs being extremely good at it. NVIDIA didn't predict the AI boom; AI happened to need exactly what NVIDIA was already building.
When a model runs inference, its weights must be loaded into GPU memory (VRAM). Not RAM. Not disk. VRAM — the fast memory physically on the GPU card. The CPU can't run the matrix multiplications; only the GPU can. So the weights have to live where the GPU can reach them instantly.
This is the hard constraint that determines what hardware you need:
A consumer GPU like an RTX 4090 has 24GB VRAM. That fits a 7B model in FP16 with some headroom for inference — but nothing larger. Running a 70B model requires either multiple high-end GPUs, quantization (reducing precision from 16-bit to 4-bit, cutting VRAM requirements by 4×), or a server-grade card like an A100 (80GB) or H100 (80GB).
Quantization is why you'll see model filenames like llama-3-70b-Q4_K_M.gguf. Q4 means 4-bit quantization — approximately 4× reduction in VRAM at the cost of some precision. Q5 and Q6 are better quality but larger. Q4 is usually the sweet spot for consumer hardware.
When someone says a model has "7 billion parameters," they mean: the model's weight matrices, combined, contain 7 billion individual numbers.
Bigger models have more weights, which means more capacity to represent complex patterns in language. But bigger also means more VRAM required, slower inference, and vastly more compute to train. The art of modern AI research is getting maximum capability out of minimum parameters — which is why small models are often more interesting than their headline numbers suggest.