The most powerful AI systems in the world are built from billions of copies of one extremely simple thing. Understanding that thing completely is the foundation for understanding everything else.
You've probably heard that artificial neural networks are modeled on the brain. That's true in the same way that airplane wings are modeled on birds — the inspiration is real, but the implementation has diverged so far from the source that the comparison can mislead as much as it helps.
A biological neuron receives electrochemical signals from other neurons through dendrites. If the combined signal is strong enough, it fires — sending its own signal down the axon to connected neurons. It's a threshold-based switch with a lot of biological complexity underneath.
An artificial neuron is a much simpler abstraction: it takes some numbers in, does a specific calculation, and produces one number out. That's the whole thing.
The brain metaphor is useful for motivation but dangerous if taken literally. Artificial neurons don't "fire," don't have memory between activations, and aren't connected in anything like the brain's architecture. The math is borrowed from neuroscience's vocabulary, not its biology.
A single artificial neuron does one thing: it computes a weighted sum of its inputs, adds a bias, then passes the result through an activation function.
Let's unpack that in plain English.
The neuron receives some number of input values — let's say three: x₁, x₂, x₃. These might represent pixel brightness values in an image, token embeddings in a language model, or any other numerical data you've fed into the network.
Each input has a corresponding weight: w₁, w₂, w₃. A weight is just a number — initially random — that controls how much influence that input has on the neuron's output. A weight of 2.0 means "this input really matters." A weight of 0.01 means "barely pay attention to this." A negative weight means "if this input is high, the output should be lower."
The weights are what the model learns. Everything else — the architecture, the training process, the hardware — exists to find good values for the weights. In a 7-billion parameter model, you have 7 billion of these numbers that have been tuned through training.
The bias is one additional number that gets added after the weighted sum. Think of it as the neuron's "default level" — how active it is when all inputs are zero. It gives the neuron more flexibility to fit patterns in the data.
The neuron computes: (x₁ × w₁) + (x₂ × w₂) + (x₃ × w₃) + bias
# A single neuron — the whole idea in four lines def neuron(inputs, weights, bias): weighted_sum = sum(x * w for x, w in zip(inputs, weights)) return weighted_sum + bias
That's it. Multiply each input by its weight, add them up, add the bias. The result is a single number.
If neurons just computed weighted sums, stacking layers of them would accomplish nothing useful. A stack of linear operations is still just a linear operation — no matter how deep you go. You'd be building an expensive straight line.
Activation functions solve this by introducing a non-linearity — a bend in the output curve. This is what allows networks to learn complex, non-linear patterns.
The most common activation function today is ReLU — Rectified Linear Unit. It couldn't be simpler:
# ReLU: if the value is negative, output zero. Otherwise, pass it through. def relu(x): return max(0, x)
That's the entire function. If the neuron's weighted sum is negative, output zero (the neuron is "not active"). If it's positive, pass it through unchanged. This simple threshold is enough non-linearity to allow deep networks to learn almost any function.
The non-linearity is doing more work than it appears. Without it, a 100-layer network is mathematically equivalent to a 1-layer network. The activation function is what makes "deep" actually mean something.
This is worth pausing on because it's easy to gloss over and it's the most important thing in this course:
The weights are just numbers. Floating point numbers stored in memory. There is no knowledge representation, no symbol system, no rule base. A trained neural network is, at the lowest level, a very large list of decimal numbers.
What makes those numbers meaningful is that they were found through a process (training) that optimized them to minimize prediction error on an enormous dataset. The "knowledge" isn't stored anywhere explicitly — it's distributed across all the weights simultaneously, in patterns that no human could read or interpret directly.
When someone says a model was "updated" or "fine-tuned," they mean: some of those numbers were adjusted. When a model is "deployed," they mean: those numbers are loaded into GPU memory and used to run inference. When a model is "quantized to 4-bit," they mean: those numbers have been compressed from 16-bit or 32-bit representations to 4-bit, trading some precision for reduced memory usage.
A 7B parameter model in 16-bit (FP16) precision requires ~14GB of VRAM just to load — because 7 billion numbers × 2 bytes each = 14GB. This is why model size directly determines hardware requirements.
A single neuron can only draw a straight line through data — it can separate inputs into "above the line" and "below the line," but it can't handle anything more complex than that.
The classic demonstration is the XOR problem. XOR returns true when exactly one of two inputs is true. You cannot draw a single straight line that separates the true cases from the false cases. It requires a curve — and curves require multiple neurons working together.
This was actually a crisis point in AI history. In 1969, Minsky and Papert's book Perceptrons proved that single-layer networks couldn't solve XOR, and the field largely concluded that neural networks were useless. It took until the mid-1980s — and the discovery of backpropagation — to realize that multiple layers solved the problem entirely.