The network starts as random noise. Training is the process of making it less wrong, repeatedly, until it's useful. The mechanism is elegant, slightly counterintuitive, and about 40 lines of the 200.
Before training begins, every weight in the network is initialized to a small random number. The model knows nothing. If you asked it to predict the next token, it would output something close to a uniform distribution — every token equally likely. It's not even wrong in an interesting way; it's just noise.
Training is the process of replacing that noise with structure — values for each weight that make the model reliably better at predicting the next token across the entire training dataset.
The question is: how do you know which direction to adjust a weight? If you have 7 billion weights and you need to make each one slightly better, how do you even know what "better" means for each individual number?
Before you can fix errors, you need to measure them. The loss function is a formula that takes the model's output and the correct answer and produces a single number representing how wrong the model was. Lower loss = less wrong.
For language modeling, the standard loss function is cross-entropy loss. In plain English: given that the model output a probability distribution over the vocabulary, how much probability did it assign to the token that was actually correct?
If the correct next token is "dog" and the model gave "dog" a 70% probability, the loss is low. If the model gave "dog" a 2% probability and distributed the other 98% across irrelevant tokens, the loss is high.
You'll often hear about "loss curves" when people discuss training runs. A loss curve plots the loss value over time. A healthy training run shows the loss steadily decreasing. A loss that plateaus too early, oscillates, or spikes signals problems with the learning rate, data quality, or architecture.
You've run the forward pass, computed the loss, and you know the model was wrong. The question now is: which weights are responsible for the error, and by how much?
This is the job of backpropagation — arguably the most important algorithm in modern AI. It runs the information backwards through the network, from the output layer to the input, computing for each weight: if I change this weight slightly, how much does the loss change?
That answer — how much the loss changes per unit change in a weight — is called the gradient of that weight. It tells you two things: which direction to adjust the weight, and roughly how sensitive the loss is to that weight.
Backpropagation computes the gradient for every single weight in the network in one backward pass. For a 7B parameter model, that's 7 billion gradients computed simultaneously, leveraging the same matrix math as the forward pass.
Backpropagation is an application of the chain rule from calculus. You don't need to understand the math to appreciate what it does: it assigns responsibility for the error to every weight in the network simultaneously, in proportion to how much each weight contributed to the mistake.
With gradients computed for every weight, training takes a step in the direction that reduces the loss. This is gradient descent.
The update rule is simple: new weight = old weight − (learning rate × gradient)
The learning rate controls the step size. Too large, and the weights overshoot the target — the loss might actually get worse. Too small, and training makes almost no progress per step and takes forever to converge. Finding the right learning rate (and adjusting it during training) is one of the key practical skills in machine learning.
# One training step — conceptually # 1. Forward pass: compute the model's prediction output = model.forward(input_tokens) # 2. Compute loss: how wrong was it? loss = cross_entropy(output, target_tokens) # 3. Backward pass: compute gradients for every weight loss.backward() # 4. Update every weight using its gradient for weight in model.parameters(): weight -= learning_rate * weight.gradient
This four-step loop runs millions of times during training. Each iteration is called a training step. A full pass through the entire training dataset is an epoch.
In practice, you don't train on one example at a time — you train on a batch of examples simultaneously. A batch might be 32, 64, or 512 sequences of text processed in parallel.
Batching serves two purposes: it makes better use of the GPU's parallelism (it's faster to process 64 sequences at once than 64 times in sequence), and it produces a more stable gradient estimate — the average gradient across a batch is less noisy than the gradient from a single example.
The batch size is another hyperparameter that affects training stability and speed. Larger batches = smoother gradients but more VRAM required.
Training a frontier model like GPT-4 or Claude involves:
This is why there are only a handful of organizations in the world capable of training frontier models from scratch — and why fine-tuning (starting from an existing model and training on a smaller, specific dataset) has become so important. You inherit all that learned structure and adjust it for your specific use case at a fraction of the cost.
Fine-tuning is not re-training. You're taking a model that already knows English, code, logic, and world knowledge — and nudging its weights toward a specific style, domain, or behavior. The expensive part (learning language from scratch) has already been done.