Both approaches continue training on your data. Full fine-tuning updates every weight in the model. PEFT freezes the model and trains a tiny set of added parameters — often less than one percent of the total. The surprise is how rarely the expensive option wins.
Full fine-tuning is exactly what the Deep Dive track described for training, restarted from a trained checkpoint: run your examples through the model, measure the error, and backpropagate updates through every weight. Nothing is off limits. The entire model reshapes itself around your data.
That completeness is its strength. When the target domain is far from what the model saw in pretraining — a new language, a specialized scientific corpus, the non-text token domains from the Beyond Text track — full fine-tuning gives the model maximum freedom to adapt. For large domain shifts, it remains the most powerful option.
It's also the reason full fine-tuning is expensive, in two distinct ways.
Training doesn't just need the weights in memory. For every weight being updated, the standard Adam optimizer also keeps a gradient plus two running statistics of its own. In practice, fine-tuning a model this way takes roughly 3–4× the memory of the model itself — and often more once you account for activations. A 7B-parameter model that runs comfortably on one GPU for inference can demand close to 100 GB of GPU memory to fully fine-tune. The hardware bill scales with every weight you allow to change.
When every weight can move, every capability can degrade. The model's general abilities — reasoning, instruction-following, everything it learned in pretraining — are stored in the same weights you're now overwriting with your narrow dataset. Push hard enough on a small dataset and the model gets better at your task while quietly getting worse at everything else. This is catastrophic forgetting, and it's insidious precisely because your task-specific evaluation won't catch it. You measure what you tuned for; the damage happens elsewhere.
A model fully fine-tuned on customer-support transcripts can become excellent at support tickets and subtly worse at the general reasoning that made it useful in the first place. Unless you evaluate broadly — not just on your task — you ship the regression without knowing it.
Parameter-efficient fine-tuning starts from an empirical observation: the change a model needs to adapt to a new task is usually far simpler than the model itself. You don't need to move all the weights — you need a small, structured nudge. So PEFT freezes the original weights entirely and trains a small number of new parameters that sit alongside them.
The dominant PEFT method is LoRA — Low-Rank Adaptation. Instead of updating a large weight matrix directly, LoRA learns the change to that matrix as the product of two thin matrices — a compressed representation of the adjustment, in the same spirit as the compression ideas from the Deep Dive track. Those thin matrices, the adapter, are the only thing trained. Typically they amount to less than 1% of the model's parameters.
The consequences follow directly:
QLoRA pushes the economics one step further. The frozen base model doesn't need full numeric precision just to sit in memory during training — so QLoRA quantizes it to 4-bit precision, shrinking its footprint by roughly a factor of four, and trains the LoRA adapter (in normal precision) on top. The result: models that once required a rack of GPUs to adapt can be fine-tuned on a single consumer card, with quality that comes remarkably close to full-precision training.
LoRA and QLoRA are why fine-tuning stopped being something only large labs do. Adapting a capable open-weights model to your task is now a workload for hardware a small team — or an individual — can actually get.
A reasonable worry: if an adapter is bolted onto the model, doesn't every request pay for the detour? No — and this is one of LoRA's most elegant properties. Because the adapter is just a learned change to the weight matrices, it can be merged: added directly into the base weights ahead of time, producing a single ordinary model. At inference there is no adapter, no extra computation, no latency penalty. The tuned model runs exactly as fast as the original.
Alternatively, you can leave adapters unmerged — and that unlocks a deployment pattern full fine-tuning can't match: one base model, many adapters. Keep a single copy of the base loaded and swap lightweight adapters per task, per customer, per domain. Ten fully fine-tuned models means ten full copies of every weight; ten LoRA adapters means one base model plus ten small files.
A support platform can serve fifty clients, each with a brand-voice adapter tuned on their own transcripts, from one GPU deployment of a single base model. The frozen base does the heavy lifting; each adapter contributes the client-specific behavior. This pattern is simply unavailable if every client requires a full copy of the model.
Full fine-tuning: every weight updates; maximum adaptation for large domain shifts; memory on the order of 3–4× the model just for the optimizer machinery; a full model copy per task; real catastrophic-forgetting risk. PEFT with LoRA/QLoRA: under 1% of parameters trained; base weights frozen and preserved; single-GPU training economics; megabyte-scale artifacts; merge for zero-latency inference or hot-swap many adapters over one base.
Given that list, the practical question isn't "which is better." It's "when is the expensive option actually justified?" That's the next page.