Loading lesson…
Fine-Tuning / Efficient Fine-Tuning
Fine-tuning without touching most of the weights.
Reviewed by Yuvaraj
Fine-tuning a large language model the naive way means updating every one of its billions of weights. That is expensive not because of the weights themselves, but because of everything training drags along with them: a full copy of gradients, plus optimizer states (Adam keeps two running averages per parameter). For a 7B model in mixed precision, the total easily exceeds 80 GB of GPU memory, out of reach for a single consumer card. Parameter-efficient fine-tuning (PEFT) asks a sharper question: can we adapt the model well while training only a tiny fraction of it? LoRA and QLoRA are the dominant answers.
LoRA (Low-Rank Adaptation) freezes the pretrained weight matrix entirely and learns a small additive update beside it. Instead of learning a dense (which would be as large as ), it factors the update into two thin matrices:
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
At inference the adapted layer computes:
where is the rank and is a scaling constant (the ratio controls how strongly the adapter speaks). Only and receive gradients; never moves. Because is just an added term, once training is done you can compute once and fold it into , the merged model has identical shape and zero extra inference latency.
Key knobs: the rank , the scaling , which modules get adapters (usually the attention projections, query, key, value, output), and adapter dropout.
For a curious beginner
Adapting a pretrained model to a new task rarely requires rewriting what it knows, it needs a nudge in a few consistent directions. A low-rank matrix is precisely "a few directions," so it can carry most of that nudge with very little capacity.
How it is actually used
You keep the frozen base as-is and attach small trainable side matrices to chosen layers. You train those, then either merge them into the weights for deployment or keep them as a swappable adapter file measured in megabytes.
The underlying mechanism
The paper's hypothesis is that the weight update has low intrinsic rank. If the useful change lives in an -dimensional subspace, factoring it as loses little, while cutting trainable parameters from to .
Take one attention projection in a 7B model with hidden size , and use rank .
| Quantity | Formula | Value |
|---|---|---|
| Full update (one matrix) | 16,777,216 | |
| LoRA update (one matrix) | 131,072 | |
| Reduction factor | 128x |
So a single matrix drops from about 16.8M trainable parameters to about 131K, a 128x reduction. Apply LoRA to the query and value projections across all 32 layers ( matrices) and you train roughly million parameters, about 0.12% of the 7B base. Full fine-tuning would train all 7,000M.
| Method | Base weights | Gradients + optimizer states | Fits on |
|---|---|---|---|
| Full fine-tune | 14 GB (FP16) | 80+ GB (FP32 grads, Adam m/v, master weights) | multi-GPU |
| LoRA | 14 GB (FP16, frozen) | ~0.1 GB (adapters only) | single 24 GB GPU |
| QLoRA | ~4 GB (4-bit NF4) | ~0.1 GB (adapters only) | single 12-16 GB GPU |
QLoRA pushes the memory win further. The frozen base is quantized to 4-bit NF4 (a "normal float" format tuned for the roughly Gaussian distribution of weights), while the LoRA adapters still train in higher precision (BF16). Gradients flow through the quantized base to update and ; the base is dequantized on the fly for each matmul but is never stored in full precision. Combined with paged optimizers, this lets you fine-tune models that otherwise would not fit at all, a 7B model on a single 12-16 GB card, or a 65B model on one 48 GB GPU. The tradeoff is a small quality gap versus full fine-tuning and somewhat slower steps from dequantization, but for most adaptation tasks the difference is minor.
Why merging matters
Because the adapter is a pure additive term, LoRA gives you both worlds: keep adapters separate to hot-swap many task-specific behaviors over one shared base, or merge one into for a single deployable checkpoint with no runtime overhead.
Common mistakes