Loading lesson…
AI Systems Engineering: Run, Operate & Evaluate / Running Models
Will this model fit, and what does quantization buy?
Reviewed by Yuvaraj
The single most practical question in applied AI is often the dullest-sounding: will this model fit? Answering it means adding up where the memory goes and knowing which knob, precision, moves the biggest number. This lesson gives you the arithmetic and the intuition, then hands you a calculator to play with.
At inference time, VRAM is consumed by three parts:
The weights are easy to compute exactly, and they are where quantization does its work.
Weight memory is just parameter count times bytes per parameter:
Answer from memory before revealing, retrieval practice is what builds durable recall.
Roughly how much weight memory does a 7B-parameter model need at fp16?
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Bytes per parameter is set by the numeric precision you store weights in:
| Precision | Bytes/param | A 7B model |
|---|---|---|
| FP32 | 4 | 28 GB |
| FP16 / BF16 | 2 | 14 GB |
| INT8 | 1 | 7 GB |
| INT4 | 0.5 | 3.5 GB |
That table is the whole reason quantization matters: it is the difference between a model needing a datacentre card and running on a laptop GPU.
For a curious beginner
Think of each weight as a number written down with a certain number of decimal places. FP32 writes it very precisely; INT4 rounds it hard. Rounding loses a little accuracy but takes far less room to store, and a big model has billions of these numbers, so the savings are enormous.
How it is actually used
You pick a precision when you load the model. FP16/BF16 is the common default for inference. INT8 and INT4 quantization roughly halve and quarter the weight memory respectively, usually with modest quality loss because modern quantization schemes calibrate the rounding per-group rather than globally. The KV cache and overhead are unaffected by weight precision, so quantization mostly buys you room for weights, which is exactly what turns "doesn't fit" into "fits."
The underlying mechanism
Quantization maps a high-precision tensor to a low-bit integer with a scale (and optional zero-point ):
The stored footprint is the bit-width of times the number of elements, plus the scales. The reconstruction error is what you trade for the smaller footprint; per-channel or per-group scales keep that error small by fitting locally.
Weights are a fixed cost you pay once at load. The KV cache is a running cost that grows as the model generates:
where is the number of layers and the hidden size. The factor of 2 is for keys and values. The consequence: a model that loads comfortably can still run out of memory at long context or high batch size. This is why a "24 GB card runs a 13B model" claim always hides an unstated context length.
Upper bound
The formula above assumes full multi-head attention. Modern models use grouped-query or multi-query attention, which share keys/values across heads and store dramatically less. Treat the naive number as a ceiling, not a measurement.
Add the three parts for a rough answer. Take a ~13B model at int4 weights, serving one request (batch 1) at a 4,096-token context, with an illustrative 40 layers and hidden size 5,120, KV in FP16:
It fits comfortably, but the KV term is the one that moves. Push context to 32k and that 3.1 GB grows roughly eightfold to ~25 GB on its own, and the same card now overflows. The weights told you it loads; the KV cache decides how far you can push it.
Open the GPU / VRAM Calculator
Reading the formulas is one thing; feeling them is another. Open the GPU / VRAM Calculator, put a ~13B model on a 24 GB card at int4, and push context and batch up until it overflows, watch which bar grows faster. That "aha" is the whole lesson.
Everything above is inference. Training adds gradients (one per weight) and optimizer state (Adam keeps two more values per weight), so a rule of thumb is that training a model in FP16 with Adam needs roughly 4× the weight memory just for those buffers, before activations. That gap is why fine-tuning techniques like LoRA and QLoRA exist: they train a small number of extra parameters instead of all of them.