Loading lesson…
GPUs & Hardware / Fitting & Scaling
Estimating whether a model actually fits.
Reviewed by Yuvaraj
Before you download a model or rent a GPU, one question settles every other decision: will the model actually fit in the card's memory? The answer is not a guess, it is arithmetic you can do on the back of an envelope. Inference memory is a budget with four line items, and once you can add them up you can predict, before spending anything, whether a given model runs on a given GPU and how much context you can afford on top of it.
When a model serves requests, VRAM is consumed by four things:
The total must sit comfortably below the card's capacity, with headroom, because you never get the full advertised number. Weights and the KV cache are the two you can compute precisely, so start there.
Weight memory is parameter count times bytes per parameter:
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Bytes per parameter is set entirely by the numeric precision you store the weights in. That single choice moves the biggest number in the budget:
| Precision | Bytes/param | Weight memory (7B) |
|---|---|---|
| FP32 | 4 | 28 GB |
| FP16 / BF16 | 2 | 14 GB |
| INT8 | 1 | 7 GB |
| INT4 | 0.5 | 3.5 GB |
That table is the whole reason quantization matters: it is the difference between a model that demands a datacentre accelerator and one that runs on a laptop GPU. FP16 or BF16 is the common default for inference; INT8 and INT4 quantization roughly halve and quarter the weight footprint.
Quantization buys memory, and it costs a little quality
Quantization rounds each weight to fewer bits. INT8 is usually near-lossless. Modern INT4 schemes (GPTQ, AWQ, NF4) calibrate the rounding per group rather than globally and lose only a little on most tasks, but the loss is real and shows up first on rare tokens and multi-step reasoning, so measure on your own eval rather than trusting a headline number. Note also what quantizing weights does not shrink: the KV cache and activations have their own dtype, so they are unaffected unless you quantize them separately.
Weights are paid once. The KV cache is a running cost that expands as the sequence lengthens:
where is the number of layers, the sequence length in tokens, the hidden size, the bytes per element (2 for FP16), and the number of concurrent sequences. The leading factor of is for storing keys and values. The consequence is the trap that catches everyone: a model that loads with room to spare can still run out of memory at long context or high concurrency.
For a curious beginner
To write each new token, the model re-reads the whole conversation so far. Rather than re-deriving what every earlier token "means" at every step, it writes that down once and keeps it. The KV cache is that notebook: memory you spend to avoid redoing work, and it gains one line per token, per layer.
How it is actually used
Per layer the model keeps two tensors, K and V, shaped roughly [batch, heads, seq, head_dim]. Memory therefore scales linearly with both context length and batch size. This is why a 7B model that loads on a small card OOMs the moment you push context or serve many users at once, and why serving stacks like vLLM pre-reserve a KV pool and page it.
The underlying mechanism
The naive formula assumes full multi-head attention, where the KV width equals . Grouped-query and multi-query attention share keys and values across heads, so replace with , which is smaller. A model with 8 KV heads instead of 32 stores a quarter as much. Treat the naive number as a ceiling.
Take a 7B model with a Llama-2-style configuration, layers, , full multi-head attention, serving one request (batch 1) at a 4,096-token context, KV stored in FP16.
First the KV cache:
Now add the four parts, with the weights quantized to INT4:
At INT4 this model is comfortable on a 24 GB card, leaving roughly 16 GB free. Even at FP16 the sum is GB, still under 24 GB. So the headline "a 7B fits on a 24 GB card" is true at short context. The question is how far you can push it.
Push the context from 4k to 32k and the KV term grows eightfold, from ~2.1 GB to ~17.2 GB, while the weights stay put at 3.5 GB. The INT4 total climbs to ~23.2 GB and the 24 GB card is nearly full. At FP16 weights the same 32k budget would be ~33.7 GB and overflow the card entirely. The weights told you it loads; the KV cache decides how far you can push it.
Common mistakes