Lab
GPU / VRAM Calculator
Estimate whether a model fits given precision and context.
Before you rent or buy a card, you can estimate on paper whether a model will fit. Three things consume VRAM at inference time: the weights, the KV cache that grows with context and batch, and framework overhead. Pick a model class, a precision and a card, then push the context up until it overflows.
Estimate, not a benchmark. The KV cache assumes full multi-head attention (an upper bound, grouped-query/MQA models store much less), and overhead is a flat 15% for the CUDA context, buffers and fragmentation. Real usage varies with the serving stack.
Weights dominate at short context: ~7B class at fp16 is 13.04 GiB of parameters (2 bytes each). Quantizing trades precision for room, dropping fp16 to int4 quarters the weight memory, which is often what turns “doesn’t fit” into “fits.”
The KV cache is the part people forget: it scales linearly with both context length and batch size. At long context on a big batch it can rival or exceed the weights, which is why a model that loads fine can still run out of memory mid-generation.
Challenge
Put the ~13B class on a 24 GB card at int4. It fits with room to spare at 4k context, now raise the context and batch until it overflows. Which grows faster, and roughly what context did it take at batch 8?