A GPU spec sheet lists two memory numbers that answer two completely different questions. Capacity, measured in gigabytes, tells you how much fits: will the weights and KV cache load at all? Bandwidth, measured in gigabytes per second, tells you how fast you can move that data between the memory and the compute cores, which sets how quickly the model actually runs. Confusing the two is the most common mistake in hardware planning, and it matters because single-stream LLM decoding is almost always limited by bandwidth, not capacity or raw compute.
Capacity and bandwidth are different specs
These two numbers are independent. A card can be capacity-rich but bandwidth-poor, or the reverse.
VRAM capacity is how many bytes the memory holds, 24 GB, 80 GB, 192 GB. It decides whether the model loads (see the companion lesson, Fitting a Model in VRAM).
Memory bandwidth is how many bytes per second can travel between VRAM and the streaming multiprocessors that do the arithmetic. It decides how fast data can be fed to those cores.
A useful mental image: capacity is the size of the warehouse; bandwidth is the width of the loading dock. A vast warehouse with a narrow dock still ships slowly. Some representative figures (approximate, and they change every generation):
GPU (class)
Memory type
Capacity
Bandwidth
RTX 4090 (consumer)
GDDR6X
24 GB
~1.0 TB/s
A100 80GB (datacenter)
HBM2e
80 GB
~2.0 TB/s
H100 SXM (datacenter)
HBM3
Ask the tutor
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
80 GB
~3.35 TB/s
Notice the 4090 and the A100 are in the same ballpark on bandwidth despite very different capacities and prices, a reminder that the two axes do not track each other.
HBM vs GDDR: two ways to build fast memory
The bandwidth of a memory system is, to first order, its bus width times the data rate per pin:
bandwidth≈bus width×data rate per pin
The two dominant technologies attack that product from opposite ends.
HBM (High Bandwidth Memory)
DRAM dies stacked vertically on the GPU package, linked by through-silicon vias over a silicon interposer
Very wide interface, 1024 bits per stack, several thousand bits across multiple stacks
Wins bandwidth through width, at a moderate per-pin clock
Highest total bandwidth and best bandwidth-per-watt; expensive to manufacture
Standard on datacenter accelerators (A100, H100, MI300)
GDDR (GDDR6 / 6X / 7)
Discrete memory chips soldered around the GPU on the circuit board
Narrower bus, typically 128–384 bits
Wins bandwidth through very high per-pin clock rates instead of width
Lower total bandwidth and capacity than HBM, but far cheaper per gigabyte
Standard on consumer and workstation cards (RTX 4090 uses GDDR6X)
The practical upshot: HBM is why datacenter parts reach 2–3+ TB/s and hold 80 GB or more on a single package, and it is a large part of why they cost what they do.
The roofline model: compute-bound vs memory-bound
To reason about whether a given kernel is limited by memory or by compute, use arithmetic intensity: the number of floating-point operations performed per byte of memory traffic, I=FLOPs/byte. Pair it with two machine numbers, peak compute π (FLOP/s) and peak bandwidth β (bytes/s), and the attainable performance of a kernel is:
P=min(π,β⋅I)
Plot that and you get the roofline. At low intensity the β⋅I term wins: performance rises with a slope equal to bandwidth, and the kernel is memory-bound, the cores wait for data. Past the ridge point at I∗=π/β (the machine's balance), the flat π roof takes over and the kernel is compute-bound.
That ridge point is surprisingly high. An H100-class accelerator does roughly 990 BF16 TFLOP/s against ~3.35 TB/s of bandwidth, so its balance is about 990/3.35≈300 FLOPs per byte. A kernel has to do 300 arithmetic operations for every byte it touches just to break even with the memory system.
A roofline diagram. The horizontal axis is arithmetic intensity (FLOPs per byte); the vertical axis is attainable performance (FLOP/s). A sloped line rising from the origin is the memory-bandwidth roof (its slope equals bandwidth β); it meets a horizontal line, the peak-compute roof (height π), at the ridge point where intensity I* = π/β ≈ 300 FLOPs/byte. Left of the ridge is the memory-bound region: batch-1 LLM decoding sits far to the left at I ≈ 1, deep in memory-bound territory. Right of the ridge is the compute-bound region, where large-batch or prefill GEMM operations land. The picture shows that the same hardware is memory-bound or compute-bound depending only on the kernel's arithmetic intensity.
Why single-stream LLM decoding is memory-bound
Autoregressive generation produces one token per forward pass, and each forward pass must read every weight from VRAM at least once. At batch size 1 the core operation is a matrix-vector product (GEMV): the weight matrix multiplies a single activation vector. Each weight is read (2 bytes in FP16), used for one multiply and one add (2 FLOPs), then discarded. Its arithmetic intensity is therefore:
I≈2bytes2FLOPs=1FLOP/byte
Against a ridge point near 300, an intensity of 1 sits roughly two orders of magnitude to the left of the balance point. The kernel spends about 0.3% of the card's compute and waits on memory the rest of the time. The weights cannot be read fast enough to keep the arithmetic units busy, that is the definition of memory-bound, and it is why decode latency tracks bandwidth, not TFLOPS.
Batching is the escape hatch, for throughput
Serve many sequences at once and the matrix-vector product becomes a
matrix-matrix product (GEMM): the weight matrix is read once but reused across
all sequences in the batch, so intensity rises roughly in proportion to batch
size, I≈batch. Push the batch toward ~300 and the kernel
finally crosses the ridge into compute-bound. This is why serving stacks batch
aggressively, it raises aggregate tokens/sec by amortizing the weight read.
It does not speed up a single stream: one user's latency is still set by how
fast that user's tokens can stream the weights.
Worked example: the tokens-per-second ceiling
Because each token must stream all the weights once, the best possible decode rate for a single stream is simply bandwidth divided by weight bytes. Take a 13B model in FP16 (≈ 26 GB of weights) on a GPU with ~1 TB/s of bandwidth:
1Weight bytes per forward pass13B × 2 bytes (FP16) = 26 GB, read once per generated token
3Time per token (floor)26 GB ÷ 1000 GB/s = 0.026 s = 26 ms, the minimum time just to read the weights
4Tokens/sec (ceiling)1000 ÷ 26 ≈ 38 tokens/sec
So roughly 38 tokens/sec is an idealized upper bound for this model on this card. It is a ceiling, not a prediction: it ignores everything except reading the weights. In reality the number is lower because real bandwidth runs at ~60–85% of the advertised peak, kernel launches and synchronization add overhead, and the KV cache and activations must also be read each step, traffic that grows with context length (at a few thousand tokens the KV reads add on the order of ~10% here; by tens of thousands of tokens they can rival the weights and roughly halve throughput). Expect real-world single-stream rates well under the ceiling.
The value of the ceiling is that it tells you which knobs move it:
Fewer weight bytes. Quantize the same model to INT8 (~13 GB) and the ceiling doubles to ~77 tokens/sec; INT4 (~6.5 GB) roughly quadruples it to ~154. Quantization improves latency, not just capacity, a point easy to miss if you only think of it as a memory-saving trick.
More bandwidth. Move to an HBM3 card at ~3.35 TB/s and the ceiling scales up with it, ~3× here.
Not more compute. Buying a card with more TFLOPS but the same bandwidth does almost nothing for batch-1 decode, because you were never compute-limited to begin with.
Common mistakes
Assuming a higher-capacity card is a faster card. Capacity decides whether the model loads; bandwidth decides how fast it runs. They are separate specs and often diverge.
Buying compute (TFLOPS) to speed up decoding. Single-stream decode is memory-bound, extra arithmetic throughput sits idle. Prioritize bandwidth (and fewer weight bytes) instead.
Comparing peak bandwidth as if you get 100% of it. Real kernels achieve perhaps 60–85% of the advertised figure; budget accordingly.
Forgetting the KV cache is memory traffic too. At long context the per-token KV reads add to the weight reads and pull the achievable tokens/sec down as the sequence grows, throughput is not constant across a long generation.
Confusing memory bandwidth with interconnect bandwidth. VRAM↔cores bandwidth (this lesson) is a different bottleneck from PCIe or NVLink bandwidth (GPU↔CPU or GPU↔GPU), which dominates only when a model or its data is split across devices.
Treating the tokens/sec ceiling as a throughput estimate. It is an upper bound that ignores overhead and KV/activation traffic; real numbers are lower.