Loading lesson…
AI Systems Engineering: Run, Operate & Evaluate / Running Models
Throughput vs. latency, continuous batching, paged KV cache.
Reviewed by Yuvaraj
Loading a model is step one. Turning that loaded model into a service that answers thousands of users without falling over, cheaply and quickly, is a different discipline. This lesson covers how inference servers actually work and the central trade-off you will negotiate forever: throughput versus latency.
An LLM generates one token at a time. Each step reads the entire set of weights from memory to produce a single next token, then appends it and repeats. Because that step moves a huge amount of data (the weights) to do relatively little arithmetic (one token), inference is usually memory-bandwidth-bound, not compute-bound. The GPU's math units are often idle, waiting on memory.
That single fact explains almost every serving optimization: they are mostly about keeping the expensive hardware busy.
If one request leaves the compute units idle, run several at once. Batching processes multiple sequences together, reading the weights once and applying them to many inputs. Throughput climbs sharply while the per-request cost falls.
The catch is latency. Naive static batching waits to collect a full batch before starting, and holds every finished sequence until the slowest one in the batch is done. A short reply gets stuck behind a long one.
Continuous batching
The key idea in modern serving is continuous (or in-flight) batching: as soon as one sequence in the batch finishes, its slot is immediately given to a waiting request instead of idling until the whole batch completes. This keeps the GPU saturated and is why production servers achieve far higher throughput than a naive loop.
Answer from memory before revealing, retrieval practice is what builds durable recall.
Why does continuous (in-flight) batching raise serving throughput?
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Recall the KV cache from the memory lesson, it stores attention keys and values so tokens are not recomputed. In a server handling many concurrent sequences of different lengths, managing that cache is the hard part. Allocating a fixed contiguous block per sequence wastes enormous memory on fragmentation.
Paged attention borrows the idea of virtual memory from operating systems: the KV cache is split into fixed-size blocks that need not be contiguous, so memory is handed out on demand. This packs far more concurrent sequences into the same VRAM, which, because more concurrency means bigger effective batches, directly raises throughput.
Serving performance is not one number. Learn to separate:
These trade off. A bigger batch raises throughput but can raise time-to-first-token as requests wait to be scheduled. The right balance depends on the product: a chat UI lives or dies on time-to-first-token; a nightly batch job cares only about throughput.
Tail latency, not average
Report percentiles (p95, p99), not the mean. An average hides the unlucky users; the p99 is what makes a product feel slow. A serving change that improves the average while worsening the p99 is often a bad trade.
Fast-moving layer
Specific inference servers, their flags, and their headline numbers change release to release. The concepts here, memory-bound generation, continuous batching, paged KV cache, the latency/throughput trade, are the durable part. Benchmark your own workload before trusting anyone's numbers, including these.