Loading lesson…
AI Systems Engineering: Run, Operate & Evaluate / Running Models
GPUs, memory, bandwidth, and parallelism.
Reviewed by Yuvaraj
A model is math, but running it is hardware. Before you can reason about cost, latency or whether a model will even load, you need a mental model of the machine underneath. This lesson is that map, the parts, why they matter for AI, and where the bottlenecks actually are.
A CPU has a handful of powerful cores optimized for doing different things quickly, one after another. A GPU has thousands of simpler cores optimized for doing the same thing to a lot of data at once. Neural networks are almost entirely large matrix multiplications, the same multiply-and-add repeated across millions of numbers, which is exactly the workload GPUs were built for.
The one-sentence version
CPUs are latency machines (finish one task fast); GPUs are throughput machines (finish a mountain of identical tasks fast). Deep learning is a mountain of identical tasks.
Answer from memory before revealing, retrieval practice is what builds durable recall.
Deep learning runs mainly on GPUs rather than CPUs because:
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Specialized accelerators push this further. Google's TPUs and various NPUs are built around matrix-multiply units and on-chip memory tuned for tensor math. The principle is the same: dedicate silicon to the operation that dominates the workload.
The specification people quote first is compute (FLOPs), but the one that most often decides whether you can run a model is VRAM, the memory physically on the accelerator. The weights have to live there, and so does everything produced during a forward pass.
Two hardware facts matter as much as capacity:
Bottlenecks move
A beginner assumes "faster GPU = faster model." In practice the limit is often bandwidth or interconnect, not raw FLOPs. Always ask what a workload is actually waiting on before spending money on it.
Large models are spread across devices using parallelism, and the vocabulary is worth knowing because it comes up constantly:
These combine. A very large training run might use all three at once, and the art is balancing compute against the communication each split adds.
Raw hardware is wrapped in layers you will meet again:
Fast-moving layer
Specific accelerators, cloud instance types and driver stacks change every year. Treat model names and price/performance numbers as perishable; treat the concepts here, capacity, bandwidth, interconnect, parallelism, as durable.
The next lessons make two of these concrete: exactly how much memory a model needs (and how quantization shrinks it), and how a serving system turns one loaded model into many concurrent answers without melting latency.