Loading lesson…
Computing Foundations for AI / Hardware Basics
Thousands of cores doing the same math at once.
Reviewed by Yuvaraj
Modern AI models run on GPUs for one concrete reason: the math inside a neural network is overwhelmingly made of matrix multiplications, and matrix multiplication is one of the most parallelizable operations in computing. A CPU is built to finish a few complex tasks quickly, one after another. A GPU is built to do the same simple arithmetic on thousands of numbers at the same time. Once you see that a neural-network layer is really just a giant pile of independent multiply-and-add operations, it becomes clear why hardware that performs thousands of those at once is the right tool.
Every layer takes a batch of inputs, multiplies them by learned weights, sums the results, and applies an activation. The multiply-and-sum step is a matrix multiplication; the activation is an elementwise operation applied to each number independently. Both are data-parallel: the same arithmetic repeated over many values that do not depend on one another.
The workhorse instruction underneath is the fused multiply-add (FMA): compute in a single hardware step. A dot product is just a chain of FMAs, and a matrix multiply is a grid of dot products. At the arithmetic level, running a neural network is performing billions of FMAs.
One instruction to rule them all
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
If you can make the FMA fast and run enough of them at once, you have made the neural network fast. Almost everything else is bookkeeping around that single operation.
Let be and be :
The product is , and each entry is the dot product of a row of with a column of :
The key observation: none of these four sums needs the result of any other. does not depend on , , or . They are independent, so four workers could compute all four at the same time. Each entry costs multiplies and adds, so the whole product is multiply-adds.
Now scale up. A single layer in a real model might multiply a matrix by a matrix. The result has about million entries, each a dot product of length , roughly billion multiply-adds. Every one of those entries is still independent. A CPU with a handful of cores grinds through them in long sequences; a GPU with thousands of cores spreads them across the chip.
GPUs use an execution model called SIMT, Single Instruction, Multiple Threads. The hardware issues one instruction (say, an FMA) to thousands of threads at once, and each thread applies it to its own data element. Map one thread to one output entry, and the multiplication happens as a wave of parallel FMAs instead of a long loop.
This is why the metric that matters for AI is throughput (total arithmetic per second across the whole chip), not single-thread speed. An individual GPU core is slower than a CPU core; there are just thousands of them. Just as important is memory bandwidth: feeding thousands of arithmetic units means streaming enormous amounts of data from GPU memory into the cores every instant, so bandwidth often limits real performance more than raw arithmetic does.
For a curious beginner
A matrix multiply is many small, independent sums. Nothing waits on anything else, so handing each sum to its own worker finishes the whole job in roughly the time of a single sum.
How it is actually used
Launch one thread per output element. All threads run the same multiply-add code at the same time, each on its own row and column, so with enough cores thousands of entries land together.
The underlying mechanism
The multiply-adds of an product have no data dependencies between output entries. Wall-clock time therefore scales with the number of available cores rather than with the total operation count, the work is embarrassingly parallel.
| Dimension | CPU | GPU |
|---|---|---|
| Core count | A few to dozens | Thousands |
| Per-core speed | Very fast | Slower individually |
| Best at | Sequential, branch-heavy logic | The same operation over huge arrays |
| Optimizes for | Latency of one task | Throughput of many tasks |
| Common AI limit | Core count | Memory bandwidth |
A GPU can only compute on data held in its own memory, called VRAM. Model weights and input batches start in system RAM and must be copied across the PCIe bus into VRAM before any kernel (a GPU program) can touch them. That transfer is slow compared with on-GPU bandwidth, so shuttling data back and forth for every small operation is a classic bottleneck. The winning pattern is to move data onto the GPU once, do as much work there as possible, and copy results back only when needed.
Modern GPUs also include tensor cores, units built specifically to multiply small matrices in a single hardware step, accelerating exactly the FMAs that dominate deep learning.
Common mistakes