CPU, GPU & TPU

The CPU: a few fat cores

A modern CPU core is a masterpiece of latency reduction. It fetches instructions ahead of time, predicts branches (guessing which way an if will go and speculatively executing down that path), reorders instructions to work around stalls, and leans on a deep cache hierarchy so that data an unpredictable program is likely to reuse sits a few nanoseconds away instead of hundreds. Each core also has modest data parallelism through SIMD vector units (SSE, AVX, AVX-512), but the design center is a small number of very capable cores, typically a handful on a laptop, up to dozens on a server.

That machinery is precisely what irregular software needs: parsing, branchy business logic, pointer-chasing data structures, coordination. It is also expensive. A representative 32-core server CPU with AVX-512, clocked around 3 GHz, tops out near 4–6 TFLOP/s of FP32 in practice, respectable, but a small fraction of what an accelerator delivers on the right workload. When the work is one long dependency chain full of branches, the CPU is unbeatable. When the work is a million independent multiply-adds, all that control logic is dead weight.

The GPU: thousands of lanes under SIMT

A GPU is organized as a set of Streaming Multiprocessors (SMs). Each SM holds many simple arithmetic lanes ("CUDA cores"), a large register file, a slice of fast on-chip memory, and a few warp schedulers. Threads are not scheduled individually, they run in lockstep groups of 32 called warps. Every thread in a warp executes the same instruction in the same cycle, each on its own data. This is SIMT: Single Instruction, Multiple Threads.

Two consequences follow directly:

• Branch divergence is costly. If threads in a warp take different sides of an if, the hardware must execute both paths and mask off the inactive lanes. Deeply branchy code wastes most of the warp, which is why control-heavy logic belongs on the CPU.
• Latency is hidden, not avoided. A single warp that reads memory stalls for hundreds of cycles. Rather than sit idle, the SM's scheduler immediately issues from another ready warp. With dozens of warps resident per SM, the arithmetic units stay busy while memory requests are in flight. The GPU does not make any one access fast; it makes sure there is always other work to do.

For scale, an NVIDIA H100 (SXM) has 132 SMs and 16,896 FP32 lanes. Its general FP32 rate is roughly 67 TFLOP/s, already about 15× a strong CPU. But the H100 also carries tensor cores: units that multiply small matrices in one hardware step, reaching about 990 TFLOP/s in BF16 (the ~1,979 figure vendors quote assumes structured sparsity, read those with care). Those tensor cores exist for exactly one reason, which the next section makes concrete.

Why dense matrix multiply fits the hardware

A neural network is, to a first approximation, a stack of matrix multiplications. Multiplying an $M \times K$ matrix by a $K \times N$ matrix produces $M \times N$ output entries, and each entry is an independent dot product, it needs none of the other entries' results. That independence is the property massively parallel hardware feeds on: map one output entry (or one tile of them) to each group of lanes, and thousands compute at once with no coordination.

Independence alone is not enough, though. The deeper reason matmul suits these chips is its arithmetic intensity, the ratio of arithmetic performed to bytes moved from memory. An $N \times N$ square matmul costs about $2N^3$ floating-point operations but touches only about $3N^2$ numbers. For $N = 4096$ in BF16 that is $\approx 137$ GFLOP against $\approx 100$ MB of data, roughly 1,300 FLOP per byte. A chip with thousands of hungry arithmetic units can only stay busy if each byte it fetches feeds hundreds of operations, and large matmul delivers exactly that.

Memory hierarchy: caches versus bandwidth

The processors also disagree about memory. A CPU builds a tall, latency-optimized hierarchy: per-core L1 (tens of KB, a few-cycle access), per-core L2 (a megabyte or two), a shared L3 (tens of MB), then DRAM at hundreds of cycles away with bandwidth of tens to a few hundred GB/s. The bet is that an unpredictable program reuses data, so keeping it close cuts latency.

A GPU makes almost the opposite bet. Per-SM caches are comparatively modest, on an H100, roughly 256 KB of register file and up to 256 KB of combined L1/shared memory per SM, plus a 50 MB shared L2, but the path to main memory is extraordinarily wide: 80 GB of HBM3 at 3.35 TB/s, an order of magnitude more bandwidth than a CPU's DRAM. The GPU tolerates high memory latency (it hides it with warps) in exchange for the raw bandwidth needed to stream operands into thousands of lanes. CPUs optimize the latency of memory; GPUs optimize its bandwidth.

The TPU: a systolic array for matrix multiply

A GPU is still a general parallel computer. A TPU (Tensor Processing Unit, Google's ML accelerator) specializes one step further, organizing its core around a systolic array: a two-dimensional grid of multiply-accumulate (MAC) cells wired directly to their neighbors. Data marches through the grid in lockstep waves, weights held stationary in the cells, activations flowing across, and each cell multiplies, adds the partial sum arriving from its neighbor, and passes the running total along. Operands are reused in space as they ripple through the array, so a matrix multiply is computed with very few trips to memory and almost no instruction-fetch overhead. The TPU's matrix unit (MXU) is a 128×128 systolic array in recent generations (the first TPU used a 256×256 array of 8-bit MACs).

1. 1Load weightsEach cell of the array is preloaded with one weight of matrix B; the weights stay put (weight-stationary dataflow).
2. 2Stream activationsA row of A enters from the left, one element per cycle, marching rightward across the grid as a wavefront.
3. 3Multiply-accumulate in placeEvery cell multiplies its stored weight by the value passing through, adds the partial sum from the cell above, and passes the total downward.
4. 4Harvest resultsAs the wavefront clears the array, each column emits one finished dot product at the bottom, with operands reused across cells instead of re-fetched from memory.

When a TPU helps versus a GPU. The systolic design is efficient precisely when the work is a large, dense matmul with static shapes, run at scale, big-batch training and inference of standard architectures, where the array stays full and its high-bandwidth pod interconnect lets thousands of chips cooperate. A TPU v4 chip delivers about 275 TFLOP/s in BF16 from that array, backed by 32 GB of HBM at ~1.2 TB/s. A GPU tends to win when you need flexibility: dynamic or irregular shapes, custom kernels, sparse or non-matmul operations, mixed workloads, and the breadth of the CUDA ecosystem. Neither is universally faster, the TPU trades generality for density on one operation, and that trade only pays when your workload really is that operation.

A worked comparison

Take one concrete kernel: multiply two $4096 \times 4096$ matrices. That is $2 \times 4096^3 \approx 137$ billion floating-point operations. Dividing the work by each chip's peak rate gives an upper bound on how fast it could finish:

Two lessons hide in that table. First, moving the same FP32 work from CPU to GPU is already ~15× faster, that is the "thousands of lanes" effect. Second, switching to the tensor cores' native BF16 buys another ~15×, that is the "purpose-built matmul silicon" effect, the same idea the TPU's array embodies. The two multipliers compound.

Further reading

• NVIDIA CUDA C++ Programming Guide, the authoritative description of the SIMT model, warps, SMs, and the GPU memory hierarchy.
• NVIDIA, GPU Performance Background, arithmetic intensity and the math-bound versus memory-bound distinction, with worked examples.
• Jouppi et al., "In-Datacenter Performance Analysis of a Tensor Processing Unit" (arXiv:1704.04760), the original TPU paper, including the systolic-array matrix unit and why it was built.
• Google Cloud, TPU System Architecture, current TPU chip organization: TensorCores, MXUs, HBM, and inter-chip interconnect.