Loading lesson…
Cloud AI / Cloud Primitives
Two ways to rent AI compute.
Reviewed by Yuvaraj
Once a model actually works, the interesting problem shifts from can it predict? to how do I run it in production without lighting money on fire? The very first fork in that road is whether to rent a dedicated GPU instance that you provision and control, or to hand inference to a serverless managed endpoint that scales itself. This one decision quietly sets your cost curve, your tail latency, and how much operational work lands on your team every week. Neither answer is universally correct, the right choice depends almost entirely on the shape of your traffic.
A dedicated instance is a virtual machine (or a Kubernetes node) with an accelerator physically attached, an A100, H100, L4, or similar. You control the whole stack: the runtime, the driver and CUDA versions, how you batch requests, and when the model weights are loaded into VRAM. Because the GPU is yours for the duration, it stays warm, weights are already resident, so the first request after a quiet minute is as fast as the thousandth.
The catch is the billing model: you pay per hour (or per second) for as long as the instance is running, whether or not it is serving a single request. A GPU idling overnight costs exactly what a GPU at full load costs. That makes dedicated instances excellent for steady, high-utilization workloads and wasteful for bursty ones. You also inherit the operational burden, autoscaling, driver patching, health checks, and keeping enough capacity warm to absorb spikes are all your responsibility.
With serverless (managed) inference, you deploy a model or a container and the platform provisions GPUs on demand. It scales replicas up and down with traffic, including scale-to-zero, where a completely idle service holds no GPUs at all, and bills you per request or per second of active compute. When nothing is happening, you pay nothing.
You give up low-level control in exchange, and you pay a cold-start penalty. When traffic arrives and no replica is warm, the platform must schedule a GPU, pull the container, and load the model weights into VRAM before the first token comes back. For a small model that might be a second or two; for a large LLM it is routinely tens of seconds. Cold starts don't hurt your average latency much, but they land squarely in your p99, and they surface exactly when a burst of users shows up at once.
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
The economics come down to a single crossover point. Work a concrete example.
Suppose a dedicated instance costs $2.00/hour. Running it always-on for a month:
Now suppose the serverless option bills $0.0004 per second of active GPU time, and each request needs about 0.5 s of GPU:
The break-even request volume is where the two monthly bills meet:
Below ~7.2M requests a month, serverless is cheaper, you simply aren't keeping the GPU busy enough to justify renting it full-time. Above that threshold, and especially at steady high utilization, the dedicated instance wins, because you stop paying the per-request premium baked into the serverless unit price. The intuition is clean: serverless trades a higher unit price for zero idle cost; dedicated trades idle waste for a low unit price. Your job is to figure out which side of the crossover your real traffic lives on.
Once you can see the crossover, matching hardware to workload is mostly pattern recognition:
The hybrid pattern is worth calling out because it captures the best of both: you get the low unit price of dedicated for the traffic you can predict, and the elasticity of serverless for the traffic you can't.
| Dimension | Dedicated instance | Serverless |
|---|---|---|
| Unit price | Low per request at high utilization | Higher per-request premium |
| Idle cost | Full price while running | Zero (scale-to-zero) |
| Cold start | None, always warm | Seconds to tens of seconds |
| Control / customization | Full stack (drivers, batching, runtime) | Limited to platform's surface |
| Scaling | You build and operate it | Automatic |
| Best for | Steady, high-volume load | Spiky, bursty, unpredictable load |
Common mistakes
Rule of thumb
Estimate your sustained requests/second, multiply out both monthly bills, and find the crossover. If you're comfortably on one side of it, the decision is made for you, the arguments only get subtle near the break-even point.