Autoscaling & Cost

When traffic can drop to nothing, scale-to-zero removes every replica so you pay for no GPU at all. The cost is the cold start: the first request after an idle period must wait for a full replica to come up, and for a large model that wait is dominated by loading weights into GPU memory.

1. 1Provision GPU nodeScheduler finds or boots a GPU host if none is warm, often the slowest step.
2. 2Init container and runtimePull the image, start the serving process, and initialize CUDA.
3. 3Load weights into VRAMCopy multi-GB weights from storage onto the GPU; time scales with model size.
4. 4Warm upCompile kernels or CUDA graphs and run a dummy pass so the first real request is fast.
5. 5Mark readyThe replica joins the pool and queued requests begin to drain.

Cold starts commonly run from a few seconds to a minute or more. Mitigations include keeping a small warm pool, snapshotting loaded state, using faster model loaders, and right-sizing down to a smaller model.

Batching for throughput

A GPU processes a batch of requests almost as cheaply as a single one, so grouping requests raises throughput per GPU dramatically. Modern LLM servers use continuous (in-flight) batching: they add and remove sequences from the running batch every step instead of waiting for a fixed batch to fill. Per-request cost falls as throughput rises:

$\text{cost per request} = \frac{\text{price per GPU-hour}}{\text{requests served per GPU-hour}}$

Batching increases the denominator, but pushing batch size too high grows latency, so there is a throughput/latency knee to tune rather than a free lunch.

What drives the bill

GPU-hours dominate the cost of inference, and a GPU costs the same whether it runs at 5% or 95% utilization, so utilization is the lever. Three tactics attack it:

• Spot / preemptible instances, often 60-90% cheaper than on-demand, but the provider can reclaim them with little notice; safe for fault-tolerant or batch work, risky for latency-critical serving without a fallback.
• Right-sizing the GPU, match VRAM to the model. Weights need roughly $\text{params} \times \text{bytes per param}$ of memory (a 7B model in FP16 needs about 14 GB), so it wastes money to run it on an 80 GB card when a smaller GPU fits.
• Autoscaling itself, the closer replica count tracks demand, the less idle GPU you pay for.

Worked example: always-on vs scale-to-zero

Take a low-traffic internal assistant that is actually used about 3 hours per business day. Assume one replica on a GPU priced at $2.00 per GPU-hour (illustrative price, real rates vary by provider, GPU model, and commitment).

Always-on (24/7):
  730 GPU-hours/month  x  $2.00  =  $1,460 / month

Scale-to-zero (~3 h/day):
  3 h/day x 30 days       =  90 GPU-hours/month
  90 GPU-hours x $2.00    =  $180 / month

Savings: $1,460 - $180 = $1,280 / month  (about 88% cheaper)

The catch is latency. The always-on replica answers instantly. The scale-to-zero replica is down most of the day, so the first request after idle pays the full cold start, perhaps 30-60 seconds, before it responds. You are buying an ~88% cost cut with occasional cold-start delay for the unlucky first caller.

Further reading

• Kubernetes Horizontal Pod Autoscaler, scaling replicas on CPU and custom metrics.
• KEDA, Kubernetes Event-driven Autoscaling, scale-to-zero and scaling on queue depth and external metrics.
• Knative Serving autoscaling, request-driven autoscaling with scale-to-zero.
• vLLM documentation, continuous batching and high-throughput LLM serving.
• NVIDIA Triton Inference Server, dynamic batching and multi-model serving.
• AWS EC2 Spot Instances, pricing and the reclamation model for preemptible capacity.