Loading lesson…
Cloud AI / Operating in the Cloud
Shipping a new version with minimal risk.
Reviewed by Yuvaraj
Shipping a new version of a cloud AI service, a fresh model checkpoint, a new serving runtime, or a rewritten inference API, is the moment where the most damage can happen, because the change is live and every user becomes a test subject. Deployment patterns exist to bound that risk: they control how many users see the new version, how fast that exposure grows, and how quickly you can undo it when the numbers go wrong. The four patterns below, blue-green, canary, rolling, and shadow, trade off rollback speed, infrastructure cost, and how much signal you gather before full cutover. For AI services these trade-offs bite harder than for an ordinary web app, because a model replica is expensive (accelerator memory), slow to start (tens of gigabytes of weights to load and warm), and can fail in subtle quality ways that no HTTP health check will ever notice.
A useful framing runs through all four: they decouple deployment from release. Pushing new weights or a new container to a fleet (deployment) is a separate act from directing user traffic at it (release). Every pattern here is really a policy for how traffic moves.
You run two identical production environments. Blue is live and serving all traffic; green is idle, running the new version. You deploy the candidate to green, smoke-test it, warm it, then flip the router, load balancer, DNS, or service-mesh weight, so 100% of traffic lands on green at once. Blue stays hot as an instant rollback target: if green misbehaves, you flip straight back.
For AI serving the headline cost is that you pay for two full fleets during the overlap window, and for GPU/TPU-backed models the accelerators are the dominant line item, you are briefly double-billed on the most expensive resource you own. Cold start is the other trap: a large model must load its weights into VRAM and warm the runtime (CUDA graph capture, KV-cache allocation, any JIT/compile step) before the first real request, or the moment you flip, users hit timeouts. Pre-warm green with synthetic traffic and gate the flip on readiness. The pattern's strength, atomic switch, seconds-to-rollback, is also its weakness: there is no gradual exposure, so a bad model reaches every user the instant you cut over.
You route a small fraction of live traffic to the new version, hold, measure, and increase the fraction in stages only while health stays within bounds. The name comes from canaries carried into coal mines: a small, expendable early-warning signal. The split is normally done by weight at the load balancer or mesh.
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
This is usually the right default for models. Because exposure grows gradually, the blast radius of a bad release is small, and, critically, you get to compare quality and guardrail metrics on real production traffic before full cutover, which offline evals never fully capture. It is also cheaper than blue-green: you run one stable fleet plus a small canary, not a full duplicate (though the canary must still be large enough to hold at least one warmed model replica). The catch is statistics: at 1% traffic, rare errors and tail-latency regressions need enough requests to become visible, so ramp on request count, not just wall-clock time.
A weighted split is straightforward to express in a service mesh. This Istio VirtualService sends 95% to stable and 5% to the canary, and, at the same time, mirrors a full copy to a shadow subset (covered below):
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: inference
spec:
hosts:
- inference.internal
http:
- route:
- destination: { host: inference, subset: stable }
weight: 95
- destination: { host: inference, subset: canary }
weight: 5
mirror: { host: inference, subset: shadow }
mirrorPercentage:
value: 100.0
You upgrade in place, replacing replicas in batches: add or restart a few replicas on the new version, wait for them to pass health checks, then continue until the whole fleet is upgraded. This is the default for Kubernetes Deployment objects, tuned with maxSurge and maxUnavailable. There is no second environment, so it is the cheapest pattern on infrastructure.
The trade-offs are real for AI. During the roll, both versions serve live users simultaneously and you cannot cleanly attribute metrics to one version, so it gives you little pre-cutover evidence. Rollback is not instant, it means rolling forward to the old version, batch by batch. And cold start interacts badly with health checks: each new replica must load model weights and warm up before it should receive traffic, so you must set generous startup and readiness thresholds or the rollout will thrash, tearing down capacity faster than the new replicas can come online. A safe config surges up without ever dropping below capacity:
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
You send a copy of live requests to the new version alongside the stable one, but you discard the shadow's responses, users only ever see the stable output. This lets you measure the candidate on the exact production input distribution with zero user impact, which makes it the standout pattern for validating an ML model: you observe its latency, error rate, and output quality on real inputs before a single prediction can reach a user.
Two disciplines make or break it. First, you still pay to run the shadow fleet and to perform inference twice, so budget for it. Second, and this is where teams get burned, the shadow must have no side effects: a mirrored request that writes to the production database, sends a notification, or calls a paid downstream/payment API is no longer harmless. Mirror reads; stub, sandbox, or no-op every write and external action. Shadowing pairs naturally with canary: shadow first to catch crashes and latency regressions safely, then canary to measure real user-facing outcomes.
A recommendation-ranking service handles roughly 200 requests/second. You are promoting a new model checkpoint behind a canary that ramps 1% → 5% → 25% → 100%, holding each stage for 30 minutes. The stable version's baseline is 0.4% error rate and 820 ms p95 latency. Promotion gates, evaluated per stage against the canary cohort only:
5xx error rate ≤ 1.0% absolute.Both fleets are pre-warmed, so readiness is not in question. The staged read-out:
| Stage | Canary weight | Canary requests | Error rate (canary vs stable) | p95 latency (canary vs stable) | Gate result |
|---|---|---|---|---|---|
| 1 | 1% | ~3,600 | 0.5% vs 0.4% | 880 ms vs 820 ms | Pass → promote |
| 2 | 5% | ~18,000 | 0.6% vs 0.4% | 905 ms vs 820 ms | Pass → promote |
| 3 | 25% | ~90,000 | 2.3% vs 0.4% | 1,180 ms vs 820 ms | Fail → roll back |
| 4 | 100% | , | not reached | not reached | Blocked by gate 3 |
At stage 3 both gates breach: 2.3% error exceeds the 1.0% cap, and 1,180 ms p95 exceeds the 1,000 ms cap. The rollback trigger fires: the controller sets the canary weight to 0% within one control-loop interval. Because the stable fleet never left service and is already warm, users see no cold start during the reversal, the blast radius was capped at the 25% cohort for the length of one observation window. The 100% stage never happens. The canary fleet is kept running (with 0% traffic) for diagnosis.
Why you must segment metrics by version
During a canary, the observed aggregate error rate blends both versions by their traffic weight :
where is the stable error rate and the canary's. At stage 1 (, ), , about 0.42%, barely above the 0.40% baseline. A broken canary is in the top-line dashboard at low weight. The gate works only because it evaluates the metrics directly, not the blend. Always split dashboards and alerts by version.
| Pattern | Traffic handling | Rollback speed | Extra cost | Best for |
|---|---|---|---|---|
| Blue-green | Flip 100% at once between two fleets | Instant (flip back) | High, full duplicate fleet | Atomic cutovers needing near-zero rollback time |
| Canary | Ramp a small weight up in stages | Fast, set weight to 0% | Low–moderate, one small extra fleet | Gathering real-traffic evidence before full exposure |
| Rolling | Replace replicas batch by batch, in place | Slow, roll forward to the old version | Minimal, no duplicate fleet | Cost-sensitive, stateless services with solid health checks |
| Shadow | Mirror a copy to the new version; discard its output | N/A, users never see it | Moderate, inference runs twice | Safely validating a model on real inputs with zero user impact |
Common mistakes
5xx and latency while its answers regress, its refusal rate climbs, or it breaches guardrails. Add quality and safety gates (eval scores, refusal rate, filter hits) to the promotion criteria.