Deployment & Rollback

Promotion: staging to production, gated by a checklist

Promotion is a registry state change guarded by pre-promotion checks. A new version does not earn the production alias by existing, it earns it by clearing every gate:

• Artifact integrity. The bundle downloads, its checksum matches the registry record, dependencies resolve, and the model actually loads on the target runtime.
• Smoke eval on a held-out (golden) set. Offline quality must clear a gate relative to current production, for example accuracy or F1 no lower than prod minus a tolerance, or an LLM eval suite whose pass rate meets a floor. The eval must run through the same preprocessing and config production will serve.
• Load and performance test. Throughput and p95/p99 latency under expected concurrency, with memory and VRAM confirmed to fit the serving hardware.
• Guardrail thresholds defined. Before any real traffic, you declare which live signals will be watched and at what limits they revert.
• Warm standby confirmed. The previous version stays loaded and reachable so the revert path is ready.

Progressive exposure, shadow, then a small canary percentage, then full, reduces blast radius, but the artifact discipline here is orthogonal to traffic splitting: what matters is which immutable version the alias resolves to and whether you can flip it back instantly.

Pin the whole bundle, not just the weights

A rollback restores prior behavior only if you pinned the entire stack together under one version identity. Reverting the weights while a preprocessing step, tokenizer, or threshold silently moved on gives you a Frankenstein that matches neither v4 nor v5. Everything below travels as one immutable bundle:

Rollback is a first-class, rehearsed operation

Rollback should be as boring and fast as a config change: repoint the alias, and the router serves the warm previous version. Two properties make that possible. First, the old version stays warm, loaded in memory or held in a warm pool, because cold-loading a large model can take minutes, and minutes are exactly what you do not have while a metric is regressing. Second, the revert path is rehearsed: run game days where you trip a guard on purpose and confirm traffic lands back on the prior version. An untested rollback is not a rollback.

Rollback triggers watch live service-level indicators against the pinned baseline: quality signals (guardrail failure rate, downstream acceptance, thumbs-down rate), operational signals (error rate, p95 latency, cost per request), and safety signals. To avoid reacting to noise, evaluate over a sustained window with a minimum sample size. A compact trigger rule: revert when the windowed live error rate exceeds the baseline by a tolerance, $\hat{e}_{W} > e_{\text{base}} + \delta$, sustained across consecutive windows that each carry enough traffic to be trustworthy:

where $\hat{e}_{W_i}$ is the measured error rate in window $i$, $e_{\text{base}}$ the pinned previous-version baseline, $\delta$ the tolerance, $k$ the number of consecutive windows required, and $n_{\min}$ the minimum requests per window.

Automatic vs manual rollback

The mature pattern uses both: automatic reverts for fast, clear operational SLIs, and a one-click manual revert for quality regressions that only surface once labels or human feedback arrive. Both resolve to the same pinned artifact.

Worked example: promoting v5 and the rollback that held the SLO

A support-ticket ranking model serves production as support-ranker@production → v4. Its live one-hour baseline is an error rate of 1.8% (responses failing schema or guardrail validation), p95 latency 410 ms, at roughly 120 req/s. The team promotes v5 behind a release manifest that pins the whole bundle and wires an automatic guard.

# release manifest, one immutable bundle, one version id
model:
  name: support-ranker
  version: v5 # immutable; resolves to a content hash
  artifact_uri: s3://models/support-ranker/v5/ # write-once
  sha256: 9f2c...e01 # integrity check at load time
pinned_with_the_weights:
  code_commit: a1b9c74 # model + serving code
  runtime: python-3.11, torch-2.3.1
  preprocessing: tokenizer-2025.06 # feature/preprocess pipeline version
  config: { max_tokens: 512, threshold: 0.61 }
promotion:
  from: staging
  to: production
  requires: [checksum_ok, golden_eval_pass, load_test_pass]
rollback:
  keep_warm: v4 # previous prod stays loaded
  auto:
    metric: error_rate_10m
    baseline: 0.018 # v4 live baseline
    trigger_above: 0.033 # baseline + 1.5pp tolerance
    sustain_windows: 2
    min_requests: 2000
    revert_to: v4

Offline, v5 looked strictly better: golden-set F1 up 0.6 points over v4, load test p95 430 ms at 150 req/s. All gates green, so at 09:00 the alias is repointed to v5 and v4 is kept warm. Then the live numbers diverge from the offline story:

The guard fired because $\hat{e}_{W} = 4.6\% > 1.8\% + 1.5\text{pp} = 3.3\%$ held across two consecutive windows that each cleared the 2000-request floor. Reverting was a pointer flip to an already-loaded model, so the error rate returned to baseline within seconds instead of the minutes a cold reload would have cost. Elevated errors lasted about 50 minutes on a bounded slice, the error budget was dented, not blown, and there was never a full outage.

Root cause: v5 shipped with an updated tokenizer (tokenizer-2025.06) that had not been the version the offline golden eval ran against, so the regression was invisible until real traffic hit the mismatched preprocessing. The fix was exactly the pinning discipline above: bind preprocessing into the version bundle, re-run the smoke eval through that exact bundle, and only then re-attempt promotion.

The promote-verify-monitor-rollback loop

1. 1PromoteRepoint the production alias to the new immutable version. Promotion is a registry state change gated by the pre-promotion checklist, not a file copy.
2. 2VerifyConfirm the artifact loads from its pinned bundle, the checksum matches, and a smoke eval on the golden set clears the quality gate before real traffic sees it.
3. 3MonitorWatch live SLIs against the pinned previous-version baseline: guardrail failure rate, error rate, p95 latency, cost per request, computed over sustained windows with a minimum sample size.
4. 4RollbackIf a guard trips, repoint the alias back to the warm previous version. Revert is a pointer flip measured in seconds; then quarantine the bad version for root-cause analysis.

Common mistakes

Further reading

• MLOps: Continuous delivery and automation pipelines in machine learning, Google Cloud
• MLflow Model Registry, versions, aliases, and stage transitions
• Amazon SageMaker Model Registry, model versioning and approval for deployment
• SageMaker inference deployment guardrails, automatic rollback via CloudWatch alarms
• The Site Reliability Workbook, Canarying Releases, Google SRE
• Rules of Machine Learning: Best Practices for ML Engineering, Google