CI/CD for Models

The pipeline stages

1. 1Triggera commit to model code, a new data version, or a schedule
2. 2Data validationcheck schema, ranges, class balance, and train/test leakage before training
3. 3Train / buildreproducible run with pinned data version, seed, and environment; log to experiment tracking
4. 4Evaluate (quality gate)score on a held-out set; fail the pipeline if a core metric regresses past a threshold
5. 5Package & registerversion the model artifact and push it to the model registry
6. 6Deploy to stagingautomated deploy behind a smoke test / shadow traffic
7. 7Promotegated promotion to production, often with human approval

The quality gate

The quality gate is the heart of model CI. Instead of the binary "tests pass / fail", the pipeline runs the candidate model against a frozen evaluation set and blocks the merge or deploy if a key metric falls below either an absolute floor or the incumbent production model. Concretely: block if accuracy on the regression suite falls below 0.90, or if it drops more than one point versus the current production model, whichever bites first.

Aggregate accuracy is not enough. A model can improve overall while quietly getting worse for an important subgroup, so the gate should include behavioral and slice tests, accuracy on specific locales, new users, or high-value segments, not just the headline number. And because model evals are stochastic (sampling, non-determinism, small eval sets), a flaky gate that randomly passes or fails is worse than none: run the evaluation enough times, or on a large enough frozen set, that the metric is stable.

evaluate:
  dataset: eval/regression_v3 # frozen, versioned
  metrics:
    accuracy:
      min: 0.90 # absolute floor
      no_regression_vs: production # must not drop > 1 point
  slices: [locale_de, new_users] # must also hold per-slice

Data and pipeline versioning

A model is reproducible only if its inputs are pinned, so the pipeline records the exact data version, code commit, hyperparameters, random seed, and container image alongside the artifact. With that lineage attached, any build can be recreated from scratch and any regression can be bisected, you can walk backward through data versions and commits to find the change that moved the metric. This is exactly what the model registry from the previous lesson is for: the registry holds the versioned weights, and the CI run stamps them with the metadata needed to trust and trace them.

Further reading

• MLOps: Continuous delivery and automation pipelines in machine learning, Google Cloud's reference for maturing from manual to automated ML pipelines.
• The ML Test Score, Breck et al., a rubric for data, model, and infrastructure tests in production ML.
• Hidden Technical Debt in Machine Learning Systems, Sculley et al., why ML systems accrue debt that code-only pipelines miss.
• Continuous Delivery for Machine Learning, the CD4ML article on Martin Fowler's site, an end-to-end pipeline walkthrough.