Experiment Tracking & Registry

If you cannot recover every argument on the right, you cannot recover the result on the left. "We got 0.91 AUC last quarter" is worthless if the notebook has since changed, the training table was overwritten in place, or nobody recorded the random seed. Reproducibility is not a nice-to-have; it is the precondition for comparing two runs at all, because a fair comparison requires that the only thing that changed between them is the thing you meant to change.

The practical discipline, then, is to pin every input: commit the code and record its git SHA, snapshot or hash the dataset, log the params and seed, and capture the environment (dependency lockfile or a container image digest). Anything left unpinned is a silent variable that will eventually make two "identical" runs disagree.

Anatomy of a tracked run

A run is a single execution of your training code. Runs are grouped into an experiment (e.g. all attempts at churn-classifier). For each run, a tracking tool records four kinds of information:

• Parameters, the fixed inputs you chose: learning_rate, n_estimators, max_depth, seed. These are write-once; a param does not change during the run.
• Metrics, the measured outputs: val_auc, log_loss. Metrics can be logged repeatedly across steps or epochs, producing a time series you can plot to spot divergence or overfitting.
• Metadata / tags, provenance that ties the run to its recipe: the git commit SHA, the dataset version, a run name, the entry point. Many trackers auto-capture the git SHA when you launch from a clean repository.
• Artifacts, the files a run produces: the serialized model, a ROC curve, a confusion matrix, feature-importance JSON, the environment lockfile.

Under the hood these split into a metadata store (small, structured, queryable, params, metrics, tags) and an artifact store (large blobs on object storage such as S3 or GCS). That separation is why you can scan thousands of runs in a table but still fetch the exact model.pkl for any one of them.

import mlflow
from sklearn.ensemble import GradientBoostingClassifier

mlflow.set_experiment("churn-classifier")

with mlflow.start_run(run_name="gbm-lr0.05-d6") as run:
    params = {"learning_rate": 0.05, "n_estimators": 400,
              "max_depth": 6, "random_state": 42}
    mlflow.log_params(params)
    mlflow.set_tag("dataset_version", "churn_v3")   # a hash or DVC/Delta version
    mlflow.set_tag("git_sha", "9f2c1a7")            # often captured automatically

    model = GradientBoostingClassifier(**params).fit(X_train, y_train)

    mlflow.log_metric("val_auc", 0.912)
    mlflow.log_metric("val_pr_auc", 0.68)
    mlflow.log_metric("val_log_loss", 0.243)

    mlflow.sklearn.log_model(model, artifact_path="model")

run_id = run.info.run_id   # the durable handle to this exact recipe + result

Tools like MLflow and Weights & Biases implement this pattern; they are illustrative here, not endorsements. The concepts, runs, params, metrics, artifacts, lineage, are tool-agnostic.

From run to registry

Tracking preserves history; a model registry manages what ships. They are complementary, not interchangeable:

When a run wins, you register its model. This creates a new immutable version under a named model, and, critically, records lineage back to the source run:

result = mlflow.register_model(
    model_uri=f"runs:/{run_id}/model",   # points into the winning run's artifacts
    name="churn-classifier",
)
# -> creates "churn-classifier" version 5, linked to run 9f2c1a7's recipe

Each version then moves through stages, conventionally None → Staging → Production → Archived. A stage is a label on a version, not the deployment itself: promoting to Production marks intent, and a serving system reads that label to decide what to load. The transition is where the promotion/approval workflow lives, automated validation in staging, then a human sign-off before anything reaches users.

from mlflow import MlflowClient
client = MlflowClient()

# 1) send the new version to staging for validation
client.transition_model_version_stage("churn-classifier", version=5, stage="Staging")

# 2) after approval, promote and auto-archive whatever was live
client.transition_model_version_stage(
    "churn-classifier", version=5, stage="Production",
    archive_existing_versions=True,   # version 4 -> Archived, no gap, no ambiguity
)

Lineage: answering "how did this get to prod?"

The payoff of registering with lineage is a chain you can walk backwards during an incident. Serving loads Production → that resolves to version 5 → which links to run 9f2c1a7 → whose tags and params reveal the exact code SHA, dataset version, hyperparameters, seed, and environment. Because every link was pinned, you can rebuild the model, diff it against the previous version, or roll back to it deterministically. Without that chain, "which model is in prod and how did we get it" becomes an archaeology project instead of a database query.

Worked example: log, compare, register, promote

The team is improving the churn classifier. A prior champion, run A, is already live as churn-classifier version 4. A new candidate, run B, is trained with a lower learning rate and deeper trees.

Both runs used the same pinned dataset and differ only in hyperparameters and code, so the comparison is fair: run B genuinely improves AUC (0.912 vs 0.897) and log loss. The team registers run B's artifact as version 5, moves it to Staging, runs the validation suite and a shadow test against live traffic, and, on a reviewer's approval, promotes version 5 to Production while auto-archiving version 4. Serving now resolves Production to version 5, and its lineage still points at run 9f2c1a7 for any future audit or rollback.

1. 1Log the runExecute training; params, metrics, git SHA, dataset version, and the model artifact are all captured under one run ID.
2. 2CompareRank the candidate against the incumbent on the agreed metric, fair only because data and code are pinned so the change is isolated.
3. 3RegisterCreate an immutable model version from the winning run's artifact; the version records lineage back to that run.
4. 4Stage to StagingMove the version to Staging; run automated validation, integration checks, and a shadow/canary test.
5. 5Approve & promoteOn human sign-off, transition the version to Production and archive the previously-live version, no gap, no ambiguity.
6. 6Serve & traceServing loads the Production version; any incident traces version → run → code + data + params for rollback or reproduction.

What to log, and why

Further reading

• MLflow Tracking documentation, runs, params, metrics, artifacts, and the tracking APIs.
• MLflow Model Registry documentation, versions, stages/aliases, lineage, and promotion workflow.
• Weights & Biases: Experiment Tracking guide, an alternative implementation of the same concepts.
• Google Cloud: MLOps, Continuous delivery and automation pipelines in ML, where tracking, registry, lineage, and reproducibility fit in an end-to-end pipeline.
• Google: Rules of Machine Learning (Best Practices), Rule 33 and the surrounding guidance on measuring and reproducing model quality.
• DVC (Data Version Control) documentation, versioning and pinning datasets so the data argument of the training function is recoverable.