Monitoring & Drift

Concept drift: $P(Y \mid X)$ itself changes. Example: "spam" evolves as senders adopt new tactics, so an email that was legitimate-looking last year is now the signature of an attack, same features, different correct label. Fraud patterns mutate the same way.

Concept drift is the harder problem. Data drift is visible in the inputs alone: you can watch $P(X)$ and see it move. Concept drift, by definition, need not disturb $P(X)$ at all, the inputs can look perfectly normal while the correct labels shift underneath them. To confirm it you need ground truth, and ground truth usually arrives late (did the loan default? did the user churn?) or never arrives at all.

Detecting drift without labels

Because labels are delayed or missing, mature teams monitor input and output distributions continuously and treat labels as confirmation rather than the primary alarm. The standard move is to compare a recent rolling window against a fixed reference window (typically the training or validation set):

• Numeric features: Population Stability Index (PSI) or a two-sample Kolmogorov–Smirnov test.
• Categorical features: KL divergence or a chi-square test on category frequencies.

PSI is the workhorse for tabular data because it produces a single interpretable number per feature:

Two more signals matter. Prediction drift, the distribution of the model's own outputs, is an early, label-free proxy: if the fraction of "high risk" scores jumps, something upstream moved even before you can measure accuracy. And whatever ground truth does trickle in should feed label-based quality metrics (accuracy, AUC, calibration) so you can confirm whether drift actually hurt performance.

The monitoring loop

1. 1InstrumentLog inputs, predictions, and (when available) outcomes, each tagged with model and data versions.
2. 2BaselineFix a reference window from training/validation data to compare everything against.
3. 3CompareOn a rolling window, compute drift metrics per feature and on the prediction distribution.
4. 4AlertThreshold on PSI/KS and on quality metrics; page a human on a breach.
5. 5DiagnoseIs it data drift, concept drift, a data-pipeline bug, or an upstream schema change?
6. 6RespondRetrain on fresh data, roll back to a known-good model, or fix the broken pipeline.

Worked example: a credit-risk model

A credit-risk scorer is trained when the median applicant income is $60k. Six months later, a marketing push draws a younger, lower-income audience. The median income in the input stream falls to $45k and the age feature's distribution shifts younger. PSI on income comes out at 0.31, a major shift. Predictions skew toward "high risk," approvals drop, and the business notices before the data team does. This is data drift: $P(X)$ moved, and the fix is to reweight or retrain on the new input mix so the model is calibrated for the population it now actually serves.

Now change the story. Suppose incomes stay the same, but a recession means a given income no longer implies the default rate it used to, the same $60k applicant is now materially more likely to default. The inputs look unchanged, so every $P(X)$-based detector stays green. Nothing fires until real default labels arrive weeks later and quality metrics collapse. This is concept drift, invisible in the inputs. The response is different too: you cannot just reweight inputs, you must relabel against the new reality and retrain the input-to-target relationship itself.

Further reading

• Google, Rules of Machine Learning, battle-tested engineering practice, including monitoring and freshness.
• Gama et al., A Survey on Concept Drift Adaptation, the canonical taxonomy of drift types and detection methods.
• Evidently AI documentation, open-source tooling for data and prediction drift reports.
• Wikipedia, Concept drift, concise conceptual overview with references.