The first time a production model drifts and you miss it because you picked the wrong detection metric, you stop debating which metric is theoretically superior and start caring about which one would have fired. Population Stability Index, KL divergence, and Wasserstein distance all measure distributional shift — they're not interchangeable, and the difference matters in practice.
This is a decision guide, not a textbook. The goal is to give you a clear answer for three scenarios: tabular input features, model output scores, and label distribution shifts.
Population Stability Index (PSI): the credit industry default
PSI was formalized in consumer credit risk modeling and has been the industry default in that domain for decades. The formula compares a baseline distribution against a current distribution across binned buckets:
import numpy as np
def psi(baseline: np.ndarray, current: np.ndarray, buckets: int = 10) -> float:
"""
Compute Population Stability Index.
baseline: reference distribution (e.g. training data)
current: observed distribution (e.g. last 7 days)
"""
breakpoints = np.percentile(baseline, np.linspace(0, 100, buckets + 1))
baseline_pcts = np.histogram(baseline, bins=breakpoints)[0] / len(baseline)
current_pcts = np.histogram(current, bins=breakpoints)[0] / len(current)
# Clip to avoid log(0)
baseline_pcts = np.clip(baseline_pcts, 1e-6, None)
current_pcts = np.clip(current_pcts, 1e-6, None)
return float(np.sum((current_pcts - baseline_pcts) * np.log(current_pcts / baseline_pcts)))
The conventional PSI thresholds come from practice rather than theory: PSI < 0.1 indicates stable distribution; 0.1–0.2 warrants investigation; > 0.2 signals significant shift requiring action. These numbers are calibrated to tabular features in credit models and should be adjusted for other domains.
PSI's strength is interpretability. A compliance team or risk officer can understand "PSI of 0.23 on feature monthly_income" without a statistics background. It bins data, which makes it robust to outliers but also means it loses information about the shape of the shift within bins.
PSI is asymmetric in its original form: PSI(A‖B) ≠ PSI(B‖A). This is usually not a practical problem because you always compare current against a fixed baseline — but it matters if you're computing it bidirectionally.
KL Divergence: information-theoretic but asymmetric
KL divergence (Kullback-Leibler) measures how much one distribution diverges from a reference distribution in information-theoretic terms. It answers: how many extra bits would you need if you encoded samples from the current distribution using a codec optimized for the baseline distribution?
KL divergence is mathematically asymmetric and undefined when the current distribution has non-zero probability mass in regions where the baseline has zero mass. In practice this means you need smoothing — the same epsilon clipping you see in the PSI implementation above.
KL divergence is sensitive to tail behavior in ways PSI is not, because it doesn't bin. If your baseline distribution had negligible probability in a region that your current distribution now frequently occupies, KL divergence will spike sharply. This makes it useful for catching rare-event emergence — a new category in a categorical feature, a novel spending pattern in a transaction feature — where PSI's binning might spread the signal across buckets and dilute it.
The practical trade-off: KL divergence is better for detecting tail shift and rare-event drift, but it requires more careful baseline estimation and is harder to explain to non-technical stakeholders.
Wasserstein Distance: the geometry of distributional shift
Wasserstein distance (also called Earth Mover's Distance) measures the minimum "work" required to transform one distribution into another — the amount of probability mass that needs to move, multiplied by the distance it needs to travel. For one-dimensional distributions it has a clean closed form:
from scipy.stats import wasserstein_distance
w_dist = wasserstein_distance(baseline_samples, current_samples)
Wasserstein is symmetric (unlike PSI and KL), meaningful even when distributions have non-overlapping support (unlike KL without smoothing), and geometrically interpretable: a Wasserstein distance of 0.4 on a feature that ranges from 0 to 1 means the distributions are 0.4 units apart in an average sense.
Wasserstein distance is particularly well-suited to continuous output score distributions — monitoring the distribution of model confidence scores or predicted probabilities. Small systematic shifts in predicted probabilities (a sign of gradual concept drift) show up clearly in Wasserstein but can be invisible in PSI if the shift is smooth and spread across bins.
When to use which: the decision matrix
After running these metrics against real production drift scenarios on tabular ML pipelines, here's the practical guidance:
Tabular input features, regulated industry (credit, insurance, healthcare): PSI is the right default. Your risk team likely already understands it, the thresholds are documented in industry practice, and the interpretability matters for internal audit trails. Run it at the feature level, not just on aggregate score.
Prediction score / output distribution monitoring: Wasserstein distance. It captures smooth shifts in predicted probability distributions without requiring binning decisions, and it's symmetric — useful when you want to compare consecutive time windows rather than always comparing against a fixed training baseline.
Detecting rare-event emergence or categorical covariate shift: KL divergence, with careful smoothing. This is where PSI's binning works against you. A new city emerging as a significant portion of your location feature, or a new merchant category in transaction data, shows up more clearly in KL divergence.
High-dimensional feature vectors or embeddings: None of the above work well in raw form in high dimensions. You need to project first (PCA to 2–5 components, or monitor per-dimension) and then apply Wasserstein per component. Multivariate Wasserstein is computationally expensive; dimensionality reduction first is the practical path.
The governance implication: different metrics need different policies
We're not saying you should pick one metric and apply it everywhere — we're saying different parts of your ML pipeline have different drift failure modes, and your retrain policy configuration should reflect that.
In a production governance setup, you might run PSI on input features (with a 0.1/0.2 threshold ladder), Wasserstein on output score distribution (threshold calibrated per model based on historical shift velocity), and a simple label drift monitor on observed outcomes where you have ground truth. Each metric fires into a different alert tier: PSI spikes trigger investigation workflow; Wasserstein drift triggers a retrain candidate proposal; label distribution shift triggers immediate human review because it might indicate labeling pipeline issues, not just input shift.
The metric choice and the threshold configuration belong in the same versioned policy document as the approval chain. If you change from PSI to Wasserstein for a given feature, that's a governance decision — it changes what signals can trigger a retrain — and it should be recorded as a policy version change, not just a code change.
Calibrating thresholds on your own data
The generic thresholds (PSI > 0.2, etc.) are starting points, not ground truth. The right approach is to compute baseline drift statistics on historical windows where you know no meaningful real-world change occurred, establish your "null distribution" of drift metric values, and set your alert thresholds at the 95th percentile of that null distribution. This means your thresholds are calibrated to your specific feature distributions and update frequencies rather than being borrowed from a credit risk textbook.
Threshold calibration is straightforward to automate during model onboarding — compute drift metrics on rolling windows of historical training data, fit a distribution to the resulting values, and emit the calibrated thresholds as part of the model's governance configuration. This is considerably more reliable than manually tuning thresholds per model, and it produces a defensible, auditable record of why each threshold was set at its value.