The first retrain policy most ML teams write is a cron job. Retrain the model every Sunday at 3am. It's simple, it's auditable in the sense that it runs on a schedule, and it works until it doesn't — until a sudden input distribution shift happens on a Wednesday and the model degrades for four days before the weekend retrain catches it. Or until the Sunday retrain runs on stale labels and produces a worse model that quietly serves bad predictions for a week before anyone notices the accuracy floor has dropped.
Cron-based retraining is not a retrain policy. It's an absence of policy dressed up as automation. A real retrain policy is a set of machine-readable rules that defines: what signals authorize a retrain, who must approve it before the pipeline runs, what the quality floor is for the new version to be eligible for promotion, and what happens if the new version doesn't clear that floor.
The anatomy of a retrain policy
A production-grade retrain policy has four components:
Trigger conditions: The quantitative signals that create a retrain candidate proposal. These can be drift-based (PSI exceeds 0.15 on feature account_age_days), performance-based (accuracy drops below 0.89 on the evaluation window), or event-based (a data pipeline version change that affects upstream features). Trigger conditions should be specific enough to fire with appropriate frequency — too sensitive and your approval queue fills with noise; too coarse and you're back to missing real drift.
Approval requirements: Who must sign off before the retrain pipeline runs. For low-risk model updates, ML Lead approval may be sufficient. For models that touch credit decisions, fraud scoring, or clinical risk, you likely need ML Lead plus Compliance Owner, possibly with a timeout SLA to prevent the approval process from becoming a bottleneck that defeats the purpose of automated triggering.
Quality gates: The criteria the new model version must pass before it's eligible for promotion. This should include both absolute floors (new version must achieve minimum F1 of 0.86 on holdout) and relative guards (new version must not be worse than the current production version by more than 0.02 on the primary metric). The relative guard matters more than people expect — a retrain triggered by drift can produce a model that's technically above the absolute floor but worse than the current version, which means you're deploying a regression.
Failure handling: What happens when the retrain pipeline completes but the new version fails quality gates. This is the part most teams skip in their initial design. Common patterns: requeue the trigger for human investigation, alert the ML Lead with the quality gate failure details, and keep the current version running — explicitly. The absence of a promotion decision should be as intentional as a promotion.
Encoding a retrain policy as YAML
Policies that live in documentation are not policies — they're aspirations. A retrain policy that can govern automated pipelines needs to be machine-readable, version-controlled, and attached to a specific model identifier. A policy YAML for a credit risk model might look like this:
model: credit-risk-classifier
version: production-strict@v3
retrain_trigger:
conditions:
- metric: psi
feature: monthly_income
threshold: 0.12
window_days: 7
- metric: accuracy
threshold: 0.88
window_days: 3
operator: OR # any condition fires a proposal
approval:
required: true
roles:
- ml-lead # 1 approver required from this role
- compliance-owner # 1 approver required from this role
sla_hours: 24
on_timeout: escalate_to_vp_ml
quality_gates:
min_accuracy: 0.89
max_regression_vs_current: 0.02
evaluation_dataset: holdout-q4-2024
on_gate_failure:
action: notify_ml_lead
keep_current_version: true
requeue_after_hours: 48
This policy is specific, auditable, and directly executable by a governance pipeline. When a retrain event fires, the governance system records: which trigger condition fired, which policy version was in effect, who the required approvers were, how long approval took, whether quality gates passed, and the final promotion decision. That record exists as an immutable audit event, not a Slack thread.
The approval chain design problem
The approval step is where most teams either over-engineer or under-specify. The common failure modes:
Single-approver chains: ML Lead approves every retrain. Works when there's one team and three models. Fails when the ML Lead is on PTO and the policy has no deputy path, blocking retrains for a week.
Approval without context: The approver receives a notification that a retrain was triggered but doesn't see the drift metrics, the current model performance, the quality gate configuration, or the last three governance events for this model. Approvals made without this context are not meaningful governance decisions.
No SLA: Without a defined SLA for approval, a human-gated workflow can become an indefinite blocker. Define timeout behavior explicitly: after 24 hours without approval, should the system escalate, auto-approve, or cancel and alert?
We're not suggesting that full automation is always wrong — for some model types in well-understood deployment contexts, automatic retraining without human approval is appropriate. But the decision to bypass human review should itself be a policy decision, documented as such, not the result of never having gotten around to adding an approval step.
Versioning policies alongside models
A retrain policy is not static. As a model matures, as you accumulate evidence about its drift behavior, and as organizational risk tolerance evolves, the policy should change. The governance implication is that the policy version must be recorded alongside every retrain event it governed.
If a model's retrain policy changed between version 2.4 and 2.5 — say, the PSI threshold was tightened from 0.15 to 0.10 after evidence of slow-burn drift going undetected — that change should be visible in the audit trail. An auditor reviewing the model's history should be able to see that version 2.5 was governed by production-strict@v4 while version 2.4 was governed by production-strict@v3, and understand what changed between those policy versions.
This is a constraint that rules out policies stored only in README files or Confluence docs. They can't be versioned with the specificity needed to associate a policy version with a model version in a queryable audit log.
Common retrain policy antipatterns
Teams that have gone through the process of formalizing retrain policies typically converge on avoiding several antipatterns:
Trigger conditions set to never fire in practice: PSI threshold of 0.25 on a feature that historically peaks at 0.08. The policy exists but never executes. Review trigger conditions against historical drift data to confirm they would have fired on past drift events.
Quality gates that don't account for data freshness: If the holdout dataset used for quality gate evaluation is from the training period, it doesn't represent the current input distribution. Quality gates should be evaluated against a recent window of production data, not historical holdout.
Policy inheritance across model versions without review: When a new model version is created, teams often copy the policy from the previous version. This is appropriate if the model architecture, feature set, and risk profile are unchanged — but should be an explicit acknowledgment, not a silent copy. Policy inheritance should require a sign-off that the inherited policy is still appropriate.
The goal of a retrain policy is not to add bureaucratic overhead to an already-burdened ML engineering team. It's to replace the implicit decisions that happen anyway — someone approves a retrain, the pipeline runs, the model is promoted — with explicit, recorded, queryable decisions that can be reviewed, audited, and improved over time.