A model swap and a software deployment are superficially similar — you're updating a versioned artifact that serves production traffic. But the failure modes are different, and the right deployment pattern for a model update depends on factors that don't apply to most software deployments: the new model's behavior on the specific distribution of requests currently in flight, whether prediction latency changes affect downstream systems, and whether there's a period where two model versions producing different predictions would create inconsistent state in dependent systems.
Three deployment patterns cover the majority of production model swap scenarios: shadow mode, canary deployment, and full swap. Each has a different risk profile, governance instrumentation requirement, and time-to-completion. Choosing the right one for a given deployment is as important as building the infrastructure to support all three.
Pattern 1: Shadow mode evaluation
Shadow mode deploys the new model version alongside the current production version but does not serve shadow model predictions to users. All requests are routed to the production model as normal; the shadow model receives the same requests in parallel, generates predictions, and logs them to a comparison store — but those predictions never reach the user or affect downstream systems.
Shadow mode is not a deployment pattern — it's a pre-deployment validation pattern. You're answering the question: how would the new model have performed on real production traffic? Shadow mode is particularly valuable when your offline evaluation dataset may not represent the full diversity of production requests, or when you want to measure prediction latency on production-scale traffic before committing to a swap.
Shadow mode instrumentation requires a traffic forking layer at the serving infrastructure level. For SageMaker, this typically means a Lambda wrapper that duplicates requests to both endpoints; for Kubernetes-based serving, it means a service mesh rule or a custom sidecar. The governance implication is that shadow model predictions need to be logged with enough metadata to compute comparison metrics: request ID (for join to production predictions), timestamp, model version identifier, and the shadow prediction value.
Typical shadow evaluation duration: 24–72 hours of production traffic, depending on traffic volume and how many distinct request types need to be represented. Shadow evaluation should be terminated and a promotion decision made before the shadow version's training data becomes stale relative to the current input distribution.
Pattern 2: Canary deployment
Canary deployment routes a small percentage of live production traffic to the new model version. Unlike shadow mode, canary predictions are real — they're served to actual users. The percentage is typically low initially (2–5%), with progressive increase contingent on meeting quality criteria: the canary version performs at least as well as the production version on the monitored metrics, with statistical confidence adjusted for the traffic proportion.
Canary deployment is the appropriate pattern when you have high confidence in the new model version (it passed shadow evaluation and quality gates) but want to limit blast radius during the initial serving period. An e-commerce recommendation model, a search ranking model, or any model where prediction quality is measurable from user engagement signals within a short feedback window is a good candidate for canary deployment.
The governance instrumentation for canary deployment tracks traffic split percentages as governance events. The progression from 5% → 25% → 50% → 100% should each be authorized decisions — either manual approval by an ML Lead or automated progression based on defined quality criteria. This progression should be recorded with timestamps so that if a quality issue surfaces at 50% traffic, you can roll back to the last-stable percentage without reconstructing what the traffic allocation was at each point.
# Canary progression policy
canary_deployment:
model: search-ranking-v4
initial_traffic_pct: 5
progression:
- at_pct: 5
hold_hours: 4
auto_advance_if:
ndcg_delta_vs_production: ">= 0"
p99_latency_ms: "<= 120"
- at_pct: 25
hold_hours: 12
require_ml_lead_approval: true
- at_pct: 50
hold_hours: 24
require_ml_lead_approval: true
- at_pct: 100
final_swap: true
Pattern 3: Full swap
Full swap replaces the production model in a single atomic operation. All traffic immediately routes to the new version. This is the fastest deployment pattern and the highest-risk one — if the new version has a problem, all production traffic is affected before the problem is detected.
Full swap is appropriate in two specific scenarios: when the deployment is a rollback to a known-good prior version (where risk is already understood and the urgency is to stop serving degraded predictions), or when the new version has been thoroughly validated through shadow evaluation and canary deployment and the final promotion to 100% traffic is a low-risk mechanical step.
Full swap without prior shadow or canary validation is the pattern most commonly associated with post-deployment quality issues in ML production systems. The cases where full swap without pre-validation is justified are narrow: genuine production incidents where the current model is causing harm and a previously validated fallback version needs to be deployed immediately.
Governance instrumentation: what each pattern records
The governance requirement for all three patterns is the same: each deployment state transition should be an immutable, structured event. What differs is the granularity of those events:
Shadow mode: evaluation start event (shadow version, production version, evaluation start timestamp), daily or per-batch comparison metrics, evaluation end event with decision (proceed to canary, proceed to full swap, reject version).
Canary: traffic split change events (from %, to %, authorized by, timestamp), quality metric snapshots at each progression step, and a final promotion event.
Full swap: a single promotion event with initiating actor, target version, source version, stated reason, and a mandatory rollback version reference.
The rollback version reference is often omitted from full swap governance records and then regretted. The promotion event should explicitly record: if this promotion needs to be reversed, what is the version to roll back to and has that version's feature schema compatibility been verified? Recording this at promotion time, when the context is fresh, is significantly more reliable than reconstructing it under incident pressure.
Infrastructure prerequisites for each pattern
Shadow mode requires: traffic duplication capability, a comparison store for shadow vs. production predictions, and a monitoring view that shows comparison metrics in near-real-time. Without the monitoring view, shadow mode produces data that nobody looks at before the next deployment cycle.
Canary deployment requires: traffic splitting capability with percentage-level granularity, per-version metric monitoring (you need to see quality metrics segmented by model version, not aggregated), and an authorized progression workflow.
Full swap requires: rollback capability with documented target version, pre-swap quality gate evaluation, and a deployment window that accounts for downstream systems that may need to accommodate the model change.
Teams that deploy all three patterns in their infrastructure typically don't use all three for every model. The governance value is having explicit policy that specifies which deployment pattern is required for which model risk tier — and a record that the specified pattern was actually used for each production swap.