Experiment tracking is, at this point, a solved problem. MLflow, Weights & Biases, Comet — these platforms capture what happened during training with remarkable fidelity: hyperparameters, loss curves, evaluation metrics, artifact checksums, dataset version hashes. The observability layer for the training phase of a model's life is rich and mature.
But experiment tracking answers a different question than the one that actually creates organizational risk. The question experiment trackers answer is: what happened when we trained model version X? The question compliance teams and platform leads actually care about is: who authorized model version X to serve production traffic, against what policy, and is there a record that proves it?
Those are not the same question. The gap between them is where enterprise ML teams get into trouble.
The governance gap, defined precisely
The governance gap is the missing chain of custody between a model artifact and its production authorization. You have W&B telling you that run exp-2024-1104-b achieved an F1 of 0.882 on your holdout set. You have MLflow storing the serialized sklearn pipeline. What you don't have is a machine-readable record of: who saw those metrics, who made the call that 0.882 was good enough for production, what policy defined "good enough," whether a compliance reviewer signed off, and what the previous version's rollback point is.
Experiment tracking records training artifacts. Governance records decisions. They are adjacent in time but structurally different objects.
Why small teams miss it until it's too late
At four people, the governance process is a Slack thread and tribal knowledge. The ML engineer who trained the model sends the metrics to the data science lead. The data science lead pings the compliance contact. The compliance contact replies "looks fine." Six months later, nobody can reconstruct that chain from Slack history — especially after org reorgs, Slack channel archiving, and personnel changes.
Consider a scenario played out at a growing fintech during an internal audit in late 2023: the model governance review for their credit-decisioning model required tracing which version was live on which date, who approved the October retrain, and whether the PSI threshold was formally documented before the retrain was triggered. The audit took eleven days and involved three people reconstructing events from git blame, MLflow run timestamps, and a spreadsheet the compliance team had partially maintained. The actual answer was recoverable — but the cost of recovery was disproportionate to the risk it mitigated.
That's the governance gap at its most concrete: not that decisions weren't made thoughtfully, but that the record of those decisions is fragile, expensive to reconstruct, and not structured for export.
What experiment trackers were built for (and aren't)
This is not a criticism of experiment tracking tools. MLflow was designed as a model artifact registry and experiment logger for data scientists iterating on training runs. W&B was designed to help researchers visualize training dynamics and compare hyperparameter configurations. Both tools do their designed jobs well. Neither was designed to answer "who authorized this model to be in production and why."
We're not saying experiment tracking is insufficient — we're saying it's insufficient as a governance system. These are different use cases that happen to share some objects (model versions, metadata) but require different data models, different access patterns, and different audit semantics.
A governance system needs immutability. Experiment tracking metadata can be updated, deleted, or reorganized without much consequence. A governance record that can be edited post-decision is not a governance record — it's a document. The immutability requirement alone is architecturally distinct.
The four missing pieces
When enterprise ML teams audit their governance posture, they consistently find the same four gaps:
Decision attribution: No structured record of who made the promotion decision and in what role. The model went from candidate to production; the event was not captured as a durable object with actor, timestamp, and context.
Policy versioning: Retrain thresholds and approval requirements exist somewhere — in a runbook, a Confluence doc, sometimes a notebook comment — but they're not versioned alongside the model artifact. When the policy changes, there's no way to know what policy governed a historical decision.
Approval chains: For regulated industries or for teams with internal risk functions, the requirement is often that multiple roles sign off: ML Lead, Compliance Owner, sometimes a Product stakeholder. These chains exist informally but aren't tracked as structured workflow state.
Lineage continuity: The ability to answer "what is the direct ancestor of the current production model, and where does the branch point to the candidate that lost the evaluation?" This DAG exists implicitly but rarely as an explicit queryable structure.
The cost of the gap as teams scale
The operational cost of the governance gap grows nonlinearly with model count. At 5 production models, a disciplined team can maintain a manual governance log in a spreadsheet. At 25 models — the typical floor for a mature ML platform org — manual maintenance is a full-time job that still produces inconsistent results. At 50+ models, the gap becomes an audit liability.
The regulatory cost is harder to quantify until something goes wrong: a model promotes an outcome with disparate impact, a rollback is needed but there's no lineage to roll back to, or an auditor under SOC 2 or GDPR asks for a complete decision trail and your team can't produce one cleanly. At that point the cost becomes legal exposure, not just engineering overhead.
Bridging it correctly: what a governance layer actually needs
A governance layer that closes the gap has a clear job description: sit between your training infrastructure (where experiment trackers live) and your serving infrastructure (where SageMaker, Kubernetes, Seldon serve predictions), and record every decision that crosses that boundary.
Each governance event should be an immutable record containing: model version identifier, the drift or performance signal that triggered the decision, the approval chain that processed it (actors, roles, timestamps), the policy version that governed the approval criteria, and the outcome — promoted, rejected, escalated. These records should be queryable in bulk for audit export and individually accessible for incident investigation.
The integration point with experiment tracking tools is the model artifact handoff: when a candidate version exits the experiment tracking system as a promotion candidate, that's when the governance record opens. When the version enters or exits production, the record closes with the full decision chain attached.
That handoff is where the gap lives — and it's also where it can be closed without disrupting your existing training infrastructure.