SageMaker is the serving layer. Inferpathio is the governance layer. Getting them to talk to each other takes three integration points: SDK initialization in your training pipeline, a Lambda-based log shipper from SageMaker endpoint invocations to Inferpathio's ingestion API, and a webhook receiver in your existing CI/CD pipeline to handle approved retrain events.
This guide assumes you have at least one active SageMaker inference endpoint, an Inferpathio account with an API key, and a training pipeline (SageMaker Pipelines, Step Functions, or similar) that produces new model artifacts.
Step 1: Register the model with Inferpathio during training
The first integration point is your training pipeline. Before the training job completes, register the model version with Inferpathio so the governance layer knows this artifact exists:
import inferpathio as ifp
import boto3
# Initialize with your API key (store in AWS Secrets Manager)
ifp.init(api_key=os.getenv('INFERPATH_API_KEY'))
# At the end of your SageMaker training job
with ifp.governance_run(
model_id='product-recommender-v2',
policy='production-standard'
) as gov:
# Register the SageMaker model artifact
gov.log_model(
artifact_uri=f"s3://{bucket}/{model_prefix}/model.tar.gz",
flavor='sklearn',
training_job_name=sagemaker_training_job_name,
dataset_version=training_dataset_version
)
gov.set_policy('production-standard')
gov.tag('team', 'recommendations')
gov.tag('env', 'candidate')
The governance_run context manager creates a governance artifact that links this model version to its training context. The training_job_name parameter creates a traceable reference back to the SageMaker training job, providing the lineage link between the Inferpathio governance record and the SageMaker artifact.
Step 2: Set up prediction log shipping from SageMaker to Inferpathio
Drift detection requires Inferpathio to see a sample of your endpoint's prediction traffic. The recommended pattern is asynchronous log shipping via SageMaker Data Capture, S3, and a Lambda trigger:
# In your endpoint deployment configuration
from sagemaker.model_monitor import DataCaptureConfig
data_capture_config = DataCaptureConfig(
enable_capture=True,
sampling_percentage=10, # sample 10% of requests
destination_s3_uri=f"s3://{bucket}/data-capture/{endpoint_name}/",
capture_options=["REQUEST", "RESPONSE"]
)
predictor = model.deploy(
initial_instance_count=1,
instance_type='ml.m5.large',
data_capture_config=data_capture_config,
endpoint_name=endpoint_name
)
Once Data Capture is writing to S3, attach a Lambda function that triggers on new S3 objects in the capture prefix and ships the captured data to Inferpathio's log ingestion endpoint:
# Lambda function: ship captured data to Inferpathio
import json, boto3, urllib3
def handler(event, context):
s3 = boto3.client('s3')
http = urllib3.PoolManager()
for record in event['Records']:
bucket = record['s3']['bucket']['name']
key = record['s3']['object']['key']
obj = s3.get_object(Bucket=bucket, Key=key)
captured_data = json.loads(obj['Body'].read())
# Forward to Inferpathio ingestion API
resp = http.request(
'POST',
'https://api.inferpathio.com/v1/ingest/predictions',
body=json.dumps({
'model_id': 'product-recommender-v2',
'endpoint_name': 'product-recommender-prod',
'records': captured_data
}),
headers={
'Content-Type': 'application/json',
'Authorization': f"Bearer {get_secret('INFERPATH_API_KEY')}"
}
)
With prediction logs flowing in, Inferpathio can begin computing drift metrics against the model's baseline distribution and firing retrain proposals when configured thresholds are crossed.
Step 3: Configure drift thresholds and approval policy
Now that the model is registered and logs are flowing, configure the drift detection policy. This lives in your inferpathio-policy.yaml, committed to the same repository as your model training code:
model: product-recommender-v2
retrain_trigger:
conditions:
- metric: psi
feature: user_session_count_7d
threshold: 0.15
window_days: 7
- metric: psi
feature: category_affinity_score
threshold: 0.18
window_days: 7
operator: OR
approval:
required: true
roles:
- ml-lead
sla_hours: 48
on_timeout: notify_ml_lead
quality_gates:
min_ndcg_at_10: 0.42
max_regression_vs_current: 0.015
on_gate_failure:
action: notify_ml_lead
keep_current_version: true
Step 4: Handle approved retrain webhooks in your pipeline
When a retrain proposal is approved, Inferpathio sends a webhook to a URL you configure. That webhook should trigger your SageMaker training pipeline. The webhook payload contains the governance context your pipeline needs:
{
"event": "retrain.approved",
"model_id": "product-recommender-v2",
"approved_by": "[email protected]",
"policy_version": "production-standard@v2",
"trigger": {
"type": "drift",
"metric": "psi",
"feature": "user_session_count_7d",
"value": 0.19,
"threshold": 0.15
},
"governance_run_id": "gov_8f3a2c1d",
"audit_url": "https://app.inferpathio.com/audit/gov_8f3a2c1d"
}Your webhook receiver (an API Gateway + Lambda, or a GitHub Actions webhook trigger) should extract the governance_run_id from the payload and pass it through to your SageMaker Pipelines execution as a parameter. This creates the traceable link between the Inferpathio approval event and the resulting SageMaker training job.
Step 5: Register the retrained version and close the governance loop
After the retraining pipeline completes and the new model version passes quality gates, register the new version with Inferpathio and reference the originating governance run:
with ifp.governance_run(
model_id='product-recommender-v2',
policy='production-standard',
parent_run_id=os.getenv('GOVERNANCE_RUN_ID') # passed from webhook
) as gov:
gov.log_model(
artifact_uri=new_model_s3_uri,
flavor='sklearn',
training_job_name=new_training_job_name,
dataset_version=current_dataset_version
)
gov.tag('env', 'candidate')
gov.submit_for_promotion() # enters the promotion approval queue
The parent_run_id links this new governance run to the drift-triggered retrain approval that authorized it. The full chain — drift detected, retrain approved by ML Lead, training job executed, new version registered, promotion evaluated — is now a single queryable lineage graph in Inferpathio.
Verifying the integration is working
After completing these steps, you can verify the integration by checking three things in the Inferpathio dashboard: the model appears in the model registry with its SageMaker endpoint reference; prediction log volume is showing in the monitoring view (confirming the Lambda shipper is working); and the drift metrics chart is populating with real data from your endpoint traffic.
The full integration, from training pipeline registration to drift-triggered governance loop, typically takes a half-day to set up for a single endpoint. The Lambda log shipper setup is usually the most time-consuming piece — particularly if your SageMaker endpoint uses a custom container where the Data Capture output format needs to be normalized before shipping.
Common integration issues and how to handle them
Data Capture output format varies by container type: Built-in SageMaker algorithm containers produce Data Capture output in a consistent JSONL format. Custom containers produce output in whatever format the inference script writes to stdout. If your inference script returns raw numpy arrays or custom JSON structures, the Lambda shipper will need normalization logic to extract feature vectors and prediction values in the format Inferpathio's ingestion API expects.
High-volume endpoints and Lambda concurrency: At significant prediction volume, a Lambda function triggered on every Data Capture S3 put can generate high Lambda concurrency. For endpoints processing more than 10,000 requests per minute with 10% sampling, consider batching the S3 puts using SageMaker Data Capture's batch aggregation configuration and processing in batch rather than per-file. The governance latency increase (minutes rather than seconds for drift metric computation) is acceptable for most retrain policy use cases, where the retrain trigger window is measured in hours or days, not minutes.
Multi-model endpoints: SageMaker multi-model endpoints host multiple models behind a single endpoint. Data Capture records include the target model name in the metadata. The Lambda shipper should extract the target model name and route prediction logs to the correct Inferpathio model ID — one multi-model endpoint may serve 20 distinct models, each with its own drift policy and governance configuration.
Maintaining the integration over time
The integration points that most commonly drift out of sync as your ML system evolves are the feature schema references in the policy configuration and the evaluation dataset version used for quality gates. When a new version of a model is trained on a different feature set, the Inferpathio policy should be updated to reflect the new expected input schema and the new evaluation dataset. This update should itself be a governance event — a policy version change — so that the audit trail shows when and why the quality gate criteria changed between model generations.
Teams that treat the Inferpathio policy configuration as a static file that they update manually when they remember find that it drifts out of sync with actual model behavior. The more reliable pattern is to generate the policy YAML programmatically from the model's training configuration as part of the training pipeline, so that every new model version automatically produces an updated policy that reflects its actual feature schema and training context.