Engineer
- governed Delta sources
- point-in-time correctness
- offline + online parity
Capability 04 / ML & MLOps
Models that survive
their first quarter.
Most enterprise ML dies in the gap between a notebook that works and a model that keeps working — unowned features, untracked experiments, a pickle file in a bucket, and nobody watching the drift. We build the lifecycle, not just the model: features, training, registry, serving, monitoring and retraining, all managed with MLflow in Unity Catalog.
The lifecycle
Seven stages. We build all of them, or none of them work.
You can skip any one of these and still get a demo. You cannot skip any one of them and get a model your business will still trust in six months.
Experiment
- baseline before boosting
- distributed when needed
- reproducible runs
MLflow
- params, metrics, artifacts
- data + code lineage
- signature and schema
Govern
- models in Unity Catalog
- aliases, not stages
- approval and audit
Serve
- batch, streaming, realtime
- shadow and canary
- rollback in one call
Watch
- drift and data quality
- inference tables
- alerts that page someone
Close the loop
- triggered, not scheduled
- challenger vs champion
- gated promotion
What we build
The unglamorous machinery around the model.
Feature engineering that is reusable
Features defined once as governed tables, with point-in-time joins so training never sees the future, and online tables so serving reads the same values the model trained on. The train/serve skew that silently destroys accuracy, designed out.
- Databricks Feature Engineering in Unity Catalog
- Point-in-time correct training sets
- Online tables for low-latency lookup
A baseline you have to beat
We start with the dumbest model that could work and make everything prove itself against it. AutoML to set the floor fast, then hand-built models where the lift is real, distributed with Spark or Ray when the data genuinely demands it.
- AutoML for a fast, honest baseline
- Hyperparameter search with Optuna
- Distributed training on Spark or Ray
MLflow as the system of record
Every run tracked with its parameters, metrics, artifacts and the exact data and code version behind it. Models registered in Unity Catalog with aliases, lineage and permissions — so “which model is in production and who approved it” has a one-line answer.
- Experiments, runs and model signatures
- Models in Unity Catalog, not a bucket
- Champion / challenger aliases
Batch, streaming and real time
The right inference mode for the decision being made — nightly batch scoring into Delta, streaming scores on arrival, or Model Serving endpoints with autoscaling behind an API. Canary and shadow traffic so a new version proves itself on real requests first.
- Mosaic AI Model Serving endpoints
- Batch and Structured Streaming inference
- Traffic splitting, canary and rollback
You find out before the business does
Inference logged to tables, input distributions and prediction quality monitored over time, and alerts wired to the people who own the model. Silent degradation is the normal failure mode of enterprise ML, and it is entirely preventable.
- Inference tables and request logging
- Data and prediction drift detection
- Alerting on quality, latency and cost
Promotion is a pipeline, not a person
Everything as code in Databricks Asset Bundles, deployed through dev, staging and prod the same way every time. A challenger only becomes champion by clearing the evaluation gate in CI, and anyone can see exactly what changed and why.
- Databricks Asset Bundles for every job
- Automated evaluation gates in CI
- Triggered retraining on drift or schedule
Experiment to production
The run that trains it also registers it.
There is no manual handoff, no exported pickle, no “can you send me the notebook”. Training logs the model with its signature and input example, registers it into Unity Catalog, and leaves a complete record of the data and code that produced it.
- Reproducible. Data version, git SHA and parameters on every run.
- Governed. The model object lives in Unity Catalog with real permissions.
- Addressable. Downstream code loads
@champion, never a path.
import mlflow
from mlflow.models import infer_signature
mlflow.set_registry_uri("databricks-uc")
MODEL = "prod.risk.churn_classifier"
with mlflow.start_run(run_name="gbt-v7") as run:
mlflow.log_params(params)
model = fit(X_train, y_train, **params)
mlflow.log_metrics({
"auc": roc_auc(y_val, model.predict(X_val)),
"recall": recall(y_val, model.predict(X_val)),
})
# Signature + example = schema enforcement at serving time.
mlflow.sklearn.log_model(
model,
name = "model",
signature = infer_signature(X_train, y_train),
input_example = X_train[:5],
registered_model_name = MODEL,
)
# Promote only after the gate passes - see evaluate.py
# client.set_registered_model_alias(MODEL, "challenger", version)from mlflow import MlflowClient
client = MlflowClient(registry_uri="databricks-uc")
champion = client.get_model_version_by_alias(MODEL, "champion")
challenger = client.get_model_version_by_alias(MODEL, "challenger")
base = score_holdout(champion)
new = score_holdout(challenger)
# A tie is not a reason to ship. Require real, measured lift.
if new["auc"] < base["auc"] + 0.005:
raise SystemExit(f"no lift: {new['auc']:.4f} vs {base['auc']:.4f}")
if new["recall_minority"] < 0.80:
raise SystemExit("fairness floor breached")
client.set_registered_model_alias(MODEL, "champion", challenger.version)
print(f"promoted v{challenger.version} - auc {new['auc']:.4f}")
# promoted v12 - auc 0.8741 (champion was 0.8663)Why a gate, not a meeting
“It looked better in the notebook” is how you ship a worse model.
Model promotion is the single highest-risk routine operation in enterprise ML, and in most organisations it is done by hand, by whoever trained it, on a metric they chose afterwards. We make it a pipeline step with thresholds agreed in advance.
- Pre-agreed. Thresholds are set before the experiment, not after.
- Held out. Both models score on the same untouched slice.
- Reversible. Rollback is moving an alias — seconds, not a redeploy.
If you cannot roll a model back in under a minute, you do not have a deployment process. You
have a hope.
ML examples
What we are usually asked to build.
These are the models that pay for themselves. Each one is a real pattern we have built, with the inference mode and the thing that actually makes it hard.
Who is leaving, and why
Propensity-to-churn scored nightly into Delta and pushed to the CRM, with per-customer reason codes from SHAP so the retention team gets an action, not a number.
- Batch inference, daily
- Hard part: label leakage in the event history
Thousands of series, not one
Hierarchical forecasts per SKU and location, trained in parallel across the cluster, reconciled so the store-level numbers add up to the regional plan.
- Batch inference, weekly with daily refresh
- Hard part: intermittent demand and promotions
Decisions in milliseconds
Real-time scoring behind a Model Serving endpoint with online feature lookup, tuned on the cost of a false positive rather than on accuracy, which is the wrong metric here.
- Real-time endpoint, p99 latency budget
- Hard part: extreme class imbalance and feedback delay
Sensor data to work orders
Streaming features off equipment telemetry, remaining-useful-life estimates, and alerts that land in the maintenance system with enough lead time to act.
- Streaming inference on arrival
- Hard part: very few true failures to learn from
Causal, not just correlated
Price elasticity estimated with the confounding handled honestly, plus guardrails so the optimiser cannot recommend something commercially absurd.
- Batch with scenario simulation
- Hard part: no experiments to learn from, only history
Retrieve, then rank
Two-stage architecture with candidate retrieval over embeddings in Vector Search and a ranking model on top, evaluated offline and then online where it counts.
- Real-time with precomputed candidates
- Hard part: cold start and offline/online metric gap
How an ML engagement runs
Decision first, model last.
Name the decision
Which decision changes because of this model, who makes it today, and what it is worth to make it better. If we cannot answer that, there is no project worth doing.
Week 1Check the data can carry it
Label quality, history depth, leakage risk and feature availability at inference time. Most ML projects are killed or saved here, before any training happens.
Week 1–2Baseline and bar
A simple model and the current process, measured side by side, with the promotion threshold agreed and written down while nobody is emotionally invested in a result.
Week 2–3Build the lifecycle
Features, training job, registry, serving, monitoring and retraining — as code in asset bundles, running in your workspace, owned by your team.
Week 3–8Hand over properly
Your engineers run the next retrain while we watch. Runbooks for drift, rollback and on-call. We are not trying to become a dependency.
Final two weeksStraight answers
The questions we always get.
Can an LLM just do this instead of training a model?
For a structured prediction on tabular data — churn, forecast, default risk — a gradient-boosted model is usually cheaper, faster, more accurate and far easier to defend to a regulator than a language model. We use LLMs where language is the problem, and we will say so when the honest answer is a small, boring model.
We already have models in production. Do you start over?
No. We wrap what exists first — get it tracked in MLflow, registered in Unity Catalog and monitored — so you can see how your current models are actually performing. That frequently reveals that one of them has been quietly degrading for a year, which is a better first win than a new model.
How do you handle model risk and audit requirements?
Lineage from training data to serving endpoint, immutable run records, model signatures, approval recorded on the alias change, and evaluation evidence stored with the version. In regulated settings we build to your model risk management policy and leave the documentation your validators actually ask for.
Do we need Databricks for this?
MLflow is open source and runs anywhere, and the patterns here are not Databricks-specific. We are opinionated about Databricks because Unity Catalog governance, Feature Engineering, Model Serving and Lakehouse Monitoring in one place removes most of the integration work — but if your platform is elsewhere, the lifecycle is still the job.
Start here
A free 60-minute architecture review
Bring your current architecture, data volumes and the workloads that hurt. You leave with a target-state diagram on Databricks, the three changes that would cut the most cost or latency, and an honest answer on whether you need us at all.
No slide deck. No procurement process. One senior architect on the call.