A model registry is not a folder

There is a question we ask early in every ML engagement, and it is deliberately boring:

Which model version is serving production right now, who approved it, and what data was it trained on?

A surprising number of competent teams cannot answer it in under a day. They have MLflow. They log runs. They have a registry. What they do not have is a lifecycle — because the registry is being used as a folder to put finished things in, rather than as the control point that decides what production runs.

The symptoms

You probably recognise at least one of these:

• The model serving production was trained from a notebook that has since been edited.
• Promotion happens when a data scientist tells an engineer, in Slack, that v9 looks good.
• Nobody knows whether the live model's accuracy has moved since launch, because nothing logs what it predicted.
• Rolling back means retraining, or finding a pickle file, or both.

Each of these is survivable alone. Together they mean your production model is an undocumented artefact that your business has quietly started depending on.

What the registry is actually for

A registry earns its keep when three things are true.

1. The model is an addressable, governed object

In Unity Catalog, a model is a first-class object with a three-level name, an owner and real permissions — not a path in a bucket that happens to contain a binary. Downstream consumers never reference a version number or a path:

model = mlflow.pyfunc.load_model("models:/prod.risk.churn_classifier@champion")

That alias indirection is the whole trick. Scoring jobs, endpoints and notebooks all resolve @champion at load time, which means promotion and rollback are metadata operations rather than code changes.

2. Registration is a build output, not a human step

The run that trains the model is the run that registers it, with its signature, input example and the data and code versions behind it:

with mlflow.start_run(run_name="gbt-v7"):
    mlflow.log_params(params)
    model = fit(X_train, y_train, **params)
    mlflow.log_metrics({"auc": auc, "recall": recall})

    mlflow.sklearn.log_model(
        model,
        name                  = "model",
        signature             = infer_signature(X_train, y_train),
        input_example         = X_train[:5],
        registered_model_name = "prod.risk.churn_classifier",
    )

The signature matters more than it looks. It is schema enforcement at serving time: the endpoint rejects a malformed request instead of quietly scoring garbage, which is how most silent production ML failures begin.

3. Promotion is a gate, not a conversation

This is the part teams skip, and it is the part that stops you shipping a worse model because it looked better in a notebook. Both candidates score on the same untouched holdout, against thresholds agreed before the experiment ran:

base = score_holdout(champion)
new  = score_holdout(challenger)

# A tie is not a reason to ship.
if new["auc"] < base["auc"] + 0.005:
    raise SystemExit(f"no lift: {new['auc']:.4f} vs {base['auc']:.4f}")

if new["recall_minority"] < 0.80:
    raise SystemExit("fairness floor breached")

client.set_registered_model_alias(MODEL, "champion", challenger.version)

Write the thresholds down while nobody is emotionally invested in a result. Afterwards, everyone has a favourite metric.

The test that tells you if it is real

Forget the architecture diagram. Ask your team to roll production back to the previous model version, right now, while you watch.

If that is an alias change and takes under a minute, you have a deployment process. If it involves finding someone, finding a file, or retraining, you have a hope — and hope has a habit of being unavailable the week it is needed.

Everything above is roughly a fortnight of work on an existing estate, and it is the cheapest insurance in enterprise ML. It is also, almost always, the engagement where we discover that a model someone shipped eighteen months ago has been degrading ever since and nobody was watching. That discovery is usually worth more than the new model they originally asked us for.