The path from a notebook to a model that users hit, with a way to train it again and see if it drifted.

How is this different from a normal deploy pipeline?

DevOps versions code. This also has to version the data and the model, and watch whether the answers got worse.

What does the pipeline include?

Data in, a training job, a registry, a way to serve the model, and an alarm when it drifts.

How do you version a model?

I version the data, the code, and the model, usually with MLflow or DVC, so last month's result can be rebuilt.

How do you lock down the model and the data?

The training data and the endpoint get real access control, and the data is encrypted. I follow the policy your security people already have.

Can you take a notebook and make it a deploy?

I look at how a model leaves the notebook today, then add the pipeline and the registry. Your team sees one training run and one deploy.

MLOps

The infrastructure around training and serving models, from the notebook to production.

Back to Services

What I do

Train it, store it, serve it

Training jobs, a place to put the model, and a way to serve it.

A model change goes through a pipeline

A model change should go through a pipeline, the same way an app change does.

Did the answers change

I watch whether the model still answers well, and whether the data going in has changed.

What you get

The infrastructure around the model

I work on the infrastructure around the model: training jobs, deploys, and the checks that tell you it drifted.

From the notebook to production

The notebook can stay. The production path cannot be 'run it on my laptop'.

You can rebuild it

I version the data, the code, and the model, so last month's result can be rebuilt.

Where it already runs

I use the cloud the training job already runs on. I don't move it for the sake of a diagram.

How the work goes

From a notebook to a user

I look at how a model gets from a notebook to something a user can hit.

Training, registry, serving

You get the training job, the registry, and how a new model gets served.

In the repo you already have

I wire it into the repo and the cloud account you already have.

One training run, one deploy

I walk through one training run and one deploy, and leave the notes beside them.

Tools I use

These show up when they fit the job.

TensorFlow

PyTorch

Kubernetes

Kubeflow

MLflow

Apache Airflow

Docker

Nvidia CUDA

Amazon SageMaker

Google Cloud AI Platform

Azure Machine Learning

TensorFlow

PyTorch

Kubernetes

Kubeflow

MLflow

Apache Airflow

Docker

Nvidia CUDA

Amazon SageMaker

Google Cloud AI Platform

Azure Machine Learning

Situations I get called in for

The model only trains on one laptop

The package versions, the data paths, and the GPU drivers live on one person's machine. When they are away, nobody can retrain.

Nobody can say which model is live

There is no registry. Production loads a pickle from S3 with a date in the file name, and nobody is sure which training run made it.

The GPU bill is high and the GPUs sit idle

Notebook instances run over the weekend. Endpoints stay sized for a peak that happens once a month. I look at what is up and what is used.

Accuracy dropped and users noticed first

An upstream system changed a field, and the inputs no longer look like the training data. Nothing alarmed, because nothing was watching the inputs.

What I check first

A pinned training environment

I put training in a container image with the framework and CUDA versions pinned. A scikit-learn model saved with pickle or joblib is not guaranteed to load in a different scikit-learn version.

A registry entry that points back

In MLflow or the SageMaker Model Registry, each model version should link to its training run, the git commit, and the dataset version. Then a rollback is a pointer change.

The same preprocessing in training and serving

When feature code is rewritten for the serving path, small differences creep in. I check that both sides call the same code, so the model sees the inputs it was trained on.

Compute that bills while idle

A SageMaker real-time endpoint bills per instance hour, with or without traffic. For training jobs that can resume, managed spot training with checkpoints is often worth a look.

Somewhere to see drift

I log the inputs and predictions next to a baseline from the training data. SageMaker Model Monitor can capture endpoint traffic and compare it for you. A plain scheduled job works too.

MLOps