Copyright Notice
These slides are distributed under the Creative Commons License.
DeepLearning.AI makes these slides available for educational purposes. You may not use or distribute
these slides for commercial purposes. You may make copies of these slides and use or distribute them for
educational purposes as long as you cite DeepLearning.AI as the source of the slides.
For the rest of the details of the license, see https://creativecommons.org/licenses/by-sa/2.0/legalcode
C1W1 Slides
The Machine Learning Project Lifecycle
Welcome
Deployment example
Edge device
Inspection
Software
API
Camera
Prediction
Server Control
Software
Predict
Photo from camera
Visual inspection example
ML in production
ML Project Code
ML Model Code
“POC to Production Gap”
The requirements surrounding ML infrastructure
Machine Resource
Data Verification
Management
Data Collection
Serving
Configuration Monitoring
ML Infrastructure
Code Analysis Tools
Feature Extraction Process Management Tools
[D. Sculley et. al. NIPS 2015: Hidden Technical Debt in Machine Learning Systems]
The Machine Learning Project Lifecycle
Steps of an ML project
The ML project lifecycle
Scoping Data Modeling Deployment
Define data Monitor &
Label and Select and Perform error Deploy in
Define project and establish maintain
organize data train model analysis production
baseline system
The Machine Learning Project Lifecycle
Case study:
speech recognition
Speech recognition: Scoping stage
Scoping Data Modeling Deployment
Define data Perform error Deploy in Monitor &
Label and Select and
Define project and establish analysis production maintain
organize data train model
baseline system
! Decide to work on speech recognition for voice search.
! Decide on key metrics:
! Accuracy, latency, throughput
! Estimate resources and timeline
Speech recognition: Data stage
Scoping Data Modeling Deployment
Define data Perform error Deploy in Monitor &
Label and Select and
Define project and establish analysis production maintain
organize data train model
baseline system
Define data
“Um, today’s weather”
! Is the data labeled consistently? “Um… today’s weather”
! How much silence before/after each clip? “Today’s weather”
! How to perform volume normalization?
Speech recognition: Modeling stage
Scoping Data Modeling Deployment
Define data Perform error Deploy in Monitor &
Label and Select and
Define project and establish analysis production maintain
organize data train model
baseline system
Code (algorithm/model)
Research/Academia
Hyperparameters ML Model
Product Team
Data
Speech recognition: Deployment stage
Scoping Data Modeling Deployment
Define data Perform error Deploy in Monitor &
Label and Select and
Define project and establish analysis production maintain
organize data train model
baseline system
Mobile phone (edge device)
Speech
API Local Software
VAD
Microphone
module
Prediction Transcript
Server
Frontend code
Search
Results
The Machine Learning Project Lifecycle
Course outline
Course outline
Scoping Data Modeling Deployment
Define data Perform error Deploy in Monitor &
Label and Select and
Define project and establish analysis maintain
organize data train model production
baseline system
1. Deployment MLOps (Machine Learning Operations) is
an emerging discipline, and comprises a
2. Modeling set of tools and principles to support
progress through the ML project lifecycle.
3. Data
Optional: Scoping
Deployment
Key challenges
Concept drift and Data drift
Speech recognition example
Training set:
! Purchased data, historical user data with transcripts
Test set:
! Data from a few months ago
How has the data changed?
Software engineering issues
Checklist of questions
! Realtime or Batch Prediction
Service
! Cloud vs. Edge/Browser
! Compute resources (CPU/GPU/memory)
! Latency, throughput (QPS)
! Logging
! Security and privacy
First deployment vs. maintenance
Scoping Data Modeling Deployment
Define data Monitor &
Label and Select and Perform error Deploy in
Define project and establish maintain
organize data train model analysis production
baseline system
Deployment
Deployment patterns
Common deployment cases
1. New product/capability
2. Automate/assist with manual task
3. Replace previous ML system
Key ideas:
! Gradual ramp up with monitoring
! Rollback
Visual inspection example
Human Human Human
ML ML ML
ML system shadows the human and runs in parallel.
ML system’s output not used for any decisions during this phase.
Sample outputs and verify predictions of ML system.
Canary deployment
! Roll out to small fraction (say 5%) of traffic initially.
! Monitor system and ramp up traffic gradually.
Blue green deployment
Phone images
Old/Blue version
Router
New/Green version
Easy way to enable rollback
Degrees of automation
Human Shadow Partial Full
Al assistance
only mode automation automation
You can choose to stop before getting to full automation.
Deployment
Monitoring
Monitoring dashboard
Fraction of non-null outputs Fraction of missing input
Server load
values
Time Time Time
! Brainstorm the things that could go wrong.
! Brainstorm a few statistics/metrics that will detect the problem.
! It is ok to use many metrics initially and gradually remove the ones you find not useful.
Examples of metrics to track
Software Memory, compute, latency, throughput, server load
metrics:
Avg input length
Avg input volume
Input metrics:
Num missing values
Avg image brightness
# times return " " (null)
Output metrics: # times user redoes search
# times user switches to typing
CTR
Just as ML modeling is iterative, so is deployment
ML model/data Deployment/Monitoring
Performance
Error analysis Experiment Traffic
analysis
Iterative process to choose the right set of metrics to monitor.
Monitoring dashboard
Fraction of non-null outputs Fraction of missing input
Server load
values
Time Time Time
! Set thresholds for alarms
! Adapt metrics and thresholds over time
Model maintenance
Scoping Data Modeling Deployment
Define data Monitor &
Label and Select and Perform error Deploy in
Define project and establish maintain
organize data train model analysis production
baseline system
! Manual retraining
! Automatic retraining
Deployment
Pipeline monitoring
Speech recognition example
Audio Transcript
Speech
Recognition
Audio Transcript
Speech
VAD
Recognition
Some cellphones might have VAD clip audio differently, leading to degraded
performance
User profile example
User Data
Recommender Product
User Profile
system recommendations
(e.g., clickstream) (e.g., own car?)
Metrics to monitor
Monitor How quickly do they change?
! Software metrics ! User data generally has slower drift.
! Input metrics ! Enterprise data (B2B applications)
can shift fast.
! Output metrics
