AW S A c a d e m y M a c h i n e L e a r n i n g F o u n d a t i o n s

Module 3: Implementing a Machine

Learning Pipeline with
Amazon SageMaker
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Module overview

Sections Demonstration
1. Scenario introduction • Training a model using Amazon
2. Collecting and securing data SageMaker
3. Evaluating your data • Optimizing Amazon SageMaker
Hyperparameters
4. Preprocessing your data
5. Training • Running Amazon SageMaker Autopilot
6. Evaluating the accuracy of the model
7. Hosting and using the model
8. Hyperparameter and model tuning Knowledge check

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 2
Module overview continued

Lab
• Guided lab: Exploring Amazon Sagemaker
• Guided lab: Visualizing data
• Guided lab: Encoding categorical data
• Guided lab: Splitting data and training a model with xgBoost
• Guided lab: Hosting and consuming a model on AWS
• Guided lab: Evaluating model accuracy
• Guided lab: Tuning with Amazon SageMaker
• Challenge Lab:

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 3
Module objectives

At the end of this module, you should be able to:

• Formulate a problem from a business request
• Obtain and secure data for machine learning (ML)
• Build a Jupyter Notebook using Amazon SageMaker
• Outline the process for evaluating data
• Explain why data needs to be preprocessed
• Use open source tools to examine and preprocess data
• Use Amazon SageMaker to train and host an ML model
• Use cross-validation to test the performance of an ML model
• Use a hosted model for inference
• Create an Amazon SageMaker hyperparameter tuning job to optimize a model’s
effectiveness
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4
Module 3: Implementing a Machine Learning Pipeline with Amazon SageMaker

Section 1: Scenario introduction

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Machine learning pipeline
Section 6
New data and retraining
Business problem Deploy model

Section 1 Section 8

Problem Yes
Tune model
formulation

Section 2 Section 3 Section 4 Section 5 Section 7

Meets
Collect and Feature Select and train
Evaluate data Evaluate model business
label data engineering model
goal?

No
Feature augmentation
Data augmentation

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 6
Machine learning pipeline

New data and retraining

Business problem Deploy model

Problem Yes
Tune model
formulation

Meets
Collect and Feature Select and train
Evaluate data Evaluate model business
label data engineering model
goal?

No
Feature augmentation
Data augmentation

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 7
Define business objective

What is the business goal?

Questions to ask:
• How is this task done today?
• How will the business measure success?
• How will the solution be used?
• Do similar solutions exist, which you might learn from?
• What assumptions have been made?
• Who are the domain experts?

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 8
How should you frame this problem?

• Is the problem a machine learning problem?

• Is the problem supervised or unsupervised? Supervised
• What is the target to predict? learning

• Do you have access to the data?

• What is the minimum performance? Unsupervised Reinforcement
learning learning
• How would you solve this problem manually?
• What’s the simplest solution?

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 9
Example: Problem formulation

You want to identify fraudulent credit card transactions so that you

can stop the transaction before it processes.

Why?
Reduce the number of customers who end their membership because
of fraud.

Can you measure it?

10%
reduction in fraud
Move from qualitative statements to quantitative statements that can
be measured.
claims in retail
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 10
Make it an ML model

Credit card transaction is either fraudulent or not fraudulent.

Fraud
Binary classification problem
Not fraud

Use historical data of fraud reports to help define your model.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 11
Wine quality dataset

Question: Based on the composition of the wine, can you predict the quality and therefore the price?

Why:
• View statistics
• Deal with outliers
• Scale numeric data

Citation
Source: UCI Wine quality dataset
Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California,
School of Information and Computer Science.
P. Cortez, A. Cerdeira, F. Almeida, T. Matos and J. Reis.
Modeling wine preferences by data mining from physicochemical properties. In Decision Support Systems, Elsevier, 47(4):547-553, 2009.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 12
Car evaluation dataset

Question: Can you use a car’s attributes to predict whether the car will be
purchased?

Why:
• View statistics
• Encode categorical data

Citation
Source: UCI Car evaluation dataset
Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California,
School of Information and Computer Science.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 13
Vertebral column dataset

Question: Based on the biomechanical features, can you predict whether a

patient has an abnormality (disk hernia or spondylolisthesis)?

Why:
• View statistics
• Encode categorical data
• Train and tune a model

Citation
Source: UCI Vertebral column dataset
Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California,
School of Information and Computer Science.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 14
 • Business problems must be
Section 1 key converted to an ML problem
takeaways • Why?
• Can it be measured?
• What kind of ML problem is it?
• Classification or regression?

15 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Module 3: Implementing a Machine Learning Pipeline with Amazon SageMaker

Section 2: Collecting and securing data

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Machine learning pipeline

New data and retraining

Business problem Deploy model

Problem Yes
Tune model
formulation

Meets
Collect and Feature Select and train
Evaluate data Evaluate model business
label data engineering model
goal?

No
Feature augmentation
Data augmentation

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 17
What data do you need?

• How much data do you have, and where is it?

• Do you have access to that data?
• What solution can you use to bring all of this data into one centralized
repository?

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 18
Data sources

• Private data: Data that customers create

• Commercial data: AWS Data Exchange, AWS Marketplace, and other
external providers
• Open-source data: Data that is publicly available (check for limits on usage)
• Kaggle
• World Health Organization
• U.S. Census Bureau
• National Oceanic and Atmospheric Administration (U.S.)
• UC Irvine Machine Learning Repository
• AWS

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 19
Observations

ML problems need a lot of data—also called observations— where

the target answer or prediction is already known.

Customer Date of Vendor Charge Was this

transaction amount fraud? Feature
ABC 10/5 Store 1 10.99 No
DEF 10/5 Store 2 99.99 Yes
GHI 10/5 Store 2 15.00 No Target
JKL 10/6 Store 2 99.99 ?
MNO 10/6 Store 1 99.99 Yes

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 20
Get a domain expert

• Do you have the data that you need to try to address this problem?
• Is your data representative?

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 21
Storing data in AWS

Amazon Simple Amazon FSx Amazon Elastic

Storage Service File System
(Amazon S3) (Amazon EFS)

Amazon Relational Amazon Redshift Amazon Timestream

Database Service
(Amazon RDS)

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 22
Extract, transform, load (ETL)

Data Catalog

Crawler
Data
Stores
Schedule or Event

Extract Load

Transform Script
Data Data
Source Target

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 23
ETL with AWS Glue

• Runs the ETL process

• Crawls data sources to create catalogs that other
systems can query
• ML functionality

AWS Glue can glue together different datasets and emit a single endpoint
that can queried.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 24
AWS Glue overview

Amazon S3 AWS Lambda

AWS Glue Data

Catalog

Endpoint
Amazon AWS Glue
Redshift

Amazon RDS
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 25
ETL with Python

import boto3, requests, zipfile, os, io

url = 'http://url.com/somezipfile.zip‘
folder='./extracts‘
Imports and variables

r = requests.get(url, stream=True)
thezip = zipfile.ZipFile(io.BytesIO(r.content)) Download and extract
thezip.extractall(folder)

s3 = boto3.client('s3')
bucket = 'bucketname'
with os.scandir(folder) as dir:
for f in dir:
if f.is_file():
Upload to Amazon S3
s3.upload_file(
Filename=os.path.join(folder,f.name),
Key=f.name, Bucket=bucket)

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 26
Securing data: AWS Identity and Access
Management (IAM) policy
IAM policies to control access:
{
"Id": "Policy1583974368597",
"Version": "2012-10-17", GetObject limits access to read only…
"Statement": [
{
"Sid": "Stmt1583974365198", …allowing the action…
"Action": ["s3:GetObject"],
"Effect": "Allow",
"Resource": "arn:aws:s3:::awsmachinelearningrepo/*", …for only this bucket…
"Principal": {"AWS": ["DataReaderRole"]}
}
] …by only this IAM role.
}

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 27
Securing data: Data encryption

The contents of many data repositories in AWS can be quickly and easily
encrypted.

Amazon S3 default encryption Amazon RDS encryption

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 28
 Securing data: AWS CloudTrail for audit

AWS CloudTrail tracks user activity and application programming interface

(API) usage.

Compliance Auditing

Operational Troubleshooting
AWS CloudTrail
Track user activity
and detect unusual Capture Store Act Review Security Analysis
API usage
Automatic Compliance
Remediation

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 29
Module 3 – Guided
Lab 1:
Exploring Amazon
SageMaker

30 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 • The first step in the machine learning
Section 2 key pipeline is to obtain data
takeaways • Extract, transform, and load (ETL) is
a common term for obtaining data for
machine learning
• AWS Glue makes it easy to run ETL
jobs from various data stores
• Securing your data includes both
controlling access and encrypting
data

31 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Module 3: Implementing a Machine Learning Pipeline with Amazon SageMaker

Section 3: Evaluating your data

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Machine learning pipeline
New data and retraining
Deploy model
Business problem

Problem Format data Yes

Tune model
formulation
Examine data types

Collect and Feature

Perform descriptive statistics
Select and train
Meets
Evaluate data Evaluate model business
label data engineering model
goal?
Visualize data
No
Feature augmentation

Data augmentation

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 33
You must understand your data

Before you can run statistics on your data, you must ensure that it’s in the
right format for analysis.
Date of Charge Was This
Customer Vendor
Transaction Amount Fraud?

“Customer:ABC,DateOfTr ABC 10/5 Store 1 10.99 No

ansation:10/5,Vendor:Store DEF 10/5 Store 2 99.99 Yes
1,ChargeAmount:10.99,Wa
GHI 10/5 Store 2 15.00 No
sThisFruad:No…”
JKL 10/6 Store 2 99.99 ?

MNO 10/6 Store 1 99.99 Yes

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 34
Loading data into pandas

• Reformats data into tabular representation (DataFrame)

• Converts common formats like comma-separated values (CSV),
JavaScript Object Notation (JSON), Excel, Pickle, and others

import pandas as pd
url = "https://somewhere.com/winequality-red.csv"
df_wine = pd.read_csv(url,';')
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 35
pandas DataFrame
Number of instances
df_wine.shape
Number of attributes

df_wine.head(5)
Columns/Attributes
Rows/Instances

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 36
Index and column names

df_wine.columns
Index(['fixed acidity', 'volatile acidity', 'citric acid', 'residual sugar',
'chlorides', 'free sulfur dioxide', 'total sulfur dioxide', 'density',
'pH', 'sulphates', 'alcohol', 'quality'],
dtype='object')

df_wine.index
RangeIndex(start=0, stop=1599, step=1)

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 37
DataFrame schema

df_wine.dtypes df_wine.info()
quality int64 <class 'pandas.core.frame.DataFrame'>
fixed acidity float64 Int64Index: 1597 entries,
volatile acidity float64 0 to 1598
citric acid float64 Data columns (total 12 columns):
residual sugar float64 quality 1597 non-null int64
chlorides float64 fixed acidity 1597 non-null float64
free sulfur dioxide float64 volatile acidity 1597 non-null float64
total sulfur dioxide float64 citric acid 1597 non-null float64
density float64 residual sugar 1597 non-null float64
pH float64 chlorides 1597 non-null float64
sulphates float64 free sulfur dioxide 1597 non-null float64
alcohol float64 total sulfur dioxide 1597 non-null float64
dtype: object density 1597 non-null float64
pH 1597 non-null float64
sulphates 1597 non-null float64
alcohol 1597 non-null float64
dtypes: float64(11), int64(1)
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
memory usage: 162.2 KB 38
Descriptive statistics

Use descriptive statistics to gain insights into your data

before you clean the data:

Overall statistics Multivariate statistics Attribute statistics

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 39
 Statistical characteristics
df_wine.describe()
total
fixed volatile residual free sulfur
citric acid chlorides sulfur pH sulphates alcohol quality
acidity acidity sugar dioxide
dioxide
count 1599.00 1599.00 1599.00 1599.00 1599.00 1599.00 1599.00 1599.00 1599.00 1599.00 1599.00

mean 8.32 0.53 0.27 2.54 0.09 15.87 46.47 3.31 0.66 10.42 5.64

std 1.74 0.18 0.19 1.41 0.05 10.46 32.90 0.15 0.17 1.07 0.81

min 4.60 0.12 0.00 0.90 0.01 1.00 6.00 2.74 0.33 8.40 3.00

25% 7.10 0.39 0.09 1.90 0.07 7.00 22.00 3.21 0.55 9.50 5.00

50% 7.90 0.52 0.26 2.20 0.08 14.00 38.00 3.31 0.62 10.20 6.00

75% 9.20 0.64 0.42 2.60 0.09 21.00 62.00 3.40 0.73 11.10 6.00

max 15.90 1.58 1.00 15.50 0.61 72.00 289.00 4.01 2.00 14.90 8.00
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 40
Categorical statistics identify frequency of values and class imbalances

df_car.head(5)
buying maint doors persons lug_boot safety class
0 vhigh vhigh 2 2 small low unacc
1 vhigh vhigh 2 2 small med unacc
2 vhigh vhigh 2 2 small high unacc
3 vhigh vhigh 2 2 med low unacc
4 vhigh vhigh 2 2 med med unacc

df_car.describe()
buying maint doors persons lug_boot safety class
count 1728 1728 1728 1728 1728 1728 1728
unique 4 4 4 3 3 3 4
top low low 2 2 big low unacc
freq 432 432 432 576 576 576 1210

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 41
Plotting attribute statistics
df_wine[‘sulphates’].hist(bins=10) df_wine[‘sulphates’].plot.box()

df_wine[‘sulphates'].plot.kde()

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 42
Plotting multivariate statistics
df_wine.plot.scatter( pd.plotting.scatter_matrix(
x='alcohol', df_wine[['citric acid',
y='sulphates') 'alcohol',
'sulphates']])

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 43
 Scatter plot with identification
high = df_wine[['sulphates','alcohol']][df_wine['quality']>5]
low = df_wine[['sulphates','alcohol']][df_wine['quality']<=5]

plt.scatter(high['sulphates'],high['alcohol'],s=50,c='blue',marker='o',label='great')
plt.scatter(x=low['sulphates'],y=low['alcohol'],s=50,c='red',marker='v',label='poor')

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 44
Correlation matrix

corr_matrix = df_wine.corr()
corr_matrix["quality"].sort_values(ascending=False)

Do alcohol and sulphates correlate to

wine quality?

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 45
Correlation matrix heat map
import seaborn as sns

correlations = df_wine.corr()
fig, ax = plt.subplots(figsize=(10, 10))

colormap =
sns.color_palette("BrBG", 10)

sns.heatmap(correlations,
cmap=colormap,
annot=True,
fmt=".2f")

ax.set_yticklabels(colum_names);

plt.show()

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 46
Module 3 – Guided
Lab 2: Visualizing
Data

47 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 • The first step in evaluating data is to
Section 3 key make sure that it’s in the right format
takeaways • pandas is a popular Python library
for working with data
• Use descriptive statistics to learn
about the dataset
• Create visualizations with pandas to
examine the dataset in more detail

48 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Module 3: Implementing a Machine Learning Pipeline with Amazon SageMaker

Section 4: Feature engineering

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Machine learning pipeline

New data and retraining

Business problem Deploy model

Problem Yes
Tune model
formulation

Meets
Collect and Feature Select and train
Evaluate data Evaluate model business
label data engineering model
goal?

No
Feature augmentation
Data augmentation

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 50
Feature selection and extraction

Feature Extraction Feature Selection

XX X

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 51
Feature extraction
• Invalid values • Transformation
• Wrong formats • Normalization
• Misspelling • Dimensionality reduction
• Duplicates
• Consistency
• Rescale
• Encode categories
• Remove outliers
• Reassign outliers
• Bucketing
• Decomposition
• Aggregation
• Combination

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 52
Encoding ordinal data

Categorical data is non-numeric data.

Categorical data must be converted (encoded) to a numeric scale.

Maintenance Costs Encoding

Low 1
Medium 2
High 3
Very High 4

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 53
Encoding non-ordinal data

If data is non-ordinal, the encoded values also must be non-ordinal.

Non-ordinal data might need to be broken into multiple categories.

… Color … Red Blue Green

… Red … 1 0 0
… Blue … 0 1 0
… Green … 0 0 1
… Blue … 0 1 0
Green … 0 0 1

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 54
Cleaning data

Types of data to clean:

Type Example Action

Variations in strings Med. vs. Medium Convert to standard text
Variations in scale Number of doors vs. Normalize to a common
number purchased scale
Columns with multiple Safe high maintenance Parse into multiple
data items columns
Missing data Missing columns of data Delete rows or impute data
Outliers Various

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 55
Finding missing data

• Missing data makes it difficult to

interpret relationships
• Causes of missing data –
• Undefined values
• Data collection errors
• Data cleaning errors
• Example pandas code to find
missing data –

df.isnull().sum() #count missing values for each column

df.isnull().sum(axis=1) #count missing values for each row

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 56
Plan for missing values

What were the mechanisms that caused the missing

values?
Before dropping
or imputing
(replacing) Are these missing values missing at random?
missing values,
ask: Are rows or columns missing that you are not
aware of?

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 57
Dropping missing values

Drop missing data with pandas

• dropna function to drop rows
• df.dropna()
• dropna function to drop columns with null values
• df.dropna (axis=1)
• dropna function to drop a subset
• df.dropna(subset=[“buying”])

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 58
Imputing missing data

• First, determine why the data is missing

• Two ways to impute missing data –
• Univariate: Adding data for a single row of missing data
• Multivariate: Adding data for multiple rows of missing data

from sklearn.preprocessing import Imputer

import numpy as np
Arr = np.array([[5,3,2,2],[3,None,1,9],[5,2,7,None]])
imputer = Imputer(strategy='mean’)
imp = imputer.fit(arr)
imputer.transform(arr)

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 59
Outliers

• Outliers can –
• Provide a broader picture of the data
• Make accurate predictions difficult
• Indicate the need for more columns
• Types of outliers –
• Univariate: Abnormal values for a
single variable
• Multivariate: Abnormal values for a
combination of two or more variables

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 60
Finding outliers

• Box plots show variation and

distance from the mean
• Example to the right shows a box
plot for the amount of alcohol in a
collection of wines
• Scatter plots can also show outliers
• Example to the right shows a scatter
plot that shows the relationship
between alcohol and sulphates in a
collection of wines

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 61
Dealing with outliers

Impute a new value for

Delete the outlier Transform the outlier
the outlier

Reduces the variation that You might use the mean of

Outlier is based on an the extreme outlier value the feature, for instance,
artificial error. causes and the outlier’s and impute that value to
influence on the dataset. replace the outlier value.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 62
Feature selection

• Filter method
• Use statistical methods

Feature Selection
• Wrapper methods
• Actually train the model
XX X
• Embedded methods
• Integrated with model

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 63
Feature selection: Filter methods

• Measures –
• Pearson’s correlation All features
• Linear discriminant analysis
(LDA)
Statistics and
• Analysis of variance
correlation
(ANOVA)
• Chi-square
Best features

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 64
Feature selection: Wrapper

• Methods – All features

• Forward selection
• Backward selection
Feature subset
Evaluate
results

Train model Train ?

Best features
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 65
Feature selection: Embedded methods

• Methods – All features

• Decision trees
• LASSO and RIDGE Train model

Feature subset
Evaluate
results
?

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 66
Module 3 – Guided
Lab 3:
Encoding
Categorical Data

67 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 • Feature engineering involves –
Section 4 key • Selection

takeaways • Extraction
• Preprocessing gives you better data
• Two categories for preprocessing –
• Converting categorical data
• Cleaning up dirty data
• Use categorical encoding to convert categorical
data
• Various types of dirty data –
• Missing data
• Outliers
• Develop a strategy for dirty data –
• Replace or delete rows with missing data
• Delete, transform, or impute new values for outliers

68 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Module 3: Implementing a Machine Learning Pipeline with Amazon SageMaker

Section 5: Training

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Machine learning pipeline

New data and retraining

Business problem Deploy model

Problem Yes
Tune model
formulation

Meets
Collect and Feature Select and train
Evaluate data Evaluate model business
label data engineering model
goal?

No
Feature augmentation
Data augmentation

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 70
File formats for machine learning

JSON
*.csv *.rec text Protobuf
lines
recordIO

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 71
Formatting the data for an algorithm

When you use CSV format, use the first column as the target variable.

Target Feature1 Feature 2 Feature 3 Feature 4 Feature 5

2 8.39 92.3 1 0 5
3 7.82 201.39 0 1 3
*.csv 3 2.39 245.21 0 0 10
2 8.48 183.92 1 1 1
1 5.80 28.2 0 1 1

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 72
Why split the data?
Y

All data that

is used for
training and
evaluation

X
Overfitting
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 73
Holdout method

10% - Test data

10% - Validation data

80% - Training data

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 74
K-fold cross validation

Model 1 Model 2 Model 3 Model 4 Model 5

Test Train Train Train Train

Train Test Train Train Train

k=5 Train Train Test Train Train

Train Train Train Test Train

Train Train Train Train Test

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 75
Shuffle your data

quality Fixed acidity sulphates

3 7.2 0.56
Test data
3 7.2 0.68
4 7.8 0.65
4 7.8 0.65
4 … …
5 … …
5 … … Training data
5 … …
6 … …
6 … …

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 76
Training models with Amazon SageMaker

Amazon SageMaker provides ML algorithms that are

optimized for speed, scale, and accuracy.

Amazon SageMaker Amazon SageMaker Amazon SageMaker AWS Marketplace

built-in algorithms supported frameworks custom frameworks algorithms

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
77
Amazon SageMaker built-in algorithms

Classification Quantitative Recommendations Unsupervised Specialized

Factorization Image Neural Topic

XGBoost XGBoost K-means
machines classification Model (NTM)

Principal Latent Dirichlet

Sequence-to-
Linear learner Linear learner Component Allocation
sequence
Analysis (LDA)

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
78
XGBoost

• Is an open-source implementation of the

gradient boosted trees algorithm
• Has performed well in machine learning
competitions
• Robustly handles various data types,
relationships, and distributions, and a
large number of hyperparameters

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
79
Linear learner

• Provides a solution for both Regression

classification and regression

problems
• Enables you to simultaneously
explore different training
objectives and choose the best
solution from your validation set Binary classification

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 80
K-means

• It attempts to find discrete

groupings within data, where
members of a group are as similar
as possible.
• The means in k-means is the
averaging of the data. This
averaging helps to find the center
of the grouping.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 81
Creating a training job in Amazon
SageMaker
2

{
{ { code comes here
code comes here code comes here code comes here
code comes here code comes here code comes here
code comes here code comes here }
} }

1 Training code
S3 bucket Helper code Training code
training data
Model training 3
on ML computer instances
Amazon Elastic
Container Registry
Amazon SageMaker (Amazon ECR)

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 82
 Demonstration: Training a Model
Using Amazon SageMaker

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 83
Module 3 – Guided
Lab 4:
Splitting Data and
Training a Model
with XGBoost

84 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 • Split data into training and testing sets –
Section 5 key • Optionally, split into three sets, including
takeaways validation set
• Can use k-fold cross validation to use all
the non-test data for validation
• Can use two key algorithms for
supervised learning –
• XGBoost
• Linear learner
• Use k-means for unsupervised learning
• Use Amazon SageMaker training jobs

85 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Module 3: Implementing a Machine Learning Pipeline with Amazon SageMaker

Section 6: Hosting and using the model

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Machine learning pipeline

New data and retraining

Business problem Deploy model

Problem Yes
Tune model
formulation

Meets
Collect and Feature Select and train
Evaluate data Evaluate model business
label data engineering model
goal?

No
Feature augmentation
Data augmentation

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 87
Is your model ready to deploy?

Trained Tuned Tested

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 88
Deployment options

Amazon SageMaker hosting services

Two options for

deployment

Batch transform

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 89
The goal of deployment

Predictions

Managed
environment hosting In production
models

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 90
Deploying with Amazon SageMaker

Client
Inference Inference
request response
Secure
Amazon SageMaker endpoint

Helper Inference
code code
Models

Helper Training
code code

Training
data Inference
code images
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 91
Creating an endpoint

You can create an endpoint in the console:

Or, you can use the Amazon SageMaker SDK:

xgb_predictor = xgb_model.deploy(initial_instance_count=1,
content_type='text/csv',
instance_type='ml.t2.medium'
)

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 92
Obtaining predictions

Data
processing Model training Predictions
steps

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 93
Module 3 – Guided
Lab 5:
Hosting and
Consuming a
Model on AWS

94 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 • Can use two options for deployment
Section 6 key • Amazon SageMaker hosting
takeaways • Batch transform
• Deploy only after you have tested
your model
• Goal is to generate predictions for
client applications
• Create an endpoint
• Single-model endpoint for simple use
cases
• Multi-model endpoint to support
multiple use cases

95 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Module 3: Implementing a Machine Learning Pipeline with Amazon SageMaker

Section 7: Evaluating the accuracy of the model

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Machine learning pipeline

New data and retraining

Business problem Deploy model

Problem Yes
Tune model
formulation

Meets
Collect and Feature Select and train
Evaluate data Evaluate model business
label data engineering model
goal?

No
Feature augmentation
Data augmentation

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 97
Evaluation determines how well your model predicts the target on future data

Testing and evaluation: 10% – Test data

To evaluate the predictive 10% – Validation data
quality of the trained model

80% – Training data

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 98
Success metric

Success metric:
Business problem:
10% reduction in fraud
Fraudulent credit Model metric
claims in 6-month
card transactions
period

The model metric must align to both the business problem and the success metric.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 99
 Confusion matrix

10% – Test data

Actual
Trained
model Cat Not cat
Predicted
Cat 107 23

Results Not cat 69 42

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 100
Confusion matrix terminology

True positive (TP)

Predicted a cat and it was a cat
Actual
False positive (FP)
Cat Not cat Predicted not a cat and it was a cat
Predicted

Cat TP FP False negative (FN)

Predicted a cat and it was not a cat
Not cat FN TN
True negative (TN)
Predicted not a cat and it was not a cat

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 101
Which model is better?

Confusion matrices from two models that use the same data:

Actual Actual
Cat Not cat Cat Not cat
Predicted

Predicted
Cat 107 23 Cat 148 53
Not cat 69 42 Not cat 28 12

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 102
Sensitivity

Sensitivity:
What percentage of cats were correctly identified?

Actual TP
sensitivity=
TP + FN
Cat Not cat
Predicted

Cat TP:107 FP:23 0.6079

Not cat FN:69 TN:42

60% of cats were identified as cats.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 103
Specificity

Specificity:
What percentage of not cats were correctly identified?

Actual TN
specificity=
TN + FP
Cat Not cat
Predicted

Cat TP:107 FP:23 0.6461

Not cat FN:69 TN:42

64% of not cats were identified
as not cats

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 104
Which model is better now?
Model A Model B

Actual Actual
Cat Not cat Cat Not cat
Predicted

Predicted
Cat 107 23 Cat 148 53
Not cat 69 42 Not cat 28 12

Sensitivity 60% Sensitivity 84%

Specificity 64% Specificity 18%

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 105
Thresholds

Models return probabilities:

Predicted
0.96 if [predicted] >= threshold It’s a cat!
0.77
0.58
0.01
0.47 if [predicted] < threshold NOT cat!
Changing the threshold can change the prediction.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 106
 Classification: ROC graph
1.0
Receiver operating
0.8
characteristic (ROC)
True-positive rate
Sensitivity

0.6

Plot points for each

0.4
TPR = FPR threshold value
0.2
Select threshold based on
0.0 business case
0.0 0.2 0.4 0.6 0.8 1.0

1 – Specificity
False-positive rate
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 107
 Classification: AUC-ROC
1.0
Area
0.8
under the
curve
True-positive rate
Sensitivity

0.6

AUC: 89% AUC-ROC is used to

0.4
easily compare one
0.2 model to another

0.0
0.0 0.2 0.4 0.6 0.8 1.0

1-Specificity
False-positive rate
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 108
Classification: Other metrics

accuracy Model’s score

precision = Proportion of positive results that

were correctly classified

precision ⋅sensitivity
F 1 score = 2· Combines precision and sensitivity
precision + sensitivity

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 109
Regression: Mean squared error
(x6,y6)

(x4,y4)

(x3,y3)

𝑛
1
MSE = ∑ ( 𝑖 𝑖 )
2
𝑦 − ỹ y (x1,y1)
𝑛 𝑖=1 (x5,y5)
(x7,y7)

(x2,y2)

x
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 110
ML tuning process

Tune model

Meets
Feature Select and train
Evaluate model business
engineering model
goal?

No
Feature augmentation

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 111
Module 3 – Guided
Lab 6:
Evaluating Model
Accuracy

112 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 • Several ways to validate the model
Section 7 key • Hold-out
takeaways • K-fold cross validation
• Two types of model evaluation
• Classification
• Confusion matrix
• AUC-ROC
• Regression testing
• Mean squared

113 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Module 3: Implementing a Machine Learning Pipeline with Amazon SageMaker

Section 8: Hyperparameter and model tuning

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Machine learning pipeline

New data and retraining

Business problem Deploy model

Problem Yes
Tune model
formulation

Meets
Collect and Feature Select and train
Evaluate data Evaluate model business
label data engineering model
goal?

No
Feature augmentation
Data augmentation

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 115
Recap: Goal for models

Balanced and generalizes well

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 116
Hyperparameter categories

Model Optimizer Data

How the model learns

Help define the model Define attributes of the data itself
patterns on data

Filter size, pooling, Gradient descent, Useful for small or homogenous

stride, padding stochastic gradient descent datasets

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 117
Tuning hyperparameters

Manually select hyperparameters

according to one’s intuition or
experience

However, this approach is often not as thorough

or efficient as needed
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 118
Amazon SageMaker offers
automated hyperparameter tuning

Clients Hyperparameter
(console, notebook, Training job
tuning job
IDEs, AWS CLI)

Training job
Tuning strategy

Objective metric:
Area under the Training job
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
curve 119
Automated hyperparameter tuning

Clients Hyperparameter
(console, notebook, Training job
tuning job
IDEs, AWS CLI)

Model Objective
Name Metric Eta Max_depth
Model3 0.8 0.7 6 Training job
Model1 0.75 0.09 5 Tuning strategy

Objective metric:
Area under the Training job
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
curve 120
Hyperparameter tuning

Note: Tuning doesn’t always improve your model.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 121
Tuning best practices

• Don’t adjust every hyperparameter

• Limit your range of values to what’s most effective
• Run one training job at a time instead of multiple jobs in parallel
• In distributed training jobs, make sure that the objective metric that you
want is the one that is reported back
• With Amazon SageMaker, convert log-scaled hyperparameters to linear-
scaled when possible

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 122
 Demonstration: Optimizing
Amazon SageMaker
Hyperparameters

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 123
Amazon SageMaker Autopilot
Inputs Outputs
Amazon SageMaker Autopilot

Test data
Analyze data
Trained models
Feature
engineering
Training data

y Model tuning
Model metrics
Target data

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 124
 Demonstration: Running
Amazon SageMaker
Autopilot

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 125
Module 3 – Guided
Lab 7:
Tuning with
Amazon
SageMaker

126 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 • Model tuning helps find the best
Section 8 key solution
takeaways • Hyperparameters
• Model
• Optimizer
• Data
• Tuning
• Use Amazon SageMaker to help tune
hyperparameters
• Use Autopilot for faster development

127 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Module 3: Implementing a Machine Learning Pipeline with Amazon SageMaker

Module wrap-up

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Module summary

• In this module, you learned how to:

• Formulate a problem from a business request
• Obtain and secure data for machine learning (ML)
• Build a Jupyter Notebook by using Amazon SageMaker
• Outline the process for evaluating data
• Explain why data must be preprocessed
• Use open source tools to examine and preprocess data
• Use Amazon SageMaker to train and host an ML model
• Use cross-validation to test the performance of an ML model
• Use a hosted model for inference
• Create an Amazon SageMaker hyperparameter tuning job to optimize a model’s
effectiveness
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 129
Complete the knowledge check

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 130
Additional resources

• <Add URLs to resources that students might find helpful for further
exploration on topics discussed in this module.>
• <Especially, link to whitepapers or other resources mentioned in the Exam
Guide of the certification exam this course is intended to help students
prepare for.>

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 131
Thank you

© 2021 Amazon Web Services, Inc. or its affiliates. All rights reserved. This work may not be reproduced or redistributed, in whole or in part, without prior written permission from Amazon
Web Services, Inc. Commercial copying, lending, or selling is prohibited. Corrections, feedback, or other questions? Contact us at https://support.aws.amazon.com/#/contacts/aws-training.
All trademarks are the property of their owners.