Introducing

Machine Learning
What is Machine Learning?

Machine learning teaches computers to do what comes naturally to

humans and animals: learn from experience. Machine learning algorithms
use computational methods to “learn” information directly from data
without relying on a predetermined equation as a model. The algorithms
adaptively improve their performance as the number of samples available
for learning increases.
More Data, More Questions, Better Answers

Machine learning algorithms find natural patterns in data Real World Applications
that generate insight and help you make better decisions
and predictions. They are used every day to make critical With the rise in big data, machine learning has become
decisions in medical diagnosis, stock trading, energy load particularly important for solving problems in areas like these:
forecasting, and more. Media sites rely on machine learning • Computational finance, for credit scoring and
to sift through millions of options to give you song or movie algorithmic trading
recommendations. Retailers use it to gain insight into their
customers’ purchasing behavior. • Image processing and computer vision, for face recognition,
motion detection, and object detection

• Computational biology, for tumor detection, drug discovery,

and DNA sequencing

• Energy production, for price and load forecasting

• Automotive, aerospace, and manufacturing, for

predictive maintenance

• Natural language processing

Introducing Machine Learning 3

How Machine Learning Works

Machine learning uses two types of techniques: supervised

learning, which trains a model on known input and output data
so that it can predict future outputs, and unsupervised learning,
which finds hidden patterns or intrinsic structures in input data.

UNSUPERVISED
LEARNING
CLUSTERING
Group and interpret
data based only
on input data

MACHINE LEARNING
CLASSIFICATION
SUPERVISED
LEARNING
Develop predictive
model based on both
input and output data
REGRESSION

Introducing Machine Learning 4

Supervised Learning

The aim of supervised machine learning is to build a model Using Supervised Learning to Predict Heart Attacks
that makes predictions based on evidence in the presence of
uncertainty. A supervised learning algorithm takes a known Suppose clinicians want to predict whether someone will have a
set of input data and known responses to the data (output) heart attack within a year. They have data on previous patients,
and trains a model to generate reasonable predictions for the including age, weight, height, and blood pressure. They know
response to new data. whether the previous patients had heart attacks within a year. So
the problem is combining the existing data into a model that can
Supervised learning uses classification and regression predict whether a new person will have a heart attack
techniques to develop predictive models. within a year.
• Classification techniques predict discrete responses—for
example, whether an email is genuine or spam, or
whether a tumor is cancerous or benign. Classification
models classify input data into categories. Typical
applications include medical imaging, speech recognition,
and credit scoring.

• Regression techniques predict continuous responses—

for example, changes in temperature or fluctuations in
power demand. Typical applications include electricity
load forecasting and algorithmic trading.

Introducing Machine Learning 5

Unsupervised Learning

Unsupervised learning finds hidden patterns or intrinsic structures in

data. It is used to draw inferences from datasets consisting of input
data without labeled responses.

Clustering is the most common unsupervised learning technique.

It is used for exploratory data analysis to find hidden patterns or
Clustering
groupings in data.
Patterns in
Applications for clustering include gene sequence analysis, market the Data
research, and object recognition.

Clustering
Patterns in
the Data
Clustering Patterns
in the Data

Clustering
Patterns in
the Data

Introducing Machine Learning 6

How Do You Decide Which Algorithm to Use?

Choosing the right algorithm can seem overwhelming—there MACHINE LEARNING

are dozens of supervised and unsupervised machine learning

algorithms, and each takes a different approach to learning.

There is no best method or one size fits all. Finding the right
algorithm is partly just trial and error—even highly experienced SUPERVISED UNSUPERVISED
data scientists can’t tell whether an algorithm will work without LEARNING LEARNING
trying it out. But algorithm selection also depends on the size and
type of data you’re working with, the insights you want to get from
the data, and how those insights will be used.

CLASSIFICATION REGRESSION CLUSTERING

Support Vector Linear Regression, K-Means, K-Medoids

Machines GLM Fuzzy C-Means

Discriminant
SVR, GPR Hierarchical
Analysis

Naive Bayes Ensemble Methods Gaussian Mixture

Nearest Neighbor Decision Trees Neural Networks

Hidden Markov
Neural Networks
Model

Introducing Machine Learning 7

When Should You Use Machine Learning?

Consider using machine learning when you have a complex task or

problem involving a large amount of data and lots of variables, but
no existing formula or equation. For example, machine learning is
a good option if you need to handle situations like these:

• Hand-written rules and equations are too complex—as in face

recognition and speech recognition.

• The rules of a task are constantly changing—as in fraud

detection from transaction records.

• The nature of the data keeps changing, and the program needs
to adapt—as in automated trading, energy demand forecasting,
and predicting shopping trends.

Introducing Machine Learning 8

Real World Examples

Creating Algorithms that Can Analyze Works of Art

Researchers at the Art and Artificial Intelligence Laboratory at

Rutgers University wanted to see whether a computer algorithm
could classify paintings by style, genre, and artist as easily as a
human. They began by identifying visual features for classifying a
painting’s style. The algorithms they developed classified the styles
of paintings in the database with 60% accuracy, outperforming
typical non-expert humans.

The researchers hypothesized that visual features useful for style

classification (a supervised learning problem) could also be used to
determine artistic influences (an unsupervised problem).

They used classification algorithms trained on Google images to

identify specific objects. They tested the algorithms on more than
1,700 paintings from 66 different artists working over a span
of 550 years. The algorithm readily identified connected works,
including the influence of Diego Velazquez’s “Portrait of Pope
Innocent X” on Francis Bacon’s “Study After Velazquez’s Portrait of
Pope Innocent X.”

Introducing Machine Learning 9

Real World Examples

Optimizing HVAC Energy Usage in Large Buildings

The heating, ventilation, and air-conditioning (HVAC) systems

in office buildings, hospitals, and other large-scale commercial
buildings are often inefficient because they do not take into
account changing weather patterns, variable energy costs, or the
building’s thermal properties.

Building IQ’s cloud-based software platform addresses this

problem. The platform uses advanced algorithms and machine
learning methods to continuously process gigabytes of information
from power meters, thermometers, and HVAC pressure sensors, as
well as weather and energy cost. In particular, machine learning
is used to segment data and determine the relative contributions
of gas, electric, steam, and solar power to heating and cooling
processes. The building IQ platform reduces HVAC energy
consumption in large-scale commercial buildings by 10% - 25%
during normal operation.

Introducing Machine Learning 10

Real World Examples

Detecting Low-Speed Car Crashes

With more than 8 million members, the RAC is one of the UK’s
largest motoring organizations, providing roadside assistance,
insurance, and other services to private and business motorists.

To enable rapid response to roadside incidents, reduce crashes,

and mitigate insurance costs, the RAC developed an onboard
crash sensing system that uses advanced machine learning
algorithms to detect low-speed collisions and distinguish these
events from more common driving events, such as driving over
speed bumps or potholes. Independent tests showed the RAC
system to be 92% accurate in detecting test crashes.

Introducing Machine Learning 11

 Learn More

Ready for a deeper dive? Explore these resources to learn more about
machine learning methods, examples, and tools.
Watch
Machine Learning Made Easy 34:34
Signal Processing and Machine Learning Techniques for Sensor Data Analytics 42:45

Read
Machine Learning Blog Posts: Social Network Analysis, Text Mining, Bayesian Reasoning, and more
The Netflix Prize and Production Machine Learning Systems: An Insider Look
Machine Learning Challenges: Choosing the Best Model and Avoiding Overfitting

Explore
MATLAB Machine Learning Examples
Machine Learning Solutions
Classify Data with the Classification Learner App

© 2016 The MathWorks, Inc. MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See mathworks.com/trademarks for a list of additional trademarks.
Other product or brand names may be trademarks or registered trademarks of their respective holders.
92991v00
Getting Started with
Machine Learning
Rarely a Straight Line

With machine learning there’s rarely a straight line from start to

finish—you’ll find yourself constantly iterating and trying different ideas
and approaches. This chapter describes a systematic machine learning
workflow, highlighting some key decision points along the way.
Machine Learning Challenges

Most machine learning challenges relate to handling your data and

finding the right model.

Data comes in all shapes and sizes. Real-world datasets can be

messy, incomplete, and in a variety of formats. You might just
have simple numeric data. But sometimes you’re combining several
different data types, such as sensor signals, text, and streaming
images from a camera.

Preprocessing your data might require specialized knowledge and

tools. For example, to select features to train an object detection
algorithm requires specialized knowledge of image processing.
Different types of data require different approaches to preprocessing.

It takes time to find the best model to fit the data. Choosing the
right model is a balancing act. Highly flexible models tend to overfit
data by modeling minor variations that could be noise. On the
other hand, simple models may assume too much. There are always
tradeoffs between model speed, accuracy, and complexity.

Sounds daunting? Don’t be discouraged. Remember that trial

and error is at the core of machine learning—if one approach or
algorithm doesn’t work, you simply try another. But a systematic
workflow will help you get off to a smooth start.

Getting Started with Machine Learning 3

Questions to Consider Before You Start

Every machine learning workflow begins with three questions:

• What kind of data are you working with?

• What insights do you want to get from it? MACHINE LEARNING
• How and where will those insights be applied?
Your answers to these questions help you decide whether to use
supervised or unsupervised learning.
SUPERVISED UNSUPERVISED
LEARNING LEARNING

Choose supervised learning if you need to train a model

to make a prediction--for example, the future value of
a continuous variable, such as temperature or a stock
price, or a classification—for example, identify makes of REGRESSION CLASSIFICATION
cars from webcam video footage.

Choose unsupervised learning if you

need to explore your data and want to
train a model to find a good internal
representation, such as splitting data up
into clusters.

Getting Started with Machine Learning 4

Workflow at a Glance

1. ACCESS and load the data. 4. TRAIN models using the

features derived in step 3.

2. PREPROCESS the data. 5. ITERATE to find the best model.

3. DERIVE features using 6. INTEGRATE the best-trained

the preprocessed data. model into a production system.

In the next sections we’ll look at the steps in more detail, using a
health monitoring app for illustration. The entire workflow will be
completed in MATLAB®.

Getting Started with Machine Learning 5

Training a Model to Classify Physical Activities

This example is based on a cell phone health-monitoring app.

The input consists of three-axial sensor data from the phone’s
accelerometer and gyroscope. The responses, (or output), are the
activities performed–walking, standing, running, climbing stairs,
or lying down.

We want to use the input data to train a classification model to

identify these activities. Since our goal is classification, we’ll be
applying supervised learning.

The trained model (or classifier) will be integrated into an app to MACHINE LEARNING
help users track their activity levels throughout the day.

Getting Started with Machine Learning 6

1 Step One: Load the Data

To load data from the accelerometer and gyroscope we do

the following:

1. Sit down holding the phone, log data from the phone,
and store it in a text file labeled “Sitting.”

2. Stand up holding the phone, log data from the phone,

and store it in a second text file labeled “Standing.”

3. Repeat the steps until we have data for each activity we

want to classify.

We store the labeled data sets in a text file. A flat file format such
as text or CSV is easy to work with and makes it straightforward to
import data.

Machine learning algorithms aren’t smart enough to tell the

difference between noise and valuable information.
Before using the data for training, we need to make sure it’s
clean and complete.

Getting Started with Machine Learning 7

2 Step Two: Preprocess the Data

We import the data into MATLAB and plot each labeled set. raw data outliers
To preprocess the data we do the following:

1. Look for outliers–data points that lie outside the

rest of the data.

We must decide whether the outliers can be ignored or whether

they indicate a phenomenon that the model should account for.
In our example, they can safely be ignored (it turns out that we
moved unintentionally while recording the data).

2. Check for missing values (perhaps we lost data

because the connection dropped during recording).

We could simply ignore the missing values, but this will reduce
the size of the data set. Alternatively, we could substitute
approximations for the missing values by interpolating or using Outliers in the activity-tracking data.

comparable data from another sample.

In many applications, outliers provide crucial information.

For example, in a credit card fraud detection app, they
indicate purchases that fall outside a customer’s usual
buying patterns.

Getting Started with Machine Learning 8

 2 Step Two: Preprocess the Data continued

3. Remove gravitational effects from the accelerometer

data so that our algorithm will focus on the movement of
the subject, not the movement of the phone. A simple high-
pass filter such as a biquad filter is commonly used for this.

4. Divide the data into two sets. We save part of the data for
testing (the test set) and use the rest (the training set) to build
models. This is referred to as holdout, and is a useful cross-
validation technique.

By testing your model against data that wasn’t used

in the modeling process, you see how it will perform
with unknown data.

Getting Started with Machine Learning 9

 3 Step Three: Derive Features

Deriving features (also known as feature engineering or feature

extraction) is one of the most important parts of machine learning.
It turns raw data into information that a machine learning algorithm
can use.

For the activity tracker, we want to extract features that capture the
frequency content of the accelerometer data. These features will
help the algorithm distinguish between walking (low frequency)
and running (high frequency). We create a new table that includes
the selected features.

Use feature selection to:

• Improve the accuracy of a machine learning algorithm
• Boost model performance for high-dimensional data sets
• Improve model interpretability
• Prevent overfitting

Getting Started with Machine Learning 10

3 Step Three: Derive Features continued

The number of features that you could derive is limited only by your imagination. However, there are a lot of techniques
commonly used for different types of data.

Data Type Feature Selection Task Techniques

Sensor data Extract signal properties from raw sensor Peak analysis – perform an fft and identify dominant frequencies
data to create higher-level information
Pulse and transition metrics – derive signal characteristics such as rise
time, fall time, and settling time

Spectral measurements – plot signal power, bandwidth, mean frequency,

and median frequency
Image and Extract features such as edge locations, Bag of visual words – create a histogram of local image features, such as
video data resolution, and color edges, corners, and blobs

Histogram of oriented gradients (HOG) – create a histogram of local

gradient directions

Minimum eigenvalue algorithm – detect corner locations in images

Edge detection – identify points where the degree of brightness changes

sharply
Transactional data Calculate derived values that enhance Timestamp decomposition – break timestamps down into components such
the information in the data as day and month

Aggregate value calculation – create higher-level features such as the total

number of times a particular event occurred

Getting Started with Machine Learning 11

 4 Step Four: Build and Train the Model

When building a model, it’s a good idea to start with something To see how well it performs, we plot the confusion matrix, a table
simple; it will be faster to run and easier to interpret. that compares the classifications made by the model with the
actual class labels that we created in step 1.
We start with a basic decision tree.

feat53<335.449 feat53>=335.449
Sitting >99% <1%

Standing <1% 99% <1%

feat3<2.50002 feat3>=2.50002 feat56<12686 feat56>=12686

Walking <1% >99% <1%

TRUE
CLASS
feat11>=0.45 Running 1% 93% 5%
feat11<0.45
Standing Sitting Walking

Dancing <1% <1% 40% 59%

St

D
Running Dancing

Ru
Si

an
an

al
tti

nn

ci
ki
di
ng

in

ng
ng
ng

g
PREDICTED CLASS

The confusion matrix shows that our model is having trouble

distinguishing between dancing and running. Maybe a decision
tree doesn’t work for this type of data. We’ll try a few
different algorithms.

Getting Started with Machine Learning 12

4 Step Four: Build and Train the Model continued

We start with a K-nearest neighbors (KNN), a simple algorithm However, KNNs take a considerable amount of memory to run,
that stores all the training data, compares new points to the since they require all the training data to make a prediction.
training data, and returns the most frequent class of the “K”
We try a linear discriminant model, but that doesn’t improve the
nearest points. That gives us 98% accuracy compared to 94.1%
results. Finally, we try a multiclass support vector machine (SVM).
for the simple decision tree. The confusion matrix looks
The SVM does very well—we now get 99% accuracy:
better, too:

Sitting >99% <1% Sitting >99% <1%

Standing 1% 99% 1% Standing <1% >99% <1%

Walking 2% 98% Walking <1% >99%

TRUE TRUE
CLASS CLASS
Running <1% 1% 97% 1% Running <1% 98% 2%

Dancing 1% 1% 6% 92% Dancing <1% <1% 3% 96%

St

St
W

W
D

D
Ru

Ru
Si

Si
an

an
an

an
al

al
tti

tti
nn

nn
ci

ci
ki

ki
di

di
ng

ng
in

in
ng

ng
ng

ng
ng

ng
g

g
PREDICTED CLASS PREDICTED CLASS

We achieved our goal by iterating on the model and trying

different algorithms. If our classifier still couldn’t reliably
differentiate between dancing and running, we’d look into ways to
improve the model.

Getting Started with Machine Learning 13

5 Step Five: Improve the Model

Improving a model can take two different directions: make the A good model includes only the features with the most
model simpler or add complexity. predictive power. A simple model that generalizes well is
better than a complex model that may not generalize or
Simplify train well to new data.

First, we look for opportunities to reduce the number of features. In machine learning, as in many other
computational processes, simplifying the
Popular feature reduction techniques include:
model makes it easier to understand,
• Correlation matrix – shows the relationship between more robust, and more computationally
efficient.
variables, so that variables (or features) that are not highly
correlated can be removed.
• Principal component analysis (PCA) – eliminates
redundancy by finding a combination of features that
captures key distinctions between the original features and
brings out strong patterns in the dataset.
• Sequential feature reduction – reduces features
iteratively on the model until there is no improvement
in performance.
Next, we look at ways to reduce the model itself. We can
do this by:

• Pruning branches from a decision tree

• Removing learners from an ensemble

Getting Started with Machine Learning 14

5 Step Five: Improve the Model continued

Add Complexity

If our model can’t differentiate dancing from running because it is

over-generalizing, then we need find ways to make it more
fine-tuned. To do this we can either:

• Use model combination – merge multiple simpler models into

a larger model that is better able to represent the trends in
the data than any of the simpler models could on their own.
• Add more data sources – look at the gyroscope data as
well as the acceleromter data. The gyroscope records the
orientation of the cell phone during activity. This data might
provide unique signatures for the different activities; for
example, there might be a combination of acceleration and
rotation that’s unique to running.
Once we’ve adjusted the model, we validate its performance on
the test data that we set aside during preprocessing.

If the model can reliably classify activities on the test data, we’re
ready to move it to the phone and start tracking.

Getting Started with Machine Learning 15

 Learn More

Ready for a deeper dive? Explore these resources to learn more about
machine learning methods, examples, and tools.

Watch
Machine Learning Made Easy 34:34
Signal Processing and Machine Learning Techniques for Sensor Data Analytics 42:45

Read
Supervised Learning Workflow and Algorithms
Data-Driven Insights with MATLAB Analytics: An Energy Load Forecasting Case Study

Explore
MATLAB Machine Learning Examples
Classify Data with the Classification Learner App

© 2016 The MathWorks, Inc. MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See mathworks.com/trademarks for a list of additional trademarks.
Other product or brand names may be trademarks or registered trademarks of their respective holders.
93014v00
Applying
Unsupervised Learning
When to Consider
Unsupervised Learning

Unsupervised learning is useful when you want to explore your data but
don’t yet have a specific goal or are not sure what information the data
contains. It’s also a good way to reduce the dimensions of your data.
Unsupervised Learning Techniques

As we saw in section 1, most unsupervised learning techniques are

a form of cluster analysis.

In cluster analysis, data is partitioned into groups based on some

measure of similarity or shared characteristic. Clusters are formed
so that objects in the same cluster are very similar and objects in
different clusters are very distinct.

Clustering algorithms fall into two broad groups:

• Hard clustering, where each data point belongs to only

one cluster
• Soft clustering, where each data point can belong to more
than one cluster Gaussian mixture model used to separate data into two clusters.
You can use hard or soft clustering techniques if you already know
the possible data groupings.
If you don’t yet know how the data might be grouped:

• Use self-organizing feature maps or hierarchical

clustering to look for possible structures in the data.

• Use cluster evaluation to look for the “best” number

of groups for a given clustering algorithm.

Applying Unsupervised Learning 3

Common Hard Clustering Algorithms

k-Means k-Medoids
How it Works How It Works
Partitions data into k number of mutually exclusive clusters. Similar to k-means, but with the requirement that the cluster
How well a point fits into a cluster is determined by the centers coincide with points in the data.
distance from that point to the cluster’s center.
Best Used...
Best Used...
• When the number of clusters is known
• When the number of clusters is known • For fast clustering of categorical data
• For fast clustering of large data sets • To scale to large data sets

Result: Cluster centers Result: Cluster centers that

coincide with data points

Applying Unsupervised Learning 4

Common Hard Clustering Algorithms continued

Hierarchical Clustering Self-Organizing Map

How it Works How It Works
Produces nested sets of clusters by analyzing similarities Neural-network based clustering that transforms a dataset
between pairs of points and grouping objects into a binary, into a topology-preserving 2D map.
hierarchical tree.
Best Used...
Best Used...
• To visualize high-dimensional data in 2D or 3D
• When you don’t know in advance how many clusters • To deduce the dimensionality of data by preserving its
are in your data topology (shape)
• You want visualization to guide
your selection

Result:
Result: Dendrogram showing Lower-dimensional
the hierarchical relationship (typically 2D)
between clusters representation

Applying Unsupervised Learning 5

Common Hard Clustering Algorithms continued

Example: Using k-Means Clustering to Site Cell Phone Towers

A cell phone company wants to know the number and placement

of cell phone towers that will provide the most reliable service. For
optimal signal reception, the towers must be located within
clusters of people.

The workflow begins with an initial guess at the number of clusters

that will be needed. To evaluate this guess, the engineers compare
service with three towers and four towers to see how well they’re
able to cluster for each scenario (in other words, how well the
towers provide service).

A phone can only talk to one tower at a time, so this is a hard

clustering problem. The team uses k-means clustering because
k-means treats each observation in the data as an object having
a location in space. It finds a partition in which objects within
each cluster are as close to each other as possible and as far from
objects in other clusters as possible.

After running the algorithm, the team can accurately determine the
results of partitioning the data into three and four clusters.

Applying Unsupervised Learning 6

Common Soft Clustering Algorithms

Fuzzy c-Means Gaussian Mixture Model

How it Works How It Works
Partition-based clustering when data points may belong to Partition-based clustering where data points come from
more than one cluster. different multivariate normal distributions with certain
probabilities.
Best Used...
Best Used...
• When the number of clusters is known
• For pattern recognition • When a data point might belong to more than
one cluster
• When clusters overlap
• When clusters have different sizes and correlation
structures within them

Result: Cluster centers Result: A

 model of
(similar to k-means) but Gaussian distributions
with fuzziness so that that give probabilities of
points may belong to a point being in a cluster
more than one cluster

Applying Unsupervised Learning 7

Common Soft Clustering Algorithms continued

Example: Using Fuzzy c-Means Clustering to Analyze

Gene Expression Data

A team of biologists is analyzing gene expression data from

microarrays to better understand the genes involved in normal and
abnormal cell division. (A gene is said to be “expressed” if it is
actively involved in a cellular function such as protein production.)

The microarray contains expression data from two tissue samples.

The researchers want to compare the samples to determine whether
certain patterns of gene expression are implicated in
cancer proliferation.

After preprocessing the data to remove noise, they cluster the data.
Because the same genes can be involved in several biological
processes, no single gene is likely to belong to one cluster only.
The researchers apply a fuzzy c-means algorithm to the data. They
then visualize the clusters to identify groups of genes that behave in
a similar way.

Applying Unsupervised Learning 8

Improving Models with Dimensionality Reduction

Machine learning is an effective method for finding patterns in As datasets get bigger, you frequently need to reduce the
big datasets. But bigger data brings added complexity. number of features, or dimensionality.

Example: EEG Data Reduction

Suppose you have electroencephalogram (EEG) data that captures

electrical activity of the brain, and you want to use this data to
predict a future seizure. The data was captured using dozens of
leads, each corresponding to a variable in your original dataset.
Each of these variables contains noise. To make your prediction
algorithm more robust, you use dimensionality reduction techniques
to derive a smaller number of features. Because these features are
calculated from multiple sensors, they will be less susceptible to
noise in an individual sensor than would be the case if you used
the raw data directly.

Applying Unsupervised Learning 9

Common Dimensionality Reduction Techniques

The three most commonly used dimensionality reduction

techniques are:

Principal component analysis (PCA)—performs a linear

transformation on the data so that most of the variance or
information in your high-dimensional dataset is captured by the
first few principal components. The first principal component
will capture the most variance, followed by the second principal
component, and so on.

Factor analysis—identifies underlying correlations between

variables in your dataset to provide a representation in terms of a
smaller number of unobserved latent, or common, factors.

Nonnegative matrix factorization—used when model terms must

represent nonnegative quantities, such as physical quantities.

Applying Unsupervised Learning 10

Using Principal Component Analysis

In datasets with many variables, groups of variables often move Each principal component is a linear combination of the original
together. PCA takes advantage of this redundancy of information variables. Because all the principal components are orthogonal to
by generating new variables via linear combinations of the original each other, there is no redundant information.
variables so that a small number of new variables captures most of
the information.

Example: Engine Health Monitoring

You have a dataset that includes measurements for different

sensors on an engine (temperatures, pressures, emissions, and so
on). While much of the data comes from a healthy engine, the
sensors have also captured data from the engine when it needs
maintenance.

You cannot see any obvious abnormalities by looking at any

individual sensor. However, by applying PCA, you can transform
this data so that most variations in the sensor measurements
are captured by a small number of principal components. It is
easier to distinguish between a healthy and unhealthy engine by
inspecting these principal components than by looking at the raw
sensor data.

Applying Unsupervised Learning 11

Using Factor Analysis

Your dataset might contain measured variables that overlap, In a factor analysis model, the measured variables depend on
meaning that they are dependent on one another. Factor a smaller number of unobserved (latent) factors. Because each
analysis lets you fit a model to multivariate data to estimate factor might affect several variables, it is known as a common
this sort of interdependence. factor. Each variable is assumed to be dependent on a linear
combination of the common factors.

Example: Tracking Stock Price Variation

Over the course of 100 weeks, the percent change in stock prices
has been recorded for ten companies. Of these ten, four are
technology companies, three are financial, and a further three
are retail. It seems reasonable to assume that the stock prices
for companies in the same sector will vary together as economic
conditions change. Factor analysis can provide quantitative
evidence to support this premise.

Applying Unsupervised Learning 12

Using Nonnegative Matrix Factorization

This dimension reduction technique is based on a low-rank nonnegative, producing models that respect features such as
approximation of the feature space. In addition to reducing the nonnegativity of physical quantities.
the number of features, it guarantees that the features are

Example: Text Mining

Suppose you want to explore variations in vocabulary and style

among several web pages. You create a matrix where each
row corresponds to an individual web page and each column
corresponds to a word (“the”,”a”,”we”, and so on). The data will be
the number of times a particular word occurs on a particular page.

Since there more than a million words in the English language,

you apply nonnegative matrix factorization to create an arbitrary
number of features that represent higher-level concepts rather than
individual words. These concepts make it easier to distinguish
between, say, news, educational content, and online retail content.

Applying Unsupervised Learning 13

Next Steps

In this section we took a closer look at hard and soft clustering

LOTS OF DATA
algorithms for unsupervised learning, offered some tips on selecting
the right algorithm for your data, and showed how reducing the
number of features in your dataset improves model performance. UNSUPERVISED
As for your next steps: LEARNING

• Unsupervised learning might be your end goal. For example,

if you are doing market research and want to segment
consumer groups to target based on web site behavior, a
DATA CLUSTERS LOWER-DIMENSIONAL
clustering algorithm will almost certainly give you the results
DATA
you’re looking for.
• On the other hand, you might want to use unsupervised RESULTS
learning as a preprocessing step for supervised learning. FEATURE
For example, apply clustering techniques to derive a smaller SELECTION
number of features, and then use those features as inputs for
training a classifier.
SUPERVISED
In section 4 we’ll explore supervised learning algorithms and LEARNING
techniques, and see how to improve models with feature selection,
feature reduction, and parameter tuning.
MODEL

Applying Unsupervised Learning 14

 Learn More

Ready for a deeper dive? Explore these unsupervised learning resources.

Clustering Algorithms Fuzzy C-Means Dimensionality

and Techniques Cluster Quasi-Random Data Using Reduction
Fuzzy C-Means Clustering Analyze Quality of Life in U.S. Cities
k-Means Using PCA
Gaussian Mixture Models
Use K-Means and Hierarchical Analyze Stock Prices Using Factor
Clustering to Find Natural Patterns Gaussian Process Regression Models Analysis
in Data Cluster Data from Mixture of Gaussian
Distributions Nonnegative Factorization
Cluster Genes Using K-Means and
Self-Organizing Maps Cluster Gaussian Mixture Data Using Perform Nonnegative Matrix
Soft Clustering Factorization
Color-Based Segmentation Using
K-Means Clustering Tune Gaussian Mixture Models Model Suburban Commuting Using
Subtractive Clustering
Hierarchical Clustering Image Processing Example: Detecting
Cars with Gaussian Mixture Models
Connectivity-Based Clustering

Iris Clustering

Self-Organizing Maps
Cluster Data with a
Self-Organizing Map

© 2016 The MathWorks, Inc. MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See mathworks.com/trademarks for a list of additional trademarks.
Other product or brand names may be trademarks or registered trademarks of their respective holders.
80823v00
Applying
Supervised Learning
When to Consider
Supervised Learning

A supervised learning algorithm takes a known set of input data (the

training set) and known responses to the data (output), and trains a
model to generate reasonable predictions for the response
to new input data. Use supervised learning if you have existing data for
the output you are trying to predict.
Supervised Learning Techniques

All supervised learning techniques are a form of classification or

regression.

Classification techniques predict discrete responses—for example,

whether an email is genuine or spam, or whether a tumor is small,
medium, or large. Classification models are trained to classify data
into categories. Applications include medical imaging, speech
recognition, and credit scoring.

Regression techniques predict continuous responses—for example,

changes in temperature or fluctuations in electricity demand.
Applications include forecasting stock prices, handwriting
recognition, and acoustic signal processing.

• Can your data be tagged or categorized? If your

data can be separated into specific groups or
classes, use classification algorithms.

• Working with a data range? If the nature of your

response is a real number –such as temperature, or
the time until failure for a piece of equipment--use
regression techniques.

Applying Supervised Learning 3

Selecting the Right Algorithm

As we saw in section 1, selecting a machine learning algorithm is

a process of trial and error. It’s also a trade-off between specific
characteristics of the algorithms, such as:

• Speed of training
• Memory usage
• Predictive accuracy on new data
• Transparency or interpretability (how easily you can
understand the reasons an algorithm makes its predictions)
Let’s take a closer look at the most commonly used classification
and regression algorithms.

Using larger training datasets often yield models that

generalize well for new data.

Applying Supervised Learning 4

Binary vs. Multiclass Classification

When you are working on a classification problem, begin by

determining whether the problem is binary or multiclass. In
a binary classification problem, a single training or test item
(instance) can only be divided into two classes—for example, if
you want to determine whether an email is genuine or spam. In
a multiclass classification problem, it can be divided into more
than two—for example, if you want to train a model to classify an
image as a dog, cat, or other animal.

Bear in mind that a multiclass classification problem is generally

more challenging because it requires a more complex model.

Certain algorithms (for example, logistic regression) are

designed specifically for binary classification problems.
During training, these algorithms tend to be more
efficient than multiclass algorithms.

Applying Supervised Learning 5

Common Classification Algorithms

Logistic Regression k Nearest Neighbor (kNN)

How it Works How it Works
Fits a model that can predict the probability of a binary kNN categorizes objects based on the classes of their
response belonging to one class or the other. Because of its nearest neighbors in the dataset. kNN predictions assume
simplicity, logistic regression is commonly used as a starting that objects near each other are similar. Distance metrics,
point for binary classification problems. such as Euclidean, city block, cosine, and Chebychev, are
used to find the nearest neighbor.
Best Used...
Best Used...
• When data can be clearly separated by a single,
linear boundary • When you need a simple algorithm to establish
• As a baseline for evaluating more complex benchmark learning rules
classification methods • When memory usage of the trained model is a lesser
concern
• When prediction speed of the trained model is a lesser
concern

Applying Supervised Learning 6

Common Classification Algorithms continued

Support Vector Machine (SVM)

How It Works
Classifies data by finding the linear decision boundary
(hyperplane) that separates all data points of one class from
those of the other class. The best hyperplane for an SVM is the
one with the largest margin between the two classes, when the
data is linearly separable. If the data is not linearly separable, a
loss function is used to penalize points on the wrong side of the
hyperplane. SVMs sometimes use a kernel transform to transform
nonlinearly separable data into higher dimensions where a linear
decision boundary can be found.

Best Used...

• For data that has exactly two classes (you can also use it
for multiclass classification with a technique called error-
correcting output codes)
• For high-dimensional, nonlinearly separable data
• When you need a classifier that’s simple, easy to interpret,
and accurate

Applying Supervised Learning 7

Common Classification Algorithms continued

Neural Network Naïve Bayes

How it Works How It Works
Inspired by the human brain, a neural network consists of A naive Bayes classifier assumes that the presence of a
highly connected networks of neurons that relate the inputs particular feature in a class is unrelated to the presence of
to the desired outputs. The network is trained by iteratively any other feature. It classifies new data based on the highest
modifying the strengths of the connections so that given probability of its belonging to a particular class.
inputs map to the correct response.
Best Used...
Best Used...
• For a small dataset containing many parameters
• For modeling highly nonlinear systems • When you need a classifier that’s easy to interpret
• When data is available incrementally and you wish to • When the model will encounter scenarios that weren’t
constantly update the model in the training data, as is the case with many financial
• When there could be unexpected changes in your and medical applications
input data
• When model interpretability is not a key concern

Applying Supervised Learning 8

Common Classification Algorithms continued

Discriminant Analysis
How It Works
Discriminant analysis classifies data by finding linear combinations
of features. Discriminant analysis assumes that different classes
generate data based on Gaussian distributions. Training a
discriminant analysis model involves finding the parameters for a
Gaussian distribution for each class. The distribution parameters
are used to calculate boundaries, which can be linear or
quadratic functions. These boundaries are used to determine the
class of new data.

Best Used...

• When you need a simple model that is easy to interpret

• When memory usage during training is a concern
• When you need a model that is fast to predict

Applying Supervised Learning 9

Common Classification Algorithms continued

Decision Tree Bagged and Boosted Decision Trees

How it Works How They Work
A decision tree lets you predict responses to data by following In these ensemble methods, several “weaker” decision trees are
the decisions in the tree from the root (beginning) down to a combined into a “stronger” ensemble.
leaf node. A tree consists of branching conditions where the
A bagged decision tree consists of trees that are trained
value of a predictor is compared to a trained weight. The
independently on data that is bootstrapped from the input data.
number of branches and the values of weights are determined
in the training process. Additional modification, or pruning, Boosting involves creating a strong learner by iteratively adding
may be used to simplify the model. “weak” learners and adjusting the weight of each weak learner
to focus on misclassified examples.
Best Used...
Best Used...
• When you need an algorithm that is easy to interpret and
fast to fit • When predictors are categorical (discrete) or behave
• To minimize memory usage nonlinearly
• When high predictive accuracy is not a requirement • When the time taken to train a model is less of a concern

Applying Supervised Learning 10

Common Classification Algorithms continued

Example: Predictive Maintenance for Manufacturing Equipment

A plastic production plant delivers about 18 million tons of plastic

and thin film products annually. The plant’s 900 workers operate
24 hours a day, 365 days a year.

To minimize machine failures and maximize plant efficiency,

engineers develop a health monitoring and predictive maintenance
application that uses advanced statistics and machine learning
algorithms to identify potential issues with the machines so that
operators can take corrective action and prevent serious problems
from occurring.

After collecting, cleaning, and logging data from all the machines
in the plant, the engineers evaluate several machine learning
techniques, including neural networks, k-nearest neighbors,
bagged decision trees, and support vector machines (SVMs). For
each technique, they train a classification model using the logged
machine data and then test the model’s ability to predict machine
problems. The tests show that an ensemble of bagged decision
trees is the most accurate model for predicting the production
quality.

Applying Unsupervised
Supervised Learning
Learning 11
Common Regression Algorithms

Linear Regression Nonlinear Regression

How it Works How It Works
Linear regression is a statistical modeling technique used to Nonlinear regression is a statistical modeling technique that
describe a continuous response variable as a linear function helps describe nonlinear relationships in experimental data.
of one or more predictor variables. Because linear regression Nonlinear regression models are generally assumed to be
models are simple to interpret and easy to train, they are parametric, where the model is described as a nonlinear
often the first model to be fitted to a new dataset. equation.

Best Used... “Nonlinear” refers to a fit function that is a nonlinear function

of the parameters. For example, if the fitting parameters
• When you need an algorithm that is easy to interpret
are b0, b1, and b2: the equation y = b0+b1x+b2x2 is a linear
and fast to fit
function of the fitting parameters, whereas y = (b0xb1)/(x+b2)
• As a baseline for evaluating other, more complex,
is a nonlinear function of the fitting parameters.
regression models
Best Used...

• When data has strong

nonlinear trends and
cannot be easily
transformed into a
linear space
• For fitting custom models
to data

Applying Supervised Learning 12

Common Regression Algorithms continued

Gaussian Process Regression Model SVM Regression

How it Works How It Works
Gaussian process regression (GPR) models are SVM regression algorithms work like SVM classification
nonparametric models that are used for predicting the value algorithms, but are modified to be able to predict a
of a continuous response variable. They are widely used in continuous response. Instead of finding a hyperplane that
the field of spatial analysis for interpolation in the presence separates data, SVM regression algorithms find a model that
of uncertainty. GPR is also referred to as Kriging. deviates from the measured data by a value no greater than
a small amount, with parameter values that are as small as
Best Used...
possible (to minimize sensitivity to error).
• For interpolating spatial data, such as hydrogeological
Best Used...
data for the distribution of ground water
• As a surrogate model to facilitate optimization of • For high-dimensional data (where there will be a large
complex designs such as automotive engines number of predictor variables)

Applying Supervised Learning 13

Common Regression Algorithms continued

Generalized Linear Model Regression Tree

How it Works How It Works
A generalized linear model is a special case of nonlinear Decision trees for regression are similar to decision trees for
models that uses linear methods. It involves fitting a linear classification, but they are modified to be able to predict
combination of the inputs to a nonlinear function (the link continuous responses.
function) of the outputs.
Best Used...
Best Used...
• When predictors are categorical (discrete) or behave
• When the response variables have nonnormal nonlinearly
distributions, such as a response variable that is
always expected to be positive

Applying Supervised Learning 14

Common Regression Algorithms continued

Example: Forecasting Energy Load

Utility analysts at a large gas and electricity company developed

models that predict energy demand for the following day. The
models enable power grid operators to optimize resources and
schedule power plant generation. Each model accesses a central
database for historical power consumption and price data,
weather forecasts, and parameters for each power plant, including
maximum power out, efficiency, costs, and all the operation
constraints that influence the plant dispatch.

Analysts looked for a model that provided a low mean absolute

percent error (MAPE) to the testing data set. After trying several
different types of regression models, it was determined that neural
networks provided the lowest MAPE due to their ability to capture
the nonlinear behavior of the system.

Applying Unsupervised
Supervised Learning
Learning 15
Improving Models

Improving a model means increasing its accuracy and predictive

power and preventing overfitting (when the model cannot
distinguish between data and noise). Model improvement involves
feature engineering (feature selection and transformation) and FEATURE SELECTION
hyperparameter tuning.

Feature selection: Identifying the most relevant features, or

variables, that provide the best predictive power in modeling your FEATURE
data. This could mean adding variables to the model or removing TRANSFORMATION
variables that do not improve model performance.

Feature transformation: Turning existing features into new

features using techniques such as principal component analysis,
nonnegative matrix factorization, and factor analysis. TRAIN MODEL HYPERPARAMETER
TUNING
Hyperparameter tuning: The process of identifying the set of
parameters that provides the best model. Hyperparameters control
how a machine learning algorithm fits the model to the data.

Applying Supervised Learning 16

Feature Selection

Feature selection is one of the most important tasks in machine

learning. It’s especially useful when you’re dealing with high-
dimensional data or when your dataset contains a large number of
features and a limited number of observations. Reducing features
also saves storage and computation time and makes your results
easier to understand.

Common feature selection techniques include:

Stepwise regression: Sequentially adding or removing features until

there is no improvement in prediction accuracy.

Sequential feature selection: Iteratively adding or removing

predictor variables and evaluating the effect of each change on
the performance of the model.

Regularization: Using shrinkage estimators to remove redundant

features by reducing their weights (coefficients) to zero.

Neighborhood component analysis (NCA): Finding the weight each

feature has in predicting the output, so that features with lower
weights can be discarded. A model is only as good as the features you
select to train it.

Applying Supervised Learning 17

Feature Transformation

Feature transformation is a form of dimensionality reduction. As we saw in section 3, the three most commonly used dimensionality
reduction techniques are:

Principal component analysis (PCA): Performs a linear

transformation on the data so that most of the variance or
information in your high-dimensional dataset is captured by the
first few principal components. The first principal component
will capture the most variance, followed by the second principal
component, and so on.

Nonnegative matrix factorization: Used when model terms must

represent nonnegative quantities, such as physical quantities.

Factor analysis: Identifies underlying correlations between variables

in your dataset to provide a representation in terms of a smaller
number of unobserved latent factors, or common factors.

Applying Supervised Learning 18

Hyperparameter Tuning

Like many machine learning tasks, parameter tuning is an iterative

process. You begin by setting parameters based on a “best guess”
of the outcome. Your goal is to find the “best possible” values—
those that yield the best model. As you adjust parameters and
model performance begins to improve, you see which parameter
settings are effective and which still require tuning.

Three common parameter tuning methods are:

• Bayesian optimization
• Grid search
• Gradient-based optimization

A simple algorithm with well-tuned parameters often

produces a better model than an inadequately tuned
complex algorithm.

Applying Supervised Learning 19

 Learn More

Ready for a deeper dive? Explore these machine learning

methods, examples, and tools.
Getting Started with Supervised Learning Regression
Classification Linear Regression

Machine Learning with MATLAB: What are Generalized Linear Models?

Getting Started with Classification Regression Trees
Introductory Classification Examples Train a Regression Ensemble to Predict the
Bayesian Brain Teaser Fuel Economy of a Car

Explore Decision Trees Interactively Feature Selection

Support Vector Machines Selecting Features for Classifying High-Dimensional Data
K-Nearest Neighbor Classification
Train a Classification Ensemble
Predicting Tumor Class from Gene Expression
Data Using Bagged Decision Trees

© 2016 The MathWorks, Inc. MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See mathworks.com/trademarks for a list of additional trademarks.
Other product or brand names may be trademarks or registered trademarks of their respective holders.
80827v00