CS 550: Machine Learning

Lecture 7: Classification

Dr. Gagan Raj Gupta

 2
Multiple Supervised Learning Formulations …
 Consider building an ML module for an e-mail client

 Some tasks that we may want this module to perform

 Predicting whether an email of spam or normal: Binary Classification
 Predicting which of the many folders the email should be sent to: Multi-class
Classification
 Predicting all the relevant tags for an email: Tagging or Multi-label Classification
 Predicting what’s the spam-score of an email: Regression
 Predicting which email(s) should be shown at the top: Ranking
 Predicting which emails are work/study-related emails: One-class Classification ?

 These predictive modeling tasks can be formulated as supervised learning

problems
 Let’s look at some basic classification algorithms and evaluation metrics
 The idea also applies to multi- 3
Learning by Examples class classification: Use one
image per class, and predict label
based on the distances of the test
 Consider a binary classification problem – cat vs dog image from all such images

 Assume training data with just 2 images – one and one

 Given a new test image (cat/dog), how do we predict its label?

 A simple idea: Predict using its distance from each of the 2 training images

d( Test
image , ) < d( Test
image
, ) ? Predict cat else dogExcellent question! Glad you
Wait. Is it ML? Seems to be Some possibilities: Use a feature asked!
like just a simple “rule”. learning/selection algorithm to Even this simple model can be
Where is the “learning” part extract features, and use a learned. For example, for the
in this? Mahalanobis distance where you feature extraction/selection part
learn the W matrix (instead of using and/or for the distance computation
a predefined W), using “distance part
metric learning” techniques
 4
Capturing Variations by using more Training Data
 Just one input per class may not sufficiently capture variations in a class

 A natural improvement can be by using more inputs per class

“cat” “dog”
“cat” “dog”
“dog”
“cat”

 We will consider two approaches to do this

 Learning with Prototypes (LwP)
 K-Nearest Neighbors (KNN)

 Both LwP and KNN will use multiple inputs per class but in different ways
 5
Learning with Prototypes (LwP)
 Basic idea: Represent each class by a “prototype” vector

 Class Prototype: The “mean” or “average” of inputs from that class

Averages (prototypes) of each of the handwritten digits 1-9

 Predict label of each test input based on its distances from the class prototypes
 Predicted label will be the class that is the closest to the test input

 How we compute distances can have an effect on the accuracy of this model
(may need to try Euclidean, weight Euclidean, Mahalanobis, or something else)
Pic from: https://www.reddit.com/r/dataisbeautiful/comments/3wgbv9/average_handwritten_digit_oc/
 6
Learning with Prototypes (LwP): An Illustration
 Suppose the task is binary classification (two classes assumed pos and neg)

 Training data: labelled examples ,,

 Assume example from positive class, examples from negative class
 Assume green is positive and red is negative

1 𝜇
∑
1
𝜇 −= 𝐱𝑛 𝜇−
+¿=
∑
¿
𝑁 − 𝑦 =−1
𝑛 𝜇+¿¿ 𝑁 +¿
𝑦 𝑛 =+ 1
𝐱 𝑛¿

For LwP, the prototype

LwP straightforwardly generalizes
vectors (and here) define
to more than 2 classes as well
(multi-class classification) – K Test example Test example the “model”
prototypes for K classes
 7
LwP: The Prediction Rule, Mathematically
 What does the prediction rule for LwP look like mathematically?

 Assume we are using Euclidean distances here

||𝝁− − 𝐱|| =||𝝁−|| +||𝐱|| − 2 ⟨ 𝝁− , 𝐱 ⟩

2 2 2
𝜇− 𝜇+¿¿
¿ ¿

Test example

Prediction Rule: Predict label as +1 if otherwise -1

 8
LwP: The Prediction Rule, Mathematically
 Let’s expand the prediction rule expression a bit more

 Thus LwP with Euclidean distance is equivalent to a linear model with

 Weight vector 2( Will look at linear models
 Bias term more formally and in more
detail later

 Prediction rule therefore is: Predict +1 if > 0, else predict -1

 9
LwP: Some Failure Cases
 Here is a case where LwP with Euclidean distance may not work well

Can use feature scaling or use

Mahalanobis distance to handle
such cases (will discuss this in
𝜇− the next lecture)
𝜇+¿¿

Test example

 In general, if classes are not equisized and spherical, LwP with Euclidean
distance will usually not work well (but improvements possible; will discuss
later)
 10
LwP: Some Key Aspects
 Very simple, interpretable, and lightweight model
 Just requires computing and storing the class prototype vectors

 Works with any number of classes (thus for multi-class classification as well)

 Can be generalized in various ways to improve it further, e.g.,

 Modeling each class by a probability distribution rather than just a prototype vector
 Using distances other than the standard Euclidean distance (e.g., Mahalanobis)

 With a learned distance function, can work very well even with very few
examples from each class (used in some “few-shot learning” models nowadays
– if interested, please refer to “Prototypical Networks for Few-shot Learning”)
 Confusion Matrix
• Accuracy is not enough to evaluate
classification algorithms
• Class imbalance problem
• One class more important than other!
• Precision=
• Recall = {True +ve rate, Sensitivity} Discuss:
• Safe Video for Kids
• Surveillance
Precision/Recall Tradeoff

Why is this not smooth?

Precision vs Recall Curve
ROC Curve

True +ve rate = Recall = {Sensitivity}

False +ve Rate = {Specificity}

 We want to choose that classifier that maximizes the area under the curve
 Precision-Recall curve preferred when the +ve class is rare
 OR when you want to reduce the false +ves.
Which classifier is better?
Multiclass Classification (MNIST digits)

Confusion Matrix Relative Errors

 Looking at this plot, it seems that your efforts should be spent on reducing the false 8s.
 More training data for digits that look like 8s (but are not) so the classifier can learn to distinguish them from real 8s.
 Engineer new features to help classifier—E.g. Count the # of closed loops (e.g., 8 has two, 6 has one, 5 has none).
 Preprocess the images (e.g., using Scikit-Image, Pillow, or OpenCV) to make some patterns stand out more, such as
closed loops.
 17
Learning with Prototypes (LwP)
1 𝜇
∑ 𝐱
1
𝜇 −= 𝜇−
+¿=
∑
¿
𝑁 − 𝑦 =−1 𝑛
𝑛
𝐰 𝜇+¿¿ 𝑁 +¿
𝑦 𝑛 =+ 1
𝐱 𝑛¿

Prediction rule for LwP

𝐰 = 𝝁+¿ −𝝁 − ¿
(for binary classification
If Euclidean distance used
with Euclidean distance)

+ For LwP, the prototype vectors (or

Decision boundary their difference) define the “model”.
(> 0 then predict +1 otherwise -1) and (or just in the Euclidean distance
(perpendicular bisector of line
case) are the model parameters.
joining the class prototype vectors)
Exercise: Show that for the bin. classfn case
𝑁 Note: Even though can be expressed in Can throw away training data after computing the
𝑓 ( 𝐱 ) =∑ 𝛼 𝑛 ⟨ 𝐱 𝑛 , 𝐱 ⟩ +𝑏 this form, if N > D, this may be more
expensive to compute (O(N) time)as
prototypes and just need to keep the model parameters
𝑛=1 for the test time in such “parametric” models
compared to (O(D) time).

So the “score” of a test point is a weighted sum of its

similarities with each of the N training inputs. Many However the form is still very useful as we will see later
supervised learning models have in this form as we will see when we discuss kernel methods
later
 18
Improving LwP when classes are complex-shaped
 Using weighted Euclidean or Mahalanobis distance can sometimes help
𝜇+¿¿ √
𝐷
𝑑𝑤 ( 𝒂 , 𝒃 )= ∑ 𝑤𝑖 ( 𝑎𝑖 −𝑏𝑖 )2
𝜇− 𝑖 =1

Use a smaller for the horizontal

axis feature in this example

 Note: Mahalanobis distance also has the effect of rotating the axes which helps
A good W will help bring
W will be a 2x2 symmetric matrix in points from same class
this case (chosen by us or learned) closer and move different
classes apart
𝑑𝑤 ( 𝒂 , 𝒃 )= √ ( 𝒂 − 𝒃 ) 𝐖 ( 𝒂 − 𝒃 )
⊤

𝜇− 𝜇+¿¿
𝜇+¿¿ 𝜇−
 19
Improving LwP when classes are complex-shaped
 Even with weighted Euclidean or Mahalanobis dist, LwP still a linear classifier

 Exercise: Prove the above fact. You may use the following hint
 Mahalanobis dist can be written as
 is a symmetric matrix and thus can be written as for any matrix
 Showing for Mahalabonis is enough. Weighted Euclidean is a special case with diag

 LwP can be extended to learn nonlinear decision boundaries if we use

nonlinear distances/similarities(more on this when we talk
Note: Modeling each about kernels)
class by not
just a mean by a probability
distribution can also help in learning
nonlinear decision boundaries. More
on this when we discuss
probabilistic models for
classification
 20
LwP as a subroutine in other ML models
 For data-clustering (unsupervised learning), K-means clustering is a popular algo

 K-means also computes means/centres/prototypes of groups of unlabeled points

 Harder than LwP since labels are unknown. But we can do the following
 Guess the label of each point, compute means using guess labels Will see K-means
in detail later
 Refine labels using these means (assign each point to the current closest mean)
 Repeat until means don’t change anymore
 Many other models also use LwP as a subroutine
 21

Supervised Learning
using
Nearest Neighbors
 22
Nearest Neighbors
 Another supervised learning technique based on computing distances
Wait. Did you say distance from
ALL the training points? That’s
 Very simple idea. Simply do the following at test time gonna be sooooo expensive! 
 Compute distance of of the test point from all the training points
 Sort the distances to find the “nearest” input(s) in training data Yes, but let’s not worry
about that at the moment.
 Predict the label using majority or avg label of these inputs There are ways to speed
up this step

 Can use Euclidean or other dist (e.g., Mahalanobis). Choice imp just like LwP

 Unlike LwP which does prototype based comparison, nearest neighbors

method looks at the labels of individual training inputs to make prediction

 Applicable to both classifn as well as regression (LwP only works for classifn)
 23

Nearest Neighbors for Classification

 24
Nearest Neighbor (or “One” Nearest Neighbor)
Decision boundary Interesting. Even with
Euclidean distances, it can learn
nonlinear decision boundaries?

Indeed. And that’s

possible since it is a
“local” method (looks at
a local neighborhood of
the test point to make
prediction)

Nearest neighbour approach

Test point induces a Voronoi
Test point
tessellation/partition of the input
space (all test points falling in a cell
will get the label of the training
input in that cell)
 25
K Nearest Neighbors (KNN)
 In many cases, it helps to look at not one but > 1 nearest neighbors

Test input = 31
How to pick the
“right” K
value?

K is this model’s
“hyperparameter”. One
way to choose it is using
“cross-validation” (will
see shortly)
Also, K should ideally be
an odd number to avoid
 Essentially, taking more votes helps! ties
 Also leads to smoother decision boundaries (less chances of overfitting on training data)
 26
-Ball Nearest Neighbors (-NN)
 Rather than looking at a fixed number of neighbors, can look inside a ball of a
given radius , around the test input So changing may change
the prediction. How to
pick the “right” value?

Test input

Just like K, is also a

“hyperparameter”. One
way to choose it is using
“cross-validation” (will
see shortly)
 27
Distance-weighted KNN and -NN
 The standard KNN and 𝜖-NN treat all nearest neighbors equally (all vote
equally)
=3
Test input

 An improvement: When 1 voting,

1 give
1 more importance to closer
In weighted approach, training
a single red training inputs
Unweighted KNN prediction:
3
+ 3 +3 = input is being given 3 times more
importance than the other two green inputs
3 1
+1
since it is sort of “three times” closer to the
Weighted KNN prediction:
5 5
+5 = test input than the other two green inputs
-NN can also be made
weighted likewise
 28
KNN/-NN for Other Supervised Learning Problems
 Can apply KNN/𝜖-NN for other supervised learning problems as well, such as
 Multi-class classification We can also try the weighted versions for
such problems, just like we did in the case
 Regression of binary classification
 Tagging/multi-label learning

 For multi-class, simply used the same majority rule like in binary classfn case
 Just a simple difference that now we have more than 2 classes

 For regression, simply compute the average of the outputs of nearest neighbors

 For multi-label learning, each output is a binary vector (presence/absence of

tag)
 Just compute the average of the binary vectors
 29
KNN Prediction Rule: The Mathematical Form
 Let’s denote the set of K nearest neighbors of an input by

 The unweighted KNN prediction for a test input can be written as

Assuming discrete labels with 5 possible
1
∑
values, the one-hot representation will be a all
𝐲= 𝐲𝑖 zeros vector of size 5, except a single 1
𝐾 𝑖∈ 𝑁 ( 𝐱 )
𝐾
denoting the value of the discrete label, e.g., if
label = 3 then one-hot vector = [0,0,1,0,0]

 This form makes direct sense of regression and for cases where the each output
is a vector (e.g., multi-class classification where each output is a discrete value
which can be represented as a one-hot vector, or tagging/multi-label
classification where each output is a binary vector)
 For binary classification, assuming labels as +1/-1, we predict )
 30
Nearest Neighbors: Some Comments
 An old, classic but still very widely used algorithm
 Can sometimes give deep neural networks a run for their money 
 Can work very well in practical with the right distance function
 Comes with very nice theoretical guarantees
 Also called a memory-based or instance-based or non-parametric method
 No “model” is learned here (unlike LwP). Prediction step uses all the training data
 Requires lots of storage (need to keep all the training data at test time)
 Prediction step can be slow at test time
 For each test point, need to compute its distance from all the training points
 Clever data-structures or data-summarization techniques can provide speed-ups
 31
Decision Trees
 A Decision Tree (DT) defines a hierarchy of rules to make a prediction
Root Node
Body
Warm temp. Cold

An Internal Node Non-mammal

A Leaf Node

Gives No
Yes
birth

Mammal Non-mammal

 Root and internal nodes test rules. Leaf nodes make predictions

 Decision Tree (DT) learning is about learning such a tree from labeled data
 32
Decision Trees for Supervised Learning
 The basic idea is very simple

 Recursively partition the training data into homogeneous regions

What do you
mean by
Even though the rule within
“homogeneous” each group is simple, we
regions? are able to learn a fairly
A homogeneous region sophisticated model overall
(note in this example, each
will have all (or a
rule is a simple
majority of) training horizontal/vertical
inputs with the classifier but the overall
same/similar outputs decision boundary is rather
sophisticated)

 Within each group, fit a simple supervised learner (e.g., predict the majority
label)
 33
Decision Trees for Classification
5
NO YES
𝑥1 >3.5 ?
4 Test input

NO 𝑥 2> 2?
YES NO 𝑥 2> 3 ?
YES
3
Feature 2 (

2
Predict Predict Predict Predict
1 Red Green Green Red

1 2 3 4 5 6
Feature 1 ( Remember: Root node
contains all training inputs
DT is very efficient at test time: To predict the Each leaf node receives a
label of a test point, nearest neighbors will subset of training inputs
require computing distances from 48 training
inputs. DT predicts the label by doing just 2
feature-value comparisons! Way more fast!!!
 34
Decision Trees for Classification: Another Example
 Deciding whether to play or not to play Tennis on a Saturday
 Each input (Saturday) has 4 categorical features: Outlook, Temp., Humidity,
Wind
 A binary classification problem (play vs no-play)
 Below Left: Training data, Below Right: A decision tree constructed using this
data

Example credit: Tom Mitchell

 35
Decision Trees for Regression Another simple
Can use any regression option can be to
model but would like a predict the average
simple one, so let’s use a output of the
constant prediction training inputs in
based regression model this region
4 NO YES
𝑥> 4 ?

3
y YES
NO Predict
2 𝑥 2> 3 ?

1
Predict Predict
1 2 3 4 5
𝐱

To predict the output for a test point, nearest

neighbors will require computing distances from 15
training inputs. DT predicts the label by doing just
at most feature-value comparisons! Way more
fast!!!
 Usually, cross- 36
Decision Trees: Some Considerations validation can be used
to decide size/shape
 What should be the size/shape of the DT?
 Number of internal and leaf nodes
 Branching factor of internal nodes
 Depth of the tree

 Split criterion at internal nodes

 Use another classifier?
 Or maybe by doing a simpler test?
Usually, constant
 What to do at the leaf node? Some options: prediction at leaf nodes
 Make a constant prediction for each test input reaching thereused since it will be very
 Use a nearest neighbor based prediction using training inputsfast
at that leaf node
 Train and predict using some other sophisticated supervised learner on that node
 37
How to Split at Internal Nodes?
 Recall that each internal node receives a subset of all the training inputs
 Regardless of the criterion, the split should result in as “pure” groups as
possible
 A pure group means that the majority of the inputs have the same label/output

 For classification problems (discrete outputs), entropy is a measure of purity

 Low entropy ⇒ high purity (less uniform label distribution)
 Splits that give the largest reduction (before split vs after split) in entropy are preferred
 38
Techniques to Split at Internal Nodes?
 Each internal node decides which outgoing branch an input should be sent to
 This decision/split can be done using various ways, e.g.,
 Testing the value of a single feature at a time (such internal node called “Decision
Stump”)
With this approach, all features DT methods based
and all possible values of each on testing a single
feature need to be evaluated in feature at each
selecting the feature to be tested internal node are
at each internal node faster and more
(can be slow but can be made popular (e.g., ID3,
faster using some tricks) C4.5 algos)

 Learning a classifier (e.g., LwP or some more sophisticated classifier)

DT methods based on learning and
using a separate classifier at each
internal node are less common. But
this approach can be very powerful
and are sometimes used in some
advanced DT methods
 39
Decision Tree Construction: An Example
 Let’s consider the playing Tennis example
 Assume each internal node will test the value of one of the features

 Question: Why does it make more sense to test the feature “outlook” first?
 Answer: Of all the 4 features, it’s the most informative
 It has the highest information gain as the root node
 Given some training
40
Constructing Decision Trees data, what’s the
“optimal” DT?
How to decide which rules
5 NO
𝑥1 >3.5 ?
YES
to test for and in what
4 order?
YES NO YES
NO
𝑥 2> 2? 𝑥 2> 3 ?
How to assess informativeness of a rule?
3
Feature 2 (

2
Predict
Red
Predict
Green
Predict
Green
Predict
Red In general, constructing DT is
1
an intractable problem (NP-
1 2 3 4 5 6 The rules are organized in the hard)
Feature 1 (
DT such that most Often we can use some “greedy”
Hmm.. So DTs are heuristics to construct a “good” DT
informative rules are tested
like the “20
first
questions” game (ask Informativeness of a rule is of related To do so, we use the training data to figure out
the most useful to the extent of the purity of the split which rules should be tested at each node
questions first) arising due to that rule. More The same rules will be applied on the test
informative rules yield more pure inputs to route them along the tree until they
splits reach some leaf node where the prediction is
made
 41
Decision Tree Construction: An Example
 Let’s consider the playing Tennis example
 Assume each internal node will test the value of one of the features

 Question: Why does it make more sense to test the feature “outlook” first?
 Answer: Of all the 4 features, it’s the most informative
 It has the highest information gain as the root node
 42
Entropy and Information Gain
 Assume a set of labelled inputs from C classes, as fraction of class c inputs
Uniform sets (all classes
 Entropy of the set is defined as roughly equally present)
have high entropy; skewed
 Suppose a rule splits into two smaller disjoint sets and sets low
 Reduction in entropy after the split is called information gain

This split has a low IG

This split has higher IG
(in fact zero IG)
 43
Entropy and Information Gain
 Let’s use IG based criterion to construct a DT for the Tennis example
 At root node, let’s compute IG of each of the 4 features
 Consider feature “wind”. Root contains all examples S = [9+,5-]
H ( S ) = −(9/14) log 2(9/14) − (5/14) log 2(5/14) = 0.94
Sweak = [6+, 2−] ⇒ H(Sweak ) = 0.811
Sstrong = [3+, 3−] ⇒ H(Sstrong) = 1
= 0.94 − 8/14 ∗ 0.811 − 6/14 ∗ 1 = 0.048

 Likewise, at root: IG(S, outlook) = 0.246, IG(S, humidity) = 0.151, IG(S,temp) = 0.029
 Thus we choose “outlook” feature to be tested at the root node
 Now how to grow the DT, i.e., what to do at the next level? Which feature to test next?
 Rule: Iterate - for each child node, select the feature with the highest IG
 44
Growing the tree

 Proceeding as before, for level 2, left node, we can verify that

 IG(S,temp) = 0.570, IG(S, humidity) = 0.970, IG(S, wind) = 0.019
 Thus humidity chosen as the feature to be tested at level 2, left node
 No need to expand the middle node (already “pure” - all “yes” training examples )
 Can also verify that wind has the largest IG for the right node
 Note: If a feature has already been tested along a path earlier, we don’t consider it again
 45
When to stop growing the tree?

 Stop expanding a node further (i.e., make it a leaf node) when

 It consist of all training examples having the same label (the node becomes “pure”)
 We run out of features to test along the path to that node
 The DT starts to overfit (can be checked by monitoring
To help prevent the tree
the validation set accuracy) from growing too
much!
 Important: No need to obsess too much for purity
 It is okay to have a leaf node that is not fully pure, e.g., this OR
 At test inputs that reach an impure leaf, can predict probability of belonging to each
class (in above example, p(red) = 3/8, p(green) = 5/8), or simply predict the majority
label
 46
Avoiding Overfitting in DTs
 Desired: a DT that is not too big in size, yet fits the training data reasonably
 Note: An example of a very simple DT is “decision-stump”
 A decision-stump only tests the value of a single feature (or a simple rule)
 Not very powerful in itself but often used in large ensembles of decision stumps
 Mainly two approaches to prune a complex DT
 Prune while building the tree (stopping early) Either can be done
 Prune after building the tree (post-pruning) using a validation set
 Criteria for judging which nodes could potentially be pruned
 Use a validation set (separate from the training set)
 Prune each possible node that doesn’t hurt the accuracy on the validation set
 Greedily remove the node that improves the validation accuracy the most
 Stop when the validation set accuracy starts worsening
 Use model complexity control, such as Minimum Description Length (will see later)
 For regression, outputs are
real-valued and we don’t 47
Decision Trees: Some Comments have a “set” of classes, so
quantities like
 Gini-index defined as can be an alternative to IG entropy/IG/gini etc. are
undefined
 For DT regression1, variance in the outputs can be used to assess purity
An illustration on the next slide
 When features are real-valued (no finite possible values to try), things are a bit more tricky
 Can use tests based on thresholding feature values (recall our synthetic data examples)
 Need to be careful w.r.t. number of threshold points, how fine each range is, etc.

 More sophisticated decision rules at the internal nodes can also be used
 Basically, need some rule that splits inputs at an internal node into homogeneous groups
 The rule can even be a machine learning classification algo (e.g., LwP or a deep learner)
 However, in DTs, we want the tests to be fast so single feature based rules are preferred

 Need to take care handling training or test inputs that have some features missing
1
Breiman, Leo; Friedman, J. H.; Olshen, R. A.; Stone, C. J. (1984). Classification and regression trees
 48
An Illustration: DT with Real-Valued Features
Test example
“Best” (purest possible) “Best”(purest possible)
Horizontal Split Vertical Split

Up Down Left Right

Between the best horizontal vs

best vertical split, the vertical split
is better (more pure), hence we
use this rule for the internal node

This illustration’s credit: Purushottam Kar

 49
Decision Trees: A Summary
.. thus helping us learn complex rule as
Some key strengths: a combination of several simpler rules
 Simple and easy to interpret
5
 Nice example of “divide and conquer” NO
𝑥1 >3.5 ?
YES

paradigm in machine learning 4

NO YES NO YES

 Easily handle different types of

𝑥 2> 2? 𝑥 2> 3 ?
3

Feature 2 (
features (real, categorical, etc.) 2
 Very fast at test time 1
Predict
Red
Predict
Green
Predict
Green
Predict
Red

 Multiple DTs can be combined

4 5 6
via ensemble methods: more powerful 1 2Feature 3
1(
Human-body
pose estimation
(e.g., Decision Forests; will see later)
 Used in several real-world ML applications, e.g., recommender systems, gaming
(Kinect)
Some key weaknesses:
 Learning optimal DT is (NP-hard) intractable. Existing algos mostly greedy heuristics