Evaluation Metrics for classifica-

tion
Confusion Matrix, Precision, Re-
call, F-score, Accuracy, ROC
curve

Machine Learning, CS354

 Slides courtesy: Nuriev Sirojiddin, Tuong

1 Le
CONTENTS

BINARY CLASSIFIER
CONFUSION MATRIX
ACCURACY
PRECISION
RECALL
ROC CURVE
- HOW TO PLOT ROC CURVE
2
 Binary Classifier

 A binary classifier produces output with two class values or labels,

such as Yes/No, 1/0, Positive/Negative for given input data
 For performance evaluation: Observed labels are used to compare
with the predicted labels after classification
 The predicted labels will be exactly the same, if the performance of
a classifier is perfect.
 But it is uncommon to be able to develop a perfect classifier.

3
 Confusion Matrix

 A confusion matrix is formed from the four outcomes

produced as a result of binary classification
 True positive (TP): correct positive prediction
 False positive (FP): incorrect positive prediction
 True negative (TN): correct negative prediction
 False negative (FN): incorrect negative prediction

4
 Confusion Matrix

 Binary Classification: Green Vs. Grey

• Rows represent what is predicted, and columns represent what

is the actual label.

5
 Confusion Matrix

True Positives False Positives

Green examples Gray examples
correctly identified Confusion Matrix falsely identified as
as green green

False Negatives True Negatives

Green examples Gray examples cor-
falsely identified as rectly identified as
grey grey

6
 Accuracy

 Accuracy is calculated as the number of all correct predictions

divided by the total number of the dataset
 The best ACC is 1.0, whereas the worst is 0.0

= (TP + TN) / (TP+ FP + TN + FN)

 Accuracy = (9+8) / (9+2+1+8) = 0.85

7
 Precision

 Precision is calculated as the number of correct positive predictions

divided by the total number of positive predictions
 The best precision is 1.0, whereas the worst is 0.0

 Precision = 9 / (9+2) = 0.81

8
 Recall

 Sensitivity = Recall = True Positive Rate

 Recall is calculated as the number of correct positive predictions
divided by the total number of positives
 The best recall is 1.0, whereas the worst is 0.0
= TP / (TP+FN)

 Recall = 9 / (9+1) = 0.9

9
 Example 1

 Example: The example to classify whether images contain either a dog

or a cat
 The training data contains 25000 images of dogs and cats;
 The training data 75% of 25000 images; (25000*0.75 = 18750)
 Validation data 25% of training data; (25000*0.25 = 6250)
Test Data, 5 cats, 5 dogs

 Precision = 2/(2 + 0) * 100% = 100%

 Recall = 2/(2 + 3) * 100% = 40%
 Accuracy = (2 + 5)/(2 + 0 + 3 + 5) * 100% = 70%
10
 Example 2 (Multi-class classification)

= = = 170/300 = .556

= = .5
= = .5
= = .667
= 0.556

= = .3
= = .6
= = 0.8
P = 0.556

11
 Precision vs Recall

 There is always a tradeoff between precision and recall.

 Higher precision means lower recall and vice versa.
 Due to this tradeoff, precision/recall alone will not be very useful.
 We need to compute both the measures to get a true picture.
 There is another measure, which takes into account both precision and
recall, i.e. the F-measure.

12
 The F-measure

 F-measure: A combined measure that assesses the precision/recall

tradeoff
 It is computed as a weighted harmonic mean of both preci-
sion and recall. 1 (  2
 1) PR
F  2
1 1  PR
  (1   )
P R
 People usually use balanced F1 measure, where F1 measure is an F measure with

β=1
 i.e., with β = 1 or α = ½
2 𝑃𝑅
 Thus, the F1-measure can be computed using the following equation:
𝐹 1=
𝑃+𝑅

 For example 1, precision was 1.0 and recall was 0.4, therefore, F1-measure can
13
 The F1-measure (Example)

 In example 1:
 Precision was 1.0 2 𝑃𝑅
𝐹 1=
 Recall was 0.4
𝑃+𝑅
 Therefore, F1-measure can be computed as:
2∗ 1.0 ∗0.4
𝐹 1=
1.0+ 0.4
0.8
𝐹 1= =0.57
1.4

 Therefore, F1-measure = 0.57

 Try yourself:
 Precision=0.8, recall=0.3
 Precision=0.4, recall=0.9

14
ROC Curve
Binary classification Confusion Ma-
trix
 ROC Curve basics

 The ROC curve is an evaluation measure that is based on two

basic evaluation measures:
- specificity and sensitivity
 Specificity = True Negative Rate,

 Sensitivity – It is the same as Recall = True Positive Rate,

 Specificity
 Specificity is calculated as the number of correct negative
predictions divided by the total number of negatives.

17
 ROC Curve basics (contd.)

 In order to understand, let us consider a disease detection (binary classification)

problem.
 Here, positive class represents diseased people and negative class represents
healthy people.
 Broadly speaking, these two quantities tell us how good we are at detecting
diseased people (sensitivity) and healthy people (specificity).
 The sensitivity is the proportion of the diseased people (T P +F N) that
we correctly classify as being diseased (T P).
 Specificity is the proportion of all of the healthy people (T N + F P) that
we correctly classify as being healthy (T N).

18
 ROC Curve basics (contd.)

 If we diagnosed everyone as healthy, we would have a specificity

of 1 (very good – we diagnose all healthy people correctly), but a
sensitivity of 0 (we diagnose all unhealthy people incorrectly)
which is very bad.
 Ideally, we would like Se = Sp = 1, i.e. perfect sensitivity and
specificity.
 It is often convenient to be able to combine sensitivity and specific-
ity into a single value.
 This can be achieved through evaluating the area under the re-
ceiver operating characteristic (ROC) curve.
19
 What is an ROC Curve?

 The ROC curve stands for Receiver Operating Characteristics

curve.
 Used in signal detection to show the tradeoff between hit rate and
false alarm rate over a noisy channel, hence the term ‘receiver’
operating characteristics.
 It is a visual tool for evaluating the performance of a binary classifier.
 It illustrates the diagnostic ability of a binary classification system
as its discrimination threshold is varied.

 An example ROC curve is shown in the figure.

 The blue line represents the ROC curve.
 The dashed line is a reference curve.

20
 ROC Curve presumption
 Before, we can plot the ROC curve, it is assumed that the classifier produces a positivity score,
which can then be used to determine a discrete label.
 E.g. in case of NB classifier, the positivity score is the probability of positive class given the
test example. Usually, a cut-off threshold v = 0.5 is applied on this positivity score for pro-
ducing the output label.
 In case of ML algorithms, that produce a discrete label for the test examples, slight modifications
can be made to produce a positivity score, e.g.
1. In KNN algorithm with K=3, majority voting is used to produce a discrete output label.
 However, a positive score can be produced by computing the probability/ratio of posi-
tive neighbors of the test example.
 i.e. if K=3, and the labels for closest neighbors are: [1, 0, 1], then the positivity score
may be (1+0+1)/3 = 2/3 = 0.66. A cut-off threshold v can be applied on this positivity
score to produce an output label, usually, v = 0.5.
 For decision tree algorithm, refer to the following link:
https://stats.stackexchange.com/questions/105760/how-we-can-draw-an-roc-curve-for-decision-trees

21
 ROC Curve presumption

 An ROC curve is plotted by varying the cut-off threshold v.

 The y-axis represents the sensitivity (true positive rate).
 The x-axis represents 1-specificity, also called the false positive rate.

 The threshold v is varied (e.g. from 0 to 1), and a pair of sensitivity/specificity

value is achieved, which forms a single point on the ROC curve.
 E.g. for v = 0.5, say sensitivity=0.8, specificity=0.6 (hence FPR=1-speci-
ficity=0.4), hence (x,y)=(0.8, 0.4). However, if v = 0.6, say sensitivity=0.7,
specificity=0.7 (hence FPR=1-specificity=0.3), hence (x,y) = (0.7, 0.3)
 The final ROC curve is drawn by connecting all these points, e.g. given below:

22
 How to Plot ROC Curve - Example

 Dynamic cut-off thresh-

olds
Cut-off = 0.020 Cut-off = 0.015 Cut-off = 0.010

23
 How to Plot ROC Curve - Example

 True positive rate (TPR) = 𝑇𝑃/(𝑇𝑃+𝐹𝑁)

and False positive rate (FPR) = 𝐹𝑃/(𝐹𝑃+𝑇𝑁)
 Use different cut-off thresholds (0.00, 0.01, 0.02,…, 1.00),
calculate the TPR and FPR, and plot them into graph.
That is receiver operating characteristic (ROC) curve.
 Example

TPR = 0.5 TPR = 1 TPR = 1

FPR = 0 FPR = 0.167 FPR = 0.667

24
 How to Plot ROC Curve

 A ROC curve is created by connecting ROC points of a classifier

 A ROC point is a point with a pair of x and y values
where x is 1-specificity and y is sensitivity
 The curve starts at (0.0, 0.0) and ends at (1.0, 1.0)

25
 ROC Curve

 A classifier with the random performance level always shows a straight

line
 Two areas separated by this ROC curve
 ROC curves in the area with the top left corner indicate good performance
levels
 ROC curves in the other area with the bottom right corner indicate poor per-
formance levels

26
 ROC Curve

 A classifier with the perfect performance level shows

a combination of two straight lines
 It is important to notice that classifiers with meaningful performance
levels
usually lie in the area between the random ROC curve and the perfect
ROC curve

27
 The AUC measure

 AUC(Area under the ROC curve) score

 An advantage of using ROC curve is a single measure called AUC score
 As the name indicates, it is an area under the curve calculated in the ROC space
 Although the theoretical range of AUC score is between 0 and 1, the actual scores of
meaningful classifiers are greater than 0.5, which is the AUC score of a random classi-
fier

 ROC curves clearly shows classifiers A

outperforms classifier B

28