Classification

INTRODUCTION

1
Classification
§ Introduction
§ Logistic Regression
§ Model Evaluation

2
Classification
Classification is the problem of identifying to which of a set of categories or label, a
new observation belongs.

Classification of the new observation is based on a training set of data containing observations (or
instances) whose category membership is known.

X1 X2 Y X1 X2 Y
10 20 100 10 20 A

15 30 150 15 30 A

5 10 75 5 10 B

Regression Classification

3
Logistic Regression
Logistic regression is a technique used for binary classification problems, where the goal is to predict one
of two possible outcomes.

Unlike linear regression, which predicts a continuous outcome, logistic regression predicts the probability
that a given input belongs to a certain class.

4
Logistic Regression
• What is the output of the logistic regression model?
• Is the output range bound?
• How is the output constrained to a range?

5
Logistic Regression
Logistic regression uses a logistic function (or sigmoid function) to model the probability of a particular
outcome. The logistic function maps any real-valued number into the range (0, 1).

z output
-5 0.01
σ=
-2 0.02

0 0.50

1 0.73

2 0.88

6
Logistic Regression

What do you notice in

the picture?

7
Logistic Regression
In the logistic function, z is a linear combination of the input features and σ gives the probability of the
output as a function of the inputs.

Z=

The objective is to find the parameters:

, that minimizes the error in predicting the class, across the training instances. The objective function is
known as the Binary Cross Entropy Function.

Minimize:
-

8
 Logistic Regression
Instance Class (yi) Probability (yi) log p(yi) Probability (1-yi) log (1-yi) - (yi*log p(yi) + (1-yi)(1-log p(yi)))
1 1 0.8 -0.22 0.2 -1.61 0.500
2 1 0.9 -0.11 0.1 -2.30 0.325
3 0 0.1 -2.30 0.9 -0.11 0.325
4 0 0.2 -1.61 0.8 -0.22 0.500
5 1 0.9 -0.11 0.1 -2.30 0.325
6 0 0.3 -1.20 0.7 -0.36 0.611
8 0 0.4 -0.92 0.6 -0.51 0.673
9 1 0.6 -0.51 0.4 -0.92 0.673
10 0 0.1 -2.30 0.9 -0.11 0.325
Obj Fn Value 0.473

9
Logistic Regression

Are the beta’s

significant?

If so, the feature

has a significant
impact on the
outcome

10
Interpretation
Positive Beta: Indicates that as the predictor
σ= increases, the probability of the outcome increases.

Negative Beta: Indicates that as the predictor

Z= increases, the probability of the outcome
decreases.

Magnitude of Beta: The larger the value of β, the

stronger the association between the predictor and
the outcome.

If β=0.5= 0.5, then e0.5 ≈ 1.65. This means the odds

of the outcome are 65% higher for each one-unit
increase in the predictor

11
Model Performance
Case Actual Class Predicted Class Type

1 1 1 True Positive

2 0 0 True Negative
3 1 0 False Negative
4 0 1 False Positive
5 1 1 True Positive

. . . .

N 0 0 True Negative

12
Model Performance
Performance Measures

Accuracy = (TP + TN) / (TP + TN + FP + FN)

(Proportion of objects rightly classified)

Recall or Sensitivity = TP / (TP + FN)

(Proportion of positives rightly classified)

Specificity = TN/(TN + FP)

(Proportion of negatives rightly classified)

Precision = TP/(TP + FP)

(Proportion of positives among all those classified a positives)

13
Thank You

14