Multiclass Receiver Operating Characteristic (roc) in Scikit Learn

The ROC curve is used to measure the performance of classification models. It shows the relationship between the true positive rate and the false positive rate. The ROC curve is used to compute the AUC score. The value of the AUC score ranges from 0 to 1. The higher the AUC score, the better the model. This article discusses how to use the ROC curve in scikit learn.

ROC for Multi class Classification

Now, let us understand how to use ROC for multi class classifier. So, we will build a simple logistic regression model to predict the type of iris. We will be using the iris dataset provided by sklearn. The iris dataset has 4 features and 3 target classes (Setosa, Versicolour, and Virginica).

Import Required Libraries

Python libraries make it very easy for us to handle the data and perform typical and complex tasks with a single line of code.

• Matplotlib/Seaborn – This library is used to draw visualisations.
• Sklearn – This module contains multiple libraries having pre-implemented functions to perform tasks from data preprocessing to model development and evaluation.

Load the Dataset

We will use the iris datasets which is one of the most common benchmark dataset for the classification models. Let’s load this dataset using the sklearn.datasets.

In this case, we will utilise the one vs rest strategy. So, there will be 3 cases.

Case 1:
Positive class - Setosa,
Negative Class - Versicolour and Virginica

Case 2:
Positive class - Versicolour,
Negative Class - Setosa and Virginica

Case 3:
Positive class - Virginica,
Negative Class - Versicolour and Setosa

Hence, we would have 3 ROC curves. We take the average of these 3 cases to report the final accuracy of the model. Now, Lets take a look at features.

Output:

array([[5.1, 3.5, 1.4, 0.2],
       [4.9, 3. , 1.4, 0.2],
       [4.7, 3.2, 1.3, 0.2],
       [4.6, 3.1, 1.5, 0.2],
       [5. , 3.6, 1.4, 0.2]])

Now, lets see the target values.

Output:

array([0, 0, 0, 0, 0])

First we need to use binarization on target values as shown below.

Output:

array([[1, 0, 0],
       [1, 0, 0],
       [1, 0, 0],
       [1, 0, 0],
       [1, 0, 0]])

Model Development and Training

We will create separate models for each case.

Output:

Model Accuracy :
model 1 - 1.0
model 2 - 0.7368421052631579
model 3 - 1.0

If we take the average of these accuracies, we get an overall accuracy of 91.2%.

Computing ROC – AUC Score

Now let’s calculate the ROC – AUC score for the predictions made by the model using the one v/s all method.

Output:

[1.0, 0.8047138047138047, 1.0]

The AUC score with Setosa as positive class is 1, with Versicolour as positive class is 0.805, and with Virginica as positive class is 1.

After taking the average we get 93.49% accuracy.

0.9349046015712682

Visualizing ROC Curve

Now by using the predictions for the three classes we will try to visualise the roc curve for each of the classes. Like one v/s all.

Output:

ROC Curve for each of the classes.

In the above graph we are able to see only two lines but that is not the case because the roc-auc score for setosa and the virginica class is same due to which the roc curve for these two classes overlap with each other.