Calculate Sensitivity, Specificity and Predictive Values in CARET

In R the caret package (Classification and Regression Training) simplifies the process of building and evaluating machine learning models. It is used for preprocessing, training and validating both classification and regression models. One of its use case is evaluating classification models using metrics such as sensitivity, specificity and predictive values derived from confusion matrix.

These metrics gives us detailed information about model's effectiveness in making correct predictions. A confusion matrix is organized in a table with rows and columns representing predicted values and actual values. It allows us to analyze four important performance measures:

1. True Positives(TP): The number of observations that are correctly predicted means the cases where the model predicted a positive outcome and the actual value is also positive.
2. True Negatives(NP): The number of observations that are incorrectly predicted as negative by the model. These are the cases where the model predicted a negative outcome and the actual value was also negative.
3. False Positives(FP): The number of observations that are incorrectly predicted as positive by the model. These are the cases where the model predicted a positive outcome but the actual value was negative.
4. False Negatives(NP): The number of observations that are incorrectly predicted as negative by the model. These are the cases where the model predicted a negative outcome but the actual value was positive.

It helps in measuring other metrics also which we will discuss further in this article. But before that we need to install and load caret package to perform calculations of these metrics.

install.packages("caret")
library(caret)

1. Accuracy

It is the overall effectiveness of a model to correctly classify both positive and negative instances and it measures the model's ability to minimize both false positives and false negatives. It is calculated as (TP + TN) / (TP + TN + FP + FN).

For example if TP = 40, TN = 30, FP = 20 and FN = 10:

accuracy = (TP + TN) / (TP + TN + FP + FN)
=> (40 + 30) / (40 + 30 + 20 + 10)
=> 70 / 100
=> 0.7 or 70%

Therefore the Accuracy of the model in this example is 70% and it can be accessed from the output of confusionMatrix() function.

actual <- factor(c(0, 1, 0, 1, 0,
                   1, 0, 1, 0, 1),
                 levels = c(0, 1))
predicted <- factor(c(0, 0, 1, 1, 0,
                      0, 1, 1, 0, 0),
                    levels = c(0, 1))


confusion_matrix <- confusionMatrix(data = predicted,
                                    reference = actual)

confusion_matrix$overall["Accuracy"]

Output:

0.5

2. Sensitivity (Recall)

It is the ability of the model to correctly identify positive cases and it measures how well the model avoids false negatives. It is calculated as TP / (TP + FN).

For example, if TP = 50 and FN = 10:

sensitivity = TP / (TP + FN)
=> 50 / (50 + 10)
=> 50 / 60
=> 0.833 or 83.3%

Therefore, the Sensitivity of the model in this example is 83.3%. The sensitivity() function in the caret package is used to calculate the sensitivity of the model.

actual <- factor(c(0, 0, 0, 1, 1, 1,
                   0, 0, 0, 1, 1, 1),
                 levels = c(0, 1))

predicted <- factor(c(1, 1, 0, 0, 1, 1,
                      0, 0, 1, 1, 0, 0),
                    levels = c(0, 1))

sens <- sensitivity(predicted, actual)
print(sens)

Output:

0.5

3. Specificity

It is the ability of the model to correctly identify negative cases and it measures the ratio of actual negative cases that are correctly identified. It is calculated as TN / (TN + FP).

For Example, if TN = 70 and FP = 30:

specificity = TN / (TN + FP)
=> 70 / (70 + 30)
=> 70 / 100
=> 0.7 or 70%

Therefore the Specificity of the model in this example is 70%. specificity() function in the caret package is used to calculate the specificity of the model.

actual <- factor(c(0, 0, 0, 1, 1, 1,
                   0, 0, 0, 1, 1, 1),
                 levels = c(0, 1))

predicted <- factor(c(1, 1, 0, 0, 1, 1,
                      0, 0, 1, 1, 0, 0),
                    levels = c(0, 1))

spec <- specificity(predicted, actual)
print(spec)

Output:

0.5

4. Precision (Positive Predictive Value)

It is the ratio of correctly predicted positive cases to all predicted positive cases and it calculates the model's ability to minimize false positive predictions. It is calculated as TP / (TP + FP).

For Example, TP = 120 and FP = 50:

Positive Predictive Value = TP / (TP + FP)
=> 120 / (120 + 50)
=> 120 / 170
=> 0.7 or 70%

Therefore, the positive predictive value for the model in this example is 70%. posPredVal() function in the caret package is used to calculate positive predictive value of the model.

actual <- factor(c(0, 0, 0, 1, 1, 1,
                   0, 0, 0, 1),
                 levels = c(0, 1))
                 
predicted <- factor(c(1, 1, 0, 0, 1,
                      1, 0, 1, 0, 0), 
                    levels = c(0, 1))

ppv <- posPredVal(predicted, actual)
print(ppv)

Output:

0.6

5. Negative Predictive Value

It is the ratio of correctly predicted negative cases to all predicted negative cases. It is particularly useful in situations where the negative class is the focus of analysis. It is calculated as TN / (TN + FN).

For Example, the TN = 70 and FN = 60:

Negative Predictive Value = TN / (TN + FN)
=> 70 / (70 + 60)
=> 70 / 130
=> 0.53 or 53%

Therefore the Negative Predicted Value of the model in this example is 53%. The negPredVal() function in the caret package is used to calculate the Negative Predicted Value of the model.

actual <- factor(c(0, 0, 0, 1, 1,
                   1, 0, 0, 0, 1), 
                 levels = c(0, 1))
                 
predicted <- factor(c(1, 1, 0, 0, 1,
                      1, 0, 1, 0, 0),
                    levels = c(0, 1))

npv <- negPredVal(predicted, actual)
print(npv)

Output:

0.4

In this article we learned how to calculate different evaluation metrics using Caret package in R programming Language and with these metrics we can evaluate how good our model is working.