Probability Calibration Curve in Scikit Learn

Probability Calibration is a technique used to convert the output scores from a binary classifier into probabilities to correlate with the actual probabilities of the target class. In this article, we will discuss probability calibration curves and how to plot them using Scikit-learn.

Probability Calibration

A probability calibration curve is a plot between the predicted probabilities and the actual observed frequency of the positive class of a binary classification problem. It is used to check the calibration of a classifier, i.e., how closely the predicted probabilities match the actual probabilities. A perfectly calibrated classifier will have a calibration curve that follows the 45-degree line, which means the predicted probabilities match the actual probabilities.

It is the process of modifying a binary classification model's predicted probability in order to correlate with the actual probabilities of the target class. The purpose of probability calibration is to ensure that the predicted probabilities are well-calibrated i.e., that a predicted probability of, say, 0.8 corresponds to an actual likelihood of the positive class being true of roughly 0.8. To put it another way, predicted probabilities have to be accurate projections of the actual possibilities.

In many classification tasks, the output of the model is a probability score between 0 and 1 indicating the confidence of the model that a given sample belongs to a particular class. However, these predicted probabilities may not always match the true probabilities of the classes, especially when the model is trained on imbalanced data or has a complex decision boundary.

Key Terminologies:

• Binary Classifier: A classifier that is trained to distinguish between two classes or categories, typically labeled as 0 and 1. It is commonly used in a variety of applications, including spam filtering, fraud detection, etc.
• Predicted Probabilities: These are the probabilities predicted by a classification model from the given set of input features.
• True Probabilities: These are the actual probabilities for the underlying classes.
• Calibration curve: This is the graph between the predicted probabilities and True Probabilities. It represents the calibration of the classification model and signifies where the model is overconfident or underconfident.
• Calibration curve: It is the difference between the predicted probabilities and True Probabilities.
• Reliability diagram: When the predicted probabilities are binned into discrete intervals, and the mean predicted probability and the true frequency of the positive class are plotted for each interval, it is known as the reliability diagram.
• Brier score: The Brier score is the mean squared error between the predicted probabilities and the true probabilities, and it ranges from 0 (perfect calibration) to 1 (worst calibration).

Scikit-learn provides two methods to plot probability calibration curves:

• CalibratedClassifierCV - It is a meta-estimator that trains a classifier on a dataset and calibrates the predicted probabilities using cross-validation.
• calibration_curve - It is a function that computes the true positive rate and the predicted positive rate for a given set of predicted probabilities.

Now, let's see how to plot a probability calibration curve using Scikit-learn.

Example:

In this example, we will use the Breast Cancer Wisconsin (Diagnostic) dataset from Scikit-learn and train a logistic regression classifier to predict if a tumor is malignant or benign.

python code

# Import necessary libraries
import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import brier_score_loss

# Load the Breast Cancer Wisconsin (Diagnostic) dataset
data = load_breast_cancer()
X = data.data
y = data.target

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    test_size=0.2,
                                                    random_state=23)

# Train a logistic regression classifier
clf = LogisticRegression(max_iter=1000, random_state=23)
clf.fit(X_train, y_train)

# Probability estimate
prob_pos = clf.predict_proba(X_test)[:, 1]

# Brier Score
b_score = brier_score_loss(y_test, prob_pos)
print("Brier Score :",b_score)

# True and Predicted Probabilities
true_pos, pred_pos = calibration_curve(y_test, prob_pos, n_bins=10)

#Plot the Probabilities Calibrated curve
plt.plot(pred_pos,
         true_pos, 
         marker='o', 
         linewidth=1, 
         label='Logistic Regression')

#Plot the Perfectly Calibrated by Adding the 45-degree line to the plot
plt.plot([0, 1], 
         [0, 1], 
         linestyle='--', 
         label='Perfectly Calibrated')


# Set the title and axis labels for the plot
plt.title('Probability Calibration Curve')
plt.xlabel('Predicted Probability')
plt.ylabel('True Probability')

# Add a legend to the plot
plt.legend(loc='best')

# Show the plot
plt.show()

Output:

Brier Score : 0.022772327739296966

In this example, we first loaded the Breast Cancer Wisconsin (Diagnostic) dataset from Scikit-learn and split it into training and testing sets. We then trained a logistic regression classifier on the training set and predicted the probabilities of the positive class (i.e., malignant tumor) for the testing set.

We then used the calibration_curve function from Scikit-learn to compute the true positive rate and the predicted positive rate for a given set of predicted probabilities. We plotted these rates using the plot function from Matplotlib and added the 45-degree line to the plot to represent a perfectly calibrated classifier.

Finally, we added a title, axis labels, and a legend to the plot and displayed it using the show function from Matplotlib. The resulting plot shows the probability calibration curve for the logistic regression classifier.

Conclusion:

In this article, we discussed probability calibration curves and how to plot them using Scikit-learn. Probability calibration is an important technique to ensure that the predicted probabilities from a binary classifier are accurate and reliable. Probability calibration curves are useful to visually inspect the calibration of a classifier and to compare the calibration of different classifiers.

We showed an example of how to plot a probability calibration curve using Scikit-learn's calibration_curve function and Matplotlib. In this example, we trained a logistic regression classifier on the Breast Cancer Wisconsin (Diagnostic) dataset and plotted its probability calibration curve.

Probability calibration curves are just one tool in the toolbox of binary classifier evaluation. They can be used in combination with other evaluation metrics such as precision, recall, and ROC curves to assess the performance of a classifier and to tune its parameters.

It is important to keep in mind that probability calibration is not always necessary or beneficial for all applications of binary classification. It depends on the specific requirements of the problem at hand, and it is always a good practice to evaluate the calibration of a classifier before using it for important decisions.

Overall, probability calibration curves are a powerful tool for binary classifier evaluation, and Scikit-learn provides an easy and efficient way to plot them.