Exp7_MLAI2
Uploaded by
ashwinitetame6Exp7_MLAI2
Uploaded by
ashwinitetame6EXPERIMENT NO.
07
Title : Evaluation metrics of Classification Model.
Aim : To analyze and compare the performance of different Classification Models using
various evaluation metrics.
Theory :
Importance of Evaluation Metrics for Classification Models:
Evaluation metrics are crucial for assessing the performance of classification models. They
provide quantitative measures to determine how well a model is performing and guide
improvements. Here’s why they are important:
1. Assess Model Accuracy
• Accuracy is the most straightforward metric, representing the proportion of correct
predictions. However, it is insufficient on its own, especially for imbalanced datasets.
Evaluation metrics provide a fuller picture of model performance.
2. Handle Class Imbalances
• In scenarios where classes are imbalanced, accuracy can be misleading. Metrics like
Precision, Recall, and F1-Score are essential for understanding how well the model
is performing on the minority class, which might be more critical in certain
applications.
3. Understand the Type of Errors
• Metrics such as Precision and Recall help in understanding the types of errors a model
makes. Precision focuses on false positives, while recall emphasizes false negatives.
This is particularly important in applications like medical diagnosis, where different
errors have different consequences.
To evaluate the performance or quality of the model, different metrics are used, and these
metrics are known as performance metrics or evaluation metrics.
In a classification problem, the category or classes of data is identified based on training
data. The model learns from the given dataset and then classifies the new data into classes
or groups based on the training. It predicts class labels as the output, such as Yes or No, 0 or
1, Spam or Not Spam, etc. To evaluate the performance of a classification model, different
metrics are used, and some of them are as follows:
o Accuracy o
Confusion Matrix
o Precision
o Recall o F1-Score
o AUC(Area Under
the Curve)-ROC
1.Accuracy
The accuracy metric is one of the simplest Classification metrics to implement, and it can
be determined as the number of correct predictions to the total number of predictions.
It can be formulated as:
To implement an accuracy metric, we can compare ground truth and predicted values in a
loop, or we can also use the scikit-learn module for this.
2.Confusion Matrix :
A confusion matrix is a tabular representation of prediction outcomes of any binary
classifier, which is used to describe the performance of the classification model on a set of
test data when true values are known.
A confusion matrix is a fundamental tool for evaluating the performance of a classification
model. It provides a summary of the prediction results on a classification problem, showing
how well the model's predictions match the actual labels. The matrix layout enables easy
identification of the types of errors the model is making.
Structure of the Confusion Matrix
For a binary classification problem, the confusion matrix is a 2x2 table that compares the
actual target values with those predicted by the model. The table is organized as follows:
True Positive (TP): The model correctly predicts the positive class.
True Negative (TN): The model correctly predicts the negative class.
False Positive (FP): The model incorrectly predicts the positive class when it's actually
negative (also known as a "Type I error").
False Negative (FN): The model incorrectly predicts the negative class when it's actually
positive (also known as a "Type II error").
Key Metrics Derived from the Confusion Matrix:
Accuracy:
It measures the overall correctness of the model's predictions.
Example :
Output:
Precision :
The precision metric is used to overcome the limitation of Accuracy. The precision
determines the proportion of positive prediction that was actually correct. It can be
calculated as the True Positive or predictions that are actually true to the total positive
predictions (True Positive and False Positive).
Example :
Output :
Recall or Sensitivity :
It is also similar to the Precision metric; however, it aims to calculate the proportion of actual
positive that was identified incorrectly. It can be calculated as True Positive or predictions
that are actually true to the total number of positives, either correctly predicted as positive
or incorrectly predicted as negative (true Positive and false negative).
Example :
Output :
F1-Score :
F-score or F1 Score is a metric to evaluate a binary classification model on the basis of
predictions that are made for the positive class. It is calculated with the help of Precision
and Recall. It is a type of single score that represents both Precision and Recall. So, the F1
Score can be calculated as the harmonic mean of both precision and Recall, assigning equal
weight to each of them.
Example :
Output :
AUC-RUC :
Sometimes we need to visualize the performance of the classification model on charts;
then, we can use the AUC-ROC curve. It is one of the popular and important metrics for
evaluating the performance of the classification model.
Firstly, let's understand ROC (Receiver Operating Characteristic curve) curve. ROC
represents a graph to show the performance of a classification model at different threshold
levels. The curve is plotted between two parameters, which are:
o True Positive Rate
o False Positive Rate
TPR or true Positive rate is a synonym for Recall, hence can be calculated as:
FPR or False Positive Rate can be calculated as:
To calculate value at any point in a ROC curve, we can evaluate a logistic regression
model multiple times with different classification thresholds, but this would not be much
efficient. So, for this, one efficient method is used, which is known as AUC.
AUC: Area Under the ROC curve
AUC is known for Area Under the ROC curve. As its name suggests, AUC calculates the
two-dimensional area under the entire ROC curve, as shown below image:
AUC calculates the performance across all the thresholds and provides an aggregate
measure. The value of AUC ranges from 0 to 1. It means a model with 100% wrong
prediction will have an AUC of 0.0, whereas models with 100% correct predictions will
have an AUC of 1.0.
Example :
Output : AUC – 0.73
Conclusion : In conclusion, the experiment utilizing a confusion matrix to evaluate the
performance of the classification model provides a comprehensive understanding of how
well the model is performing. By breaking down the predictions into true positives, true
negatives, false positives, and false negatives, we can clearly identify the strengths and
weaknesses of the model.
