ML Model Building Process Flow

Performance Parameters

• Regression Metrics
• R2 score (Coefficient of determination)
• It is a statistical measure used to evaluate the goodness of fit of a regression model.
• It tells you how well the independent variables in your model explain the variability in the dependent
variable.
• The R2 score is a value between 0 and 1, and it can also be expressed as a percentage between 0% and
100%.
• R2 = 0: The model explains none of the variability in the dependent variable, indicating a poor fit.
• R2 = 1 (or 100%): The model explains all the variability in the dependent variable, indicating a perfect fit.
• In practice, R2 values typically fall between 0 and 1, and a higher R2 score indicates a better fit of the model to
the data.
 Performance Parameters

• Regression Metrics
• R2 score (Coefficient of determination)
• Mathematically, R2 is calculated as follows:
 Performance Parameters

• Regression Metrics
• Adjusted R2 score (Coefficient of determination)
• Mathematically, Adjusted R2 is calculated as follows: 1- ([(1- R2)(n-1)]/[n-k-1])
• Here, n is the number of rows, k is the total no of independent features, and R2 is the ordinary R
squared.
• The adjusted R-squared (Adjusted R2) is an improvement over the regular R-squared (R2) in multiple
regression analysis, as it considers the number of independent features in the model. Some advantages
of using the adjusted R-squared over the regular R-squared:
• One major advantage of the adjusted R2 is that it penalizes the addition of unnecessary variables to the
model. The regular R2 tends to increase as more variables are added, even if they do not significantly
contribute to explaining the variance in the dependent variable. Adjusted R2 adjusts for this, providing a more
realistic measure of the model's goodness of fit.
• The adjusted R2 helps guard against overfitting by penalizing models that include too many variables, which
may not generalize well to new data.
 Performance Parameters

• Classification Metrics
• Accuracy: Accuracy is a measure of how many of the total predictions made by the model are
correct.
• It is calculated by the formula TP+TN / TP+TN+FP+FN
 Performance Parameters

• Classification Metrics
• Confusion matrix: A confusion matrix is a table that summarizes the performance of a
classification model by showing the counts of true positive, true negative, false positive, and
false negative predictions.
 Performance Parameters

• Classification Metrics
• Precision:
• It gives an indication of how many of the instances predicted as positive are positives in actual.
• It is calculated by the formula TP / TP+FP.
• Precision is especially important when false positives are costly or undesirable.
 Performance Parameters

• Classification Metrics
• Recall (Sensitivity or True Positive Rate):
• It gives an indication of how well the model can correctly identify positive instances.
• This can also be defined as how many of the actual positive instances are predicted as positive.
• It is calculated by the formula = TP / TP+FN
• Recall is particularly important when false negatives are costly or detrimental. E.g., if model
unable to predict the patient of cancer as cancer.
 Performance Parameters

• Classification Metrics
• False Positive Rate (fall-out or false alarm ratio):
• It is a metric used in binary classification to quantify the proportion of negative instances that
are incorrectly predicted as positive by a model.
• E.g., person is not suffering from any disease, but model is saying that he has, or mail is not
spam but model is saying its spam.
• This can also be defined as how many of the actual negative instances are predicted as
positive.
• It is calculated by the formula FP / FP+TN
 Performance Parameters

• Classification Metrics
• True Negative Rate (Specificity):
• It is a metric used in binary classification to quantify the proportion of negative instances that
are correctly predicted as negative by a model.
• E.g., person is not suffering from any disease, and model is also saying that he is not suffering
from the disease.
• This can also be defined as how many of the actual negative instances are predicted as
negative.
• It is calculated by the formula TN / FP+TN
 Performance Parameters

• Classification Metrics
• F score: is a metric used to evaluate the performance of a machine learning model. It combines
precision and recall into a single score.
• F-score is a measure of a test's accuracy.
• It is given by the formula

• When β is 1 then it becomes:

It indicates that we are giving equal weightage to precision and recall.

• When β is 2 then it becomes:

Less weight on precision, more weight on recall.

• When β is .5 then it becomes:

More weight on precision, less weight on recall.
 Performance Parameters

• Classification Metrics
• AUC ROC Curve
• AUC ROC curve helps us visualize how well our machine learning classifier performs. It works only for
binary classification problems.
• ROC Curve (Receiver Operating Characteristic Curve):
• A graphical plot illustrating the trade-off between True Positive Rate and False Positive Rate at various
classification thresholds.
• It helps us see how the model makes decisions at different levels of certainty.
• The curve has two lines: one for how often the model correctly identifies positive cases (true positives) and
another for how often it mistakenly identifies negative cases as positive (false positives).
• By looking at this graph, we can understand how good the model is and choose the threshold that gives us the
right balance between correct and incorrect predictions.
• In other words, it shows the performance of a classification model at all classification thresholds.
 Performance Parameters

• Classification Metrics
• AUC ROC Curve
• AUC ROC curve helps us visualize how well our machine learning classifier performs. It works only for
binary classification problems.
• AUC (Area Under the Curve):
• A single metric representing the overall performance of a binary classification model based on the area under
its ROC curve.
• The Area Under the Curve (AUC) is the measure of the ability of a binary classifier to distinguish between
classes and is used as a summary of the ROC curve.
• The higher the AUC, the better the model’s performance at distinguishing between the positive and negative
classes.
 Performance Parameters

• Classification Metrics
• AUC ROC Curve

• When AUC = 1, the classifier can correctly distinguish between all the Positive and the Negative class points.
• If, however, the AUC had been 0, then the classifier would predict all Negatives as Positives and all Positives
as Negatives.
• When 0.5<AUC<1, there is a high chance that the classifier will be able to distinguish the positive class values
from the negative ones. This is so because the classifier can detect more numbers of True positives and True
negatives than False negatives and False positives.
• When AUC=0.5, then the classifier is not able to distinguish between Positive and Negative class points.
Meaning that the classifier either predicts a random class or a constant class for all the data points.
• So, the higher the AUC value for a classifier, the better is the performance of the model.
 Performance Parameters

• Classification Metrics
• How Does the AUC-ROC Curve Work?
• In an AUC-ROC curve, a higher X-axis value indicates a higher number of False positives than True
negatives.
• While a higher Y-axis value indicates a higher number of True positives than False negatives.
• So, the choice of the threshold depends on the ability to balance False positives and False negatives
naturally.
 Performance Parameters

• Classification Metrics
• Example
• We can try and understand this graph by generating a confusion matrix for
each point corresponding to a threshold and talk about the performance of
our classifier:
 Performance Parameters

• Classification Metrics
• Example
• Point A is where the Sensitivity is the highest and Specificity the lowest. This
means all the Positive class points are classified correctly, and all the
Negative class points are classified incorrectly.

• In fact, any point on the blue line corresponds to a situation where the True
Positive Rate is equal to False Positive Rate.

• All points above this line correspond to the situation where the proportion
of correctly classified points belonging to the Positive class is greater than
the proportion of incorrectly classified points belonging to the Negative
class.
 Performance Parameters

• Classification Metrics
• Example
• Although Point B has the same Sensitivity as Point A, it has a higher
Specificity.
• The number of incorrectly Negative class points is lower than the previous
threshold.
• This indicates that this threshold is better than the previous one.
 Performance Parameters

• Classification Metrics
• Example
• Between points C and D, the Sensitivity at point C is higher than point D for
the same Specificity.
• This means, for the same number of incorrectly classified Negative class
points, the classifier predicted a higher number of Positive class points.
• Therefore, the threshold at point C is better than point D.
 Performance Parameters

• Classification Metrics
• Example
• Point E is where the Specificity becomes highest. Meaning the model
classifies no False Positives. The model can correctly classify all the Negative
class points! We would choose this point if our problem was to give perfect
song recommendations to our users.
• Going by this logic, can you guess where the point corresponding to a perfect
classifier would lie on the graph?
• Yes! It would be on the top-left corner of the ROC graph corresponding to
the coordinate (0, 1) in the cartesian plane. Here, both the Sensitivity and
Specificity would be the highest, and the classifier would correctly classify all
the Positive and Negative class points
 Performance Parameters

• Classification Metrics
• Receiver Operating Characteristic (ROC) Curve:
• The receiver operating characteristic (ROC) curve is frequently used for evaluating the performance of
binary classification algorithms.
• It provides a graphical representation of a classifier’s performance, rather than a single value like most
other metrics. It is a graphical representation of the model's performance at various thresholds.
• The ROC curve provides a visual representation of the trade-off between TPR & FPR at different
classification thresholds values and helps in selecting an appropriate threshold based on the specific
objectives of the classification task.
 Performance Parameters

• Classification Metrics
• Area Under Curve (AUC):
• The Area Under the Curve (AUC) is a metric commonly used to evaluate the performance of a binary
classification model, particularly in the context of a Receiver Operating Characteristic (ROC) curve.
• The AUC represents the area under the ROC curve and provides a single scalar value that summarizes
the model's ability to discriminate between positive and negative classes across different threshold
settings.
• A higher AUC indicates that the model is better at distinguishing between the positive and negative
classes, even when the threshold for classification changes.
• The AUC is especially useful when comparing the performance of multiple models. The model with a
higher AUC is generally considered to have better discrimination ability and superior overall
performance in terms of correctly classifying instances.
• The AUC ranges between 0 and 1, where a higher value indicates better model performance.

• Here, n is the number of threshold points in the ROC curve.

 Performance Parameters

• Classification Metrics
• Area Under Curve (AUC):
• Suppose we have the following TPR (True
Positive Rate) and FPR (False Positive Rate)
values at different threshold points:
Threshold TPR FPR
0.1 0.10 0.00
0.3 0.35 0.05
0.5 0.65 0.15
0.7 0.80 0.25
0.9 0.95 0.35