4/22/25, 8:52 PM DeepLearningLab2.

ipynb - Colab

keyboard_arrow_down Experiment 2
Deploy the Confusion matrix and simulate for Overfitting

Objective:

1.To understand and implement the Confusion Matrix as a performance evaluation metric for
classification models.

2.To simulate and analyze the effects of overfitting on model performance.

Prerequisites:

Basic understanding of Machine Learning concepts, especially classification.

Familiarity with Python programming and libraries like NumPy, Pandas, Scikit-learn, and
Matplotlib.
A working Python environment (e.g., Jupyter Notebook, Google Colab).

Key Terms:

Confusion Matrix: A table summarizing the performance of a classification model,

showing the counts of true positive, true negative, false positive, and false negative
predictions.
True Positive (TP): Correctly predicted positive instances.
True Negative (TN): Correctly predicted negative instances.
False Positive (FP) (Type I Error): Incorrectly predicted positive instances (actually
negative).
Overfitting: A phenomenon where a model learns the training data too well, including
noise, and performs poorly on unseen data.
Regularization: Techniques used to prevent overfitting (e.g., L1/L2 regularization, dropout).

Experimental Setup:

1.Dataset: Choose a suitable dataset for classification (e.g., Iris, Breast Cancer, or a synthetic
dataset). If using a real-world dataset, ensure proper data preprocessing (handling missing
values, encoding categorical features).

2.Model Selection: Select a classification algorithm (e.g., Logistic Regression, Decision Tree,
Random Forest, Support Vector Machine).

3.Train-Test Split: Divide the dataset into training and testing sets (e.g., 80% for training, 20% for
testing).

https://colab.research.google.com/drive/1msqPjYlywX54WK-50VU6J7bddOsjXsci#printMode=true 1/7
4/22/25, 8:52 PM DeepLearningLab2.ipynb - Colab

4.Model Training: Train the chosen model on the training data.

5.Prediction: Use the trained model to make predictions on the testing data.

6.Confusion Matrix Calculation: Calculate the confusion matrix based on the actual and
predicted labels.

7.Performance Metrics Calculation: Calculate accuracy, precision, recall, and F1-score from the
confusion matrix.

8.Overfitting Simulation: Train the model with increasing model complexity (e.g., increasing tree
depth in a decision tree, using a polynomial kernel of higher degree in SVM) or by training on a
smaller subset of the training data. Observe the performance on both training and testing sets.

Theory:

The confusion matrix provides a detailed breakdown of the model's performance beyond simple
accuracy. Understanding the different components (TP, TN, FP, FN) is crucial for evaluating the
model's strengths and weaknesses, especially in imbalanced datasets. Overfitting occurs when
a model becomes too complex and starts memorizing the training data, leading to poor
generalization to new data. This is often indicated by high accuracy on the training set but low
accuracy on the testing set.

Experimental Procedure:

Step 1: Import neccesary libraries

import tensorflow as tf

from tensorflow.keras.datasets import cifar10

from tensorflow.keras.models import Sequential

from tensorflow.keras.layers import Dense, Flatten
from tensorflow.keras.utils import to_categorical
from sklearn.metrics import confusion_matrix

import numpy as np
import matplotlib.pyplot as plt

Step 2: Load and preprocess the CIFAR10 dataset:

(x_train, y_train), (x_test, y_test)= cifar10.load_data()

x_train= x_train/255.0
x_test= x_test/255.0
y_train= to_categorical(y_train)
y_test= to_categorical(y_test)

https://colab.research.google.com/drive/1msqPjYlywX54WK-50VU6J7bddOsjXsci#printMode=true 2/7
4/22/25, 8:52 PM DeepLearningLab2.ipynb - Colab

Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz

170498071/170498071 ━━━━━━━━━━━━━━━━━━━━ 2s 0us/step

Step 3: Build a simple neural network model:

model = Sequential([
Flatten(input_shape=(32,32,3)),
Dense(128, activation='relu'),
Dense(10, activation='softmax')
])

/usr/local/lib/python3.11/dist-packages/keras/src/layers/reshaping/flatten.py:37: Use
super().__init__(**kwargs)

 

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

Step 4: Train the model:

history = model.fit(x_train, y_train, epochs=10, validation_data=(x_test, y_test))

Epoch 1/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 14s 8ms/step - accuracy: 0.2936 - loss: 1.9858 - val_a
Epoch 2/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.3719 - loss: 1.7501 - val_a
Epoch 3/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 21s 8ms/step - accuracy: 0.3964 - loss: 1.6945 - val_a
Epoch 4/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 20s 8ms/step - accuracy: 0.4142 - loss: 1.6468 - val_a
Epoch 5/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 8ms/step - accuracy: 0.4188 - loss: 1.6235 - val_a
Epoch 6/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 22s 8ms/step - accuracy: 0.4286 - loss: 1.6072 - val_a
Epoch 7/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 21s 9ms/step - accuracy: 0.4331 - loss: 1.5939 - val_a
Epoch 8/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.4404 - loss: 1.5734 - val_a
Epoch 9/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.4396 - loss: 1.5682 - val_a
Epoch 10/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 14s 9ms/step - accuracy: 0.4414 - loss: 1.5599 - val_a

 

Step 5: Simulate overfitting by increasing the number of epochs significantly:

history_overfit = model.fit(x_train, y_train, epochs=50, validation_data=(x_test, y_test)

Epoch 1/50 
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.4951 - loss: 1.4221 - va
Epoch 2/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 20s 8ms/step - accuracy: 0.4984 - loss: 1.4052 - va
Epoch 3/50
https://colab.research.google.com/drive/1msqPjYlywX54WK-50VU6J7bddOsjXsci#printMode=true 3/7
4/22/25, 8:52 PM DeepLearningLab2.ipynb - Colab
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 20s 8ms/step - accuracy: 0.4942 - loss: 1.4143 - va 
Epoch 4/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 20s 8ms/step - accuracy: 0.4988 - loss: 1.4059 - va
Epoch 5/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 24s 10ms/step - accuracy: 0.4914 - loss: 1.4096 - v
Epoch 6/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 8ms/step - accuracy: 0.5000 - loss: 1.4002 - va
Epoch 7/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 21s 8ms/step - accuracy: 0.4972 - loss: 1.4054 - va
Epoch 8/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 20s 8ms/step - accuracy: 0.4983 - loss: 1.4019 - va
Epoch 9/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 8ms/step - accuracy: 0.5004 - loss: 1.3984 - va
Epoch 10/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 8ms/step - accuracy: 0.5004 - loss: 1.3976 - va
Epoch 11/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 14s 9ms/step - accuracy: 0.4976 - loss: 1.4114 - va
Epoch 12/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 19s 8ms/step - accuracy: 0.4991 - loss: 1.4040 - va
Epoch 13/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 8ms/step - accuracy: 0.5010 - loss: 1.3918 - va
Epoch 14/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 20s 7ms/step - accuracy: 0.5021 - loss: 1.3986 - va
Epoch 15/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 8ms/step - accuracy: 0.5008 - loss: 1.3991 - va
Epoch 16/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 21s 8ms/step - accuracy: 0.5011 - loss: 1.3922 - va
Epoch 17/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.4981 - loss: 1.4032 - va
Epoch 18/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 19s 8ms/step - accuracy: 0.5066 - loss: 1.3870 - va
Epoch 19/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 21s 8ms/step - accuracy: 0.4995 - loss: 1.3968 - va
Epoch 20/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.5004 - loss: 1.3967 - va
Epoch 21/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 20s 8ms/step - accuracy: 0.5016 - loss: 1.3896 - va
Epoch 22/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 22s 9ms/step - accuracy: 0.5042 - loss: 1.3908 - va
Epoch 23/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 14s 9ms/step - accuracy: 0.4991 - loss: 1.3968 - va
Epoch 24/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 19s 8ms/step - accuracy: 0.5009 - loss: 1.3936 - va
Epoch 25/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 8ms/step - accuracy: 0.5007 - loss: 1.3906 - va
Epoch 26/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 21s 8ms/step - accuracy: 0.5046 - loss: 1.3803 - va
Epoch 27/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.5050 - loss: 1.3869 - va
Epoch 28/50
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 20s 8ms/step - accuracy: 0.5027 - loss: 1.3956 - va
Epoch 29/50

Step 6: Evaluate the model and generate the confusion matrix:


loss,accuracy = model.evaluate(x_test, y_test, verbose=0) 
print('Test Loss:', loss)

https://colab.research.google.com/drive/1msqPjYlywX54WK-50VU6J7bddOsjXsci#printMode=true 4/7
4/22/25, 8:52 PM DeepLearningLab2.ipynb - Colab

print('Test Accuracy:', accuracy)

Test Loss: 1.6039667129516602

Test Accuracy: 0.4422000050544739

y_pred = model.predict(x_test)
y_pred_classes = np.argmax(y_pred, axis=1)
y_true_classes = np.argmax(y_test, axis=1)
cm = confusion_matrix(y_true_classes, y_pred_classes)
print("Confusion Matrix: ",cm)

313/313 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step

Confusion Matrix: [[628 44 73 23 12 23 36 61 32 68]
[ 73 518 28 19 10 33 48 46 17 208]
[103 18 340 58 73 90 180 109 6 23]
[ 49 21 91 204 31 250 223 91 8 32]
[ 64 7 182 46 230 61 231 163 7 9]
[ 40 8 93 129 39 404 139 122 6 20]
[ 11 11 74 71 55 59 656 48 4 11]
[ 43 12 66 43 55 103 51 586 6 35]
[300 79 27 27 17 50 46 21 342 91]
[ 93 157 21 35 1 34 40 85 20 514]]

Step 7: Plot training and validation to visualize overfitting:

plt.plot(history_overfit.history['accuracy'])
plt.plot(history_overfit.history['val_accuracy'])
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
plt.show()

https://colab.research.google.com/drive/1msqPjYlywX54WK-50VU6J7bddOsjXsci#printMode=true 5/7
4/22/25, 8:52 PM DeepLearningLab2.ipynb - Colab

Precautions and Sources of Error:

Data Leakage: Ensure that information from the testing set does not leak into the training
set.

Imbalanced Datasets: If the dataset has a skewed class distribution, consider using
appropriate evaluation metrics (e.g., precision, recall, F1-score) and techniques (e.g.,
oversampling, undersampling).

Random Initialization: Be mindful of the random initialization of model parameters and use
a fixed random seed for reproducibility.

Insufficient Data: Limited training data can exacerbate overfitting.

Result:

The output should include:

1.The confusion matrix.

2.Calculated performance metrics (accuracy, precision, recall, F1-score).

3.Plots showing the training and testing accuracy/loss as a function of model complexity or
training data size, demonstrating the effects of overfitting.

Short Questions:

https://colab.research.google.com/drive/1msqPjYlywX54WK-50VU6J7bddOsjXsci#printMode=true 6/7
4/22/25, 8:52 PM DeepLearningLab2.ipynb - Colab

1.What are the advantages of using a confusion matrix over simple accuracy?

2.Explain the difference between Type I and Type II errors.

3.How does overfitting affect the model's performance on training and testing data?

4.What are some common techniques for preventing overfitting?

5.How can you interpret the values in a confusion matrix to understand the model's behavior?

https://colab.research.google.com/drive/1msqPjYlywX54WK-50VU6J7bddOsjXsci#printMode=true 7/7