Project to perform image classification on the CIFAR-10 dataset using four AI models: Naive Bayes, a decision tree, a multi-layer perception, and a convolutional neural network (CNN) using Python and Pytorch to detect the 10 object classes in the CIFAR-10 dataset. While applying basic evaluation metrics: accuracy, confusion matrix, precision, and recall.
models/: A folder containing all saved trained models.
graphs/: A folder containing all graphs, confusion matrices.
main.py: Where we will be running our program. This file contains code on transforming and loading our data from CIFAR-10.
utils.py: Contains utility functions: select_n_img() and extract_feature_vectors(). select_n_img() will select 500 training images per class and 100 testing images per class. extract_feature_vectors() takes in the data along with a model and will generate the PCA-reduced feature vectors.
mlp.py: This is the file where we have defined our multilayer perceptron model and its variants. Training, testing and evaluation are done in this same file and metrics printed to the console.
cnn.py: This is the file where we have created our Convolutional Neural Network. Training, testing and evaluation code included.
naive_bayes.py: Contains the Naive Bayes algorithm implementation.
decision_tree.py: Contains the decision tree algorithm implementation and a help function to plot the calculated metrics (training accuracy, testing accuracy, precision, recall and f1-score).
graphs/Decision_Tree.png: A graph plotting the metrics (training accuracy, testing accuracy, precision, recall and f1-score) against the max_depths of the local implemented decision tree.
graphs/Scikit_Decision_Tree.png: A graph plotting the metrics (training accuracy, testing accuracy, precision, recall and f1-score) against the max_depths of the Scikit's decision tree.
graphs/DT_10.png: Confusion matrices for locally implemented Decision Tree algorithm with maximum depth of 10. There are multiple but the file name follows this naming convention. DT_{depth}, where depth is the decision tree's maximum depth.
graphs/Scikit_DT_10.png: Confusion matrices for Scikit's Decision Tree algorithm with maximum depth of 10. There are multiple but the file name follows this naming convention. Scikit_DT_{depth}, where depth is the decision tree's maximum depth.
.gitignore: Ignores folders cifar-10-batches-py/ and __pycache__/ since we don't want to commit these.
MLP: This is a folder that contains our pre-trained MLP models. The confusion matrix for the main model and its variants are also stored inside. .pth are model saves and .png are the confusion matrices (file is respective filename).
To run the program, simply run the script main.py in Interactive Window. This will ensure that the graphs are generated.
Running the main.py script will automatically preprocess the images. Below is a detailed explanation:
image-datasetsis how we are getting our datasets, we are saving the image batches in our root directory sincedata_dir = "./. We make sure to check whether we have already downloaded the dataset before.data_transformscontains instructions on how we are transforming our images. In this case for bothtrainandtestwe are resizing these images to 224x224 and then normalizing them.
- We then use the utility function
select_n_imgto select the first 500 images of each class for the training datasets, and 100 images of each class for the testing datasets. - We then use
dataloadersto load data into our model, with training batch_size = 500 and testing batch_size=100.
- Using the pretrained RestNet-18 model, we removed the last layer (restnet18.fc) to use it as a fixed feature extractor.
- We then pass in the modified model, the dataloaders and our device (cuda:0 or cpu) to the helper function
extract_feature_vectorsto get our PCA-reduced training feature vectors, testing feature vectors, training labels, testing labels.
-
Train
To train our locally implemented model. Uncomment line
70-72. This will train the model and save it in./models/gaussian_naive_bayes.pthTo train Scikit's model. Uncomment line82-84. This will train the model and save it in./models/scikit_gaussian_naive_bayes.pth -
Evaluate
To evaluate the models and get our metrics, we are using the helper function
get_naive_bayes_metrics(). This function simply calculates the metrics and print them to the console. We can choose which model we want to use via the paramget_naive_bayes_metrics(..., useScikit = True/False), ifTruewe will use Scikit's model,Falsewe will use our locally implemented model. -
Apply
To apply the model, we simply load a trained model and pass in our test feature vectors
test_features_pca. We can then calculate our metrics using the predictions the model returnsnb_predictions. To get the model prediction we are using.predict()
-
Train
To train our locally implemented model. Uncomment line
95-100. Since we are training this model at differentmax_depthwe will save it in./models/decision_tree_model_{depth}.pthTo train Scikit's model. Uncomment line112-117. Since we are training this model at differentmax_depthwe will save it in./models/scikit_decision_tree_model_{depth}.pth -
Evaluate
To evaluate the models and get our metrics, we are using the helper function
plot_decision_tree_metrics(). This function calculates the metrics and plot them (metrics vs tree depth). We can choose which model we want to use via the paramplot_decision_tree_metrics(..., useScikit = True/False), ifTruewe will use Scikit's model,Falsewe will use our locally implemented model. -
Apply
To apply the model, we simply load our trained models into a list. We can then iterate over this list to calculate our metrics for each model. Each model metrics are then store inside the following lists:
train_accuracies, test_accuracies, precisions, recalls, f1_scores. For example,train_accuracies[0]will store the training accuracy for decision tree model withmax_depth = 10.train_accuracies[len(loaded_dtcs)]will store the training accuracy for decision tree model withmax_depth = 50. To get the model prediction we are using.predict()
-
Open the
mlp.pyfile. -
Click the run button and wait for the program to finish executing.
-
Train
line
69, 223, 387, 563, 713are where the model definition starts, first one being main and rest the variants. The models will be trained using the following training loop. -
Test
The models are tested the using the following loop. The testing loop is same for all the models.
-
Evaluation
For evaluating the results the following code snippet is used. Precision, recall and f1 measure are calculated using the sickit-learn functions. For generating the confusion matrix we use the seaborn, numpy, pandas and matplotlib libraries.
-
Open the
cnn.pyfile -
Click run and wait for program to execute
-
The Evaluation metrics for the model and printed










