Volume 9, Issue 7, July – 2024 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24JUL929

SVM, KNN, and Neural Networks Investigated for

Machine Learning in Written Word Decoding
Gottipati Ajay1 Srungavarapu Bhuvanesh Babu2
Department of Computer Science and Enginnering, Department of Computer Science and Engineering,
Koneru Lakshmaiah Education Foundation, Koneru Lakshmaiah Education Foundation,
Vaddeswaram, AP, India Vaddeswaram,AP, India

Madala Narasimha Rao3 Magam Satya Siva Krishna4

Department of Computer Science and Engineering, Department of Computer Science and Engineering,
Koneru Lakshmaiah Education Foundation, Koneru Lakshmaiah Education Foundation,
Vaddeswaram, AP, India Vaddeswaram, AP, India

Dr. M. D Gouse5
(Professor)
Department of CSE,
Koneru Lakshmaiah Education Foundation,
Vaddeswaram, AP, India

Abstract:- The capacity of a device to recognise and II. RELATED WORK

understand legible handwriting input from a variety of
origins, including written material, snap shots, displays, Several research studies have contributed significantly
and other electronics, is known as handwritten to the advancement of handwriting recognition in various
reputation. In this study, we investigate three fields. In a paper titled "Feature Set Evaluation for Offline
classification algorithms: Support Vector Machines Handwriting Recognition Systems: Application of Recurrent
(SVM), K-Nearest_Neighbours (KNN), and Neural Neural Networks" Yusuf Chherawala, Partha Pratim Roy,
Networks for handwritten character recognition, and we and Mohammad Cheriet thus emphasize the important role of
will identify the best one among these three. feature extraction in handwriting recognition systems if they
emphasize that the systems are effective rely heavily on
Keywords:- Handwritten popularity, SVM, Neural Network, extracting features from handwritten word pictures[3]. While
K-Nearest Neighbor; there are a variety of feature extraction methods, identifying
the most promising ones often requires more than a simple
I. INTRODUCTION comparison based on detection rates To overcome this
challenge a system of propose a list of items based on a
Handwriting perception empowers computers and common design. Their methodology predicts the significance
devices to understand handwritten input, whether it is from of various feature types, demonstrating the robustness and
bodily files, pictures, or direct touch display screen input. universality of the technique through an integrated idea of
This era has end up fundamental to diverse gadgets together embedded RNN (recurrent neural network) classifiers.
with smartphones, drugs, and PDAs, allowing users to
rapidly enter numbers and textual content using a stylus or In another research entitled "Handwriting Digit
finger. The versatility of handwriting reputation has caused Detection and K-Nearest Neighbor Classification Using
its widespread adoption in numerous packages these days. Local Binary Pattern Variance", Nurul Ilmi, Tjokorda Agung
Among the strategies used for handwriting recognition is Budi W, and Kurniawan Nur R focus on digit detection in
Optical Character Recognition (OCR), which extracts textual handwriting using local binary pattern variance method in
content from scanned documents and translates it into a addition to K-Nearest over Neighbors (KNN)
format that computers can process. In this paper, we explore classification[2]. Their study has focused exclusively on the
three category algorithms—Support Vector Machine (SVM), C1 form that Indonesia's General Electoral Commission uses.
Neural Network, and K-Nearest_Neighbor (KNN) to beautify Through the experiment, it was observed that although the
handwriting recognition competencies. We delve into the LBP variance method achieved an impressive accuracy of
details of those algorithms later inside the paper, discussing 89.81 percent in identifying the handwriting characters in the
their strengths and packages in recognizing handwritten text. MNIST dataset, when applied to the data from the C1 form in
which the accuracy remained promising at 70.91 percent [2] .

IJISRT24JUL929 www.ijisrt.com 1773

Volume 9, Issue 7, July – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24JUL929

Researchers Kutscher, T., Dietze, M., Bönninger,I.

Travieso, C. M., Dutta, M. K., & Singh, A. conducted a study
titled "Online handwriting verification with safe password
and increasing number of features "[4]. Their primary
objective is to reduce processing time or reduce volume
while maintaining or improving distribution performance.
Their analysis shows that the Bayes net classifier
outperformed the others, achieving 100 percent correct
accuracy in terms of lead time, speed andcorrelation, with a
false acceptance rate (FAR) of only 3.13 [4].

In a provocative study titled "Identifying Human

Personality Parameters Based on Handwriting Using Neural
Networks", Behnam Fallah and Hasan Khotanlo investigate
the possible relationship between handwriting characteristics
and human personality traits is a hidden Markov model of
neural networks (MLP) are concerned for handwriting- Fig 1 SVM Categorization
personality based classification in their approach[9]. f
combination is employed integrating dependent and The SVM implementation in scikit-learn, namely
independent text features into the feature extraction process LinearSVC, works well with both sparse and dense data
Their identity recognition system proposed makes it more sources. With the use of the liblinear library and a linear
accurate and reliable Furthermore, their automated method kernel, LinearSVC provides more options for selecting loss
eliminates the need for segmentation time a feature extraction functions and penalties. Because of this, it is especially
is eliminated, while techniques such as Gaussian discriminant effective at managing huge datasets like MNIST, which has a
analysis (GDA) are used to optimize class separation huge collection of handwritten digit pictures. There will be
two reports in this category: generate_Classifier.Py for the
Collectively, these research efforts also highlight the category and perform_Recognition.Py for the type
importance of handwriting recognition technology for a experiment. There are three steps that we can complete in the
variety of applications, ranging from increased classification generate_Classifier.Py, including: a. Determine the HOG
accuracy to strong correlations with handwriting characteristics for each sample in the database. B. Use each
characteristics searching between human minds. sample's HOG capabilities at the side of the matching label to
train a multi-class linear SVM. C. Store the classifier in a
III. RECOMMENDED SYSTEMS data file.

We utilised sklearn, openCV, and python for this

gadget's categorization and dataset examination. For training
and type evaluation, we used the MNIST dataset. An
assessment tool for machine learning styles on handwritten Fig 2 A c q u i r e M I N S T D a t a s e t .
digit problems is the MNIST problem dataset. The dataset was
created by combining several scanned document datasets that The MNIST dataset will be downloaded, as
were made accessible by the National Institute of Standards demonstrated in Distinct 2. Then, we may extract the MNIST
and Technology (NIST). There are 748 total pixels in each dataset's characteristic using Histogram of Oriented
image on this dataset, measuring 28 by 28 pixels [5]. In the Gradients (HOG) feature extraction. The digit photos from
dataset, there are seventy thousand images that can be utilised the dataset may be stored in a numpy array with labels that
for both training and inspection of the gadget. Three (3) match. The HOG functions for every snapshot can then be
categorization sets of rules, namely Support Vector Machine computed and saved in a different numpy array. The coding
(SVM), Multi Layer Perceptron and K-Nearest_Neighbour, is displayed in figure three below.
were employed in this proposed gadget to determine its
popularity.

A. Support Vector Machine (SVM)

SVM is a collection of supervised learning techniques
used in classification, regression, and outlier identification. In
SVM, the price of each feature is the fee of a certain
coordinate. Each record item can be plotted as a factor in n-
dimensional space (n = number of feature). The method used
to complete the type is to identify the hyper-plane that
separates the two (2) instructions[6].

IJISRT24JUL929 www.ijisrt.com 1774

Volume 9, Issue 7, July – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24JUL929

Fig 3 Features Detection

In order to compute HOG functions, we first set the size nine. This suggests that the HOG function vector may have a
of cells to be 14 × 14. Since the MNIST dataset is 28 by 28 dimension of 4 x 9 = 36. The classifier can then be kept in a
pixels in size, we can have four (4) blocks or cells that are report, as seen in parent four below, once we have created a
each 14 by 14. The vector of orientation is approximately Linear SVM object and completed the dataset's education.

Fig 4 Linear SVM Categorization

B. K-Nearest_Neighbor (KNN)
The K-nearest_neighbours classifier is a popular
technique in the realm of image classification due to its
simplicity and effectiveness. Unlike more complex
algorithms that require extensive training processes, KNN
operates on a different principle. It doesn't undergo
traditional learning iterations with labeled data. Instead, KNN
relies on the inherent structure of the data itself.

The idea of closeness or proximity between feature

vectors is fundamental to KNN. When a new, unlabeled data
point is introduced, KNN looks around it by calculating the
separations between the known data points that are the
closest to it. The categorization of the new data point is
mostly dependent on these neighbours, whose characteristics
are most similar to it.
Fig 5 KNN Classification
Suppose that every data point in the dataset corresponds
 To train an image classifier, we go through five steps:
to an image that has different properties, including texture
patterns, colour distributions, or pixel values. Based on how
comparable these features are, KNN finds the nearest  Prepare the dataset: We organize the data for training and
neighbours of a new image that is included for classification. evaluation.
 Divide the data: The dataset is split into sections for
This is where the idea of "voting" is relevant. In a sense, testing and training.
every neighbour "votes" for the class to which it belongs. For  Extract features: We identify important characteristics
example, the new data point is likely to be classed as from the images.
belonging to Class A if the majority of its closest neighbours  Train the model: We teach the classifier to recognize
are also in Class A. By using a democratic procedure, the patterns in the features.
classification decision is made based on the opinions of all  Evaluate the model: We check how well the classifier
nearby neighbours, as opposed to just one particular criterion. performs using the testing data.

IJISRT24JUL929 www.ijisrt.com 1775

Volume 9, Issue 7, July – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24JUL929

We import the MNIST dataset and divide it into two parts: 25% for testing and 75% for training. Ten percent of the training
data were reserved for validation.

Fig 6 Divided Validation, Testing, and Training.

The classifier is then trained, and the optimal value for k—a parameter used in the K-nearest_neighbours (KNN)
algorithm—is ascertained. Furthermore, we compute the classifier's accuracy. We cycle over a range of k numbers, from 1 to 15,
in this process. The classifier's performance is next evaluated in order to confirm its efficiency, as shown in Figure seven below.

Fig 7 Classifier for Training and Validation

Afterward, we apply the trained classifier to the testing dataset, utilizing a k value set to 1. Following this, we conduct the final
evaluation of the classifier's performance, as illustrated in Figure seven.

Fig 8 Testing Classifier

Next, we will choose five (5) photos at random from the testing dataset in order to investigate each unique prediction. We'll
analyze the classifier's output for each of these images.

IJISRT24JUL929 www.ijisrt.com 1776

Volume 9, Issue 7, July – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24JUL929

Fig 9 Analysing the Classifier

C. Multi-Layer Perceptron (MLP) Two files will be used to perform the classification
The Multi-layer Perceptron (MLP) is a supervised process: perform_Recognition.py for classification evaluation
learning algorithm that aims to learn a function: f(·) : Rm → and generate_Classifier-nn.py for handling the actual
Ro classification.

The number of input dimensions, denoted by m, and the  Within generate_Classifier-nn.py, We'll take three crucial
number of output dimensions, denoted by o, are used to train steps:
the function on a dataset. For either regression or
classification problems, the MLP may learn a non-linear  Calculating HOG Features: For every sample in the
function approximator given a target y and a set of features X database, this entails calculating the Histogram of
= x1, x2,... xm. By adding one or more non-linear layers— Oriented Gradients (HOG) features.
also referred to as hidden layers—between the input and  Training a Multi-class MLP Neural Network: Utilising
output layers, it differs from logistic regression. Neural the HOG features of every sample together with their
network topologies called Multi-layer Perceptrons (MLPs) respective labels, we will train a Multi-Layer
may identify complex patterns in data. Multiple hidden layers Perceptron neural network which is multi-class.
with non-linear activation functions are a feature of MLPs, in  Saving the classifier: Once the classifier is trained, we'll
contrast to more straightforward models like logistic save it into a file for later use.
regression. Consequently, they perform remarkably well in
fields such as natural language processing, picture
identification, and financial forecasting. Throughout training,
MLPs iteratively modify weights and biases to improve their
representations, enabling them to generalise and produce Fig 11 Download MNIST Dataset
precise predictions on fresh data. Because of their versatility,
MLPs are often used for a wide range of supervised learning Initially, we will download the MNIST dataset, which is
tasks involving intricate, non-linear interactions. made up of handwritten numbers. Next, we'll use a method
known as HOG (Histogram of Oriented Gradients) to take
these digit snapshots and extract features. The digit images
and the labels that go with them will be arranged into numpy
arrays. Next, for every image, we will compute the HOG
features and store them in a different numpy array. This
procedure's detailed coding instructions can be found in the
figure 12 beneath.

Fig 10 The Hidden Layer of MLP

IJISRT24JUL929 www.ijisrt.com 1777

Volume 9, Issue 7, July – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24JUL929

Fig 12 Extraction of Features

In order to calculate the HOG features, we will use a 14 Following that, we'll instantiate a MLP (Multi-Layer
× 14 cell size. Since the MNIST dataset images are 28 x 28 Perceptron) Neural Network object & train it using the
pixels, we'll divide them into four blocks or cells, each with a dataset. As seen in Figure 13, after training, we'll save the
size of 14X14 pixels. We'll define the orientation vector to 9, classification model into a file.
resulting in a feature vector of HOG size of 4 x 9 = 36.

Fig 13 Neural Network Classification using MLP

D. Experiment After loading the image intended for testing the

In order to evaluate the classifiers developed using the classifier, we'll preprocess it. This preprocessing involves
MLP (Multi-Layer Perceptron) neural networks and Support converting the image to grayscale and applying Gaussian
Vector Machine (SVM), we'll utilize the perform_ filtering to reduce noise. Subsequently, we'll apply
Recognition.py script. Within this script, we'll load the thresholding to segment the image and identify contours
classifier files that were previously saved, as demonstrated in within it. Following this, we'll extract the rectangles that
Figure 14. encompass each contour, as illustrated in Figure 15.

Fig 14 Call Classifier File

Fig 15 Preprocessing the Image

IJISRT24JUL929 www.ijisrt.com 1778

Volume 9, Issue 7, July – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24JUL929

The classifier will then be applied to the test image.

This entails computing the image's HOG features and
applying them to forecast the digit that the image represents.
The final result and the anticipated number will then be
shown, as seen in Figure 17. Fig 16 Testing image

Fig 17 Examining the Classifier

For KNN (K-Nearest_Neighbors), we'll conduct

experiments similar to those described in the previous
sections for MLP Neural Network and SVM. We'll evaluate
the classification model using random images from the
testing dataset. This process involves testing the classifier on
a variety of images to assess its performance across different
digit representations.

IV. RESULT

The experiment shows consistent findings for K-Nearest

Neighbours (KNN) for a variety of values of the parameter k,
which vary from 1 to 15. The accuracy obtained from all
these values is a remarkable 99.26 percent, meaning that the
accuracy stays high no matter which value of k is selected
within this range. In the following figure, this observation is
illustrated. Fig 18 The classifier's accuracy

The testing data evaluation is depicted in the figure 19.

IJISRT24JUL929 www.ijisrt.com 1779

Volume 9, Issue 7, July – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24JUL929

The next figure 22 displays the MLP Neural Network

classification result.

Fig 22 MLP Neural Network Result

V. CONCLUSION
Fig 19 Assessment of the test data
The outcomes show that the dataset is accurately
classified by both KNN (K-Nearest_Neighbours) and SVM
(Support Vector Machine) classifiers. But there are certain
inaccuracies with the MLP (Multi-Layer Perceptron) Neural
Network, especially when it comes to forecasting the number
nine (9). This disparity results from the fact that MLP uses a
non-linear function, whereas KNN and SVM predict based
only on the features that have been retrieved. Multiple local
minima result from the non-convex loss functions of MLPs,
particularly those with hidden layers. As a result, different
random weight initializations may produce varied validation
accuracy. This issue can be mitigated by employing
Convolutional Neural Networks (CNNs) with frameworks
like Keras, which are better suited for capturing complex,
non-linear relationships in data.

REFERENCES

[1]. Shakoor, U., Mim, S. S., & Logofatu, D. (2023). Use

of machine learning algorithms to analyze the digit
recognizer problem in an effective manner. Artificial
Neural Networks and Machine Learning – ICANN
2023, 496-507. https://doi.org/10.1007/978-3-031-
44201-8\_40
[2]. Ilmi, N., Budi, W. T., & Nur, R. K. (2016).
Handwriting digit recognition using local binary
pattern variance and k-nearest neighbor classification.
2016 4th International Conference on Information and
Communication Technology (ICoICT).
Fig 20 Analysing the Outcome https://doi.org/10.1109/icoict.2016.7571937
[3]. Chherawala, Y., Roy, P. P., & Cheriet, M. (2016).
As seen in the figure 20, the analysing classifier uses Feature set evaluation for offline handwriting
five (5) random images from the training dataset. recognition systems: Application to the recurrent
neural network. IEEE Transactions on Cybernetics,
The SVM classification result is displayed in the figure 46(12), December 2016.
21. [4]. Kutzner, T., Dietze, M., Bönninger, I., Travieso, C.
M., Dutta, M. K., & Singh, A. (2016). Online
handwriting verification with safe password and
increasing number of features. 2016 3rd International
Conference on Signal Processing and Integrated
Networks (SPIN).
[5]. Yao, Y., & Cao, J. (2017). An adaptive scheduling
mechanism for analytical workflow model.
Communications in Computer and Information
Science, 31-45. https://doi.org/10.1007/978-981-10-
3996-6\_3
Fig 21 SVM Classification Model Output

IJISRT24JUL929 www.ijisrt.com 1780

Volume 9, Issue 7, July – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24JUL929

[6]. Belavagi, M. C., & Muniyal, B. (2016). Performance

evaluation of supervised machine learning algorithms
for intrusion detection. Procedia Computer Science,
89, 117-123. https://doi.org/10.1016/j.procs.2016.06.
016
[7]. El-Bendary, N., Zawbaa, H. M., Daoud, M. S.,
Hassanien, A. E., & Nakamatsu, K. (2010). ArSLAT:
Arabic sign language alphabets translator.2010
International Conference on Computer Information
Systems and Industrial Management Applications
(CISIM). https://doi.org/10.1109/cisim.2010.5643519
[8]. Seiderer, A., Flutura, S., & André, E. (2017).
Development of a mobile multi-device nutrition
logger. Proceedings of the 2nd ACM SIGCHI
International Workshop on Multisensory Approaches
to Human-Food Interaction. https://doi.org/10.1145/
3141788.3141790
[9]. Fallah, B., & Khotanlou, H. (2016). Identify human
personality parameters based on handwriting using
neural network. 2016 Artificial Intelligence and
Robotics (IRANOPEN). https://doi.org/10.1109/rios.
2016.7529501

IJISRT24JUL929 www.ijisrt.com 1781