2019 International Conference on Computer, Control, Electrical, and Electronics Engineering

(ICCCEEE)

Arabic-Numbers Recognition System for Car Plates

Yasir Elhadi Omar Abdalshakour Sharief Babiker
Department of Electrical and Department of Electrical and Department of Electrical and
Electronic Engineering Electronic Engineering Electronic Engineering
University of Khartoum, Faculty of University of Khartoum, Faculty of University of Khartoum, Faculty of
Engineering Engineering Engineering
Khartoum, Sudan Khartoum, Sudan Khartoum, Sudan
[email protected] [email protected] [email protected]

Abstract— In the modern era, computers are used to relies on some handcrafted image features that capture
automate most of the operations used to be done by humans. certain morphological, color or textural attributes of the
This leads to higher performance and low cost of operations. license plate [2]. These features can be sensitive to image
The field of traffic management is not an exception. In this
paper, we propose an automatic license plate recognition noises, and may result in many false positives under
system which can extract the license plate number of the complex backgrounds or under different illumination
vehicles using machine learning models and image processing conditions.
algorithms. License plate recognition systems pass through In recent years, convolutional neural networks (CNN)
three successive stages and differ in the techniques used to have been successfully used in many challenging tasks of
achieve the output. In this paper, Locating the license plate in a object detection and recognition. CNNs have demonstrated
given image is the first stage. It is achieved using Faster
Region-Based Convolutional Neural Network, which is capable impressive performance on various tasks including image
of identifying regions with objects based on regions proposal classification, object detection, semantic segmentation, etc.
network. It is also capable of classifying the detected objects. [3]
Second stage is the segmentation of the plate using image Segmentation is depending on image processing
processing algorithms and final stage is to recognize the algorithms to crop every character, which is fed to the
segmented digits using convolutional neural network model. recognition stage where neural networks is the most
Results reveal that the presented system successfully detects
and recognizes the vehicle number plate on real images, commonly used technique for character classification.
achieving an overall accuracy of 93% with a total of 100 In this paper, we investigate the use of neural networks
images. and image processing algorithms for license plate
recognition system. We expecting better results than most of
Index Terms—Convolutional neural networks (CNN), Image the work done in this area.
processing, License plate recognition system. The structure of this paper is as follows. Section II
describes the problem dealt with in this work and discusses
I. INTRODUCTION previous work. Section III describes the developed methods
Due to rapidly growing number of vehicles, traffic has and algorithms used un the system. Experimental results are
presented in Section IV. Section V contains the overall
become more complex to manage. Stolen cars, speed
conclusion.
violation, unregistered cars and parking management are all
examples of problems that are used to be solved manually II. BACKGROUND
by humans. License plate recognition provides a more 1. License plate recognition problem
efficient solution to these problems by using the computer In this section we will have a sight about some of the
to automate required processes. work done in this problem.
The license plate recognition system works in three steps, A. License Plate Detection
the first step is to detect and locate the plate inside a given
License plate detection is the process of locating the
digital image or a video frame, the second step is to
region in the image that contain the license plate and the
segregate the digits of the plate to be fed to the third steps process of extraction and normalization of the plate.
which is to recognize the digits and get a string as an output. Usually, the detection process is designed only to work in
Detection stage is considered the most difficult task in the certain conditions. For example, most of the systems are
license plate recognition systems since plates can be designed to obtain the plate of specific country or region
different in size, shapes and colors, also the angle of the where all plates have the same size and colors, others also
image and the orientation of the plate make it even more constraint the image of the vehicle to be in certain distance
difficult [1]. The existing algorithms for extracting and and angle.
detection of the license can be either done using image The existing algorithms for extracting and detection of
processing techniques or by using neural networks models. the license can be either done using image processing
Image processing algorithms for detection process usually techniques or by using neural networks models. Image

978-1-7281-1006-6/19/$31.00 ©2019 IEEE

Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 16:30:45 UTC from IEEE Xplore. Restrictions apply.
processing algorithms for detection process can be roughly classification, where the input character image is classified
classified into four categories [4]: edge-based, color-based, to the best matching character.
texture-based, and character-based. Most of license plates systems use template matching
Edge-based approaches try to find regions with higher algorithm to classify images to correct labels [11].
edge density than elsewhere in the image as license plates. Modifications to template matching technique is done to
Method [5] uses edge detector combining with some increase the efficacy and make it more robust to noisy and
morphological operations to find the rectangles which are distorted input images. That is done in Hui Wu and Bing Li
regarded as a candidate license plate. work [12], where neural networks used to train the system
Edge-based methods are fast in detection speed. for better and faster results.
However, they cannot be applied to complex images as they
are too sensitive to unwanted edges. It is also difficult to 2. RCNN, Fast RCNN & Faster RCNN
find license plates if they are blurry. A typical CNN can only tell you the class of the objects
Color-based approaches are based on the observation that but not where they are located. It is actually possible to
color of the license plate is usually different from that of the regress bounding boxes directly from a CNN but that can
car body. HSI color model is used in [6] to detect candidate only happen for one object at a time. In R-CNN the CNN is
license plate regions, which are later verified by position forced to focus on a single region at a time. Regions in the
histogram. Color-based methods cannot distinguish other R-CNN are detected by one of the region proposal
objects in the image with similar color and size as the algorithms followed by resizing so that the regions are of
license plates. equal size before they are fed to a CNN for classification
Texture-based approaches try to detect license plates and bounding box regression.
according to the unconventional pixel intensity distribution The most straightforward way to generate smaller sub-
in plate regions. Zhang et al. [7] propose a license plate regions (patches) is called the Sliding Window approach.
detection method using both global statistical features and However, the sliding window approach has several
local Haar-like features. Texture-based methods use more limitations. These limitations are overcome by a class of
discriminative characteristics than edge or color, but result algorithms called the “Region Proposal” algorithms.
in a higher computational complexity. Selective Search is a region proposal algorithm used in
Character-based approaches regard the license plate as a object detection. It is based on computing hierarchical
string of characters and detect it via examining the presence grouping of similar regions based on color, texture, size and
of characters in the image. Lin et al. [8] propose to detect shape compatibility. Selective Search starts by over-
license plate based on image saliency. Character-based segmenting the image based on intensity of the pixels using
methods are more reliable. However, the performance is a graph-based segmentation method by Felzenszwalb and
affected by the general text in the image background. Huttenlocher [13].
One approach is described by Chen et al. [9]. Their
method employs a CNN with a single convolutional layer,
applied in a sliding window fashion over small image sub-
region. The CNN was trained to classify individual
characters as text/non-text, with the outputs being merged
and then tested based on aspect ratio and size.
Unfortunately, the authors do not provide enough details
about the algorithm and the evaluation, so it is not possible Figure 1:Selective search algorithm
to reproduce their method for comparison purposes.
RCNN model is based on generating 2000 sub-regions
B. Character Segmentation
using selective search algorithm and then feed them to
Character segmentation is the process of partitioning the ConvNets for generating feature maps which is then fed to
image of the license plate into individual characters that can SVM’s classify them depending on the given classes.
be recognized by the system. The segmentation importance Bounding box regressors is added to predict the object
comes from the fact that a more efficient and accurate coordinates.
segmentation will result easier and more accurate character Due to the large number of regions generated, training is
recognition. expensive in space and time and object detection is slow.
In Karthikeyan, V.J. Vijayalakshmi, and P. Jayakumar
work [10], connected components algorithm is used to
segment the characters based on thresholded binary image.
This algorithm is accurate in case of characters that are
constructed from one component and free of dots and
dashes. Also, this can take more computations than the
projection techniques but will produce a better result in case
of noisy images.
C. Character Recognition
Character recognition is the process of assigning a label to Figure 2: RCNN
a character image representing that character. It is a sort of

Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 16:30:45 UTC from IEEE Xplore. Restrictions apply.
 Fast R-CNN was able to solve the problem of speed, in
this model, the image is first-fed through a ConvNet, After the RPN, there a final classifier and regressor to get
features of the region proposals are obtained from the last the score of the given object classes and the final box
feature map of the ConvNet. coordinates.

III. METHODOLOGY

Figure 6: Sudanese license plate regions

License plate detection and localization:

Figure 3: Fast RCNN
The output to be expected from this stage is the
The speed of detection is further enhanced in the Faster coordinates of a box bounding the license plate in a digital
RCNN model which uses region proposal network (RPN) image.
for generating region proposals. Faster-rcnn_inception_v2_coco configuration model will
be used for training the dataset. The model architecture is
shown below:

Figure 4: Faster-RCNN

RPN uses the concept of anchors, Anchor is the central

point of the sliding window. Classifier determines the Figure 7: Faster-RCNN model architecture
probability of a proposal having the target object.
Regression regresses the coordinates of the proposals. For There are some assumptions regarding the detection
any image, scale and aspect-ratio are two important process:
parameters, aspect ratio = width of image/height of image, • We captured the training images from distances in the
scale is the size of the image. The developers chose 3 scale
range of [1-4] meters.
and 3 aspect-ratio. So, total of 9 proposals are possible for
• We tested the system against images in the ranges of [1-
each pixel, this is how the value of k is decided, K=9 for this
8] meters.
case. For the whole image, number of anchors is W*H*K.
• Training images are captured by maximum angle of
30o.
These anchors are assigned label based on two factors:
The flow of events being done before training the dataset
are as follows:
1. The anchors with highest Intersection-over-union - Gathering the dataset:
overlap with a ground truth box. The dataset in our case is a collection of a mobile
2. The anchors with I-o-U Overlap > 0.7.
captured images of cars located at different
distances. Images is then compressed because
mobile captured images have sizes of 5MB or more
which could result in a memory exhaustion. We
gather around 500 different car images from
different angles and distances. The dataset is
divided into training set and test set, 400 for
training and 100 for test.
- Create label-boxes for every image in the dataset:
LabelImg (a graphical image annotation tool) is
used for labelling training images with bounding
Figure 5: Region proposal network [14] box around the license plate.

Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 16:30:45 UTC from IEEE Xplore. Restrictions apply.
 - Generate the TFRecords that serve as input data to Model description:
the TensorFlow training model. x We constructed a sequential model with a first
- Create label map: convolutional layer which has 32 filters and kernel
The label map tells the trainer what each object is size of (3*3). The input shape is fixed at
by defining a mapping of class names to class ID (64*64*3). The activation function used is
numbers. rectified-linear unit (ReLU).
- Configure training: x The second layer is a pooling layer with a pool size
Faster_rcnn_inception_v2 is the model used for of (2*2).
training. The model folder which contains the x The ANN needs individual features in order to be
checkpoints file. Checkpoints will save the weights able to classify thus we need to flatten the output of
of the training after every certain patch. the CNN network into 1D.
- Run the training: x Then we added a hidden layer after the output of
Batch size is set to (1) with a total steps number of the convolutional neural-network. It consists of 64
200,000. Each step of training reports the loss. units and the activation function is also a rectified-
- After the training is complete, the last step is to linear unit.
generate the frozen graph, which contains the x Last layer is the output layer, consists of 11 units
object detection classifier. with a “SoftMax” activation function. The number
- The last step is to use the object-detection classifier of units represents the numbers from 0-9 plus a unit
to test it out on an image, either captured image, to recognize the region which contains “ ‫ﺟﻤﻬﻮﺭﻳﺔ‬
live image using webcam or a video. ‫ ”ﺍﻟﺴﻮﺩﺍﻥ‬then printing zero if it appeared.

Character segmentation: Training dataset:

The input to this process is the part of the resultant The problem with getting large dataset in our case is that
image, which is bounded by the green box. The output is a there are no much available Arabic digit images, specifically
segmented image with each character in a separated image. by the shape founded in the Sudanese license plate.
Assumptions: The proposed methodology of gathering large number of
x The pixel position from which we begin the Arabic digits is as follows:
segmentation process is found by trial on many x Create a for-loop to iterate through a successive set
different images. We found it to be [400] for the X- of license plate images, and begin the detection
direction. The “Y” points are specified according to process until the output appears.
the algorithm shown below. x After the first output of the segmentation process,
x We didn’t count the part of “4 ‫ ”ﺥ‬which located in we did cv2.imwrite() function, which saves every
region-2 of the plate. image in its folder. (names of the folders from 0-9).
x The part of “‫ ”ﺟﻤﻬﻮﺭﻳﺔ ﺍﻟﺴﻮﺩﺍﻥ‬may be counted in the x Repeat the process.
segmentation, but it’s not counted in the By following the above procedure, we created a dataset of
recognition. more than 5000 images for the defined classes.
Algorithm:
1. Specify the initial (X, Y) point. Should be located
in a white vertical line. IV. RESULTS
2. From (1), we start a loop to check for a vertical line Testing data:
with a majority of black pixels (more than 20
pixels is the convergence we’re searching for). Testing data is a collection of images, each containing a
3. If the threshold in (2) is met, it means we started license plate and other objects. This data is collected by
the first digit region. capturing images of Sudanese private cars using a digital
4. From (3), begin another loop to check for a fully camera of Huawei Mate 10 Lite smartphone.
white line (end of the digit). Testing images is chosen to reflect all situations the
5. Go to (1), but with updating the (X, Y) value. system is capable of handling. For examples, image
6. Specify end point for the loop. captured from more than 10 meters distance, images with
different levels of illumination and images rotated with
The initial and end points for the above loops is found by specific angles
feeding a lot of images for the detection phase, and Four of the successful results are selected here to
searching for a common point to start from. represent the system’s step-by step outputs. We took care of
diversity in the testing images chosen:
Character recognition:
The input to this process is the segmented images from
the segmentation process. The output is the number seen in
the image in a string form.
We use a convolutional neural-network model for
recognizing the input image.

Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 16:30:45 UTC from IEEE Xplore. Restrictions apply.
 Figure 12: Validation loss & validation accuracy for the recognition model

Figure 8: Sample of the test images

The overall accuracy of the system is calculated by
feeding 100 images from the testing data to the system, and
The above images are inserted to the detection stage, and then checking the correlation between the output and the
the resultant output is as follows: input as seen by us. The system has recorded an overall
accuracy of 93%, showing mistakes in a total of 7 images
due to errors in different stages.

Table 1: Sample of testing data and its results

Figure9: Detection output

V. CONCLUSION
The above images with bounding boxes giving the A license plate recognition system used for identification
boundaries of the X and Y direction is cropped and and recognition of Sudanese private cars is presented by this
converted to binary images. Showed below: paper. The developed system makes use of machine learning
and digital image processing methods to operate.

Detection and localization of license plate is performed

using the concept of region-based convolutional neural
networks. Using faster-RCNN notably outperformed pure
image processing techniques in terms of accuracy.
Character Segmentation is performed to separate each
number in the license plate alone. Accurate results are
Figure 10: Enhancement of the detection output before segmentation produced for good quality noiseless images. Image
enhancement is used to provide such images.
These images are then segmented using the segmentation Convolutional neural network model is constructed to
algorithm explained above in the methodology chapter. The recognize the numbers of the license plate. Using the CNN
segmented numbers are illustrated below: model increases the robustness of the system and make it
more time efficient.
This system has potential to be implemented in Sudan in
areas such as parking management systems. We naturally
envisage that future work regarding this system that it
should be able to identify all regions of the Sudan’s license
Figure 11: Segmented characters plate, so it could be used inside the speed radars systems.

These segmented numbers are fed to a CNN model to

recognize the numbers in a string form. The representation REFERENCES
of the validation loss and accuracy is shown in the next [1] Hui Li, Chunhua Shen, “Reading Car License Plates Using Deep
figure: Convolutional Neural Networks and LSTMs”, arXiv:1601.05610v1,
2016.

Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 16:30:45 UTC from IEEE Xplore. Restrictions apply.
[2] S. Du, M. Ibrahim, M. Shehata, and W. Badawy, “Automatic license
plate recognition (alpr): A state-of-theart review,” IEEE Trans.
Circuits Syst. Video Technol., vol. 23, no. 2, pp. 311–325, 2013.
[3] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature
hierarchies for accurate object detection and semantic segmentation,”
in Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2014. [Online].
Available: http://arxiv.org/abs/1311.2524
[4] S. Du, M. Ibrahim, M. Shehata, and W. Badawy, “Automatic license
plate recognition (alpr): A state-of-theart review,” IEEE Trans.
Circuits Syst. Video Technol., vol. 23, no. 2, pp. 311–325, 2013.
[5] H. Bai and C. Liu, “A hybrid license plate extraction method based on
edge statistics and morphology,” in Proc. IEEE Int. Conf. Patt.
Recogn. pp. 831–834, 2004.
[6] K. Deb and K. Jo, “Hsi color based vehicle license plate detection,”
in Proc. Int. Conf. Cont. Autom. Syst., pp. 687–691, 2008.
[7] H.Zhang, W.Jia, X.He, and Q.Wu, “Learning-based license plate
detection using global and local features,” in Proc. IEEE Int. Conf.
Patt. Recogn., pp. 1102–1105, 2006.
[8] K. Lin, H. Tang, and T. Huang, “Robust license plate detection using
image saliency,” in Proc. IEEE Int. Conf. Patt. Recogn., pp. 3945–
3948, 2010.
[9] Ying-Nong Chen, Chin-CHuan Han, Cheng-Tzu Wang, Bor- Shenn
Jeng, and Kuo-Chin Fan, “The application of a convolution neural
network on face and license plate detection,” in International
Conference on Pattern Recognition (ICPR), pp. 552–555,2006.
[10] V. Karthikeyan, V. J. Vijayalakshmi, and P. Jeyakumar, “License
Plate Segmentation Using Connected Component Analysis,” J.
Electron. Commun. Eng., vol. 4, no. 5, pp. 18–24, 2013.
[11] P. Kulkarni, A. Khandebharad, D. Khope, and P. U. Chavan, “License
Plate Recognition: A review,” Fourth Int. Conf. Adv. Comput., pp. 1–
8, 2012.
[12] H. Wu and B. Li, “License plate recognition system,” 2011 Int. Conf.
Multimed. Technol. ICMT 2011, no. 0, pp. 5425–5427, 2011.
[13] Pedro F. Felzenszwalb and Daniel P. Huttenlocher. Efficient Graph-
Based Image Segmentation.
[14] Faster R-CNN: Towards Real-Time ObjectDetection with Region
Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick, and
Jian Sun

Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 16:30:45 UTC from IEEE Xplore. Restrictions apply.