AI Problem Statement EIA

License Plate Recognition System

CT/DT Number: DT20195359939

Contestant Name: Jaskirat Singh
College Name:HMR Institute Of Technology And Management

Tata Consultancy Services Confidential 1

AI Problem Statement EIA

1. Background
The problem statement is to Identify the License Plate in the image using object detection and do an
Optical Character Recognition to extract the characters from the detected license plate. India had the
third largest road network in the world and the total number of vehicles in the year 2016 stood at 230
Million . Their is a need to develop Expert Automatic License Plate Recognition systems in India
beacuse of the tremendous increase in the number of vehicles flying on the roads. Therefore,
there is a need to develop Automatic Number Plate Recognition (ANPR) system as it would help in
proper tracking of the vehicles , traffic monitoring ,tracking stolen cars, managing parking toll, red
light volation enforcement, border and customs checkpoints.Yet it’s a very challenging problem, due
to the diversity of plate formats, different scales, rotations,lightning conditions and non-uniform
llumination conditions during image acquisition. The dataset which would be used for implementing
ALPR is available in Kaggle Inc and it’s known as Vehicle Number Plate Detection provided by
DataTurks. The Dataset is available in json format and contains 353 items of which 229 items
have been manually labeled. The labels are divided in single category which is number_plate. The
datset contains web links of the vehicle images , points of the bounding boxes in each image, width
and the height of each the image.The whole project is divided into three phases – 1) Object Detection
of License Plate using Neural Networks ,deep learning, Tensorflow Object Detection API’s 2)
Preprocessing of the detected License plate using OpenCV 3) Optical Character Recognition using
PyTesseract from thesegmented license plate.

2. Your Understanding
According to my understanding firstly we would extract all the vechile images through their URL
Links from the json format dataset . After extracting the images our goal is to label the License plate
in the images that are extracted. Point to be noted is that we have to label the license plate with the
points of the bounding boxes that are mentioned in the dataset . Their are four parameters which
represents the corresponding ratios of the coordinates of the rectangle (bounding box) of the License
Plate in an image and other attributes in the dataset are the width and the height of the image.We have
to mutiply the corresponding ratios of the coordinates of the bounding box (diagonally opposite (x,y)
coordinate ratios ) with the width and the height of the image to extract the (x,y) coordinates of the
top and the bottom of the bounding box. Now that we have acchievd the coordinates we have to
label the License plate with those coordinates in each image. For this task I preffer to go with
LabelImg which is a graphical image annotation tool and label object bounding boxes in images.With
LabelImg we can label the bounding boxes with the specified coordinates mentioned in the dataset
and mention the class of the labelled image which is number_plate as mentioned in the dataset .
Whenver we label each and every image using LabelImg tool the Coordinates of the bounding
box(xmin,ymin,xmax,ymax), Width/Height of the image and the Class name of the labelled image
which is number_plate is sored in XML format. After Annotating the images we have to split the data
into train/test samples(20/80 percentage). Now we have to convert all these XML files into singular
CSV files that can be then converted to the TFRecord files. Their would be two csv files one for test
images a nd other for train images. Same is the case for TFRecord files. The TFRecord files contain
the data about the annotated coordinates ,width,height and the label name of each image.TFRecords
serves as the input data to the TensorFlow training model.Finally, the object detection training
pipeline must be configured. It defines which model and what parameters will be used for training.

Tata Consultancy Services Confidential 2

AI Problem Statement EIA

This is the last step before running training.Their are a number of configs files like Faster RCNN
Inception V2 pets config ,SSD Mobilenet V2 COCO, SSD Inception V2 COCO config etc .
The config files containd the learning rate, number of steps ,path to fine_tune_checkpoint, number of
classes to detect , num_examples , train and test record paths , label map path etc. Now it’s time to
run the model for training .If everything has been set up correctly, TensorFlow will initialize the
training. The initialization can take up to 30 seconds before the actual training begins. Each step of
training reports the loss. It will start high and get lower and lower as training progresses. For my
training on the Faster-RCNN-Inception-V2 model, it started at about 3.0 and quickly dropped below
0.8.I recommend allowing the model to train until the loss consistently drops below 0.05, which will
take about 40,000 steps, or about 2 hours (depending on how powerful the CPU and GPU are).The
loss numbers will be different if a different model is used. MobileNet-SSD starts with a loss of about
20, and should be trained until the loss is consistently under 2.We can view the progress of the training
job by using TensorBoard.

Tata Consultancy Services Confidential 3

AI Problem Statement EIA

LOSSES GRAPH USING TENSORBOARD

The TensorBoard page provides information and graphs that show how the training is progressing.
One important graph is the Loss graph, which shows the overall loss of the classifier over time.
Training the model takes 3-4 hours for GPU enabled OS . After that training is complete, the last step

Tata Consultancy Services Confidential 4

AI Problem Statement EIA

is to generate the frozen inference graph (.pb file).The .pb file contains the License Plate detection
classifier. The final trained model can detect and extract license plates from the test images or
videos.These extracted images of License plates are then passed to the next stage for extraction of
individual characters.For this stage, I wrote my own preprocessing algorithm which involved passing
the extracted License plates through a series of binarizing filters with various thresholds followed my
finding countours.What makes this Algorithm different is that it works very robustly for different
lightining condition and even when some part of the license plate is covered in shadows, and also that
this algorithm works very efficeintly for motor bike and car license plates which are 2 lined.For the
next stage we would implement OCR on the segmented License plate using Python-Tesseract , or
simply PyTesseract, library which is a wrapper for Google’s Tesseract – OCR Engine. Tesseract
Engine is used to recognize the charactes from the detected plate.I considered this because it is
completely open-source and being developed and maintained by the giant that is Google. The
advantage of using Pytesseract module for OCR is that it can recognise alphanumeric characters as
well recognise characters of different languages like English, French , Spanish, japanese, Arabic.
Though Tesseract supports Indic scripts, the approach tesseract takes to train models for languages
like Hindi,Tamil, Malayalam, Oriya, Gujarati, Kannada and Telugu is same as those for English,
French or Spanish.

3. Scope
Elements being considered in the Solution.
TensorFlow
TensorFlow is an open-source software library (Deep learning) for dataflow programming across a
range of tasks. It is a symbolic math library, and also used for machine learning applications such as
neural networks. So I have planned to use it for License plate detection.
IMAGE LABELING
Extracted the set of 237 images (Cars,bikes along with number plate) from the url links provided in
the Dataset . Then annotated the set of images by drawing the boundary box over the number plates
to send it for the training phase.
 The Annoation gives the co-ordinates of license plates such as (xmin, ymin, xmax, ymax)
 Then the co-ordinates are saved into a XML file
 All the XML files are grouped and the Co-ordinates are saved in CSV file.
 Then the CSV file is converted into TensorFlow record format.

Tata Consultancy Services Confidential 5

AI Problem Statement EIA

GPU TRAINING
By using the Tensorflow-gpu version, the set of annotated images were sent into the Convolutional
neural network called as Faster RCNN where the metrics such as model learning rate, batch of
images sent into the network and evaluation configurations were set. The training phase of the model
took several days. At last the model came around with the positive result and detected the number
plate over the input images.

License Plate Recognition (Accuracy =99%)

Tata Consultancy Services Confidential 6

AI Problem Statement EIA

OCR PART
Then the detected number plate is cropped using Tensorflow, By using the Google Tesseract-OCR
(Package originally developed to scan hard copy documents to filter out the characters from it) the
picture undergoes some coversions using computer vision package then the charcters are filtered out.
CROPPED LICENSE PLATE

Tata Consultancy Services Confidential 7

AI Problem Statement EIA

AFTER PREPROCESSING

LICENSE PLATE RECOGNITION AND DETECTION

Tata Consultancy Services Confidential 8

AI Problem Statement EIA

4. Out of Scope
In the First phase I have used Faster RCNN and SSD Mobilenet V2 COCO Model for training the
TFRecord files.In future i will be using SSD InceptionV2 COCO, SSD Mobilenet V1 0.75 Depth
COCO ,Faster RCNN Inception NAS, mask_rcnn_resnet50_atrous_coco , Mask RCNN Inception V2
Coco, Mask RCNN Inception Resnet V2 atrous Coco, SSD Mobilenet V1 Coco , Faster RCNN
Inception Resnet V2 Atrous Coco , Faster RCNN Resnet 50 Coco, SSD Resnet 50 FPN COCO models
etc.

In the second phase , sometimes preprocessing donot extract License Plate characters in low light
conditions. In the Future this problem will be removed.

In the Third phase , I have not tried OCR using a trained model with support vector machine or 5
layered Convolutional neural network. In the future I will train two models on a 68,000 dataset of
alphanumeric characters the first model was a support vector machine with a linear kernel and the
other model with a 5 layered Convolutional neural network.

Tata Consultancy Services Confidential 9

AI Problem Statement EIA

5. Assumptions
During Extraction of the images from the json format dataset out of 353 url links of the images only
237 were active url links .

Talking about the dataset when the coordinates of the bounding box were calculated for each image
,they appear to be of float datatype beacuse the ratios of the coordinates mentioned in the dataset are
also in decimals. But when we label the License Plate with the specific coordinates in float values
and save them the Xmin,Ymin, Xmax,Ymax values are having coordinates as string values in the
generated XML file. When we convert XML files to singular CSV files we have to convert the string
type coordinates to integer. Now if we typecast string values to integer it gives an error saying invalid
literal for int() with base10.After going through stackoverflow I tried to typecast the string values to
float but again their is an error saying you cannot convert string to float because the string is having
decimal places. So finally I thought to manually change the floating type coordinates to their nearest
integer .Thus after editing all the XML files i successfully converted floating point coordinates to
their most nearest integer values .

Their were 7 images in the dataset that were of unknown format due to which the training was not
able to start because the expected image formats while training are jpg/jpeg/png. so i excluded those
images from the dataset.

6. Solution Approach

Tata Consultancy Services Confidential 10

AI Problem Statement EIA

Working of ALPR

1. Extracting The Vehicle Images

2. Labelling the Image

Labelling the image using LabelImg Tool graphical tool.

Tata Consultancy Services Confidential 11

AI Problem Statement EIA

3.Splitting The annotated images to test and train images

Out Of 237 Images 55 Images (20%) were used as test Images and the rest were used as training
images.

4.Converting XML Files to Singular CSV files

Tata Consultancy Services Confidential 12

AI Problem Statement EIA

Two CSV files are created as follows:

a) train_labels.csv

b) test_labels.csv

5.Creating TFRecord Files

Tata Consultancy Services Confidential 13

AI Problem Statement EIA

6. Training The Model

7.Export Inference Graph

Tata Consultancy Services Confidential 14

AI Problem Statement EIA

8. License Plate Detection using Tensorflow API

9. License Plate Preprocessing using OpenCV

a) Converting image into GreyScale.

Greyscale conversion is nothing but converting an image into black and white view with greyshades
. Colored images doesn't help us to identify the important edges and other features.so the image needs
tobe converted in Greyscale format.

b) Binarization
Otsu Thresholding method is used to convert the grey scale image to monochrome.This method
reduces the complexity of captured image (input).

c) MedianBlur
After Binarization Image is Blurred.

d) Morphological operation on Greyscale image

This operation starts with the dilation of an image then erosion needs to be performed after that by
applying median filtering noise can be reduced.

Before Preprocessing

Tata Consultancy Services Confidential 15

AI Problem Statement EIA

After Preprocessing

10. OCR using Pytesseract Python Module

Tata Consultancy Services Confidential 16

AI Problem Statement EIA

PROJECTION OF TRAINED DATA

Tata Consultancy Services Confidential 17

AI Problem Statement EIA

Algoritms used/Models Used for License Plate Detection

Convolutional Neural network algorithm and Faster RCNN Inception V2 Model was used for License
plate detection. Faster RCNN Inception V2 Model has a speed of 58 ms and COCO mAP[^1] of 28.
Faster RCNN is an object detection architecture presented by Ross Girshick, Shaoqing Ren, Kaiming
He and Jian Sun in 2015, and is one of the famous object detection architectures that uses convolution
neural networks like YOLO (You Look Only Once) and SSD ( Single Shot Detector).
Let’s explain how this architecture works,
Faster RCNN is composed from 3 parts

Tata Consultancy Services Confidential 18

AI Problem Statement EIA

Part 1 : Convolution layers

In this layers we train filters to extract the appropriate features the image, for example let’s say that
we are going to train those filters to extract the appropriate features for a human License Plate, then
those filters are going to learn throught training shapes and colors that only exist in the License Plate.

Convolution networks are generally composed of Convolution layers, pooling layers and a last
component wich is the fully connected or another extended thing that will be used for an appropriate
task like classification or detection.

We compute convolution by sliding filter all along our input image and the result is a two dimension
matrix called feature map.
We compute convolution by sliding filter all along our input image and the result is a two dimension
matrix called feature map.

Tata Consultancy Services Confidential 19

AI Problem Statement EIA

Pooling consists of decreasing quantity of features in the features map by eliminating pixels with low
values.

Tata Consultancy Services Confidential 20

AI Problem Statement EIA

And the last thing is using the fully connected layer to classify those features wich not our case in the
Faster RCNN.

Part 2 : Region Proposel Network (RPN)

RPN is small neural network sliding on the last feature map of the convolution layers and predict
wether there is an object or not and also predict the bounding box of those objects.

Tata Consultancy Services Confidential 21

AI Problem Statement EIA

Classes and Bounding Boxes prediction

Now we use another Fully connected neural networks that takes as an inpt the regions proposed by
the RPN and predict object class ( classification) and Bounding boxes (Regression).

Tata Consultancy Services Confidential 22

AI Problem Statement EIA

Training
To train this architecture, we use SGD to optimize convolution layers filters, RPN weights and the
last fully connected layer weights.

7. Implementation Framework
General Implementation Approach
Includes Hardware and Software Details
Software Packs Needed
 Anaconda 3 (Tool comes with most of the required python packages along with python3 &
spyder IDE)
 Tesseract Engine (Must need to be installed)

Dependencies:
 Python v3.6.+ (64bits)
 NumPy v1.14.+
 Pillow v5.+
 OpenCV v3.14.+ (for cv2)
 MatPlotLib v2.1.+
 Tensorflow v1.0+
 PIL
 Protobuf 3.0.0
 Python-tk
 Pillow 1.0+
 lxml
 tf Slim (which is included in the "tensorflow/models/research/" checkout)
 Jupyter notebook
 Cython
 contextlib2
 cocoapi
For detailed steps to install Tensorflow, follow the Tensorflow installation instructions. A typical user
can install Tensorflow using one of the following commands:
# For CPU
pip install tensorflow
# For GPU
pip install tensorflow-gpu

In Anaconda Environment
# for GPU
conda install -c anaconda tensorflow-gpu
#for CPU
conda install -c conda-forge tensorflow
The remaining libraries can be installed on Ubuntu 16.04 using via
apt-get:

Tata Consultancy Services Confidential 23

AI Problem Statement EIA

sudo apt-get install protobuf-compiler python-pil python-lxml python-tk

pip install --user Cython
pip install --user contextlib2
pip install --user jupyter
pip install --user matplotlib

Alternatively, users can install dependencies using pip:

pip install --user Cython
pip install --user contextlib2
pip install --user pillow
pip install --user lxml
pip install --user jupyter
pip install --user matplotlib

Note: sometimes "sudo apt-get install protobuf-compiler" will install

Protobuf 3+ versions for you and some users have issues when using
3.5. If that is your case, try the manual installation.

Protobuf Compilation
The Tensorflow Object Detection API uses Protobufs to configure model and training parameters.
Before the framework can be used, the Protobuf libraries must be compiled. This should be done by
running the following command from the tensorflow/models/research/ directory:
# From tensorflow/models/research/
protoc object_detection/protos/*.proto --python_out=.

Manual protobuf-compiler installation and

usage
If you are on linux:
Download and install the 3.0 release of protoc, then unzip the file.
# From tensorflow/models/research/
wget -O protobuf.zip
https://github.com/google/protobuf/releases/download/v3.0.0/protoc-3.0.0-linux-
x86_64.zip
unzip protobuf.zip

Run the compilation process again, but use the downloaded version
of protoc
# From tensorflow/models/research/
./bin/protoc object_detection/protos/*.proto --python_out=.

Add Libraries to PYTHONPATH

When running locally, the tensorflow/models/research/ and slim directories should be appended to
PYTHONPATH. This can be done by running the following from tensorflow/models/research/:

Tata Consultancy Services Confidential 24

AI Problem Statement EIA

# From tensorflow/models/research/
export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim

Note: This command needs to run from every new terminal you start. If you wish to
avoid running this manually, you can add it as a new line to the end of your
~/.bashrc file, replacing `pwd` with the absolute path of
tensorflow/models/research on your system.

Python Packages Needed

 Tensorflow
 OpenCV
 Pytesseract
 LalebImg
 urlib
 pandas
 requests
 Keras
 Sci-kit learn

Hardware Details
Lenovo Ideapad 330 Intel Core i5 8th Gen , 8 GB Ram , 1TB Hardisk , Ubuntu 19.04

8. Solution Submission
Share the link with the code uploaded on GitHub.\\

https://github.com/jaskirat111/Automatic-License-Plate-Recognition.git

9. Appendix
Common Errors
It appears that the TensorFlow Object Detection API was developed on a Linux-based operating
system, and most of the directions given by the documentation are for a Linux OS. So I have used
Ubuntu based OS. There are many little snags that I ran in to while trying to set up tensorflow-gpu to
train an object detection classifier on Ubuntu 19.04. This Appendix is a list of errors I ran in to, and
their resolutions.
1) ModuleNotFoundError: No module named 'deployment' or No module named 'nets'
This error occurs when you try to run ALPR.ipynb or train.py and you don’t have the PATH and
PYTHONPATH environment variables set up correctly.
Solution:
sudo apt-get install protobuf-compiler python-pil python-lxml python-tk

Tata Consultancy Services Confidential 25

AI Problem Statement EIA

# From tensorflow/models/research/
protoc object_detection/protos/*.proto –python_out=.

# From tensorflow/models/research/

When running locally, the tensorflow/models/research/ and slim directories should be appended to
PYTHONPATH. This can be done by running the following from tensorflow/models/research/:

export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim

Note: This command needs to run from every new terminal you start. If you wish to avoid running
this manually, you can add it as a new line to the end of your ~/.bashrc file, replacing `pwd` with the
absolute path of tensorflow/models/research on your system.

Also, make sure you have run these commands from the \models\research directory:

setup.py build
setup.py install

2)ImportError: DLL load failed: The specified procedure could not be found. (or other DLL-related
errors)

This error occurs because the CUDA and cuDNN versions you have installed are not compatible with
the version of TensorFlow you are using. The easiest way to resolve this error is to use Anaconda's
cudatoolkit package rather than manually installing CUDA and cuDNN. If you ran into these errors,
try creating a new Anaconda virtual environment:

conda create -n tensorflow2 pip python=3.6

Then, once inside the environment, install TensorFlow using CONDA rather than PIP:
conda install tensorflow-gpu

3) No module names pytesseract

If you’re using the Ubuntu operating system, simply use apt-get to install Tesseract OCR:

$ sudo apt-get install tesseract-ocr

$sudo apt install libtesseract-dev

10. References

Tata Consultancy Services Confidential 26

AI Problem Statement EIA

References used in the Solution Approach

https://github.com/EdjeElectronics/TensorFlow-Object-Detection-API-Tutorial-Train-Multiple-
Objects-Windows-10
https://github.com/tensorflow/models/tree/master/research/object_detection
https://pdfs.semanticscholar.org/bb77/489c2186d622482bf0886341d387124963d4.pdf
https://github.com/msiddique1436/Automatic_License_plate_recognition
https://github.com/stevefielding/tensorflow-anpr
https://github.com/Dharun/Tensorflow-License-Plate-Detection
https://matthewearl.github.io/2016/05/06/cnn-anpr/
https://www.kaggle.com/questions-and-answers/50622
https://github.com/rafariva/ANPR-Tensorflow
https://github.com/Deevoluation/ALPR
https://github.com/Dharun/TensorFlow-ANPR
https://www.kaggle.com/dataturks/vehicle-number-plate-detection
https://pythonprogramming.net/introduction-use-tensorflow-object-detection-api-tutorial/
https://github.com/MicrocontrollersAndMore/OpenCV_3_KNN_Character_Recognition_Python/bl
ob/master/TrainAndTest.py
https://towardsdatascience.com/faster-rcnn-object-detection-f865e5ed7fc4

Tata Consultancy Services Confidential 27