Ho Chi Minh City University of Technology and Education

High Quality Training Faculty

Digital Image Processing

Topic: Detect and Recognize Vehicle’s License Plate

Class: 18110CLA4

Instructors: Hoàng Văn Dũng

Students: Nguyễn Minh Phú 18110038

Võ Minh Huy 18110020

Ho Chi Minh city - 2020

Digital Image Processing

Table of contents:
Contents
Table of contents:........................................................................................................................................1
Teacher comment.......................................................................................................................................3
Categories and abbreviated terms..............................................................................................................4
The introduce:.............................................................................................................................................5
1. Heading...............................................................................................................................................6
1.1. An overview of the topic..............................................................................................................6
1.2. Target..........................................................................................................................................6
1.3. Subject scope...............................................................................................................................6
2. Theoretical basis..................................................................................................................................8
2.1. Theory..........................................................................................................................................8
2.2. Tools and libraries:.......................................................................................................................8
3. Topic execution...................................................................................................................................9
3.1. Methods of implementation........................................................................................................9
3.2. Block diagram:...........................................................................................................................10
3.3. Implementation process............................................................................................................10
3.3.1. Detection License Plate with Wpod-Net............................................................................10
3.3.2. Plate character segmentation............................................................................................16
3.3.2.1. Image processing.......................................................................................................16
3.3.2.2. Determine contour of License characters..................................................................18
3.3.2.3. Visualize our segmented characters..........................................................................20
3.3.3. Recognize plate license characters with OpenCV and Deep Learning................................21
3.3.3.1. Dataset.......................................................................................................................21
3.3.3.2. Data Pre-processing...................................................................................................23
3.3.3.3. Initialize MobileNets architecture with pre-trained weights......................................25
3.3.3.4. Train and evaluate model..........................................................................................26
3.3.4. Combining everything........................................................................................................29
4. Design the theme..............................................................................................................................31
5. Conclusion and development direction.............................................................................................34
5.1. Conclude:...................................................................................................................................34
5.1.1. Result:................................................................................................................................34

Page 1
Digital Image Processing

5.1.2. Limitations:........................................................................................................................34
5.2. Development:............................................................................................................................34
References.................................................................................................................................................36

Page 2
Digital Image Processing

Teacher comment
............................................................................................
............................................................................................
............................................................................................
............................................................................................
............................................................................................
............................................................................................
............................................................................................
............................................................................................
............................................................................................
............................................................................................
............................................................................................
............................................................................................
............................................................................................
............................................................................................
............................................................................................
............................................................................................

Page 3
Digital Image Processing

Categories and abbreviated terms

Terminology and acronyms Explain

OpenCV Open Source Computer Vision

CNN Convolutional neural network

ALPR Automatic License Plate Recognition

OCR Optical Character Recognition

ML Machine Learning

Page 4
Digital Image Processing

The introduce:
Image processing is an area of interest and has become a specialized subject
of engineering staff, the bachelor's degree in information technology, as well as
other engineering disciplines in engineering universities. information, statistical
theory, artificial intelligence, identification, edge separation etc. Image processing
research all things related to image manipulation such as: Image rendering, image
acquisition, enhancement and restore the image to achieve the desired image. Each
problem requires some algorithmic processing in each pixel or image element.
Learning helps us to understand more deeply about the mechanism, principles of
image processing, color adjustment, brightness, contrast, .. make the image more
beautiful and artistic. Therefore, image processing is a field of great interest and
has become the passion of many students.
In this report, I would like to give the topic: Detect and Recognize Vehicle’s
License Plate
Due to limited knowledge, this essay material may still lack many
shortcomings, looking forward to the contribution of teachers and readers to make
the document more complete.
We sincerely thank!

Page 5
Digital Image Processing

1. Heading
1.1.An overview of the topic

- In this day and age, everything a human can do, almost any machine can do.
It is because programmers have turned computers and its peripherals into
intelligent systems that can do anything.

- As simple as parking a car, just a few years ago when we went to any
parking lot we saw that all cars had to be treated with paper by the staff,
which was inconvenient and much the risk of paper is very easy to be
damaged, torn or sticky ... not only that, the recording of paper cannot avoid
errors when the employee looks at the car for a moment, distracting and
incorrectly recording, so it is very time consuming.

- But now that is not a problem when we have technology to recognize license
plates with magnetic cards. This work consists of 3 main phases: Take the
image containing the license plate, process the image to get the code number
on the license plate, finally save them on the card.

- In which the most important stage is to collect the license plate number from
the image. In this topic we will present the method how to recognize that
code.
1.2.Target
- In the process of topic, the group sets out 2 main goals.
- Have basic knowledge of Python programming. Because this is an extremely
popular and important language, especially in the field of AI and Machine
Learning.
- Learn about the OpenCV library through which its application can identify
the license plate relatively accurately.
- Building an automatic license plate recognition system.
1.3.Subject scope
- Our report is created to give you step-by-step instructions on how to deploy
a typical ALPR system using pre-trained model Wpod-Net, Computer

Page 6
Digital Image Processing

Vision techniques with OpenCV and Character Recognition with Neural

Network.
- Identify license plates of 10 different countries(Germany, Vietnam, Japan,
Thailand, Saudi, Russia, Korea, USA, India, China).
- Motorcycle license plates are for businesses or individuals

Page 7
Digital Image Processing

2. Theoretical basis
2.1.Theory
- Using pre-trained mode Wpod-Net, Neural Network mode, MobileNets
architecture
2.2.Tools and libraries:
- Python 3.7
- Keras 2.3.1
- Tensorflow 1.14.0
- Jupyter Notebook
- Numpy 1.17.4
- Matplotlib 3.2.1
- OpenCV 4.1.0

Page 8
Digital Image Processing

3. Topic execution
3.1.Methods of implementation
 Introducing the python programming language
- Python is a multi-purpose programming language created in the late 1980s
as an open source language with a feature of syntax that is easy to read and
understand. today, Python is one of the most popular programming
languages in the world besides C #, Java…. Especially in the field of AI and
Machine Learning, python shows its superiority.
 Introduction to OpenCV.
- OpenCV is a leading open source library for computer vision, image
processing and machine learning, and GPU acceleration features in real-time
operations.
 Introduce WPOD-Net
- Pre-trained model named Wpod-Net to detect and extract License Plates of
vehicle
 Introduce CNN
- Convolutional Neural Network (CNNs) is one of the advanced Deep
Learning models. It helps us to build intelligent systems with high accuracy
today.

Page 9
Digital Image Processing

3.2.Block diagram:

The photo Locate and cut the

contains the sea with Wpod-Net
license plate

Plate character
segmentation with
OpenCV

Train a Neural
Network to predict Characters
segmented
characters

3.3.Implementation process
3.3.1. Detection License Plate with Wpod-Net
- Open a new notebook and import all required libraries and packages as
shown below
import
cv2
import numpy as np
import matplotlib.pyplot as plt
from local_utils import detect_lp
from os.path import splitext, basename
from keras.models import
model_from_json
import glob
%matplotlib.inline

- We will need to load our model from the pre-trained one.

Def load_model(path):
try:

Page 10
Digital Image Processing

path = splitext(path)[0]
with open('%s.json' % path, 'r')
as json_file:
model_json = json_file.read()
model =
model_from_json(model_json,
custom_objects={})
model.load_weights('%s.h5' % path)
print("Loading model
successfully...")
return model
except Exception as e:
print(e)

wpod_net_path = "wpod-net.json"
wpod_net = load_model(wpod_net_path)

- Subsequently, we need to create a function name preprocess_image to read

and pre-process our plate images. This function basically reads the parsing
image , converts it to RGB and normalizes the image data to 0–1 range so it
can be compatible with matplotlib. Additionally, we can set resize = True to
resize all images to same dimension of (width = 224, height = 224) for
visualizing purpose in the next step.

- Def preprocess_image(image_path,resize=False):
img = cv2.imread(image_path)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img = img / 255
if resize:
img = cv2.resize(img, (224,224))
return img

Page 11
Digital Image Processing

- A function named get_plate which processes the raw image, send it to our

model and return the plate image (LpImg) and its coordinates (cor). If there
is no plate founded, the program would warn you with an error “No License
plate is founded!”. In this case, try to increase Dmin value to adjust the
boundary dimension. It should be mentioned that Wpod-Net is able to detect
only standard plates(black character, white background), and if the image is
too blurry or the plate is hidden by obstacles, the model might fail to predict
it.
def get_plate(image_path, Dmax=608, Dmin=256):
vehicle = preprocess_image(image_path)
ratio = float(max(vehicle.shape[:2])) /
min(vehicle.shape[:2])
side = int(ratio * Dmin)
bound_dim = min(side, Dmax)
_ , LpImg, _, cor = detect_lp(wpod_net,
vehicle, bound_dim, lp_threshold=0.5)
return LpImg, cor

# Obtain plate image and its coordinates from an

image
test_image = image_paths[0]
LpImg,cor = get_plate(test_image)
print("Detect %i plate(s)
in"%len(LpImg),splitext(basename(test_image))
[0])
print("Coordinate of plate(s) in image: \n",
cor)

# Visualize our result

plt.figure(figsize=(10,5))
plt.subplot(1,2,1)
plt.axis(False)
plt.imshow(preprocess_image(test_image))

Page 12
Digital Image Processing

plt.subplot(1,2,2)
plt.axis(False)
plt.imshow(LpImg[0])

3.3.2. Plate character segmentation

3.3.2.1. Image processing

- We will applies several processing techniques to reduce noise

and emphasize key features of license characters.
- The techniques we will use is shown below:
- Convert to 255 scale. Extracted license image from Wpod-Net
is interpreted as 0–1 scale, thus we need to convert it to 8-bit
scale as standard image.
- Convert to grayscale. Color plays an negligible role to
understand the license plate, thus we can remove it to optimize
computational power.
- Blur image. Blur technique is performed to remove noise and
irrelevant information. In the example code, I used Gaussian
Blur with kernel size of (7,7) but this value can be tuned
depending on your image. The higher of the kernel size, the less
noise but more information is lost.
- Image thresholding. We set a threshold so that any smaller
pixel value than it would be converted to 255 and vice versa.
This type of thresholding is called inverse binary

Page 13
Digital Image Processing

thresholding . In the example code, I used threshold value of

180, this value can be modified to be more compatible with
your image.
- Dilation. This is a technique to increase the white region of the
image. By implementing dilation, we want to enhance the white
contour of each character,
- Code for this part is demonstrated as below:

if (len(LpImg)): #check if there is at least one

license image
plate_image = cv2.convertScaleAbs(LpImg[0],
alpha=(255.0))

# convert to grayscale and blur the image

gray = cv2.cvtColor(plate_image,
cv2.COLOR_BGR2GRAY)
blur = cv2.GaussianBlur(gray,(7,7),0)

# Applied inversed thresh_binary

binary = cv2.threshold(blur, 180, 255,
cv2.THRESH_BINARY_INV +
cv2.THRESH_OTSU)[1]
## Applied dilation
kernel3 =
cv2.getStructuringElement(cv2.MORPH_RECT, (3, 3))
thre_mor = cv2.morphologyEx(binary,
cv2.MORPH_DILATE, kernel3)

Page 14
Digital Image Processing

3.3.2.2. Determine contour of License characters

-  We will use find Contours function of OpenCV to identify the
coordinates of license character. This function is based on a
simple theory: contours is simply curve joining all continuous
point (along the boundary) sharing the same color and intensity.
I created a function called sort_contours which basically sorts
founded contours from left to right. This is essential since we
want not only to recognize the character but also arrange them
in correct order. As can be seen in the example code as below,
the ratio, is equal to the height divided by the width of the
contour. Since we know that our license character usually have
greater height than width, we can filter the irrelevant contours
by select only contours with ratio between 1 to 3.5 . We also
know that each license character should have its height greater
than half of plate height, thus we can add another contour
filter. Later, we draw bounding with all contours that passes
these filters, applies binary thresholding on each determined
contours and append them to list crop_characters
- This is will be shown below:

Page 15
Digital Image Processing

# Create sort_contours() function to grab the contour of each

digit from left to right
def sort_contours(cnts,reverse = False):
i = 0
boundingBoxes = [cv2.boundingRect(c) for c in cnts]
(cnts, boundingBoxes) = zip(*sorted(zip(cnts,
boundingBoxes),
key=lambda b: b[1][i],
reverse=reverse))
return cnts

cont, _ = cv2.findContours(binary, cv2.RETR_EXTERNAL,

cv2.CHAIN_APPROX_SIMPLE)

# creat a copy version "test_roi" of plat_image to draw

bounding box
test_roi = plate_image.copy()

# Initialize a list which will be used to append charater image

crop_characters = []

# define standard width and height of character

digit_w, digit_h = 30, 60

for c in sort_contours(cont):
(x, y, w, h) = cv2.boundingRect(c)
ratio = h/w
if 1<=ratio<=3.5: # Only select contour with defined ratio
if h/plate_image.shape[0]>=0.5: # Select contour which
has the height larger than 50% of the plate
# Draw bounding box arroung digit number
cv2.rectangle(test_roi, (x, y), (x + w, y + h), (0,
255,0), 2)

Page 16
Digital Image Processing

# Sperate number and gibe prediction

curr_num = thre_mor[y:y+h,x:x+w]
curr_num = cv2.resize(curr_num, dsize=(digit_w,
digit_h))
_, curr_num = cv2.threshold(curr_num, 220, 255,
cv2.THRESH_BINARY + cv2.THRESH_OTSU)
crop_characters.append(curr_num)

print("Detect {} letters...".format(len(crop_characters)))

3.3.2.3. Visualize our segmented characters

- We already have all segmented characters storing
in crop_characters. We can visualize the with matplotlib using
the example code as below

fig = plt.figure(figsize=(14,4))
grid =
gridspec.GridSpec(ncols=len(crop_characters),nrows=1,figure=fig
)

for i in range(len(crop_characters)):
fig.add_subplot(grid[i])

Page 17
Digital Image Processing

plt.axis(False)
plt.imshow(crop_characters[i],cmap="gray")

The result:

3.3.3. Recognize plate license characters with OpenCV and Deep

Learning
3.3.3.1. Dataset
- The dataset contain of 34575 images divided into 36 classes of
characters (A->Z) (0->9).  The below snippet code was used to
visualize our data:
import
glob
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
import numpy as np

dataset_paths = glob.glob("dataset_characters/**/*.jpg")

cols=4
rows=3
fig = plt.figure(figsize=(10,8))
plt.rcParams.update({"font.size":14})
grid = gridspec.GridSpec(ncols=cols,nrows=rows,figure=fig)

# create a random list of images will be displayed

np.random.seed(45)

Page 18
Digital Image Processing

rand =
np.random.randint(0,len(dataset_paths),size=(cols*rows))

# Plot example images

for i in range(cols*rows):
fig.add_subplot(grid[i])
image = load_img(dataset_paths[rand[i]])
label = dataset_paths[rand[i]].split(os.path.sep)[-2]
plt.title('"{:s}"'.format(label))
plt.axis(False)
plt.imshow(image)

3.3.3.2. Data Pre-processing

- We need to conduct several data pre-processing techniques as:

Page 19
Digital Image Processing

- Arrange input data and their corresponding labels. The original

size of input image for MobileNets is 224x224, however, since
COLAB kept crashing when I tried to stack the input data, I
reduce the image size to 80x80 as you can code below. This
configuration managed to achieve around 98% accuracy, but
still, you can try to play with larger image size if your computer
is powerful enough.
X=[]
labels=[]

for image_path in dataset_paths:

label = image_path.split(os.path.sep)[-2]

image=load_img(image_path,target_size=(80,80))
image=img_to_array(image)

X.append(image)
labels.append(label)

X = np.array(X,dtype="float16")
labels = np.array(labels)

- Convert our labels as 1D-array to one-hot encoding labels. This gives us

better representation among our labels. The label classes record
need to be save locally, so it can be used latter for inverse
transformation.
lb.fit(labels)
labels = lb.transform(labels)
y = to_categorical(labels)

# save label file so we can use in another

script
np.save('license_character_classes.npy',
lb.classes_)

Page 20
Digital Image Processing

- Split our dataset into training set (90%) and validation set
(10%). This allows us monitoring our model accuracy and
avoiding overfitting.
(trainX, testX, trainY, testY) =
train_test_split(X, y, test_size=0.10, stratify=y,
random_state=42)

- Create a data augmentation method with some basic transform

techniques such as rotating, shifting, zooming, etc. Be careful
that you don’t overused these techniques, cause a vertically
flipping number “6” would give us number “9” — a totally
different one.

image_gen =
ImageDataGenerator(rota
tion_range=10,
width_shift_range=0.1,

height_shift_range=0.1,

shear_range=0.1,

zoom_range=0.1,

fill_mode="nearest"

3.3.3.3. Initialize MobileNets architecture with pre-trained weights

Page 21
Digital Image Processing

- We construct our MobileNets architecture with pre-trained

weights on imagenet dataset. The model can be imported
directly from Keras Application package. There are several
crucial elements that require to be noticed here:
- Discard the last output layers of default MobileNets
architecture and replace it with our desired output layer. Out
output layer would contains 36 nodes associating to 36
characters. Recall that the shape of our input images are
(80,80,3), we need to config our input layer as the same
dimension
- If training=True, defining all layers in base_model as trainable
layers (weight can be updated during training); initializing
learning rate and decay value; and compiling our model with
losses and metrics as categorical cross entropy and accuracy
respectively

def create_model(lr=1e-4,decay=1e-4/25,
training=False,output_shape=y.shape[1]):
baseModel = MobileNetV2(weights="imagenet",
include_top=False,
input_tensor=Input(shape=(80, 80,
3)))

headModel = baseModel.output
headModel = AveragePooling2D(pool_size=(3, 3))(headModel)
headModel = Flatten(name="flatten")(headModel)
headModel = Dense(128, activation="relu")(headModel)
headModel = Dropout(0.5)(headModel)
headModel = Dense(output_shape, activation="softmax")
(headModel)

model = Model(inputs=baseModel.input, outputs=headModel)

if training:
# define trainable lalyer
for layer in baseModel.layers:
layer.trainable = True

Page 22
Digital Image Processing

# compile model
optimizer = Adam(lr=lr, decay = decay)
model.compile(loss="categorical_crossentropy",
optimizer=optimizer,metrics=["accuracy"])

return model

# initilaize initial hyperparameter

INIT_LR = 1e-4
EPOCHS = 5

model = create_model(lr=INIT_LR,
decay=INIT_LR/EPOCHS,training=True)

3.3.3.4. Train and evaluate model

- We can start training our model. Depending on your system
specification, you can either increase or decrease your
BATCH_SIZE value. The trade-off between small and large
batch is explained very clearly in this post. Basically, larger
BATCH_SIZE means less samples are trained each epoch,
which might decline your model performance but reduce the
computational expense.
- In an attempt to avoid time-wasting of insufficient training, I
implement some callbacks function to monitor the
metric. EarlyStopping would stop training process if val_loss
doesn’t improve after 5 epochs, while ModelCheckpoint
save weight of our model every time val_loss is improved.
The model acquired ~95% accuracy after 5 epochs and as the
end of training process, it achieved ~99% accuracy (Fig. 3) on
validation set.
BATCH_SIZE = 64
my_checkpointer = [

EarlyStopping(monitor='val_loss', patience=5,

Page 23
Digital Image Processing

verbose=0),

ModelCheckpoint(filepath="License_character_recognitio
n.h5", verbose=1, save_weights_only=True)
]

result = model.fit(image_gen.flow(trainX, trainY,

batch_size=BATCH_SIZE),
steps_per_epoch=len(trainX) //
BATCH_SIZE,
validation_data=(testX, testY),
validation_steps=len(testX) //
BATCH_SIZE,
epochs=EPOCHS,
callbacks=my_checkpointer)

Page 24
Digital Image Processing

- Finally, model architecture need to be save so later we do not have to

construct our model architecture from scratch. This act can be done with
the snippet code as below

# save model architectur as json file

model_json = model.to_json()
with open("MobileNets_character_recognition.json", "w") as json_file:
json_file.write(model_json)

3.3.4. Combining everything

-  To summarize, we manged to extract the plate license from vehicle image,
segment its character and train a CNN model to predict characters from
images. To deliver prediction, first we would need to rebuild our model
architecture, load saved weight after training phase and origin label classes.

# Load model architecture, weight and labels

json_file = open('MobileNets_character_recognition.json',
'r')
loaded_model_json = json_file.read()
json_file.close()
model = model_from_json(loaded_model_json)
model.load_weights("License_character_recognition_weight.h5")

Page 25
Digital Image Processing

print("[INFO] Model loaded successfully...")

labels = LabelEncoder()
labels.classes_ = np.load('license_character_classes.npy')
print("[INFO] Labels loaded successfully...")

- Remember that we configure the input layer of our model to receive input
images with the shape of (80,80,3). Thus we would need to convert our
image to the appropriate size and correct channel .We implement the loaded
label classes to inverse one-hot encoding label obtained from model to
digital characters. A loop is generated over each character image
in crop_characters, which stores all predictions from model in to
final_result, and plot each image with its corresponding predictions as
shown as Fig. 4.

# pre-processing input images and pedict with model

def predict_from_model(image,model,labels):
image = cv2.resize(image,(80,80))
image = np.stack((image,)*3, axis=-1)
prediction =
labels.inverse_transform([np.argmax(model.predict(image[np
.newaxis,:]))])
return prediction

fig = plt.figure(figsize=(15,3))
cols = len(crop_characters)
grid = gridspec.GridSpec(ncols=cols,nrows=1,figure=fig)

final_string = ''
for i,character in enumerate(crop_characters):
fig.add_subplot(grid[i])
title =
np.array2string(predict_from_model(character,model,labels)

Page 26
Digital Image Processing

)
plt.title('{}'.format(title.strip("'[]"),fontsize=20))
final_string+=title.strip("'[]")
plt.axis(False)
plt.imshow(character,cmap='gray')

print("Achieved result: ", final_string)

#plt.savefig('final_result.png', dpi=300)

4. Design the theme

- Overall of the program

Page 27
Digital Image Processing

- Browse the right image file (click the browse image button)

- If you don’t choose the image, the program will raise a warning message
box

Page 28
Digital Image Processing

- Get the plate image and give prediction on it (Click on the get result button)
- If the image quality is good and detect contour correctly it will give out the
result like this

Page 29
Digital Image Processing

- The get result has 4 processes:

- The first one is getting the plate
- The second is plotting the image plate under many different filters
- The third is detecting the letter by drawing the bounding box around the
digit letter and plot it using matplotlib.
- Last one is loading the machine model and use that to give a predict base on
the plot image.

5. Conclusion and development direction

5.1.Conclude:
5.1.1. Result:
- In general, the group has achieved the initial goals of completing the topic of
identifying license plates and learning about Python and OpenCV
programming languages. improve in the next execution.
5.1.2. Limitations:
- The first limitation: the code is still too long to be optimized to be shorter,
which makes it difficult to read and understand, programming knowledge is
not well versed, so there are still many errors in programming, and cannot
save RAM and CPU. Getting program errors when there is a problem
missing
- Second limitation: recognizing the plate is still not good, because there are
many types of plates with different sizes, we need to adjust Dmin and Dmax
the boundary dimension of different license plates.
- The third limitation: can not recognize characters with the number plates that
are too blurred and have a lot of noise.
5.2.Development:
- Need to collect more photos to train the model in many different
environmental conditions
- Programming knowledge should be enriched. Refer to the programmers
Other good at finding errors for the program and improving processing
speed.
Page 30
Digital Image Processing

- From the topic we can apply it directly to the identification of number plates
in parking lots.
- If it can be developed, it can be applied in the field of artificial intelligence
such as robot reading. Or develop into a character detector to assist the deaf,
blind ....

Page 31
Digital Image Processing

References
[1] Big exercise in image processing: identifying license plates - Ho Chi Minh City
University of Technology
https://www.123doc.net/document/5196410-bai-tap-lon-xu-ly-anh-nhan-dien-bien-
so-xe-dai-hoc-bach-khoa-tphcm.htm
[2] https://medium.com/@quangnhatnguyenle/detect-and-recognize-vehicles-
license-plate-with-machine-learning-and-python-part-1-detection-795fda47e922
[3] https://medium.com/@quangnhatnguyenle/detect-and-recognize-vehicles-
license-plate-with-machine-learning-and-python-part-2-plate-de644de9849f
[4] https://medium.com/@quangnhatnguyenle/detect-and-recognize-vehicles-
license-plate-with-machine-learning-and-python-part-3-recognize-be2eca1a9f12

Page 32