2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC)

Windsor Oceanico Hotel, Rio de Janeiro, Brazil, November 1-4, 2016

LICENSE PLATE RECOGNITION BASED ON TEMPORAL REDUNDANCY

Gabriel Resende Gonçalves1 , David Menotti2 , William Robson Schwartz1

1
Smart Surveillance Interest Group, Computer Science Department,
Universidade Federal de Minas Gerais, Minas Gerais, Brazil
2
Department of Informatics, Universidade Federal do Paraná, Paraná, Brazil

ABSTRACT computers with high processing power, usually provided by GPU

Recognition of vehicle license plates is an important task in several cards. In addition, they need huge set of examples for training.
real applications. Most approaches first detect a vehicle, locate Therefore, we decided to not utilize Deep Learning approaches in
the license plate and then recognize its characters. However, the our ALPR system.
focus relies on performing these tasks using only a single frame This work proposes a temporal redundancy approach to perform
of each vehicle in the video. Therefore, such approaches might ALPR based on multiple frames instead of selecting only a single
have their recognition rates reduced due to noise present in that frame (see Figure 1) that can be executed in real-time. Whereas
particular frame. Instead of selecting a single frame to perform the redundancy aggregation is a well-known technique in the machine
recognition, we propose a novel real-time approach to automatically learning community, to the best of our knowledge, this is the first
detect the vehicle and identify (locate/recognize) its license plate time it is applied to improve results of an ALPR pipeline. We
based on temporal redundancy information. To achieve further also develop two post-processing steps to improve the results of
improvements, we also propose two post-processing techniques the recognition/identification considering that there is a database of
by querying a license plate database. The experimental results, registered license plates and vehicle models. The first is based on
performed in a dataset composed of 300 on-track vehicles acquired vehicle appearance classification (VAC) and the second is based on
on an urban road, demonstrate that it is possible to improve a search tree containing valid license plates. Finally, we introduce
the vehicle recognition rate in 15.3 percentage points using our a public dataset of vehicles classified/labeled according to their
proposal temporal redundancy approach. Additional 7.8 percentage appearance.
points are achieved by querying registered license plates on a The main contributions of this work can be pointed as follows:
database by the vehicle appearance, leading to a final recognition • a new real-time framework to perform ALPR using spatio-
rate of 89.6%. Furthermore, the technique is able to process 34 temporal information;
frames per second, which characterizes it as a real-time approach. • two post-processing techniques to improve the final accuracy

Index Terms— automatic license plate recognition, vehicle clas- of the ALPR system;
• a public dataset of vehicles classified/labeled according to
sification, novel dataset, computer vision, machine learning
their appearance.
I. I NTRODUCTION Our experiments were performed using a novel dataset composed
Recognition of an on-road vehicle using its license plate is of 5, 200 samples of 300 on-track vehicles acquired on an urban
an important task performed by several intelligent transportation road in Brazil. The results demonstrate an improvement of around
systems around the world. This task is known as Automatic License 15 percentage points in recognition rate when temporal redundancy
Plate Recognition (ALPR) and plays an important role in many information, considering the vehicle tracking is employed. More-
real application scenarios such automatic toll collection, access over, we show that we can achieve an additional increase of 7.8
control in private parking lots, stolen vehicles identification and percentage points when we correct the ALPR results using post-
traffic surveillance. Recently new approaches have been proposed to processing steps, leading to a final recognition rate of 89.6%, in
perform ALPR in an efficient way [1], [2]. However, we believe that contrast to 66.3%, achieved by the baseline approach.
there are still many problems that can be explored using modern The remainder of this paper is organized as follows. Section II
techniques, e.g., simultaneous recognition of multiple vehicles and reviews works related to the goal of this paper. In Section III,
vehicle recognition in low-light environments and in high speed the proposed method using spatial-temporal approach to perform
highways with low quality samples. the ALPR is presented as well as post-processing steps. Then,
ALPR approaches are commonly subdivided into multiple Section IV presents the experiments conducted to evaluate the
smaller and simpler tasks that are executed sequentially [3]: (i) proposed approach and the achieved results are described and
image acquisition; (ii) vehicle location; (iii) license plate detection; discussed. Finally, Section V concludes this work and discusses
(iv) character segmentation; and (v) optical character recognition perspectives for future works.
(OCR). Although some approaches perform vehicle tracking [4],
II. R ELATED W ORK
[5], [6], they do not use all captured information to recognize the
characters. Instead, they select only a single frame to perform the In this section, we present a brief literature review describing
recognition, based on some defined rule [6], [7], making the method some ALPR-related works. The outlined papers are divided in the
more sensitive to noise and prone to recognition errors. following groups: vehicle detection, license plate detection, license
In this work, we propose an approach to perform ALPR in real- plate character segmentation, optical character recognition, full
time. One of our main concerns is to avoid the need to embedding ALPR pipeline and works addressing other car-related problems.
high-cost computers on the highways. This could make the system The preliminary tasks performed in ALPR are vehicle and
unfeasible to be employed in the real-world applications. Although license plate detection, which are usually solved using connected
there are some works in the literature providing outstanding results components labeling (CCL) [11], [12], template matching [13],
in computer vision tasks using techniques based on deep learn- background separation [14], and more often machine learning
ing [8], [9], [10], these are too computationally expensive and need techniques [15], [16]. In the latter approach, a window is slided

978-1-5090-1889-5/16/$31.00 ©2016 IEEE 2577

on the image and classified as whether containing or not a license vehicle license plate in static images by sliding window approach to
plate (or a vehicle) according to feature descriptors extracted from detect the license plate, to counting the vertical points in the license
each image location. plate region to segment the characters and to recognize them using
The work described in Sivaraman and Trivedi [17] compares a self-organizing neural network. Bremananth et al. [6] proposed
three methods to perform vehicle detection using active learning. a technique to select the best frame to recognize the license plate
Furthermore, there are others important works in the field such as using a SVM trained to identify the less blurred frame. They also
the one by Chen et al. [18] that proposes a new system to perform segment the characters using histogram analysis and recognize them
night time vehicle detection and by Kembhavi et al. [19] which using an OCR based on template matching.
proposes an approach to detect vehicles on aerial cameras using Nowadays, many researchers have turned their attention to
Partial Least Squares. vehicle-related problems, such as vehicle classification according to
Since the license plate images might contain artifacts such a set of characteristics, other than license plate identification [27],
as skew transform, shadows and blurring, generated during the [29], [30], [31], [32]. Shin and Wang [27] proposed a technique
image acquisition process, one of the most challenging tasks in to recognize the vehicle using its appearance instead of the license
ALPR is the character segmentation [20], [21], [22]. Araújo et plate characters. Dong et al. [29] presented an approach to classify
al. [20] proposed a technique to segment characters using CCL vehicles into six categories: bus, microbus, minivan, SUV, sedan
and showed that the OCR results are greatly affected by the and truck. Their approach was able to achieve 91.6% of recognition
character segmentation step. For instance, while they achieved rate using a dataset with 227 images of eight different vehicle
recognition rates of 95.59% for manually segmented license plates, models. Hsieh et al. [30] and Hu et al. [9] propose approaches
only 71.15% was obtained when automatic segmentation was to classify vehicles according to its color. The former applies a
performed. Such behavior is corroborated by our current work correction to reduce the effect of the lightning change, and the
(see Table I). The approach proposed by Soumya et al. [21] latter utilizes a deep learning technique for this purpose.
performed character segmentation by counting the black pixels in Recently, Yang et al. [32] introduced a new large dataset called
the horizontal and vertical direction of each license plate region. CompCars containing 136, 727 car images to be used on car-related
Finally, Wang et al. [22] employed a sequence of techniques to problems. The authors argue that there are still many problems that
improve the segmentation based on vertical projection and a A* have not been well-explored by the research community, i.e. fine-
pathfinding algorithm. grained classification and vehicle attribute prediction. In this sense,
The last step of the ALPR performs optical character recognition our work proposes a new approach to recognize the vehicle frontal
(OCR) to identify each letter and digit composing the license appearance and perform a query on a dataset to reduce the domain
plate. Note that in the license plate recognition scenario, an OCR of possible license plates. In contrast, Duan et al. [33] employed
approach has to work as close as possible to the optimality (100% Scale Invariant Transform Feature (SIFT) and color histograms to
of recognition rate) since a single mistake may imply in an incorrect identify characteristics of the vehicles (e.g., headlight, tire color
identification of the vehicle. To achieve this goal, there are works and wheel shape), which are used then to classify the vehicles
in the literature that produce outstanding results using artificial according to their model. In this work, we utilize frontal appearance
neural networks techniques [1], [11], [23] In addition, when there is to classify vehicles and improve the recognition results at the end.
prior knowledge of a specific license plate layout, the lexicon size We also propose a novel approach to filter out unlikely license plate
can be diminished (for instance, plates with 4 letters and 5 digits candidates using a simple algorithm based on a tree search.
in a sequence) and the classification accuracy can be improved.
However, although most works utilizes learning-based techniques, III. P ROPOSED A PPROACH
there are also works producing promising results that use template
matching to perform it [20], [24]. This section describes the proposed improvements for license
plate recognition. First, we briefly overview the ALPR pipeline
Note that in the license plate recognition scenario, an OCR has
(Section III-A) being employed in this work. Then, we define
to work as close as possible to the optimality (100% of recognition
the proposed temporal redundancy aggregation (Section III-B)
rate) since a single mistake in the character recognition task may
and the two post-processing techniques based on the assumption
imply in an incorrect identification of the vehicle.
that we have access to a database containing all issued license
Although many works propose approaches to solve only a single plates (Sections III-C and III-D), e.g., a database of a Department
subtask at a time, there are also works proposing techniques to of Motor Vehicle. Figure 1 illustrates the recognition pipeline,
perform the entire ALPR pipeline [1], [2], [6], [25], [26], [27], [28]. described in the next sections.
The work proposed by Guo and Liu [25] detects the license plate
using template matching, segments the characters using vertical and
A. ALPR Pipeline
horizontal projections and recognizes Dutch license plates using
Hotelling transform and Euclidean distance. The approach pre- Vehicle and license plate detection are crucial tasks on ALPR
sented in Donoser et al. [28] utilizes analysis of Maximally Stable system. We first detect the vehicle and then its license plate, located
Extremal Region (MSER) to detect the license plate, track the inside the vehicle patch. To solve both tasks, we employ a sliding
vehicle and segment its characters. The characters are recognized window approach composed of a classifier based on Support Vector
using a SVM-based OCR. They also combine multiple detections Machines (SVM) and Histograms of Oriented Gradient (HOG) [34]
in order to make the recognize robust to noises presented in a single as feature descriptors. Afterwards, we track the vehicles over the
frame. Wang et al. [26] proposed a technique to locate the license multiple frames employing the approach described by Kalal et
plate using horizontal scans of contrast changes, segment the plate al. [35] to group temporally detections belonging to the same
using lateral histogram analysis and recognize the characters using vehicle.
an Artificial Neural Network for Italian license plates. Kocer and Once the license plate has been located, we need to segment
Cevik [2] proposed a work to locate the region of the image with the the image into multiple patches containing license plate characters
most transition points assuming that it corresponds to the license (LPCS). For such aim, we developed a straightforward iterative
plate. The characters are then segmented using a blob coloring technique to perform LPCS on real scenarios. In this approach,
method and the characters are recognized using a multi layered instead of using a single threshold to perform license plate bi-
perceptron. Rao [1] proposed an automatic approach to recognize narization using the Otsu method, we consider a set of different
vehicles in multiple cameras of a surveillance system aiming at values. Otsu’s approach assumes that the pixels of the image
performing the recognition in several points to estimate the vehicle belong to one of the two classes: foreground and background.
path. In addition, the author described a new methods to recognize a Therefore, it calculates the optimal threshold that separate the

2578
 Vehicle
Detect Vehicle
Track License Plate
Track
Video Frames Vehicles Patches Vehicles Tracklets
Detection
Vehicles Tracking
Vehicles Detection
Vehicles

W ACV
WACV -6
- 201
Proposed 2016
Approaches License Plates ABC-1234

Patches
Tree-Based DACV - 201
D ACV
2016
–6
QWE-2016
Query ABC -
1234

Temporal Optical Segment

QWE -
Single Frame Cropped Characters
Track
Vehicle
2016
Redundancy Character Characters
Recognition Characters Segmentation
Vehicles
Appearance Aggregation Recognition
Recognition

Fig. 1: Sequence of tasks performed by the ALPR. The approaches proposed in this work are highlighted in the rectangle.

ABD - 3110

ACC -2111
Fig. 2: Samples of the license plate considering different thresholds, ABC - 2110
5 and 10 on the top images and 20 and 30 on the bottom images.
ABC-1100
classes by minimizing the intra-class variance. For more details,
see Gonzalez [36].
The algorithm assumes that the image contains two classes of TBC - 2110
pixels following bi-modal histogram (foreground pixels and back-
ground pixels), it then calculates the optimum threshold separating
the two classes so that their combined spread (intra-class variance) Fig. 3: The proposed approach combine results of multiple frames
is minimal Starting from a threshold equals 5, we binarize the to improve the vehicle recognition rate.
image as we increase this threshold until we have the number
of connected components equals to the number of license plate
characters. Figure 2 illustrates this process. By doing this, we are classifier score from all recognitions and assumes that the class
trying to avoid the problem where two adjacent characters are with the highest score is the correct. In the case of Support Vector
touching each other due to noise, by some noise pixel. This could Machines, the classifier confidence/score is given by the distance
happen because a binarization starting from small thresholds tends from the projected instance to the separation hyperplane, where
to set most pixels to the maximum value, resulting in fewer noises large and small distances indicates higher and lower confidences,
connecting two adjacent characters. Note that when the threshold respectively. In preliminary experiments, we also evaluated the
is too small, we tend to have more connected components due to use of the Ranking Aggregation technique proposed by Stuart et
sliced characters and when the threshold is too large, we have few al. [37], but the results were not satisfactory.
connected components due to presence of touching characters. C. Vehicle Appearance Classification
The OCR employed is an one-against-all SVM classifier using
HOG features. As a result, we have 36 trained SVMs, one for Once we have the vehicle location in multiple frames, we
each character of the Latin alphabet and one for each digit. It is recognize its appearance, which is used then to query the license
important to note that by knowing the layout of the license plate plate database, and retrieve the license plates belonging to vehicles
beforehand (in our case, it has three letters followed by four digits), with that appearance. The use of vehicle appearance instead of
only the appropriate models can be applied to each character (10 the recognized license plate itself to select candidates can help
SVM models for digits and 26 SVM models for letters), which the ALPR to discard those candidates that have license plates
reduces the incorrect classification. similar to the correct one but belong to different vehicles models.
Therefore, we hypothesize that fewer candidate license plates have
to be evaluated, reducing the ALPR recognition error.
B. Temporal Redundancy Aggregation
The main challenge of this approach is that several vehicles
Since the proposed approach aims at exploring the temporal from the same manufacturer might have the same frontal (or back)
redundancy information, we hypothesize that the combination of appearance, making the distinction of those vehicles a very complex
individual results belonging to the same vehicle should improve task, even for humans (Figure 4 shows two different models that
the recognition of its license plate, as illustrated in Figure 3. have very similar frontal appearance). Thus, we decided to classify
We combine the individual recognition results using two main vehicles according to their appearance instead of their actual model.
approaches: (i) majority voting and (ii) average of the classifier To recognize the vehicle appearance, we employ a standard
confidence. The use of majority voting was already employed in recognition approach using SVM based on SIFT features and Bag
Donoser et al. [28]. While the former takes all predictions for of Visual Words (BoVW) [38]. The only difference between the
each frame and assumes that the most predicted character for conventional approaches and the proposed one is in the feature
every license plate position is the correct, the latter averages the space quantization step of the BoVW. In this work, instead of

2579
Fig. 4: Two different vehicle models presenting very similar frontal
appearance. Voyage (left) vs. Gol (right).
Fig. 5: Sample of a frame in the dataset. Each frame might have
creating a single global dictionary, we build a dictionary per class more than one vehicle.
and append all codewords generating a large BoVW. Although this
approach can generate high-dimensional feature space, it signifi- Table I: Recognition rates achieved by the proposed approach
cantly improves the final recognition rate. Furthermore, since our compared to the baseline using manual and automatic character
approach considers multiple frames of each vehicle, we recognize segmentation. Segmentation
Approach
Manual Automatic
the vehicle appearance for each frame and combine all answers Bremananth et al. [6] (without redundancy) 78.3% 66.3%
using the ranking aggregation technique proposed by Stuart et redundancy with OCR average 93.6% 77.9%
al. [37]. redundancy with majority voting 94.6% 81.8%
D. Tree-Based Query
We propose this technique based on the fact that there are
millions of character combinations that do not correspond to any from surveillance videos with 300 on-road moving vehicles (17.33
(in-use) license plate. For instance, according to the Brazilian frames per vehicle on average) recorded in Brazil. The vehicles
Department of Transportation, there are 87 million different license license plates have size of 120 × 42 pixels and aspect ratio of 2.86
plates currently being used in Brazil1 . However, the combination on average. The third set, used for vehicle classification by appear-
of three letters followed by four numbers provides more than 175 ance, contains 1,000 samples divided in 48 classes corresponding
million possibilities. to an average of 20.83 vehicles per class. Even though we could
Once the license plate has been recognized by the temporal have used the dataset proposed in Yang et al. [32], we chose to
redundancy ALPR, we sort the recognized characters by the OCR collect our own samples due to the fact that all Brazilian vehicles
confidence and, from the most to the least confident character, we used in our experiments must present a corresponding appearance
filter those license plates that do not have that same character on class within our dataset, which is not available in their data.
that particular position. If we find a group having only a single Although we developed our method using images of vehicles
license plate, we assume that this is the correct license plate. with Brazilian license plate models, we can also use the proposed
Otherwise, if we do not have any license plate at some iteration, approach on different models. For this purpose, we only have to
we return one level of the filtering and choose a license plate that train the license plate detector with examples of the new model
is the most likely to the correct one using the OCR confidence. and adapt the LPCS technique to work properly with the model
We implement this technique using a tree. In this case, the root concerned. Furthermore, in case of license plate being located in the
node contains all possible license plates and this amount is reduced vehicle rears, we can also train the vehicle detection to recognize
at every level of the tree until convergence to a single license the back of the vehicle using new appropriate examples.
plate at a leaf node. The edge connecting two nodes represents B. Temporal Redundancy Aggregation
the filtering of the license plate from the parent to the child by a
specific character. However, it is not feasible to generate the entire To evaluate the contribution of employing temporal redundancy
tree due to its high branching factor. Instead, we can use the OCR to the ALPR pipeline, we compare our proposed approach with the
confidence to dynamically build the tree using only the required technique proposed in Bremananth et al. [6]. Their method selects
nodes, ignoring branches with low confidence characters. the best frame using a machine learning technique that classifies
the frame as blurred or non-blurred assuming that the less blurred
IV. E XPERIMENTAL E VALUATION frame is, the most reliable to perform the recognition. We report
the results of our approach using two techniques to combine the
This section presents the results achieved using the technique results: majority voting and average OCR confidence. Furthermore,
described in Section III. We use an approach to recognize vehicle we perform both automatic and manual segmentation to evaluate
using a single frame per vehicle as baseline to evaluate the the influence of the character segmentation on the final recognition
improvement achieved by the addition of redundancy. Furthermore, results.
this section presents the results achieved when we employ the post- According to the results shown in Table I, the proposed approach
processing techniques to perform vehicle appearance classification using automatic segmentation was able to outperform the baseline
and the tree-search. in 11.6 percentage points (p.p.) using average OCR confidence
A. Datasets and 15.5 p.p. using majority voting. This fact corroborates the
hypothesis that combining the results of multiple vehicle detections
We collected three sets of data to validate the proposed ap- can provide better recognition rates than using just a single frame.
proaches2 . The first set, used to train vehicle and license plate
detectors, contains 650 images of on-road vehicles used as positive C. Post Processing Approaches
examples to both detectors. Figure 5 shows an example frame of Once the best results were achieved using majority voting, we
this dataset. The second set, used to evaluate the entire pipeline, utilize the results of this approach as input to both post-processing
contains 5,200 frames, with size of 1920 × 1080 pixels, extracted techniques.
1) Vehicle Appearance Classification
1 http://www.denatran.gov.br/frota2015.htm (in Portuguese) To evaluate our vehicle appearance classification model, we
2 The data used to validate the proposed methods will be made publicly employed a 5-fold cross-validation in the third set of images
available to the research community once the paper is accepted. described earlier.

2580
 0 e
1 e

019
049
018
ekeeeeeeeeeeeekeeeekeeeekeeeekekkee

000esesesesss

keeeekeeeekekkee
017 000esesesesss 048
600esesesesss
900esesesesss
016
0000esesesesss 047
0500esesesesss
015

046
010

010
045
010
044 e
010 1 3 5 10 15 00
kkkk
0e
5 00 05 00 05
kkkk
00 05 00 05 Fig. 7: Percentage of license plates correctly recognized as a
function of the amount of license plates evaluated according to
Fig. 6: Recognition rates as a function of the top rank positions. rank.

Bremananth et al. [6], does not provide such high recognition rate
Figure 6 illustrates the achieved results of the proposed classifier as the combination of all images of the same vehicle does. Further-
model for different number of codewords per class. It is possible to more, although the results using manual (i.e., perfect) segmentation
see that there are no improvements on the classification when we (Table I) are only theoretical, it is worth noticing the impact of
use more than 900 visual words per class (final dimensionality of segmentation on the ALPR system. A manual segmentation can
43, 200). In the best case, the model was capable of predicting cor- improve the results by 12.8p.p. using majority voting and 15.7
rectly around 48% of the test vehicle images in rank-1. Nonetheless, p.p. using average OCR confidence, reaching a recognition rate of
the model returned the correct class in 80% and 91% of the cases 94.6%.
using ranking 10 and 20, respectively. Therefore, the use of ranks Focusing on real-world applications, we evaluated the computa-
higher than 1 can reduce the search space significantly without tional cost of our approach in a Linux Ubuntu 14.04 with 32GB
degrading much the recognition rate. The model was capable to of RAM and a Intel(R) Xeon(R) X5670 CPU. All techniques were
recognize all license plates only using the first 35 classes in the implemented using the C++ language supported by the OpenCV 3.0
best case. library. Our implementation in this scenario achieved a processing
We performed an experiment varying the rank of classes used to rate of 34 frames per second, i.e. the realization of a real-time
predict the license plate, using 900 codewords per class. According system.
to the results shown in Figure 7, the approach achieved 88.9% of Both post-processing approaches were able to improve the
recognition rate using the top 10 classes, which is an improvement results of the temporal redundancy approach by querying a dataset
of 7.1 p.p. compared to the original proposed ALPR approach, as of all possible license plates. It is important to point out that the
shown in the third row of Table I (81.8%). This supports the claim vehicle appearance classification is computationally expensive due
that classifying a vehicle using its appearance and performing a to the high dimensionality of the feature vector. Therefore, it should
query on a database can help to improve the ALPR results. Note be used in systems with high computational processing power,
that the use of more than 10 top predicted classes does not bring otherwise, it may compromise the ALPR system, once such system
significant improvements to the classification. should be able to run in real-time. Furthermore, when we combined
2) Tree-Based Query both approaches, we observed a gain of 7.8 p.p. compared to the
To execute the experiment using the tree-based approach, we proposed temporal redundancy approach, which is a significant
generate a database containing 80 million random license plates improvement and justifies the combined use of both post-processing
to simulate a real vehicles scenario. The approach was capable of approaches.
improving the results obtained using only the temporal redundancy
information in 4.8 percentage points, leading to a recognition rate V. C ONCLUSIONS
of 86.2%. This demonstrates that, once we have access to the In this work, we proposed a new approach to perform real-time
database of all registered vehicles (i.e. the Department of Motor ALPR exploring temporal redundancy information from detected
Vehicles database), we can correct erroneous recognitions, even vehicles. We also proposed two post-processing techniques to
when this database is very large. improve the final recognition accuracy of the ALPR pipeline by
querying a license plate database. The former approach classifies
D. Combined Results the vehicle according to its appearance and verifies whether the
In this last experiment, we combine the three approaches pro- recognized plate corresponds to a valid license plate of a vehicle
posed in this paper. To the best of our knowledge, this is the first with that appearance. The latter performs a tree-based search on
work applying temporal redundancy aggregation to recognize on- the database to verify whether the recognized license plate is valid
track vehicles. First, we recognized the vehicle combining multiple or not. Both approaches can be used by an agent/system that
frames employing the temporal redundancy approach. Then, we has access to the enrolled vehicles (their license plates) in the
performed the vehicle appearance classification to filter those scenario, e.g., the Department of Motor Vehicle of a country/state.
candidate license plates to be used in the next step. Finally, we We demonstrated that we can improve the results by 15.5 p.p. using
executed the tree-based query approach in the set of license plates multiple frames to identify the vehicle. In addition, we showed that
filtered by the VAC model. The combined approach achieved a it is possible to achieve 89.6% of recognition rate using the both
recognition rate of 89.6%, an increase of 7.8 p.p. compared to the post-processing proposed approaches. As future directions, we plan
results obtained considering only temporal redundancy. to employ a Vehicle Model Classification trained with more classes
and a larger dataset to make the filtering process more effective.
E. Discussion
The proposed temporal redundancy aggregation approach was ACKNOWLEDGMENTS
able to significantly outperform the baseline. One can observe The authors would like to thank the Minas Gerais Research
that the use of the most reliable frame, approach proposed by Foundation – FAPEMIG (Grants APQ-00567-14 and PPM-00025-

2581
15), the Coordination for the Improvement of Higher Education surveillance,” Industrial Electronics, IEEE Transactions on,
Personnel – CAPES (DeepEyes Project) and the National Council pp. 2030–2044, 2011.
for Scientific and Technological Development (Grant 307010/2014- [19] A. Kembhavi, D. Harwood, and L. Davis, “Vehicle detection
7). using partial least squares,” TPAMI, pp. 1250–1265, 2011.
VI. R EFERENCES [20] L. Araújo, S. Pio, and D. Menotti, “Segmenting and recog-
nizing license plate characters,” in WUW-SIBGRAPI, 2013,
[1] Y. Rao, “Automatic vehicle recognition in multiple cameras pp. 251–270.
for video surveillance,” The Visual Computer, vol. 31, no. 3, [21] K. Soumya, A. Babu, and L. Therattil, “License plate
pp. 271–280, 2015. detection and character recognition using contour analysis,”
[2] E. Kocer and K. Cevik, “Artificial neural networks based ve- IJATCSE, vol. 3, no. 1, pp. 15–18, 2014.
hicle license plate recognition,” Procedia Computer Science, [22] R. Wang, G. Wang, J. Liu, and J. Tian, “A novel approach
vol. 3, pp. 1033–1037, 2011. for segmentation of touching characters on the license plate,”
[3] S. Du, M. Ibrahim, M. Shehata, and W. Badawy, “Automatic in ICGIP, 2013, pp. 876847–876847.
license plate recognition (ALPR): A state-of-the-art review,” [23] D. Menotti, G. Chiachia, A. Falcão, and V. Oliveira-Neto,
Circuits and Systems for Video Technology, IEEE Transactions “Vehicle license plate recognition with random convolutional
on, vol. 23, no. 2, pp. 311–325, 2013. networks,” in SIBGRAPI, 2014, pp. 298–303.
[4] K. Suresh, M. Kumar, and AN Rajagopalan, “Superresolution [24] T. Shuang-Tong and L. Wen-Ju, “Number and letter character
of license plates in real traffic videos,” ITS, IEEE Transactions recognition of vehicle license plate based on edge hausdorff
on, vol. 8, no. 2, pp. 321–331, 2007. distance,” in PDCAT, 2005, pp. 850–852.
[5] N. Sirikuntamat, S. Satoh, and T. Chalidabhongse, “Vehicle [25] JM Guo and YF Liu, “License plate localization and character
tracking in low hue contrast based on camshift and back- segmentation with feedback self-learning and hybrid binariza-
ground subtraction,” in JCSSE. IEEE, 2015, pp. 58–62. tion techniques,” Vehicular Technology, IEEE Transactions
[6] R. Bremananth, A. Chitra, V. Seetharaman, and V. Nathan, on, vol. 57, no. 3, pp. 1417–1424, 2008.
“A robust video based license plate recognition system,” in [26] ML Wang, YH Liu, BY Liao, YS Lin, and MF Horng,
Intelligent Sensing and Information Processing, 2005. Pro- “A vehicle license plate recognition system based on spa-
ceedings of 2005 International Conference on. IEEE, 2005, tial/frequency domain filtering and neural networks,” in
pp. 175–180. ICCCI, pp. 63–70. Springer, 2010.
[7] V. Oliveira-Neto, G. Cámara-Chávez, and D. Menotti, “To- [27] HC Shih and HY Wang, “Vehicle identification using
wards license plate recognition: Comparying moving objects distance-based appearance model,” in AVSS. IEEE, 2015, pp.
segmentation approaches,” in IPCV, 2012, pp. 447–453. 1–4.
[8] A. Krizhevsky, I. Sutskever, and GE Hinton, “Imagenet [28] Michael Donoser, Clemens Arth, and Horst Bischof, “De-
classification with deep convolutional neural networks,” in tecting, tracking and recognizing license plates,” in Asian
Advances in neural information processing systems, 2012, pp. Conference on Computer Vision. Springer, 2007, pp. 447–456.
1097–1105. [29] Z. Dong, M. Pei, Y. He, T. Liu, Y. Dong, and Y. Jia, “Vehicle
[9] C. Hu, X. Bai, L. Qi, P. Chen, G. Xue, and L. Mei, “Vehicle type classification using unsupervised convolutional neural
color recognition with spatial pyramid deep learning,” Intelli- network,” in ICPR. IEEE, 2014, pp. 172–177.
gent Transportation Systems, IEEE Transactions on, vol. 16, [30] JW Hsieh, LC Chen, SY Chen, DY Chen, S. Alghyaline, and
no. 5, pp. 2925–2934, 2015. HF Chiang, “Vehicle color classification under different light-
[10] Y. Sun, X. Wang, and X. Tang, “Deep learning face rep- ing conditions through color correction,” Sensors Journal,
resentation from predicting 10,000 classes,” in Proceedings IEEE, vol. 15, no. 2, pp. 971–983, 2015.
of the IEEE Conference on Computer Vision and Pattern [31] Y. Tang, C. Zhang, R. Gu, P. Li, and B. Yang, “Vehicle
Recognition, 2014, pp. 1891–1898. detection and recognition for intelligent traffic surveillance
[11] H. Caner, S. Gecim, and A. Alkar, “Efficient embedded system,” Multimedia Tools and Applications, vol. online, pp.
neural-network-based license plate recognition system,” Ve- 1–16, 2015.
hicular Technology, IEEE Transactions on, vol. 57, no. 5, pp. [32] L. Yang, P. Luo, CC Loy, and X. Tang, “A large-scale car
2675–2683, 2008. dataset for fine-grained categorization and verification,” in
[12] Y. Wen, Y. Lu, J. Yan, Z. Zhou, K. von Deneen, and CVPR, 2015, pp. 3973–3981.
P. Shi, “An algorithm for license plate recognition applied [33] K. Duan, L. Marchesotti, and DJ Crandall, “Attribute-based
to intelligent transportation system,” ITS, IEEE Transactions vehicle recognition using viewpoint-aware multiple instance
on, vol. 12, no. 3, pp. 830–845, 2011. svms,” in WACV. IEEE, 2014, pp. 333–338.
[13] M. Betke, E. Haritaoglu, and L. Davis, “Real-time multiple [34] N. Dalal and B. Triggs, “Histograms of Oriented Gradients
vehicle detection and tracking from a moving vehicle,” MVA, for human detection,” in CVPR, 2005, pp. 886–893.
vol. 12, no. 2, pp. 69–83, 2000. [35] Z. Kalal, K. Mikolajczyk, and J. Matas, “Forward-backward
[14] A. Jazayeri, h. Cai, J. Zheng, and M. Tuceryan, “Vehicle error: Automatic detection of tracking failures,” in ICPR.
detection and tracking in car video based on motion model,” IEEE, 2010, pp. 2756–2759.
ITS, IEEE Transactions on, vol. 12, no. 2, pp. 583–595, 2011. [36] R. Gonzalez, Digital image processing, Pearson Education
[15] R. Prates, G. Cámara-Chávez, W. Schwartz, and D. Menotti, India, 2009.
“Brazilian License Plate detection using histogram of oriented [37] J. Stuart, E. Segal, D. Koller, and S. Kim, “A gene-
gradients and sliding windows,” IJCSIT, vol. 5, no. 6, pp. 39– coexpression network for global discovery of conserved ge-
52, 2013. netic modules,” Science, vol. 302, no. 5643, pp. 249–255,
[16] X. Wen, L. Shao, W. Fang, and Y. Xue, “Efficient feature 2003.
selection and classification for vehicle detection,” TCSVT, [38] J. Yang, YG Jiang, AG Hauptmann, and CW Ngo, “Eval-
pp. 508–517, 2015. uating bag-of-visual-words representations in scene classifi-
[17] S. Sivaraman and M. Trivedi, “Active learning for on-road cation,” in Proceedings of the international workshop on
vehicle detection: A comparative study,” MVA, pp. 1–13, Workshop on multimedia information retrieval. ACM, 2007,
2014. pp. 197–206.
[18] YL Chen, BF Wu, HY. Huang, and CJ Fan, “A real-
time vision system for nighttime vehicle detection and traffic

2582