Traffic Sign Detection using Deep Learning

Dr Vinita Tiwari
(Mentor)

Ashutosh Mazumdar Suryansh Kumar Jaiswal Saurav Suman

(2018kuec2069) (2018kuec2064) (2018kuec2053)

Abstract- With the further and faster urbanization, here challenging subject and also a valuable subject in traffic
comes the advent and development of intelligent public engineering research. A variety of traffic-sign
transportation systems. The identification of the traffic signs, as identification technologies have been developed. Deep
a key component of the intelligent transit system, has a
prospect of widespread use in self-driving and driver CNN is trained with a big data set, and then effective
assistance systems.Programmed detection and recognition of regional convolutional neural network (RCNN) detection
traffic signs plays a crucial role in management of the is obtained through a spot of standard traffic training
traffic-sign inventory. It gives precise and ideal approach to
oversee traffic-sign inventory with a minimal human effort. In
examples. Proper management of traffic-sign inventory
the computer vision community the recognition and detection is an important task in ensuring safety and efficiency of
of traffic signs is a well-researched problem.A larger part of the traffic flow. Most often this task is performed anually.
existing methodologies perform well on traffic signs required Traffic signs are captured using a vehicle-mounted
for cutting edge drivers assistance and self-sufficient
frameworks. In this paper, we address the issue of detecting camera and manual localization and recognition is
traffic-sign categories for which we adopt a convolutional performed off-line by a human operator to check for
neural network (CNN) approach, the Mask R-CNN. consistency with the existing database. However, such
Keyword: Deep learning, Traffic-sign detection, Traffic-sign manual work can be extremely timeconsuming when
dataset, Mask R-CNN.
applied to thousands of kilometers of roads. Automating
this task would significantly reduce the amount of
INTRODUCTION manual work and improve safety through quicker
Traffic sign recognition system can be grouped into two detection of damaged or missing traffic signs.
technologies, traffic-sign detection and traffic-sign In this paper, we address the issue of learning and
recognition. The correctness of detection will directly detecting a large number of traffic-sign categories for
lead to the final identification results. Traffic signs road-based trafficsign inventory management. As our
contain necessary messages about vehicle safety and main contribution, we propose a deep-learning-based
they show the latest traffic conditions, define road rights, system for training a large number of traffic-sign
forbid and allow some behaviors and driving routes, cue categories using convolutional neural networks. We
dangerous messages and so on. They can also help base our system on the state-of-the-art detector Mask
drivers identify the condition of the road, so as to RCNN , which demonstrated great accuracy and speed
determine the driving routes. Traffic signs have some in the field of object detection. In contrast to traditional
constant characteristics that can be used for detection approaches with handcrafted features, the convolutional
and classification, among them, color and shape are approach is applied to a broad set of categories, where
important attributes that can help drivers obtain road individual traffic-sign instances are not only subject to
information. The colors used in traffic signs in each change in lighting conditions, scale, viewing angle, blur,
country are almost similar, usually consisting of simple and occlusions, but also to significant intra-category
colors (red, blue, yellow, etc.) and fixed shapes (circles, appearance variations as well as low intercategory
triangles, rectangles, etc.). The image of traffic signs is variations.
often affected by some external factors, such as weather
conditions. Therefore, traffic-sign recognition is a
 LITERATURE SURVEY
An enormous amount of literature exists on the topics
of TSR and TSD, and several review papers are
available. In general, it is very difficult to decide which
approach gives better overall results, mainly due to the
lack of a standard publicly available benchmark dataset
that would contain an extensive set of various traffic-sign
categories, as emphasized in several recent studies.
Some approaches sample images from multiple
datasets to perform the evaluation. On the other hand, a
vast number of authors use their own private datasets.

Journals studied for literature survey

 How Successful is Mask R-CNN?

Various methods have been employed in TSD.

Traditionally hand-crafted features have been used, like ● Mask R-CNN is simple to train and adds
histogram of oriented gradients (HOG), scale invariant only a small overhead to Faster R-CNN,
feature transform (SIFT), local binary patterns (LBP) or running at 5 fps.
integral channel features. A wide range of machine ● Mask R-CNN outperformed all existing,
learning methods have also been employed, ranging single model entries on every task,
from support vector machine (SVM), logistic regression , including the COCO 2016 challenge
and random forests , to artificial neural networks in the winners.
form of an extreme learning machine (ELM). In TSR, ● It gives bounding boxes and segmentation
CNNs have been used to automatically learn feature masks on each and every object leading to
representations as well as to perform the final instance segmentation.
classification. Our method, which is based on Mask
R-CNN, uses region proposal network instead of using a
separate method for generating region proposals, and
we employ deeper networks based on ResNet-101
architecture (ResNet-101 is a convolutional neural Image Segmentation using Mask RCNN
network that is 101 layers deep).
The first step is to take an image and extract features
using the ResNet 101 architecture. These features act
TRAFFIC-SIGN DETECTION WITH MASK R-CNN as an input for the next layer.
Now, we take the feature maps obtained in the previous
We present our system for traffic-sign detection using step and apply a Region Proposal Network (RPN). This
the Mask R-CNN. basically predicts if an object is present in that region or
not. In this step, we get those regions or feature maps
Mask R-CNN which the model predicts contain some object.

The Mask R-CNN network is an extension of Faster The regions obtained from the RPN might be of different
R-CNN , both of which are composed of two modules. shapes. Hence, we apply a pooling layer and convert all
The first module is deep fully convolutional network, a the regions to the same shape.
so-called Region Proposal Network (RPN), that takes an Next, these regions are passed through a fully
input image and produces a set of rectangular object connected network so that the class label and bounding
proposals, each with an objectness score. The second boxes are predicted.
module is a region-based CNN, called Fast R-CNN, that In addition to this, Mask R-CNN also generates the
classifies the proposed regions into a set of predefined segmentation mask.
categories. Fast R-CNN is highly efficient, since it
shares convolutions across individual proposals. It also
performs bounding box regression to further refine the For that, we first compute the region of interest so that
quality of the proposed regions. The entire system is a the computation time can be reduced. For all the
single unified network, in which RPN and Fast R-CNN predicted regions, we compute the Intersection over
are merged by sharing their convolutional features. Union (IoU) with the ground truth boxes. We can
Mask R-CNN improves the system by combining the
computer IoU like this:
underlying network architecture with a Feature Pyramid
Network (FPN). With the FPN, the detector is able to
improve the performance on small objects, since FPN IoU = Area of the intersection / Area of the union
extracts features from lower layers of the network,
before the down-sampling removes important details in
Now, only if the IoU is greater than or equal to
small objects.
0.5, we consider that as a region of interest.
Otherwise, we neglect that particular region.

Once we have the RoIs based on the IoU values, we

can add a mask branch to the existing architecture.
 Conclusion images: The German traffic sign detection benchmark,”
in IJCNN. Ieee, aug 2013, pp. 1–8. 1, 2, 3
Here we have discussed the traffic sign detection [11] A. Mogelmose, M. M. Trivedi, and T. B. Moeslund,
system (TDS) and have stated the various methods “Vision-Based Traffic Sign Detection and Analysis for
used in TDS. We have also done a brief study of the Intelligent Driver Assistance Systems: Perspectives and
Mask R-CNN based on the papers we read in our Survey,” Transactions on Intelligent Transportation
literature survey. Therefore, it can be concluded that Systems, vol. 13, no. 4, pp. 1484–1497, 2012. 1, 2
despite excellent performance of the proposed approach [12] F. Zaklouta and B. Stanciulescu, “Real-time
there is still room for improvement. On the basis of our traffic-sign recognition using tree classifiers,” IEEE
study we can say that the ideal performance is still not Transactions on Intelligent Transportation Systems, vol.
achieved, mostly due to several missed detections that 13, no. 4, pp. 1507–1514, 2012. 1, 3
are being lost by the classification network. Future [13] Z. Zhu, D. Liang, S. Zhang, X. Huang, B. Li, and S.
improvements should focus on improving this part of the Hu, “Traffic-Sign Detection and Classification in the
system. Wild,” in CVPR, 2016, pp. 2110– 2118. 1, 2, 3
[14] H. Kaiming, G. Gkioxara, P. Dollar, and R. Girshick,
“Mask R-CNN,” in International Conference on
References Computer Vision, 2017, pp. 2961–2969. 2, 3, 12
[1] V. Balali, A. Ashouri Rad, and M. Golparvar-Fard, [15] S. B. Wali, M. A. Hannan, A. Hussain, and S. A.
“Detection, classification, and mapping of U.S. traffic Samad, “Comparative Survey on Traffic Sign Detection
signs using google street view images for roadway and Recognition: a Review,” Przeglad Elektrotechniczny,
inventory management,” Visualization in Engineering, vol. 1, no. 12, pp. 40–44, 2015. 2
vol. 3, no. 1, p. 15, 2015. 1 [16] A. Ellahyani, M. E. Aansari, and I. E. Jaafari, “Traffic
[2] K. C. Wang, Z. Hou, and W. Gong, “Automated road Sign Detection and Recognition using Features
sign inventory system based on stereo vision and Combination and Random Forests,” IJACSA, vol. 7, no.
tracking,” Computer-Aided Civil and Infrastructure 1, pp. 6861–6931, 2016. 2, 3
Engineering, vol. 25, no. 6, pp. 468–477, 2010. 1 [17] R. Timofte, K. Zimmermann, and L. V. Gool,
[3] V. Balali and M. Golparvar-Fard, “Evaluation of “Multi-view traffic sign detection, recognition, and 3D
Multiclass Traffic Sign Detection and Classification localisation,” in WACV, 2009, pp. 1–8. 2, 5
Methods for U.S. Roadway Asset Inventory [18] S. Segvic and K. Brkic, “A computer vision assisted
Management,” Journal of Computing in Civil geoinformation inventory for traffic infrastructure,” in 13th
Engineering, vol. 30, no. 2, p. 04015022, 2016. 1 International IEEE Conference on Intelligent
[4] J. M. Lillo-Castellano, I. Mora-Jimenez, C. Transportation Systems (ITSC), 2010, pp. 66–73. 2
Figuera-Pozuelo, and J. L. Rojo-Alvarez, “Traffic sign [19] Z. Huang, Y. Yu, J. Gu, and H. Liu, “An Efficient
segmentation and classification using statistical learning Method for Traffic Sign Recognition Based on Extreme
methods,” Neurocomputing, vol. 153, pp. 286–299, Learning Machine,” IEEE Transactions on Cybernetics,
2015. 1, 2 no. 99, pp. 1–14, 2016. 2, 3
[5] M. Haloi, “A novel pLSA based Traffic Signs [20] F. Larsson and M. Felsberg, “Using fourier
Classification System,” CoRR, vol. abs/1503.0, 2015. 1, descriptors and spatial models for traffic sign
3 recognition,” Image Analysis, no. May, pp. 238– 249,
[6] Y. Zhu, C. Zhang, D. Zhou, X. Wang, X. Bai, and W. 2011. 2
Liu, “Traffic sign detection and recognition using fully
convolutional network guided proposals,”
Neurocomputing, vol. 214, pp. 758–766, 2016. 1, 2, 3, 7,
8
[7] R. Timofte, V. A. Prisacariu, L. J. V. Gool, and I. Reid,
“Combining Traffic Sign Detection with 3D Tracking
Towards Better Driver Assistance,” in Emerging Topics in
Computer Vision and its Applications, 2011, pp.
425–446. 1
[8] A. Mogelmose, “Visual Analysis in Traffic &
Re-identification,” Ph.D. dissertation, Faculty of
Engineering and Science, Aalborg University, 2015. 1
[9] J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel,
“Man vs. computer: Benchmarking machine learning
algorithms for traffic sign recognition,” Neural Networks,
vol. 32, pp. 323–332, 2012. 1, 2
[10] S. Houben, J. Stallkamp, J. Salmen, M. Schlipsing,
and C. Igel, “Detection of traffic signs in real-world