Autonomous vehicles

Abstract
Image recognition is susceptible to interference from the external environment. It is challenging
to recognize traffic lights accurately and reliably in all weather conditions. This article proposed
an improved vision-based traffic light recognition algorithm for autonomous driving, integrating
deep learning and multi-sensor data fusion assist (MSDA). We introduce a method to
dynamically obtain the best size of the region of interest (ROI), including four aspects. First,
based on multi-sensor data (RTK BDS/GPS, IMU, camera, and LiDAR) acquired in a normal
environment, we generated a prior map that contained sufficient traffic light information. And
then, by analyzing the relationship between the sensors' error and the optimal ROI size, the
adaptively dynamic adjustment (ADA) model was built. Furthermore, according to the multi-
sensor data fusion positioning and ADA model, the optimal ROI can be obtained to predict the
location of traffic lights. Finally, YOLOv4 is employed to extract and identify the image
features. We evaluated our algorithm using a public data set and an actual city road test at night.
The experimental results demonstrate that the proposed algorithm has a relatively high accuracy
rate in complex scenarios and can promote the engineering application of autonomous driving
technology.
Keywords
Traffic lights recognition, ADA model, autonomous driving, multi-sensor data fusion, deep
learning
Introduction
In recent years, the rapid development of autonomous driving has attracted more and more
attention worldwide. Simultaneously, with the help of artificial intelligence, significant
technological breakthroughs have been made, and industrialization has been deduced to an
unprecedented height.1 The autonomous driving system mainly includes three main functional
modules: perception, cognition, and execution. The perception module of the system mainly
consists of sensors such as LiDAR, RADAR, camera, IMU, BDS/GPS, and odometer. The
difference in measurement principle determines the good complementarity between multimodal
sensing data.2 Multimodal data fusion is an effective method to improve the performance of the
perception module.3,4 According to the multimodal sensors data fusion and its state, the
autonomous driving vehicle realizes the driving trajectory planning.5 Furthermore, it uses the
control module to finally obtain control instructions for the steering wheel, accelerator, and
brake.6
Traffic light recognition is an essential part of the perception system.7 Although the cooperative
vehicle infrastructure system (CVIS) based on V2I communication8 is a reliable and accurate
scheme, it still has disadvantages of difficulty in implementation, and the communication easily
interferes. Therefore, vision-based traffic light recognition is an indispensable key technology for
the autonomous driving system. However, due to the inherent small object, easily interfered with
by external light sources, and other factors, vision-based traffic light recognition has always been
a technical challenge.9 Especially at night, with extreme weather conditions, intersections,
multiple traffic lights, and other complex road conditions, this problem will become more
difficult.10 The traffic light recognition technology has become a bottleneck, restricting the
engineering application of autonomous driving.
Essentially, vision-based traffic light recognition is a problem of image object detection and
classification. Based on traditional recognition methods, the technical realization process is
generally divided into two stages: object detection and object recognition.11 In the object
detection stage, the main task is to obtain the ROI and determine the traffic light’s location in the
image. Besides, the main goal of the object recognition stage is to achieve accurate object
classification through manual feature extraction and machine learning classification. The typical
manual features have good immutability, such as SIFT,12 HOG,13,14 Haar-like,13,14 but the
recognition accuracy is significantly reduced when the quality of the image is worse.
Simultaneously, the commonly used machine learning algorithms such as AdaBoost, and 9
SVM10 run fast, while the recognition accuracy is difficult to satisfy the practical requirements
of autonomous driving.
With the introduction and application of deep learning15,16 in recent years, the accuracy of
traffic lights recognition has been greatly improved, especially since the successful application of
AlexNet17 deep convolution neural network in the ILSVRC challenge,18 a variety of excellent
improved CNN models such as VGGNet,19 GoogleNet,20 ResNet21 have been proposed one
after another. As early as 2011, Sermanet and LeCun22 applied deep learning to traffic sign
recognition for the first time, which surpassed the level of human recognition and attracted
widespread attention. As Faster-CNN,23 YOLO,24, and other better algorithms were proposed
and applied, the accuracy of traffic light recognition based on deep learning is significantly
higher than that of traditional algorithms.25 However, it is slightly insufficient in real-time.26 In
addition, the accuracy of the algorithm is still too hard to satisfy the actual requirements in
extreme illumination, severe weather, and complex road conditions.
In the actual application scenarios of autonomous driving, vision-based traffic light recognition
is challenging due to the influence of illumination and other light source interference.27 The
performance of any single recognition algorithm is difficult to satisfy the needs of the actual
application. Therefore, the fusion of multiple technologies based on deep learning has become an
important research field.28 On one hand, combining traditional feature extraction and
recognition methods with deep learning can make full use of their complementarity to improve
system performance.29 On the other hand, integrating real-time positioning and prior map data
to obtain ROI and reduce environmental interference is helpful to improve the performance of
the traffic lights recognition algorithm.30
Accurate and reliable traffic light recognition at all times and in all weather is essential to the
safety of autonomous vehicle systems. However, due to the susceptibility to interference from
the external environment, the visual-based traffic light recognition algorithm is complicated. The
existing results mainly studied the image recognition of ordinary illumination and weather,
which is difficult to meet the actual application requirements of autonomous driving. To solve
this question, we proposed an improved traffic light recognition algorithm based on the multi-
sensor data fusion assist (MSDA). The main contributions of this article are summarized as
follows.
1. Based on the prior map and multi-sensor data, we propose a method of dynamically
adjusting the size of ROI based on different positioning precision (RTK, degraded,
outages), achieving the optimal auxiliary effect on the image recognition algorithm.
2. By analyzing the relationship between the sensor error and the optimal size of the ROI,
we built an ADA conversion model from sensor parameters to the best ROI.
3. A priori map labeling method based on multi-sensor fusion is proposed, which uses
various information such as area size, center point coordinates, semantics, road attributes,
and sensor working status.
The following content of this article was organized as follows. The second section introduces
related research progress of traffic light recognition from two aspects of deep learning and prior
maps and briefly analyzes the shortcomings and research fields of existing achievements. In the
third section, the prior map acquisition method was described in detail. At the same time, the
principle and technical implementation scheme of our algorithm was introduced.
Experimentation and validation of the algorithm were introduced in the fourth section. Compared
with the traditional method, the superiority of our algorithm was also analyzed. In the fifth
section, the main innovations and achievements of this article were summarized, and further
research work was also discussed.
Related work
In our algorithm, multi-sensor data fusion assistance and a prior map were used to obtain the
ROI in the image. Furthermore, the ADA conversion model is used to dynamically optimize the
ROI online, to improve the accuracy of traffic light recognition in the complex scenario. At
present, deep learning and prior knowledge assistance are two main directions in the field of
traffic light recognition. Therefore, this part mainly introduces the progress of the related
research in the above two aspects.

Deep learning–based traffic lights recognition

AlexNet model is a classical deep learning algorithm in the field of image recognition. On that
basis, the DeepTLR31 algorithm for traffic light recognition was proposed, which achieved
significantly better performance than manual features without relying on any prior knowledge.
With the help of the concise and efficient AlexNet model, Gao and Wang32 combined deep
learning and traditional recognition technology, implemented image segmentation in HSV space,
extracted HOG manual features, and merged SVM-based traditional detectors with deep learning
detectors. After the iterative development of RCNN to Faster-RCNN, the two-step deep learning
algorithm has made significant progress in image recognition accuracy and computational
efficiency. Kim et al.33,34 applied Faster-RCNN to traffic light recognition. The latter also
analyzed and compared the recognition performance of six color spaces such as RGB, HSV, and
three deep learning models composed of Faster-RCNN, R-FCN, Resnet-101, etc. The results
showed that in the RGB color space, the recognition accuracy of the deep learning algorithm
composed of Faster R-CNN and Inception-Resnet-v2 was better than other combinations.
The one-step algorithms (YOLO, SSD) have more advantages in computational efficiency. They
can be used in autonomous vehicle application scenarios with high real-time requirements and
limited computing resources compared with the two-step deep learning algorithm. Based on the
YOLO algorithm, Behrendt et al.35 achieved a detection speed of 10 frames per second for 1280
× 720 resolution images and achieved traffic light tracking through the combination of a stereo
camera and an onboard odometer. At the same time, combining the high-efficiency advantages
of YOLO with the high/low exposure dual-channel36 architecture has apparent advantages in a
highly dynamic environment. For the weakness of the one-step deep learning algorithm in small
object detection, TL-SSD37 has improved the SSD algorithm to achieve simultaneous
improvement in the recognition precision and recall rate of traffic lights. Ouyang et al.38
combined the specially designed lightweight convolutional neural network RTTLD (Real-Time
Traffic lights Detector) and heuristic ROI detector, which achieved the accuracy rate of 99.3%
and 99.7%, respectively, at 10 Hz on an NVIDIA Jetson TX1/TX2 hardware platform. Saini et
al.39 combined a convolutional neural network algorithm with traditional HOG feature
extraction and SVM classifier to achieve efficient recognition of traffic lights.
Most of the above-mentioned traffic light recognition algorithms were studied for applications in
conventional scenarios (illumination, weather, and traffic conditions). However, in actual
application, there are many extreme illuminations (strong lights, backlight, night), severe
weather (rain, snow, fog, haze), and complex road conditions with various crossroads, and the
performance of conventional algorithms is greatly limited. Thus, it is more important to acquire a
high-precision map, positioning, attitude, and LiDAR information and fusion with vision
information in severe circumstances. The multi-sensor data fusion–assisted improved traffic
lights recognition algorithm proposed in this article combines deep learning and prior knowledge
assistance, precisely and robustly obtains the ROI through an online adaptive adjustment to
improve the recognition accuracy in complex scenarios.

Prior map-based traffic lights recognition

Whether it is traditional or deep learning-based image detection and recognition, with the
assistance of real-time positioning and prior maps, the accuracy of traffic signal recognition can
be significantly improved. Based on Google’s autonomous driving vehicles, on one hand,
Fairfield and Urmson30 used differential GPS, IMU, and LiDAR to complete a prior map
acquisition. On the other hand, with the help of high-precision positioning, vehicle heading and
attitude, and prior knowledge, it improved the performance of the traditional traffic light
recognition algorithm. At the same time, through the semantic information of the prior map, the
relevant traffic lights were extracted from the image, achieving a recognition accuracy of 99%
and a recall rate of 62% under normal lighting conditions. More importantly, there were no false
positives of green lights.
Furthermore, Levinson et al.10 analyzed the error sources in the three stages of prior map
matching, positioning, and recognition. With the aid of prior knowledge, the comprehensive
recognition accuracy of traditional image detection and recognition algorithms in intersection
scenes can reach 94% in the three time periods of noon, evening, and night. However, the
recognition accuracy rate of green lights and yellow lights was low, and the false positive
probability of green lights reaches 1.35%. Besides, the performance of traditional feature
extraction and recognition algorithms can be further improved by fully considering the impact of
road slope factor40 when using object location information for auxiliary traffic light recognition.
MSDF-AlexNet28 integrates the prior map and multi-sensor auxiliary information and
effectively improves image recognition accuracy by acquiring ROI. Those mentioned above
prior knowledge-assisted solutions all rely on Real-Time Kinematic GPS (RTK GPS) positioning
and high-precision IMU, while the expensive equipment price brings a huge economic burden.
Barnes et al.41 implemented a prior map-aided scheme based on low-cost GPS positioning,
which improved the accuracy of HOG feature extraction and SVM classification recognition
algorithm by 40%.
Compared with the traditional algorithm, the prior knowledge-aided algorithm based on deep
learning has a better development prospect. Based on the GPS signal-assisted acquisition of the
ROI and using the convolutional neural network for feature extraction and classification, John et
al.9 proposed two image recognition schemes suitable for normal light and low light conditions,
respectively. Rossetti et al.42 Combined the newly proposed deep learning algorithm and prior
knowledge assistance and used a prior map to select relevant traffic lights of vehicle driving
behavior from the recognition results of YOLOv3.
Those mentioned above prior knowledge-assisted traffic lights recognition algorithms mainly
focus on obtaining the ROI through online positioning and assisting traditional, deep learning, or
hybrid architecture image recognition solutions. However, they do not consider the impact of
sensor errors on the acquisition of ROI.
Different from the existing research achievement of deep learning-based methods such as
AlexNet,31,32 Faster-RCNN,33,34 SSD,37 and YOLO,35,36 this article combines multi-sensor
data and prior maps to further improve the algorithm performance by obtaining ROI. At the same
time, different from the achievement of a traditional prior map based9,10,41,42 traffic lights
recognition methods, we analyzed the relationship between sensor error and the size of the ROI
and built an ADA transition model from the performance parameters of the sensors to the
optimum ROI. Through adaptive adjusting and matching online in different scenarios, the best
ROI was obtained, which effectively improves the environmental adaptability and robustness of
the algorithm.
The improved traffic lights recognition method proposed in this paper improves the recognition
accuracy by using a prior map and multi-sensor fusion data. In practical applications, if the size
of the ROI is too large, it will bring less improvement to the performance of the algorithm.
Conversely, if the size of the ROI is too small, it will easily cause the object to miss detection.
Therefore, obtaining the optimal size of the ROI is crucial to the improvement of image
recognition accuracy. It is known that BDS/GPS receiver is easy to be disturbed by multiple
factors such as signal block, electromagnetic interference, and multipath effect.43 Especially in
the urban areas, due to environmental factors such as buildings and trees, BDS/GPS signals are
often interrupted or the positioning precision degraded,44,45 which means that the sensor’s
working environment will undoubtedly bring a significant influence on acquiring ROI. Different
from the existing methods, based on the traditional prior knowledge-assisted traffic lights
recognition algorithm, this article’s method can adaptively adjust and match to obtain the best
ROI according to the actual performance of the sensors.
Prior map-based traffic lights recognition
Whether it is traditional or deep learning-based image detection and recognition, with the
assistance of real-time positioning and prior maps, the accuracy of traffic signal recognition can
be significantly improved. Based on Google’s autonomous driving vehicles, on one hand,
Fairfield and Urmson30 used differential GPS, IMU, and LiDAR to complete a prior map
acquisition. On the other hand, with the help of high-precision positioning, vehicle heading and
attitude, and prior knowledge, it improved the performance of the traditional traffic light
recognition algorithm. At the same time, through the semantic information of the prior map, the
relevant traffic lights were extracted from the image, achieving a recognition accuracy of 99%
and a recall rate of 62% under normal lighting conditions. More importantly, there were no false
positives of green lights.
Furthermore, Levinson et al.10 analyzed the error sources in the three stages of prior map
matching, positioning, and recognition. With the aid of prior knowledge, the comprehensive
recognition accuracy of traditional image detection and recognition algorithms in intersection
scenes can reach 94% in the three time periods of noon, evening, and night. However, the
recognition accuracy rate of green lights and yellow lights was low, and the false positive
probability of green lights reaches 1.35%. Besides, the performance of traditional feature
extraction and recognition algorithms can be further improved by fully considering the impact of
road slope factor40 when using object location information for auxiliary traffic light recognition.
MSDF-AlexNet28 integrates the prior map and multi-sensor auxiliary information and
effectively improves image recognition accuracy by acquiring ROI. Those mentioned above
prior knowledge-assisted solutions all rely on Real-Time Kinematic GPS (RTK GPS) positioning
and high-precision IMU, while the expensive equipment price brings a huge economic burden.
Barnes et al.41 implemented a prior map-aided scheme based on low-cost GPS positioning,
which improved the accuracy of HOG feature extraction and SVM classification recognition
algorithm by 40%.
Compared with the traditional algorithm, the prior knowledge-aided algorithm based on deep
learning has a better development prospect. Based on the GPS signal-assisted acquisition of the
ROI and using the convolutional neural network for feature extraction and classification, John et
al.9 proposed two image recognition schemes suitable for normal light and low light conditions,
respectively. Rossetti et al.42 Combined the newly proposed deep learning algorithm and prior
knowledge assistance and used a prior map to select relevant traffic lights of vehicle driving
behavior from the recognition results of YOLOv3.
Those mentioned above prior knowledge-assisted traffic lights recognition algorithms mainly
focus on obtaining the ROI through online positioning and assisting traditional, deep learning, or
hybrid architecture image recognition solutions. However, they do not consider the impact of
sensor errors on the acquisition of ROI.
Different from the existing research achievement of deep learning-based methods such as
AlexNet,31,32 Faster-RCNN,33,34 SSD,37 and YOLO,35,36 this article combines multi-sensor
data and prior maps to further improve the algorithm performance by obtaining ROI. At the same
time, different from the achievement of a traditional prior map based9,10,41,42 traffic lights
recognition methods, we analyzed the relationship between sensor error and the size of the ROI
and built an ADA transition model from the performance parameters of the sensors to the
optimum ROI. Through adaptive adjusting and matching online in different scenarios, the best
ROI was obtained, which effectively improves the environmental adaptability and robustness of
the algorithm.
The improved traffic lights recognition method proposed in this paper improves the recognition
accuracy by using a prior map and multi-sensor fusion data. In practical applications, if the size
of the ROI is too large, it will bring less improvement to the performance of the algorithm.
Conversely, if the size of the ROI is too small, it will easily cause the object to miss detection.
Therefore, obtaining the optimal size of the ROI is crucial to the improvement of image
recognition accuracy. It is known that BDS/GPS receiver is easy to be disturbed by multiple
factors such as signal block, electromagnetic interference, and multipath effect.43 Especially in
the urban areas, due to environmental factors such as buildings and trees, BDS/GPS signals are
often interrupted or the positioning precision degraded,44,45 which means that the sensor’s
working environment will undoubtedly bring a significant influence on acquiring ROI. Different
from the existing methods, based on the traditional prior knowledge-assisted traffic lights
recognition algorithm, this article’s method can adaptively adjust and match to obtain the best
ROI according to the actual performance of the sensors.
Furthermore, Levinson et al.10 analyzed the error sources in the three stages of prior map
matching, positioning, and recognition. With the aid of prior knowledge, the comprehensive
recognition accuracy of traditional image detection and recognition algorithms in intersection
scenes can reach 94% in the three time periods of noon, evening, and night. However, the
recognition accuracy rate of green lights and yellow lights was low, and the false positive
probability of green lights reaches 1.35%. Besides, the performance of traditional feature
extraction and recognition algorithms can be further improved by fully considering the impact of
road slope factor40 when using object location information for auxiliary traffic light recognition.
MSDF-AlexNet28 integrates the prior map and multi-sensor auxiliary information and
effectively improves image recognition accuracy by acquiring ROI. Those mentioned above
prior knowledge-assisted solutions all rely on Real-Time Kinematic GPS (RTK GPS) positioning
and high-precision IMU, while the expensive equipment price brings a huge economic burden.
Barnes et al.41 implemented a prior map-aided scheme based on low-cost GPS positioning,
which improved the accuracy of HOG feature extraction and SVM classification recognition
algorithm by 40%.
Compared with the traditional algorithm, the prior knowledge-aided algorithm based on deep
learning has a better development prospect. Based on the GPS signal-assisted acquisition of the
ROI and using the convolutional neural network for feature extraction and classification, John et
al.9 proposed two image recognition schemes suitable for normal light and low light conditions,
respectively. Rossetti et al.42 Combined the newly proposed deep learning algorithm and prior
knowledge assistance and used a prior map to select relevant traffic lights of vehicle driving
behavior from the recognition results of YOLOv3.
Those mentioned above prior knowledge-assisted traffic lights recognition algorithms mainly
focus on obtaining the ROI through online positioning and assisting traditional, deep learning, or
hybrid architecture image recognition solutions. However, they do not consider the impact of
sensor errors on the acquisition of ROI.
Different from the existing research achievement of deep learning-based methods such as
AlexNet,31,32 Faster-RCNN,33,34 SSD,37 and YOLO,35,36 this article combines multi-sensor
data and prior maps to further improve the algorithm performance by obtaining ROI. At the same
time, different from the achievement of a traditional prior map based9,10,41,42 traffic lights
recognition methods, we analyzed the relationship between sensor error and the size of the ROI
and built an ADA transition model from the performance parameters of the sensors to the
optimum ROI. Through adaptive adjusting and matching online in different scenarios, the best
ROI was obtained, which effectively improves the environmental adaptability and robustness of
the algorithm.
The improved traffic lights recognition method proposed in this paper improves the recognition
accuracy by using a prior map and multi-sensor fusion data. In practical applications, if the size
of the ROI is too large, it will bring less improvement to the performance of the algorithm.
Conversely, if the size of the ROI is too small, it will easily cause the object to miss detection.
Therefore, obtaining the optimal size of the ROI is crucial to the improvement of image
recognition accuracy. It is known that BDS/GPS receiver is easy to be disturbed by multiple
factors such as signal block, electromagnetic interference, and multipath effect.43 Especially in
the urban areas, due to environmental factors such as buildings and trees, BDS/GPS signals are
often interrupted or the positioning precision degraded,44,45 which means that the sensor’s
working environment will undoubtedly bring a significant influence on acquiring ROI. Different
from the existing methods, based on the traditional prior knowledge-assisted traffic lights
recognition algorithm, this article’s method can adaptively adjust and match to obtain the best
ROI according to the actual performance of the sensors.
Proposal of an improved traffic lights recognition algorithm based on MSDA
Traditional traffic light recognition methods mainly include manual feature extraction and
machine learning algorithm classification. The accuracy and environmental adaptability are
difficult to satisfy the requirements of autonomous driving. With the development and
application of deep learning technology in recent years, feature extraction and classification
based on convolutional neural networks have made significant progress in traffic light
recognition technology. However, due to the complexity of the road and the external
environment, the performance of the algorithm is still far from the demand for autonomous
driving in actual scenarios. Any single traffic lights recognition algorithm is difficult to meet the
practical requirements. Therefore, various methods combining traditional technologies, deep
learning, prior knowledge, and multi-sensor data have been proposed. In this article, an improved
traffic light recognition scheme was proposed to apply autonomous driving in Complex
Scenarios. Based on a prior map and multi-sensor data, the best ROI can be obtained through
online adaptive matching and adjustment. The implementation scheme mainly includes joint
calibration of the sensors, prior map generation, ADA model built, relevant traffic lights
selection in the complex crossroad, and feature extraction and recognition based on deep
learning.
Overall scheme construction
The vision-based traffic light recognition algorithm is greatly affected by the external
environment. For example, it is easy to cause the algorithm to miss detection in extreme
illumination and severe weather conditions. The interference of similar external light sources
(car taillights and neon lights) can also easily cause algorithm detection and recognition
mistakenly. Considering the actual application requirements of autonomous driving in Complex
Scenarios, an improved traffic light recognition algorithm was proposed in this article. The
architecture mainly includes 4 parts: (1) Sensors’ calibration: joint calibration of the sensors such
as camera, LiDAR, IMU, BDS/GPS receiver, unified to the vehicle body coordinate system. (2)
Prior map generation: autonomous vehicles integrated with multiple sensors were used to collect
and label the traffic light data (position, bounding box, semantic, attribute, etc.). (3) ADA model
built: analyzing the relationship between the sensor’s error and the size of ROI, and building an
ADA model from the performance of the sensors to the size of ROI. Based on the real-time
sensor data and the prior map, we select and locate relevant traffic lights according to the
trajectory planning parameters and control commands of autonomous driving. (4) ROI
acquisition: selecting relevant traffic lights in complex crossroads based on planning decision
commands and selecting the best ROI based on the ADA model adaptively; (5) Features
extraction and state recognition based on deep learning. The algorithm framework of this article
was shown in Figure 1.

Figure 1. The framework of the algorithm. According to a prior map and multi-sensor data, we
get the ROI of the image first, then use the ADA model to optimize it. Finally, traffic light
recognition is performed.
As shown in Figure 1, the principle of our proposed traffic signal recognition algorithm is as
follows.
1. First, we used the collected multi-sensor data to generate a prior map offline, which
contained rich traffic light information (see section “Prior map generation” for details).
By analyzing the relationship between the sensor error and the optimal size of the ROI,
we built an ADA conversion model from sensor parameters to the best ROI (see section
“Adaptively Dynamic Adjustment (ADA) model” for details).
2. Second, the autonomous driving system preprocessed multi-sensor data (LiDAR point
cloud, image, attitude, heading, latitude, longitude, height). According to the prior map,
we utilized multi-sensor fusion positioning data to calculate relative positional
 relationships of the camera and traffic lights (see section “ROI center point acquisition”
for details)
3. Third, according to the decision command of the autonomous driving system, the related
traffic lights were obtained (see section “Research on selection rules of relevant traffic
lights” for details). And then, we used sensor parameters to calculate the best interest area
size online (see section “Adaptively Dynamic Adjustment (ADA) model” for details).
4. Finally, we adopted the state-of-the-art deep learning algorithm YOLOv4 to recognize
traffic lights (see section “Feature extraction and recognition algorithm based on deep
learning” for details).
References:
1. Liu Y, Li Z. Key technology and application of intelligent connected patrol vehicles for
security scenario. Telecomm. Sci. 2020; 36(4): 53–60.
2. Muresan MP, Giosan I, Nedevschi S. Stabilization and validation of 3D object position
using multimodal sensor fusion and semantic segmentation. Sensors 2020; 20(4): 1110.
3. Nie J, Yan J, Yin H, et al. A multimodality fusion deep neural network and safety test
strategy for intelligent vehicles. In: IEEE transactions on intelligent vehicles,
https://ieeexplore.ieee.org/document/9207961
4. Jin X-B, Yu X-H, Su T-L, et al. Distributed deep fusion predictor for a multi-sensor
system based on causality entropy. Entropy 2021; 23: 219.
5. Kim H, Cho J, Kim D, et al. Intervention minimized semi-autonomous control using
decoupled model predictive control. In: 2017 IEEE intelligent vehicles symposium (IV), Los
Angeles, CA, 31 July 2017, pp.618–623. New York: IEEE.
6. Arıkan A, Kayaduman A, Polat S, et al. Control method simulation and application for
autonomous vehicles. In: 2018 international conference on artificial intelligence and data
processing (IDAP), Malatya, Turkey, 28–30 September 2018, pp.1–4New York: IEEE.
7. Diaz M, Cerri P, Pirlo G, et al. A survey on traffic lights detection. In: International
conference on image analysis and processing, Genoa, Italy, 7–11 September 2015, pp.201–208.
New York: Springer.
8. 3GPP. Study on enhancement of 3GPP support for 5G V2X services: TR22.886, v.15.1.0,
2017, https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?
specificationId=3108
9. John V, Yoneda K, Qi B, et al. Traffic lights recognition in varying illumination using
deep learning and saliency map. In: Proceedings of the IEEE Conference on Intelligent Transport
Systems (ITSC), Qindao, China, 8–11 October 2014, pp.2286–2291. New York: IEEE.
10. Levinson J, Askeland J, Dolson J, et al. Traffic light mapping, localization, and state
detection for autonomous vehicles. In: 2011 IEEE international conference on robotics and
automation (ICRA), Shanghai, China, 9–13 May 2011, pp.5784–5791. New York: IEEE.
11. Mogelmose A, Trivedi MM, Moeslund TB. Vision-based traffic signs detection and
analysis for intelligent driver assistance systems: perspectives and survey. IEEE Trans Intel
Transport Syst 2012; 13(4): 1484–1497.
12. Ren F, Huang J, Jiang R, et al. General traffic sign recognition by feature matching. In:
2009 24th international conference image and vision computing, Wellington, New Zealand, 23–
25 November 2009, pp.409–414. New York: IEEE.
13. Liang M, Cui X, Song Q. Traffic sign recognition method based on HOG-Gabor feature
fusion and Softmax classifier. J Traffic Transport Eng 2017; 17(3): 151–158.
14. Lee S, Kim J, Lim Y, et al. Traffic lights detection and recognition based on Haar-like
features. In: International conference on electronics, information, and communication, Honolulu,
HI, 24–27 January 2018, pp.1–4. New York: IEEE.
15. Greenhalgh J, Mirmehdi M. Real-time detection and recognition of road traffic signs.
IEEE Trans Intel Transport Syst 2012; 13(4): 1498–1506.
16. Ge H, Stehyw O. A fast learning algorithm for deep belief nets. Neural Comput 2006;
18(7): 1527–1554.
17. Krizhevsky A, Sutskever I, Hinton G. Imagenet classification with deep convolutional
neural network. Adv Neural Informati Process Syst 2012; 60: 1097–1105.
18. Russakovsky O, Deng J, Su H. ImageNet large scale visual recognition challenge. Int J
Comp Vision 2015; 115(3): 211–252.
19. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image
recognition. Computer Ence, 2014, https://arxiv.org/abs/1409.1556
20. Szegdy C, Liu W, Jia Y, et al. Going deeper with convolutions. In: Proceeding of the
IEEE conference on computer vision and pattern recognition, Boston, MA, 7–12 October 2015,
pp. 1–9. New York: IEEE.
21. He K, Zhang X, Ren S, et al. Deep residual learning for image recognition. In:
Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV,
2730 June 2016, pp.770778. New York: IEEE.
22. Sermanet P, LeCun Y. Traffic sign recognition with multi-scale convolutional networks.
In: 2011 international joint conference on neural networks, San Jose, CA, 31 July 5 August 2011,
pp.28092813. New York: IEEE.
23. Ren S, He K, Girshick R, et al. Faster R-CNN: towards real-time object detection with
region proposal networks. Adv Neural Informat Process Syst 2015, pp.9199,
https://arxiv.org/abs/1506.01497
24. Redmon J, Farhadi A. Yolov3: an incremental improvement. arXiv, 2018,
https://arxiv.org/abs/1804.02767
25. Jensen MB, Philipsen MP, Moelmose A, et al. Traffic lights detection at night:
comparison of a learning-based detector and three model-based detectors. In: 11th symposium on
visual computing, Las Vegas, NV, 1416 December 2015, pp.774783. New York: Springer.
26. Hu Q, Paisitkriangkrai S, Shen C, et al. Fast detection of multiple objects in traffic scenes
with a common detection framework. IEEE Trans Intel Transport Syst 2015; 17(4): 1002–1014.
27. Siogkas G, Skodras E, Dermatas E. Traffic lights detection in adverse conditions using
color, symmetry and spatiotemporal information. In: International conference on computer vision
theory and applications (VISAPP), Rome, Italy, 2426 February 2012. New York: Springer.
28. Li Z, Zeng Q, Zhang S, et al. An image recognition algorithm based on multi-sensor data
fusion assisted AlexNet model. J Chinese Inertial Tech 2020; 28(2): 219225.
29. Wu Y, Geng K, Xue P, et al. Traffic lights detection and recognition algorithm based on
multi-feature fusion. In: 2019 IEEE 4th international conference on image, vision and computing
(ICIVC), Xiamen, China, 57 July 2019. pp.427432. New York: IEEE.
30. Fairfield N, Urmson C. Traffic lights mapping and detection. In: 2011 IEEE international
conference on robotics and automation, Shanghai, China, 1113 May 2011, pp.54215426. New
York: IEEE.
31. Weber M, Wolf P, Zöllner JM. DeepTLR: a single deep convolutional network for
detection and classification of traffic lights. In: 2016 IEEE intelligent vehicles symposium,
Gothenburg, 2922 June 2016, pp.342348. New York: IEEE.
32. Gao F, Wang C. A hybrid strategy for traffic lights detection by combining classical and
learning detectors. IET Intel Transp Syst 2020; 14(7): 735–741
33. Kim J, Cho H, Hwangbo M, et al. Deep traffic lights detection for self-driving cars from
a large-scale dataset. In: 2018 21st international conference on intelligent transportation systems,
Maui, HI, 47 November 2018, pp.280285. New York: IEEE.
34. Kim H-K, Park JH, Jung H-Y. An efficient color space for deep-learning based traffic
light recognition. J Adv Transport 2018; 2018: 2365414.
35. Behrendt K, Novak L, Botros R. A deep learning approach to traffic lights: detection,
tracking, and classification. In: International conference on robotics and automation (ICRA),
Singapore, 29 May 3 June 2017. New York: IEEE.
36. Wang J, Zhou L. Traffic lights recognition with high dynamic range imaging and deep
learning. IEEE Trans Intel Transport Syst 2019; 20(4): 1341352.
37. Müller J, Dietmayer K. Detecting traffic lights by single shot detection. In: 2018 21st
international conference on intelligent transportation systems (ITSC), Maui, HI, 47 November
2018, pp.266273. New York: IEEE.
38. Ouyang Z, Niu J, Liu Y. Deep CNN-Based Real-Time Traffic lights Detector for Self-
Driving Vehicles. IEEE Trans Mobile Comp 2020; 19: 300313.
39. Saini S, Nikhil S, Konda KR, et al. An efficient vision-based traffic lights detection and
state recognition for autonomous vehicles. In: 2017 IEEE Intelligent Vehicles Symposium, Los
Angeles, CA, 1114 June 2017, pp.606–611. New York: IEEE.
40. Jang C, Cho S, Jeong S, et al. Traffic lights recognition exploiting map and localization at
every stage. Expert Syst Appl 2017; 88: 290–304.
41. Barnes D, Maddern W, Posner I. Exploiting 3D semantic scene priors for online traffic
lights interpretation. In: 2015 IEEE intelligent vehicles symposium (IV), Seoul, South Korea, 28
June 1 July 2015, pp.573578. New York: IEEE.
42. Possatti LC, Guidolini R, Cardoso VB, et al. Traffic lights recognition using deep
learning and prior maps for autonomous cars. arXiv, 2019,
https://arxiv.org/abs/1906.11886#:~:text=Traffic%20Light%20Recognition%20Using%20Deep
%20Learning%20and%20Prior%20Maps%20for%20Autonomous%20Cars,-Lucas
%20C.&text=Autonomous%20terrestrial%20vehicles%20must%20be,identify%20the
%20relevant%20traffic%20lights
43. Meng Q, Hsu L-T. Integrity monitoring for all-source navigation enhanced by kalman
filter based solution separation. IEEE Sens J. Epub ahead of print 23 September 2020. Crossref
44. Meng Q, Hsu L-T. Integrity for autonomous vehicles and towards a novel alert limit
determination method. Proc IMechE, Part D: J Automobile Engineering 2021; 235(4): 996–1006.
45. Meng Q, Liu J, Zeng Q, et al. Improved ARAIM fault modes determination scheme
based on feedback structure with probability accumulation. GPS Solutions 2019; 23: 16.
46. Zeng Q, Chen W, Liu J, et al. An improved multi-sensor fusion navigation algorithm
based on the factor graph. Sensors 2017; 17(3): 641.
47. AQSIQ and SAC. Specifications for road traffic lights setting and installation: GB14886-
2016, 2016, https://www.chinesestandard.net/PDF/English.aspx/GB14886-2016
48. Bochkovskiy A, Wang C, Liao HM. Cv4: optimal speed and accuracy of object detection.
arXiv, 2020, https://arxiv.org/abs/2004.10934
49. Jensen MB, Philipsen MP, Møgelmose A, et al. Vision for looking at traffic lights: issues,
survey, and perspectives. IEEE Trans Intel Transport Syst 2015; 17: 18001815.