Deep Learning for Lane
Detection in Adverse
Conditions
09.04.20XX
─
Your Name
Your Company
1
Abstract
This research proposes a novel method of accurate and robust lane detection in adverse
conditions such as low lighting at night and snowy roads that are partly covered. We collect
a data set of images where each clear image has corresponding images in adverse
circumstances. For example if we have an image of a road at day, we also have a
corresponding image of the same portion of the road from the same perspective at night
time. This data set is collected using a realistic open world virtual simulation. In the first
step; Computer Vision techniques are applied on images of roads with prominent lane lines
to get lane boundary coordinates. Then we use these output coordinates in combination
with the corresponding images in adverse conditions to train a Convolutional Neural
Network. Once trained the CNN will be able to detect lanes accurately in adverse
conditions.
Introduction
Need
Lane detection is present in almost all modern vehicles and is an indispensable part of
today’s semi-automatic vehicles. It is also quintessential in development of self-driving
vehicles. In today’s vehicle’s lane detection is used as a warning system to notify the driver
when they are about to get into a dangerous predicament, such as, if they fall asleep or
simply not paying attention to the road ahead.
In semi-automatic vehicles it can be a great tool for safety and path planning. In terms of
safety, it allows the vehicle to determine the safe zone where it should lie in while driving
and behave in a manner in which it is less likely to collide with other vehicles. The lanes also
guide the planning algorithm of the vehicle, allowing it localize itself on the road and
helping it make decisions about overtaking other vehicles or staying in one’s own lane.
In order to improve performance of current systems and get closer to level 5 autonomous
vehicles lane detection needs to be flawless. In that direction we have seen that deep
learning algorithms although require a lot of training data at first but when trained have
show superhuman results on image recognition problems. We want to apply these same
algorithms to the domain of detecting lanes, and in order to overcome the problem of
getting data, we have devised a process for generating labeled images with a diverse range
of scenarios on roads with lane markings.
2
We are confident that this data is enough to make a highly robust system which can detect
lanes accurately in a broad range of weather and road wear.
Goals
The goals of the project will focus on producing efficient solutions for robust lane detection
in adverse conditions using combined techniques of computer vision and deep learning in
tensorflow/ OpenCV. The objectives will be:
1. Collection of data from GTA-V Desktop Game in two forms with same perspective: 1)
Normal Sunny conditions and 2) Adverse conditions.
2. Refinement of the produced data to convert it into standard data-set for testing of
lane algorithms.
3. Detection of lanes in two forms of data-sets using well known lane detection
algorithms used for standard detection of lanes.
4. Detection of lanes in two forms of data-set using deep learning algorithm for lane
detection.
5. Calculation of mean-squared error using the results from the step 2 and 3 and
determination of accuracy of detected lanes in deep learning model. Afterwards,
these result are compared other state of the art lane detection systems.
Literature Review
[1] Robust Lane Detection Based On Convolutional Neural Network and Random
Sample Consensus, [Jihun Kim, Minho Lee, Neural Information Processing. ICONIP
2014]
Random sample consensus is combined with CNN's to get a robust way of detecting lanes
lines. First edges are extracted from the image and then if the road scene is simple only the
RANSAC algorithm is applied but if it is complex than a CNN is applied before and after
doing RANSAC.
[2] DeepLanes: End-To-End Lane Position Estimation using Deep Neural Networks
[Alexandru Gurghian, Tejaswi Koduri, Smita V. Bailur, Kyle J. Carey, Vidya N. Murali
Ford Research and Innovation Center, July 2016]
This research proposes a deep learning approach for lane detection. Camera setup is
different from traditional front camera behind the windscreen. Instead two lateral cameras
are mounted on the side of the car:
3
Fig 1. Laterally-mounted camera looking downward onto the lane markers.
Each image from the side camera is split into rows and the row which is closest to the
bottom (closest to the car) is considered as the correct estimation. The problem is setup as
a classification problem where we classify each row from the image, (y1, y2, y3, ….y316
where yk is the probability for the kth being the row containing the lane line), as either the
lane or not lane. The network is give a 240x360x3 image as an input and it gives out
softmax probabilities for each row of the image as an output. Images containing no lane
markings are classified to a default class.
Fig 2: The label ti ∈ [0, . . . , 316] for image Xi corresponds to the row with the pixel of the lane marking
that is closest to the bottom border of the image. The arrows point to the image row containing the border
of the marking facing the car.
[3] Deep Learning Lane Detection for Autonomous Vehicle Localization [Joel
Pazhayampallil Kai Yuan Kuan December 13, 2013]
This research discusses a methodology for using lane detection and GPS estimations to
make a high accuracy map of Roads. Basically the lane data, High Accuracy GPS data of the
current position of the vehicle and data from the internal sensor of the vehicle are fused to
create a high accuracy map of a road.
4
[4] Lane Estimation for Autonomous Vehicles using Vision and LIDAR by [Albert S.
Huang February 2010]
The research defines a completed methodology for detection of lanes with special focus on
detecting multiple curves instead of just two side curves which is commonly done,
moreover the part of the images that consist of obstacles such as other vehicles, traffic
signs, building e.t.c. are detected through LIDAR and removed for getting just that part of
the image which we are concerned with.
[5] REVIEW OF LANE DETECTION AND TRACKING ALGORITHMS IN ADVANCED DRIVER
ASSISTANCE SYSTEM [Ammu M Kumar and Philomina Simon] [August 2015]
This is a comparison research of different researches in lane detection methods and their
pros and cons. Various conditions are considered such as bad lighting, rain, snow, e.t.c.
[6] Real time Detection of Lane Markers in Urban Streets [Mohamed Aly
Computational Vision Lab Electrical Engineering California Institute of Technology]
Lanes are detected using Computer Vision techniques. Top view of the road is gotten using
a perspective transform of a road then Gaussian filters are applied and the Hough
Transform is used to extract prominent lines, afterward various curve fitting techniques are
applied to get the lane lines. The algorithm is designed have a high frame rate to be used in
Real-Time applications.
[7] Z.Kim, "Robust lane detection and tracking in challenging scenarios," IEEE Trans.
Intell. Transp. Syst. , [vol. 9, no. 1, pp] [16-26, Mar. 2008]
Lane regions are Identified using gradient detector and intensity bump detector. Then
Artificial Neural Networks are applied to these regions to detect lane pixels and then
RANSAC algorithm is used to get the lane curve.
[8] H.Y. Cheng, C.C. Yu, , C. C. Tseng, K. C. Fan,J. N. Hwang, and B. S. Jeng, "Hierarchical
lane detection for different types of roads," In Acoustics, Speech and Signal
Processing (ICASSP), 2008 IEEE International Conference on [pp. 1349-1352. IEEE, 2008]
A hierarchical approach is used for robust detection of lanes first of all the road image is
classified into two different types; structured roads and unstructured roads. Structured
roads have obvious lane markings while unstructured roads don’t have obvious lane
markings or lane boundaries. For structured roads, edge/intensity based lane detection
techniques are used, while for unstructured roads color/texture based methods are
employed.
[9] ROBUST LANE DETECTION AND TRACKING WITH RANSAC AND KALMAN FILTER
Amol Borkar, [Monson Hayes Center for Signal and Image Processing (CSIP) Georgia
Institute of Technology Atlanta, Mark T. Smith Institute for Till]
5
Bird's eye view of the road is taken using a perspective transform. Hough Transform and
random sample consensus are used to detect lane and Kalman filter is used to smoothly
detect lanes over multiple frames.
[10] C.W.Lin, H.Y.Wang, D. C. Tseng, "A robust lane detection and verification method
for intelligent vehicles," In Intelligent Information Technology Application, [2009. IITA
2009]
2D and 3D geometric information of the lane lines is taken into consideration. The positive
and negative second difference map of the image is used to emphasis the lane markings.
After detection verification is performed to judge the correctness of detected lane marker.
The verification is done to check the slopes and intercept relationships, actual road width
and position of vanishing point.
[11] A.Borkar, M. Hayes, M. Smith, and S. Pankanti, "A layered approach to robust
lane detection at night," in [Proc. IEEE Workshop Comput. Intell. Vehicles Veh. Syst.,
2009]
First a region of interest is defined in the Image and sky and other irrelevant regions are
ignored. Then in the next step the image is converted to grayscale. Temporal blurring is
used to elongate dashed lines. Then adaptive thresholding is used to extract bright object
and Hough Transform is used to extract lines. Finally a Gaussian kernel is used to define
lane lines.
[12] G.Liu, F.Wörgötter, and I.Markeli´c, "Combining statistical hough transform and
particle filter for robust lane detection and tracking," in [Proc. IEEE IV, Jun. 2010]
Perspective is mapped to a different one. Properties such as color, position, gradient are
used to detect lane markers. Then various other techniques such as Statistical Hough
Transform, Sobel Edge extraction and Particle Filters are used to extract lines.
[13] A.Borkar, M.Hayes, and M.T.Smith, "Polar randomized hough transform for lane
detection using loose constraints of parallel lines." Acoustics, Speech and Signal
Processing (ICASSP), [2011 IEEE International Conference on. IEEE, 2011]
Takes into consideration the parallel nature of lane lines. Top View of road is obtained
using IPM(Inverse perspective mapping). Polar Randomized Hough Transform is used to
detect lines and their parallel nature is verified. A tolerance window is incorporated for
loosening the parallel constraint slightly.
[14] T.T. Tran, H.M.Cho, S.B.Cho, "A robust method for detecting lane boundary in
challenging scenes." [Information Technology Journal 2011]
6
Region of Interest is defined in the lower region of the image where the road lies. Then, this
algorithm uses Canny Edge Detector and then applies random selection
consensus(RANSAC) to detect lane lines.
[15] S.Zhou, Y.Jiang, J. Xi, J. Gong, G. Xiong, and H. Chen, "A novel lane detection based
on geometrical model and gabor filter," in Intelligent Vehicles Symposium (IV ), [2010
IEEE , June 2010]
Vanishing point of the road is detected based on the Gabor Texture analysis. After
vanishing point is obtained the width and orientation of the lane is estimated. Road
boundaries are detected using Canny Edge Detection and Hough Transform. Multiple lane
model are compared to the the obtained boundaries and the one with the most votes is
selected.
[16] P.Daigavane and P. Bajaj, "Road Lane Detection with Improved Canny Edges
Using Ant Colony Optimization," in [3rd International Conference on Emerging Trends
in Engineering and Technology International Journal of Computer Science &
Information Technology (IJCSIT) 2015]
The images are converted to grayscale and then Canny Edge Detection is used. The output
is fed into an Ant Colony Optimization algorithm which properly connects small breaks
from the Canny Edge detector. Finally Hough Transform is applied and lines with the
highest peaks are selected as lane lines.
[17] C.Guo, S.Mita, and D.McAllester, "Lane detection and tracking in challenging
environments based on a weighted graph and integrated cues," in [Proc. Int. Conf. on
IEEE/RSJ Intelligent Robots and Systems,Taipei, Taiwan, Oct. 2010]
A stereo camera is used and inverse perspective mapping is applied on these images, after
that Canny Edge detection is applied and a sophisticated lane feature detector is applied
for coping with challenging environments such as bad weather e.t.c. An Artificial Neural
Network neural to extract the geometry of the lanes. A weighted graph is constructed by
integrating the result of the best fit from the ANN, particle filtering, Bayesian network and
maximum likelihood estimations. All of these processes are applied to get cues about the
lane boundaries. The lane lines are estimated using splines.
[18] Y.-C. Leng and C.-L. Chen, "Vision-based lane departure detection system in
urban traffic scenes," in [Control Automation Robotics Vision (ICARCV ), 2010]
Two feature of the lanes, lane width and lane continuity are used to remove false detection
of lanes in Urban environments. Any lane has a min width and max with, each region
should have a width between these two limits. Also the safety signs on the road don’t fulfill
the criteria of continuity ( they should be parallel to the lanes to be considered part of the
lane lines).
7
[19] R.Gopalan, T.Hong, M.Shneier, and R. Chellappa, "A Learning Approach Towards
Detection and Tracking of Lane Markings," [Intelligent Transportation Systems, IEEE,
2012]
Lane detection is done using a learning approach. It is treated as a classification problem
where each block falls in one of two classes. “Lane” and “Non-Lane”. A pixel hierarchy based
descriptor is used to analyse features such as intensity, edges, patterns and textures on the
region surrounding each pixel. For tracking lane markings in several frames, a particle filter
is used.
[20] G.Liu, F.Wörgötter, and I.Markeli´c, "Stochastic Lane Shape Estimation Using
Local Image Descriptors," [IEEE Transactions on Intelligent Transportation Systems
2013]
Particle Filters are used to estimate the position of the lane lines. The measurement model
is selected such that the particles model uncertainties of local cues regarding lane
detection with Gaussian Kernels.
[21] H.Jung, J. Min, and J.Kim,"An efficient lane detection algorithm for lane departure
detection," In [Intelligent Vehicles Symposium (IV ), 2013]
The algorithm achieves 90.16% detection rate. First candidate points are extracted to make
a hypothesis about where the lines are, this gives a mist like image which is very small in
size and can be efficiently operated on. This hypothesis is the validated via knowledge
about the position of the camera on the car. Afterwards, lane departure detection is done.
[22] M.Tan, B. Paula, and C.R.Jung, "Real-time detection and classification of road lane
markings," [In Graphics, Patterns and Images (SIBGRAPI), 2013 IEEE]
Lane pixels are assumed to have a higher intensity than the pavement. A small patch of the
image is used to sample lane and pavement color intensities. This sample is then used to
differentiate lanes from non-lanes. Afterwards, the lane are classified into 5 different
classes, namely; dashed, dashed-solid, solid dashed, single solid, double solid.
[23] S.C.Tsai, B.Y.Huang, Y.H.Lin, C. W. Lin, C. S. Tseng, and J. H. Wang, "Novel
boundary determination algorithm for lane detection," [In Connected Vehicles and
Expo (ICCVE), 2013 IEEE International Conference]
A Sobel filter is used for find the local gradient direction. Histograms of oriented gradients
are used to detect the highest gradient which is used are gradient for that section of the
image. (2 citations)
Most of the rest of the researches discussed in this paper [3] are image processing based
techniques with similar methodologies as discussed earlier. In order to it brief we have
avoided to mention their details.
8
[24] H.Yoo, U.Yang, and K. Sohn, “Gradient-enhancing conversion for
illumination-robust lane detection,” [IEEE Transactions on Intelligent Transportation
Systems, 2013]
[25] Zhao, H., Teng, Z., Kim, H.H., and Kang, D. J. , "Annealed Particle Filter Algorithm
Used for Lane Detection and Tracking." [Journal of Automation and Control
Engineering 2013]
[26] U.Ozgunalp, and N.Dahnoun, "Robust lane detection and tracking based on novel
feature extraction and lane categorization," [In Acoustics, Speech and Signal
Processing (ICASSP) International Conference, IEEE, 2014]
[27] Y.Li, A.Iqbal, and N.R.Gans, “Multiple lane boundary detection using a
combination of low-level image features.” [In Intelligent Transportation Systems
(ITSC), 2014 IEEE 17th International Conference. IEEE, 2014]
[28] V.S.Bottazzi, P.V.Borges, B.Stantic,“Adaptive regions of interest based on HSV
histograms for lane marks detection.” [In Robot Intelligence Technology and
Applications.Springer International Publishing, 2014]
[29] J.Wang, T.Mei, B. Kong, and H.Wei, “An approach of lane detection based on
Inverse Perspective Mapping,” [In Intelligent Transportation Systems (ITSC), 2014 IEEE
17th International Conference]
[30] H.Tan, Y. Zhou, Y.Zhu, D.Yao, and K. Li, “A novel curve lane detection based on
Improved River Flow and RANSAC,” [In Intelligent Transportation Systems (ITSC), 2014
IEEE 17th International Conference]
[31] An Adaptive Method for Detecting Lane Boundary in [Night Scene Trung-Thien
Tra , Jin-Ho Son , Byung-Jae Uk , Jeong-Hwa Lee , and Hyo-Moon Cho]
A region of interest image is extracted and converted to grayscale. Then maximum value in
R, G, B are identified and that value is used and on each row the pixels of maximum
intensity are located, and finally the road boundary is detected using the Hough Transform.
9
Working
The scope of the project covers tools of OpenCV and TensorFlow to design a deep learning
model for detection of lanes on a road in highly adverse conditions with high accuracy. To
determine the results, the project was completed in five major steps.
1) Synthetic Data Generation
One of the major challenges in deep learning research is the acquirement of
desired data-set for the research problem. In the light of this research, the data-set
requirements were exceptional. The nature of research problem demanded a
data-set of frames with same perspective of road lanes in variety of conditions. To
meet this condition, collection of data from real life brought forth immense
challenges i.e. geographical restraints of weather conditions as well as appealed for
more time since adverse conditions rarely occur. To counter such challenges, virtual
setting was sought to collect frames for the data-set in which data was collected
with more ease in comparison to the alternative.
The best option in many virtual settings for collection of road lanes is the game “GTA
V” created by the gaming studio named RockStar Games. The game is designed
with a highly detailed surroundings to capture real life experience. Resultantly, the
roads are created with fine texture.
GTA V comes with another great feature called “Modding” that provides variety of
maneuvering in the game settings. Certain mods were designed and implemented
to set the game environment to a dashcam recording the driving course with a
number of frames in alternative conditions in a unit of time. To increase the texture
and detail of the road lanes, few extra mods were also implemented. Hence, a real
life capture of the road lanes was made possible with achievement of desired goals:
collection of lane frames with same perspective in normal and adverse conditions.
2) Refinement of Data
The data generated in step 1 is then refined by cutting unnecessary frames
and parallel streams of images with favourable and adverse road conditions are
made. In these parallel streams a set of images with multiple environmental
conditions has lanes at the exact same position. The streams are optimized such
that each set has one image on which a computer vision lane detection algorithm
can be used for labelling and others are of adverse nature where current computer
vision algorithms would fail.
10
3) Lane Extraction Using Computer Vision Techniques
As discussed in 2, each set of images has one image in which lanes are very clear
and easily detectable. This image is fed to a computer vision lane detection system.
This system consists of the following steps:
- Camera Calibration:
OpenCV has tools through which we calibrated the camera using 20
images of a 10x7 chessboard. Object points and image points are extracted
from the 20 images and fed to the calibrateCamera() function of OpenCV to
get calibration and distortion matrices. Then the undistort() function is
applied to all the lane image data.
- Perspective Transform:
All the images are captured from a dash cam view but the ideal situation for
our computer vision algorithm is a top view of lanes, so in order get that we
apply a perspective transform to the images in order to get a bird’s eye view
of the road.
- Emphasizing Lane Lines:
Using a combination of color and gradient threshold we get a binarized
image where lanes are emphasized and rest of the image is faded out.
11
- Detecting the Lane Pixels:
Using Histogram on vertical levels of the image, I find the maximum
values in the Histogram and define a region around them and consider those
pixels to be lanes.
Afterwards a 2nd order polynomial is fit through the left and right lane points
to get the estimated lanes.
12
The curvature of the lanes is found by calculating the radius of curvature
from the two polynomials.
An in detail explanation of this algorithm with accompanying code is
available on github.
4) Training Model for Adverse Conditions:
The detected lanes in the previous step are used to train a deep
convolutional network with corresponding adverse condition images. Note that
lanes are detected from the clear image and the training is done using the image
which has adverse conditions. By doing this the network is able to learn features
which help it detect lane in the most adverse conditions.
5) Evaluation and Comparison with Current Lane Detection Systems:
13
14
Code
https://github.com/usmanwardag/bad_lanes
