Vision-Based Lane/Road Detection Methods

Artificial Vision in Road Vehicles (Bertozzi, et al, 2002) http://www.dx.doi.org/10.1109/JPROC.2002.801444 - General Advice 1. Limiting the search for road/lane markers to a specific region of the image speeds things up. This can be done using knowledge of where they were in the last image, or general assumptions of where they usually should be. 2. Assuming continuity and parallelism of lane markers along with smoothly varying lane widths can help speed things up. Once you find a piece of a marker, you can limit the search to nearly-parallel copies of it (extended along it, offset by a lane width + some amount to account for slowly varying widths, or offset by a line spacing width if you are looking for double lines). 3. Reconstructing road geometry can be made much simpler by assumptions on its shape. a. I think that this means assuming it is a flat plane, typically. b. Clothoids have been used. They are a parametric plot which blends a straight segment into a curved segment using integrals of sin(x2) and cos(x2). See http://en.wikipedia.org/wiki/Clothoid for a quick

explanation. The plot looks like: 4. If you have a specific camera calibration and assume a flat road without bumps, then the mapping between image pixels and world coordinates becomes much easier. - Key Difficulties 1. Dealing with shadows. 2. Occlusion of the road and lane markers due to cars and other obstacles. - Example Algorithms 1. ALVINN [9]: Uses neural net training with a large database of training images to automatically follow roads without any explicit feature recognition. 2. SCARF [10] and POSTECH PVR II [11]: Both use color cameras and the assumption that the road will be homogeneously colored to extract the road region from images even on completely unstructured roads.

3.

4.

5.

6.

7.

o This sounds promising it could still find the road even if lane markers arent present in all segments and maybe even in off-road stretches. PVR II [11]: analyzes two parallel stripes of the image at a time looking for edges using Gaussian masks and zero crossings. The result is compared to a typical model and the quality of the fit indicates a confidence as to whether or not it is a lane marking. Toyota Central R&D Labs [12]: pattern-based technique which was developed to handle situations where visibility of the lane markers is poor. It extracts edges and then matches edge points to various patterns of lane markers which are generated to model typical roads. The pattern which best matches the edge points found in the image is selected as the most likely model of the lane markers. MOSFET [13]: color-based technique uses a color segmentation algorithm that maximizes the contrast between lane markings and the road. Uses parabolas to model the lane markings. Does the fitting with a simplified Hough transform. o Hough transform basically a method where candidate points are all allowed to vote for any line on which they could belong. The lines with the most number of votes at the end are considered to be the best fits to the points. A wikipedia article located here: http://en.wikipedia.org/wiki/Hough_transform gives a good brief introduction. It explains that the candidate points are usually identified via edge detection. If you use gradient information from this process to limit the directions in which candidate lines can point, the algorithm can be sped up significantly. It also mentions the Generalized Hough Transform which is supposed to be able to find any shape which can be succinctly parameterized (circles, ellipses, etc.). A cool web page with an applet showing how the transform works is http://www.cs.tu-bs.de/rob/lehre/bv/HNF.html . They also give a link to the Java source code. (We could probably find MATLAB code for Hough transforms somewhere with a little looking.) LAKE and SAVE autonomous vehicles [14] only analyze the image portion corresponding to 12 m ahead of the vehicle where it is usually safe to assume that the lane markers will not be obscured by other vehicles. This still gives enough time to maneuver on highways and ramps. RALPH [15] uses radar-based obstacle detection to limit the portion of the image to be analyzed for lane markings. The image portions where the lane markings are obscured can be ignored (for finding lanes) this way. The system always limits the search to a given distance ahead of the vehicle and removes the perspective effect from that portion of the image so that the lane markers in the image can be compared directly to standard models in which they are parallel (not slanted and/or leaning towards each other). o Sounds cool. How do you take the perspective out of the image?

8. General Dynamics Robotics Systems and Ohio State University method [16]: uses a gray-level histogram to segment and classify objects as road, lane marking candidates or obstacle candidates. Early version of Ohio States algorithm uses polynomials to represent the lane markings [24]. To narrow in on candidate lane markings, they put the bright regions in a vector list and then use parameters such as convergence at infinity and lane width to decide which ones to keep. Using a low-order polynomial curve allows dashed lines to be represented. 9. Laboratoire Central des Ponts-et-Chaussees de Strasbourg [17]: Assumes that there should always be a chromatic contrast between road and off-road areas/obstacles in at least one color channel. Uses the concept of chromatic saturation to separate the components. o What is chromatic saturation? 10. LASMEA [18,33]: uses a dynamic window of interest in which lane markings are sought. The window is selected using statistical methods based on the current state and previous windows. The lane model is updated continuously and the overall system allows the process to be very robust to noise and poor visibility. 11. Robert Bosch GmbH [19]: uses a model of both the road and the vehicle dynamics to predict where lane marking should be found. 12. Universitt der Bundeswehr [20], Daimler-Benz [21], and Robert Bosch GmbH [22]: Base their road models on clothoids. In a clothoid the curvature depends linearly on the curvilinear reference. Simplifies representation of the road, describing the lane markings, and computing position and heading within the lane. 13. University of Michigan [26]: proposed using concentric circles as a model for lane markings since that is actually how they are laid out and it is a better fit than polynomial approximations. 14. CyCab electric vehicle [28]: uses an edge linking process based on Contour Chains and Causal Neighborhood Windows to piece edges together. Then the longest chains with angles close to 45 and 135 are found and picked as the most probably candidates for left and right lanes. 15. Transportation College of Jilin University of Technology [29]: uses a linear lane model and represents road markers as sequences of straight lines. 16. GOLD system [32]: exploits the flat, planar road assumption to map the image into another domain representing a birds eye view of the road (takes out the perspective like the RALPH system). 17. ARGO [34]: restricts the search for lane markings to the nearest neighborhood of the markings previously detected. 18. Istituto Elettrotecnico Nazionale G. Ferraris [31]: Uses results for lane markings from previous frames to validate the results for the current one.

Video-Based Lane Estimation and Tracking for Driver Assistance: Survey, System, and Evaluation (McCall and Trivedi, 2006) http://www.dx.doi.org/10.1109/TITS.2006.869595 - General Advice/Observations 1. It is important to identify the objectives for a lane tracking system and develop relevant performance metrics. A method which works well for one set of objectives may not be suited for another. You need the right performance metrics to tell you whether or not you are getting the job done. 2. Figure 2(a)-(f) illustrates that lane markers can vary widely in their style and visibility. It also shows that the road surface may not be uniform, especially in regions where transitions from asphalt to cement occur, like in construction zones. Another problem shown is the drastic change in road appearance that can occur due to shadows from trees and overpasses. 3. Generalized framework for lane detection methods: a. A road model is the first component (a vehicle model can be included, too, if data is available and that is of interest). The lane detection system will build around this model. Models used have been as simple as straight line segments, piecewise constant curvatures [16,21], or more complex, like parabolas, hyperbolas, splines [17,22], or even clothoids [16]. b. There is typically a first shot processing method which extracts some kind of features from the raw images. These may be edges, bright patches, motion vectors, textures, or something else. This first step wont single out the lane markers entirely. They should, obviously, be included in the results, but further processing is needed to isolate them. The road/vehicle model and information from past frames can assist in this step by helping to narrow in on the region of interest for processing rather than just plowing through the whole image. c. This first processing attempt is then refined using additional information about general lane marking characteristics and/or information from the road model and past frames. This post processing is where most of the difficulty and opportunity for innovation lies. You try and glean just the lane markers from the initial feature extraction rejecting everything else. This step can be made much more robust when combined with the road model and tracking information temporally. Methods which have been used include: i. The Hough transform [26, 27]. ii. Orientation based enhancement/attenuation [23]. iii. Dynamic programming [28], which was shown to remove outliers more effectively than the Hough transform. iv. Cue scheduling [29]. (Im not sure what that is.) 4. Common Simplifying Assumptions: (These make the task much easier, but must be kept in mind as potential causes for failure when they are violated.) a. The road/lane texture is constant. b. The road/lane width is locally constant.

c. Road markings follow strict rules for appearance or placement. d. The road is a flat plane or follows a strict model for elevation change. Planar assumption causes errors in estimated road curvature on hills. 5. Lane tracking techniques: turns feature extraction and position tracking into a closed-loop feedback system. The tracked lane position (combined with information from the vehicle model?) defines an a priori estimate of the location and orientation of the extracted features. Methods used include: a. Kalman filtering [12, 16] b. Particle filtering [29, 30] c. Nonlinear systems [31] - Example Algorithms 1. Bertozzi and Broggi [20]: Assumed road markings form parallel lines in an inverse-perspective warped image. 2. Recovering 3D Road Shape [32]: Uses the assumption of constant lane width to estimate changes in elevation of the road. Obviously, this will fail in locations where the lane width isnt constant. 3. GOLD [20]: Uses an inverse-perspective road image and combines laneposition tracking with obstacle detection. 4. Springrobot [26]: Uses an adaptive randomized Hough transform for processing detected edges. 5. LOIS [34]: Uses edge magnitude and orientation with a maximum a posterior estimator to find lane position. (Not sure what that means.) Got results with a standard deviation of 13 cm. 6. Huge comparison table on next page.

- Their Algorithm VioLET (VIdeO-based Lane Estimation and Tracking) 1. They were trying to come up with something that would be unprecedented in its robustness to varying road surface, lane marker types, shadows, and lighting. 2. They wanted accuracy for at least 30-40 m ahead of the vehicle, which they say can be attained with a simple parabolic road model (second order polynomial).

a. 3. Basic feature extraction: Steerable Filters a. They create a basis set of responses from three separable 2-D Gaussians:

b. Then, the second-derivative response at any angle can be calculated via [39]: c. Setting the derivative equal to zero, the angles of the maximum and minimum responses can be found by:

d. They used a filter kernel roughly three times the expected width of the lane markers and applied it in the inverse-perspective warped image so that a kernel of constant size could be used everywhere.

e. For circular markers, the maximum and minimum response should be roughly equal. So, you can pick those out by looking for areas where there is a small difference between the max and min. f. For lane markers, the maximum response should be in the direction of the lane marker (assumed known from the previous frame?), and the minimum response should be low. g. You can also get results which are tuned to a specific angle (like the assumed or last known lane marker angle) by formula (6). This lets you attenuate the response for objects/edges which are at the wrong orientation. 4. Road curvature estimation. Basically, they get a set of points of the estimate of the center of the road at certain distance increments ahead of the vehicle, then fit the parabola to those points. a. How they get the points used in this estimation is very, very poorly described. Quite disappointing. But, it sounds like their technique is based upon the following: i. They throw out points which arent near the estimated lane markings according to their road model and prior knowledge. ii. Somehow they take first and second moments of the candidate points. There is supposed to be a high variance in the direction of the lane markings and a lower variance in other directions since points from the lane markings should be spread out more in the direction of the road. iii. A Hough transform also helps find markings for solid/striped (non-circular reflector) lane markings. iv. They also use a road center template and squared intensity error measure to try and come up with the center of the road. It looks like they pick a template which excludes the lane markings and depends on their being some recognizable dark patch down the center of the lane or something. (All this is so remarkably unclear in their descriptions.) 5. Results: Again, their description of their performance metrics is weak, but the results look pretty good overall. An important note they make is the following: a. In most of the situations where the tracking was lost, vehicles changed lanes directly in front of the test bed. The specular highlights and lines of the vehicles caused false positives in the feature extraction.