The goals / steps of this project are the following:

• Compute the camera calibration matrix and distortion coefficients given a set of chessboard images.
• Apply a distortion correction to raw images.
• Use color transforms, gradients, etc., to create a thresholded binary image.
• Apply a perspective transform to rectify binary image ("birds-eye view").
• Detect lane pixels and fit to find the lane boundary.
• Determine the curvature of the lane and vehicle position with respect to center.
• Warp the detected lane boundaries back onto the original image.
• Output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position.

• [Jupyter notebook][https://github.com/jmsktm/CarND-Advanced-Lane-Lines/blob/master/Advanced_Lane_Lines.ipynb]
• I have reused much of the boilerplate code from quizzes where available, with some tweaks in places.

The code comprises of two methods:

• get_distortion_vars(): This function steps through the list of calibration images, and finds their object points and image points to perform camera calibration.
• undistort(image): This function takes an image, and undistorts it using the calibration attributes returned by the above function.

In this section, I have used the functions in code section 1.1 to calibrate and undistort one of the calibration image.

In this section, I have used the functions from Code section 1 to undistort an image of a road image.

I have used the image /test_images/test2.jpg for the purpose of testing the functionalities below.

The difference between the original and undistorted images isn't quite evident when seen separately. So I have created an overlapped image.

I have splitted the image into H, L and S channels to check which one depicts the lane more prominently.

We see that the lanes are more prominent on the S-channel. In the sections below, we will perform further operations on the S-channel image for lane detection.

In the quizzes, the combined threshold was generated as:
combined[((gradx == 1) & (grady == 1)) | ((mag_binary == 1) & (dir_binary == 1))] = 1

However, I observed that the s-channel provided a pretty good and strong signal about the lanes. So, I have factored that in during the generation of combined threshold. combined[((gradx == 1) & (grady == 1)) | ((mag_binary == 1) & (dir_binary == 1)) | (s_channel_binary == 1)] = 1

I calibrated for straight road using the provide image straight_lines_2.jpg and traced lines over the lanes to obtain the endpoints for use in subsequent steps.

Firstly, I overlapped the corners obtained from straight lane calibration over the test image.

Then I warped the image along the four coordinates. The red lines here (and below) serve as a reference to the straight lanes obtained from straight lane calibration above.

I then performed a series of threshold operations on the warped image.

We'll be using the combined threshold image for lane detection in the subsequent steps.

We are using sliding window approach for lane detection. Here is the result:

I have then used the coordinates obtained from sliding windows to polyfill the lane on the warped image.

The layer is then unwarped back to the real world perspective.

The unwarped layer is then overlaid on top of the real image to get the final image.

Here, I am simply using the formula from the lesson to calculate lane radius based from the position (y-coordinate on the image), and the coefficients of the second-degree curves representing the lanes.

Here, we are using the unwarped lane image for the straight road as a reference to map dimensions from pixels to meters.

Here are some calculations:

• Dashed line width (in px): 81
• Standard length of dashed line (in meters): 3.0
• meters per pixel in y dimension: 0.037037037037037035
• Lane width in px: 720.0
• Standard width of lanes (in meters): 3.7
• meters per pixel in x dimension: 0.005138888888888889

Hence,

• meters/pixel along x-axis: 0.0051
• meters/pixel along y-axis: 0.0370

Here, I have written a few helper functions for calculation of lane curvature and vehicle offset in pixels, and to convert it to meters scale.

For calculation of offset, I have compared the offset at the bottom of the images between:

1. The center of the lane obtained from the image used to find end coordinates of a straight road.
2. The mid-point of the detected lane.

I then used the multiplication factor (meters/pixel along x-axis: 0.0051) to find the offset in meters.

I have done it in step #6

Here's how it finally looks like after overlaying the lane and displaying the lane curvature and vehicle offset from the center of the road:

Here's the code for the pipeline, with some comments added to explain each step:

def process_image(img):
    # Undistorting the image frame
    undistorted = undistort(img)

    # Warping the image based on the four corners obtained beforehand.
    pts = endpoints_for_perspective_transform()
    warped, M, dst = warp(undistorted, pts)

    # Thresholding on the warped image. The function get_thresholds() returns multiple
    # thresholds:
    # 1. Gradient along x-axis
    # 2. Gradient along y-axis
    # 3. Magnitude of the gradients
    # 4. Direction of the gradients
    # 5. Threshold from S-channel of HLS
    # 6. Combined threshold from the above.
    gradx, grady, mag_binary, dir_binary, s_channel_binary, combined = get_thresholds(warped)
    binary_warped = combined
    
    # Curve fitting on the binary, warped image frame.
    # out_img = image with the output of curve fitting
    # left_fit = coefficients of second degree polynomial obtained from curve fitting for the left lane
    # right_fit = coefficients of second degree polynomial obtained from curve fitting for the right lane
    # left_fitx = points on the second degree curve obtained from curve fitting for the left lane
    # right_fitx = points on the second degree curve obtained from curve fitting for the right lane
    out_img, left_fit, right_fit, left_fitx, right_fitx, ploty = fit_polynomial(binary_warped)
    
    # Create a blank layer with the lane polyfilled w.r.t. the warped image.
    layer = np.zeros_like(undistorted)
    coordinates_left = np.vstack((left_fitx.astype(int), ploty.astype(int))).T
    coordinates_right = np.flipud(np.vstack((right_fitx.astype(int), ploty.astype(int))).T)
    coordinates = np.concatenate((coordinates_left, coordinates_right), axis=0)
    cv2.fillPoly(layer, np.array([ coordinates ]), color=(0, 255, 0))
    
    # Unwarping the above image (polyfilled lane) back to real-world perspective.
    unwarped, M, dst = unwarp(layer, pts)

    # Overlay the polyfilled layer on the original image.
    overlay = weighted_img(unwarped, undistorted, 0.95, 0.6, 0)
    
    # Overlay text on the image frame
    text = get_overlay_text(binary_warped, left_fit, right_fit)
    overlay_text(overlay, text)
    
    car_offset_text = find_car_offset_text(img, left_fit, right_fit)
    overlay_text(overlay, car_offset_text, pos=(50, 100))

    # Return the composite image (with the polyfilled image overlaid on original image)
    return overlay

Here is the link to the Youtube video generated using the above pipeline. Click on the image to play video.

The pipeline performed pretty decently for the project video, other than for 1 second at 40s. But it does't perform well with the Challenge video. I couldn't take care of it because of time constraints.

• I would make changes to ignore sudden changes over short durations. If the change is persistent over longer than some specified duration (say, a couple of seconds), I would display some warning message.
• I would make changes to support lane detection on highly curved (low radius) roads
• I would make changes to save the result as a serialized data (instead of image/image) which can be overlaid on the the image as needed. That would make it more portable.
• Not directly related to lane detection, but the process of lane-detection happens sequentially frame-by-frame and is a time consuming process. I would use python's multiprocessing to run it in parallel for each frame.