Capstone Design (SOC4150–004)

Final Assignment project report

Devs_Team
Team mebers:
1. Mirzakhmedov Shokhrukhmirzo – u1610230(Team leader)
2. Abbosjon Kudratov - u1610001
3. Solikh Mirzaev – u1610234
4. Boburjon Borataliyev – u1610053

Lane Detection for Autonomous vehicle

Problem statement

To detect the lane lines in a road sample video and show the detected lines in the frame.
Make prediction according to detected lines about turns car is about make.

Algorithm

The process of lane lines detection can be divided into the following steps:

1) Image denoising

2) Edge detection from binary image

3) Mask the image

4) Lines detection using Hough transform technique

5) Left and right lines separation

6) Drawing the complete line

7) Predict the turn

Algorithm implementation steps (code explanation)

Pre-implementation steps:

We used in this project OpenCV library to do main tasks along with the help of other libraries
like numpy, imutils and matplotlib

import cv2
import numpy as np
import imutils
import matplotlib.pyplot as plt

We defined the function to process the frames of video. As video is a sequence of image
frames, first we need to load the video and we can process each frame as individual image.

cv2.VideoCapture() function returns 2 variables – first of them ret is a state boolean

variable(true/false). We will open the video while ret is true and process frames by sending
them to function process_image()

# open the video file

videofile = cv2.VideoCapture('./road.mp4')

while True:
# ret = frame capture result
# frame = captured frame
ret, frame = videofile.read()
frame = process_image(frame)
try:
# cv2.imshow("video", frame)
cv2.imshow('with lines', frame)
except:
pass

If the variable ret becomes false it means we reached the end of the video. To quit from the
loop we define waitKey() function with trigger some key pressed to break:

if ret is False:
print('end of video!!: break!')
break

q = cv2.waitKey(1) & 0xff

if q == ord('q'):
print('q is pressed: quit')
break
1) Image denoising

All code about processing the frame image we defined inside the function process_image()

Passed as argument image has is shape property. As it is colored image frame it has inside
shape property height, width and channel count. With help of resize() function of cv2 we
resize and store the image and load to another variable:

def process_image(img):
# resize the frame image:
if img is not None:
h, w, d = img.shape
frame = cv2.resize(img, (int(w / 1.4), int(h / 1.4))) # store the resized image in another variable

To do denoising of image – one of the best ways to make it using Gaussian filter or
Gaussian Blur with kernel matrix 5x5:

lane_img = np.copy(frame)
# apply Gaussian filter to smooth the image:
blurred = cv2.GaussianBlur(lane_img, (5, 5), 1)

Note: We applied the GaussianBlur to the copy of the image

Original image frame and Blurred

2) Edge detection from binary image

Before detecting edges we need to convert our colored 3-channel blurred image to 1-
channel grayscale image using cvtColor() function of cv2

# convert to grascale
gray = cv2.cvtColor(blurred, cv2.COLOR_BGR2GRAY)

Resulting grayscale image

Now to binarize the image we apply thresholding technique. This code determines the
threshold automatically from the image using Otsu's method and returns tuple of binarized
image with its threshold value:

(thresh, im_bw) = cv2.threshold(gray, 128, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)

Binarization takes all values till 128 – converts into black and other values greater than 128 till 255 ->
into white
 Binarized image

To detect edges we use Canny edge technique. We need 1-channel image to detect edges.

# apply canny edge detection with binarized image

canny = auto_canny(im_bw)

We defined auto_canny() function to make the edge detection better that gives us additional
parameters (sigma and median values), but main thing here is Canny() of cv2:

def auto_canny(image, sigma=0.33):

# compute the median of the single channel pixel intensities
v = np.median(image)
# apply automatic Canny edge detection using the computed median
lower = int(max(0, (1.0 - sigma) * v))
upper = int(min(255, (1.0 + sigma) * v))
edged = cv2.Canny(image, lower, upper)
# return the edged image
return edged
Display the found edges:

3) Mask the image

We are interested in only some defined region of the image. To find our ROI (region of
interest) we need points to crop accordingly.

We display the image using matplotlib to find some points on it to define ROI. Find the
approximate locations of these red dots to crop road ROI:

ROI_frame = [
(200,500),
(800,500),
(450,350) ]

To mask the image means leave out only the ROI and cover other useless parts. To do so
we have defined the function ROI_area_masking()
Call this function and store the returned image:

cropped_masked_img = ROI_area_masking(canny, np.array([ROI_frame], np.int32))

This function is defined very simple. We need image and vertices(points) as arguments.
Channel count is not defined for our canny edge detected image so we comment it out. With
cv2.fillPoly() we fill the polynomial defined by edges and then make ‘bitwise AND’ to apply
as a mask:

def ROI_area_masking(img, vertices):

mask = np.zeros_like(img)
# channel_count = img.shape[2]
match_mask_color = (255,) # * channel_count
cv2.fillPoly(mask, vertices, match_mask_color)

# we can use cv2.bitwise_and() to do masking:

masked_img = cv2.bitwise_and(img, mask)
return masked_img

So we receive the masked cropped image like this with only detected edges (because we
passed to it our canny edge detected image) in our ROI:
4) Lines detection using Hough transform technique

Expressing lines in the Polar system by using Hough transform.

We should convert Lines from x,y space to polar mc space. But first we use HoughLinesP()
probabilistic function to determine points for us and store in lines variable:

lines = cv2.HoughLinesP(cropped_masked_img, rho=4, theta=np.pi / 180,

threshold=3, minLineLength=130, maxLineGap=40)

With function avg_slope_intercept() we collect 4 points: x1,x2,y1,y2. For input argument we

pass lines from HoughLinesP and our lane image:

averaged_lines = avg_slope_intercept(lane_img, lines)

So the function avg_slope_intercept() is defined like this: it takes 4 points from passed line
and fits into polyfit from where it gets the slope and intercept as parameters array:

def avg_slope_intercept(img, lines):

left_points = []
right_points = []
if lines is not None:
for line in lines:
x1, y1, x2, y2 = line.reshape(4)

parameters = np.polyfit((x1, x2), (y1, y2), 1)

slope = parameters[0]
intercept = parameters[1]

# we can differentiate lines whether on right or left by their slope values:

5) Left and right lines separation.

The slope helps us to identify which points are on the left or right:

if slope < 0: # points on the left have negative slope

left_points.append((slope, intercept)) # so we add these points to our left_points array
else:
right_points.append((slope, intercept)) # else these points are on the right side

To be able to draw the continuous line we have to find averages of these left or right points
arrays:

left_points_avg = np.average(left_points,
axis=0) # axis should be 0 because we want averages of columns in array
right_points_avg = np.average(right_points, axis=0)
And to draw them we will also need the coordinates and then we can return the arrays of
lines left and right. For making coordinates we will have to define another function
make_coordinates()

# we need the coordinates to draw the line:

right_line = make_coordinates(img, right_points_avg)
left_line = make_coordinates(img, left_points_avg)

return np.array([left_line, right_line])

make_coordinates() function helps us to make coordinates for the line with given slope and
intercept:

def make_coordinates(img, line_parameters):

try:
slope, intercept = line_parameters
except TypeError:
slope, intercept = 0.001, 0

# slope, intercept = line_parameters

y1 = img.shape[0]
y2 = int(y1 * (3 / 4.3))
x1 = int((y1 - intercept) / slope)
x2 = int((y2 - intercept) / slope)

return np.array([x1, y1, x2, y2])

6) Drawing the complete line

To draw the lines use the function draw_the_lines() :

image_with_lines = draw_the_lines(lane_img, averaged_lines)

Defined like this - it takes the argument image and averaged lines with defined points and
draws using cv2.line() function:

def draw_the_lines(img, lines):

img = np.copy(img)
blank_image = np.zeros((img.shape[0], img.shape[1], 3), dtype=np.uint8)
if lines is not None:
 for line in lines:
x1, y1, x2, y2 = line.reshape(4)
# for x1, y1, x2, y2 in line:
try:
cv2.line(blank_image, (int(x1), int(y1)), (int(x2), int(y2)), (255, 255, 0), thickness=5)
except OverflowError:
pass

img = cv2.addWeighted(img, 0.8, blank_image, 1, 0.0)

return img

We enclosed cv2.line() function inside try statement to avoid OverflowError (as we

encountered this during runtime)

For better effect of line detection:

We defined function morph_closing() to refine lines – make them thicker and connect by
using the effect of morphological closing:

def morph_closing(image):
# P.S. erosion and dilation => ONLY APPLIED TO GRAY IMAGES!
# create 5x5 kernel with 1s
kernel = np.ones((5, 5), np.uint8)
kernel2 = np.ones((7, 7), np.uint8)

# apply dilation
#morphological closing = dilation , then erosion
frame_dilation = cv2.dilate(image, kernel2, iterations=2)
frame_erosion = cv2.erode(frame_dilation, kernel2, iterations=2)

#or alternatively
closing = cv2.morphologyEx(image,cv2.MORPH_DILATE,kernel2,iterations=2)

cv2.imshow('morpholofical closing effect',closing)

return closing

The canny edge

detected lines and these lines after morphological closing(refining) effect
The RESULT

The resulting image with drawn detected lines: we have a problem identifying points to draw
lines simultaneously for right and left sides with function cv2.HoughLinesP() and its
parameters minLineLength and maxLineGap – we could not find the optimal values.

lines = cv2.HoughLinesP(cropped_masked_img, rho=4, theta=np.pi / 180,

threshold=3, minLineLength=130, maxLineGap=40)
The written source code can be found inside the archive and it is named:
“capstone-team53.py”

The video demonstration has been uploaded to Youtube and the link is the following:
https://www.youtube.com/watch?v=d497UY59Uok

The source code also has been uploaded to Github for pulic use. Link to the repository:
https://github.com/abbosjon-kudratov/line_detection_opencv