REPORT

ON

Vision Based Lane Detection

For
Unmanned Ground Vehicle (UGV)

1
 TABLE OF CONTENTS

Sr. TITLE
No.
1 Problem statement

2 Brief Theory

3 Project work

4 Project Status

PROBLEM STATEMENT
2
One of the important features of an autonomous Unmanned Ground Vehicle

(UGV) is the ability to navigate in outdoor environment condition. To

accomplish this, task of autonomous navigation on a concrete road is first taken

up. The vehicle should be able to detect the lane on the road and navigate. The

vehicle should be able to negotiate the obstacles and incoming vehicles

accordingly. Autonomous navigation on an off road track and a rough terrain

will be the next target after this task.

BRIEF THEORY

3
 For navigation in outdoor environment the possible approaches are ROS

based navigation using 3D LIDAR and computer vision based navigation

(Image processing and machine learning). The approach adopted for lane

detection is Computer vision. Computer Vision (CV) approach is to develop

techniques so that computers can see and understand the environment in the

form of digital images such as photographs and videos. Machine Learning is an

application of artificial intelligence (AI) that provides systems the ability to

automatically learn and improve from experience without being explicitly

programmed.

Lane Detection Architecture:

Vision-based lane detection systems consist of three main Components:
 Image preprocessing
 Feature Extraction and model fitting
 lane tracking

Figure 1: Lane Detection Architecture

1. Pre-processing: Images taken directly from the camera often need to

undergo certain preparations and transformations to make information

4
 extraction easier. These can include the use of one or multiple of the

following techniques:

1.1 Region of interest (ROI) selection: ROI selection means cropping the
received image to have only required information and remove all
unnecessary information. The main reason for focusing on ROI is to
increase computation efficiency and reduce false lane detections. The
figure below shows a typical example of ROI selection.

Figure 2: ROI Selection

1.2 Grey scale image Conversion: A color image consisting of three

channels (Red, Green and Blue) is converted into single channel Grey
scale image.

Figure 3: Grey Scale image Conversion

1.3 Noise reduction and blurring: There is noise present in the captured

image due to environmental factors like rain, Fog, Shadows etc. The

developed algorithms must either be noise tolerant or the noise must be

5
 eliminated. The mean filter is one of the techniques used to blur an

image in order to remove noise. It involves determining the mean of the

pixel values within n x n kernel. The pixel intensity of the center element

is then replaced by the mean. This eliminates some of the noise in the

image and smoothes the edges of the image. A typical De-noising action

is shown in figure below.

Figure 4: Noisy and De-noised image

1.4 Inverse Perspective Mapping (IPM): IPM transforms image’s

perspective into bird’s eye view. This view makes the width of the road

equal size at any distance. Otherwise in one point perspective view, the

width of the road diminishes the further you look.

2.1 Feature Extraction: This is information extraction phase. Typically, the

intent is to extract features that resemble lane markings, this could be achieved

by using the following algorithms:

6
2.1.1 Canny Edge Detection: As its name implies, the algorithm aims to detect
all the edges in an image. We can recognize an image as a matrix or an array of
pixels. A pixel contains the light intensity at some location in the image. Each
pixel's intensity is denoted by a numeric value that ranges from 0 to 255, an
intensity value of zero indicates no intensity if something is completely black
whereas 255 represents maximum intensity something being completely white.
A gradient is the change in brightness over a series of pixels. A strong gradient
indicates a steep change whereas a small gradient represents a shallow change.
The figure below explains the theory.

Figure 5: Sharp gradient: Edge detected

Canny detection may require threshold tuning to achieve the desired
range of edges. The figure below gives an illustration of output of edge
detection using canny algorithm.

Figure 6: Canny Edge detection

2.1.2 Hough Transform: In an image for detecting arbitrary shapes, such as

straight lines, Hough Transform algorithm is used. It uses the edge pixel

7
images. It is one of the most widely used algorithms for lane detection. This

transform is used to identify the lane lines.

It is possible to associate with each line of the image a pair . The plane is

sometimes referred to as Hough space for the set of straight lines in two

dimensions. In polar coordinates the equation of our line is:

Slope of line is given then the line can be plotted in the Hough Space as a single

dot. There are many possible lines that can pass through this dot. Now in order

to identify the lines, we will first split our Hough space into a grid. Each bin

inside the grid corresponding to the slope and y intercept value of the line. For

every point of intersection in a Hough Space bin we're going to cast a vote

inside of the bin that it belongs to. The bin with maximum number of votes will

be our line. However, this method is designed to detect straight lines in the

beginning and is not efficient in curve lane detection.

8
 Figure 7: Left lane identification by Hough Transform

2.2 Model fitting: It is the approximation of extracted features into a smooth

path using geometric models. The generated path is later used for various

decision making processes typically for trajectory or heading calculations. The

aim of this model is to fit a curve as accurately as possible with given features.

The most common ones used are:

2.2.1 RANSAC: Random sample consensus (RANSAC) is an iterative method

to estimate parameters of a mathematical model from a set of observed data that

contains outliers, when outliers are to be accorded no influence on the values of

the estimates. Therefore, it also can be interpreted as an outlier detection

method.

A simple example is fitting a line in two dimensions to a set of

observations. Assuming that this set contains both inliers (points which

approximately can be fitted to a line) and outliers (points which cannot be fitted

to this line). RANSAC attempts to exclude the outliers and find a linear model

that only uses the inliers in its calculation.

9
 Figure 8: Fitted line with RANSAC; outlier does not affect the result.

2.2.2 Kalman Filter: It is a process that filters out noise from given noise data.

It is based on a set of mathematical equations that provide a recursive mean to

estimate a process and minimizes the mean of squared error, predicting a future

state from the previous ones. An illustration of Kalman Filter is shown below.

Figure 9: Kalman Filter illustration

10
2.2.3 Polynomial Fitting: Curve/polynomial fitting is a mathematical technique

that is widely used in engineering applications. It consists of fitting a set of

points to a curve using Lagrange Interpolation Polynomial. The principal of this

method is that given a set of points, the aim is to fit them in a smooth curve that

passes through the aforementioned points and the order of the function depends

on the number of points. It can be used to approximate complicated curves.

However, a high-degree interpolation may also result in a poor prediction of the

function between points. The Figure below shows an illustration of polynomial

fitting.

Figure 10: Polynomial fitting

3. Lane Tracking: It can also be understood as trajectory calculation. Once the

vehicle receives the coordinates for desired heading, the trajectory is derived on

a 2D plane. Further it needs to translate to correspond to real world coordinates

11
- 3D plane. It includes estimation of lane curvature, Lane offset and estimated

speed. An illustration of lane tracking is shown below.

Figure 11: Lane tracking

12
 PROJECT WORK
Project work is being carried out for lane detection and tracking for UGV
using image processing and machine learning. The software used are as
mentioned below.
 Operating System: Ubuntu 16.04 LTS

 Programming Language: Python with Anaconda and PyCharm IDE

 Software Library used:

◦ OpenCV for image processing
◦ Tensorflow and Keras for machine learning
The project is divided into various tasks which are discussed below.

Task 1: Detect lane from a given image.

Task 2: Detect lane from a given video.
Task 3: Machine learning along with image processing for efficient operation.
Task 4: Lane detection and tracking in real time environment.
1. Detecting lane from a given image.
First task is to detect the lane in a still image using image processing. For the
task a sample road image (1200x700 pixels) is taken as shown below.

Figure 12: Sample image for lane detection

13
Stage 1: preprocessing the Image which includes ROI calculation and Grey
scale conversion.
Source Code:

import
matplotlib.pylab as
plt
import
cv2
import
numpy as np

image =
cv2.imread('road.jp
g')
image =
cv2.cvtColor(image,
cv2.COLOR_BGR2RGB
)

print(image.
shape)
height =
image.shape[0]
width =
image.shape[1]

region_of_interest_
vertices = [
(0,
height),
(width/2,
height/2),
(width,
height)
]

def

14
region_of_interest(im
g, vertices):
mask =
np.zeros_like(im
g)
channel_count =
img.shape[2]
match_mask_color =
(255,) * channel_count
cv2.fillPoly(mask,
vertices,
match_mask_color)
masked_image =
cv2.bitwise_and(img,
mask)
return
masked_imag
e

cropped_image =
region_of_interest(ima
ge,
np.array([region_of_inter
est_vertices], np.int32),)

plt.imshow(cropp
ed_image)
plt.show()
Result: The result of source code execution is shown below.

15
 Figure 13: Sample image with ROI shown

Stage 2: incorporating feature extraction like edge detection and lane detection
on the image

Source Code:
import
matplotlib.pylab
as plt
import
cv2
import
numpy as np

def
region_of_interest(im
g, vertices):
mask =
np.zeros_like(im
g)
#channel_count =
img.shape[2]
match_mask_c
olor = 255
cv2.fillPoly(mask,
vertices,
match_mask_color)

16
masked_image =
cv2.bitwise_and(img,
mask)
return
masked_imag
e

def
drow_the_lines(im
g, lines):
img =
np.copy(img
)
blank_image =
np.zeros((img.shape[0],
img.shape[1], 3),
dtype=np.uint8)

for line in
lines:
for x1, y1, x2,
y2 in line:
cv2.line(blank_image,
(x1,y1), (x2,y2), (0, 255,
0), thickness=10)

img =
cv2.addWeighted(img,
0.8, blank_image, 1, 0.0)
return
img

image =
cv2.imread('road.j
pg')
image =
cv2.cvtColor(image,
cv2.COLOR_BGR2RGB
)
print(image.

17
shape)
height =
image.shape[0]
width =
image.shape[1
]
region_of_interest
_vertices = [
(0,
height)
,
(width/2,
height/2),
(width,
height)
]
gray_image =
cv2.cvtColor(image,
cv2.COLOR_RGB2GRA
Y)
canny_image =
cv2.Canny(gray_image,
100, 200)
cropped_image =
region_of_interest(canny
_image,
np.array([region_of_inter
est_vertices], np.int32),)
lines =
cv2.HoughLinesP(crop
ped_image,
rho=
6,
theta=np.pi/
180,
threshold=
160,
lines=np.arra
y([]),
minLineLengt
h=40,

18
maxLineGap
=25)
image_with_lines =
drow_the_lines(image, lines)
plt.imshow(image_wi
th_lines)
plt.show()

Result: The result of source code execution is shown below. The detected lane is
highlighted with green color edges.

Figure 14: Sample image with lane detected

PROJECT STATUS
Finished: Task 1 of detecting lane in the image.

In Progress: Task 2 of detecting lane in the video.

Further Action: Machine learning algorithm will be implemented along with

image processing to make the lane detection and tracking more efficient.

Final Goal: Real time video based lane detection trails to be conducted with

on-board UGV camera.

19