Contents
Declaration i
Certificate by the Supervisor ii
Acknowledgements iv
Abstract vi
Certificate by the Examiners ix
List of Figures xv
List of Tables xviii
List of Abbreviations xix
1 Introduction 1
1.1 Overview of Autonomous Vehicle . . . . . . . . . . . . . . . . . . . . 2
1.2 Benefits of Autonomous Vehicles to the society . . . . . . . . . . . . 3
1.3 Historical Background . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3.1 The beginning . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3.2 Series of grand challenges . . . . . . . . . . . . . . . . . . . . 6
1.3.3 Commercialization . . . . . . . . . . . . . . . . . . . . . . . . 7
1.4 Levels of Autonomy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.5 Sensors technologies used in Autonomous Vehicles . . . . . . . . . . . 9
1.5.1 Exteroceptive Sensors . . . . . . . . . . . . . . . . . . . . . . 10
1.5.2 Proprioceptive Sensors . . . . . . . . . . . . . . . . . . . . . . 12
x
xi
1.6 Challenges of driverless vehicles in urban environment . . . . . . . . . 15
1.7 Barriers: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
1.8 Organization of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . 19
2 Literature Review 21
2.1 Related Work: Phases of intelligent vehicle system . . . . . . . . . . . 22
2.1.1 Road Detection . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.1.2 Lane Detection . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.1.3 Pothole Detection . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.1.4 Speed Bump Detection . . . . . . . . . . . . . . . . . . . . . . 32
2.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.3 Objectives of the Proposed Research . . . . . . . . . . . . . . . . . . 37
3 Design and development of A.I. based intelligent vehicle system for
detection and classification of road surfaces 38
3.1 Road classification (RCNet) . . . . . . . . . . . . . . . . . . . . . . . 39
3.1.1 Network Architecture . . . . . . . . . . . . . . . . . . . . . . . 39
3.2 Experimental Results and Discussion (RCNet) . . . . . . . . . . . . . 46
3.2.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.2.2 Impact of Optimizer on Classification model . . . . . . . . . . 47
3.2.3 Analysis of computational time . . . . . . . . . . . . . . . . . 48
3.2.4 Performance evaluation of RCNet . . . . . . . . . . . . . . . . 50
3.2.5 Evaluation of RCNet in real time and comparison with state-of-the-art
methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.3 Road detection (DSLnO + FCN) . . . . . . . . . . . . . . . . . . . . 54
3.3.1 Semantic Contour Map . . . . . . . . . . . . . . . . . . . . . . 54
3.3.2 Siamesed FCN (s-FCN) . . . . . . . . . . . . . . . . . . . . . 56
3.3.3 Integrating Location Priors in s-FCN . . . . . . . . . . . . . . 56
3.3.4 Super Pixel Segmentation . . . . . . . . . . . . . . . . . . . . 57
3.3.5 DSLnO Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 59
3.4 Experimental Results and Discussion (DSLnO + FCN) . . . . . . . . 60
3.4.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
xii
3.4.2 Performance analysis . . . . . . . . . . . . . . . . . . . . . . . 61
3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4 Design and development of lane detection model for various lane
scheme using computer vision approach 68
4.1 Lane Detection Model (LDM) . . . . . . . . . . . . . . . . . . . . . . 69
4.1.1 Image Signature Conversion . . . . . . . . . . . . . . . . . . . 69
4.1.2 Region of interest (ROI) . . . . . . . . . . . . . . . . . . . . . 69
4.1.3 Inverse perspective mapping (IPM) . . . . . . . . . . . . . . . 70
4.1.4 Binary level thresholding . . . . . . . . . . . . . . . . . . . . . 72
4.1.5 Edge Detection . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.1.6 Lane Detection . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.1.7 Evaluation Measures . . . . . . . . . . . . . . . . . . . . . . . 75
4.2 Experimental Results and Discussion - Lane Detection Model (LDM) 76
4.2.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.2.2 Vehicle Prototype Configuration . . . . . . . . . . . . . . . . . 76
4.2.3 Driving Behavior Analysis against the Detected Lane . . . . . 77
4.2.4 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . 78
4.3 Vision-based lane region detection (VLDNet) using semantic segmen-
tation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.3.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.4 Experimental Results and Discussion – VLDNet . . . . . . . . . . . . 87
4.4.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.4.2 Performance assessment of VLDNet and comparison with state-
of-the-art techniques . . . . . . . . . . . . . . . . . . . . . . . 87
4.4.3 Performance comparison of VLDNet with state-of-the-art tech-
niques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
5 Design and development of pothole detection system in dynamic
environment using deep learning approach 94
5.1 Strategies for pothole detection . . . . . . . . . . . . . . . . . . . . . 95
xiii
5.2 Architectural Design . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.3 Experimental results and discussion . . . . . . . . . . . . . . . . . . . 98
5.3.1 Dataset and annotation . . . . . . . . . . . . . . . . . . . . . 98
5.3.2 Batch Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
5.3.3 Versions of PDNet . . . . . . . . . . . . . . . . . . . . . . . . 100
5.3.4 Evaluation measures . . . . . . . . . . . . . . . . . . . . . . . 100
5.3.5 Performance evaluation of the proposed model . . . . . . . . . 103
5.3.6 Detection Performance and comparison with the state-of-the-
art techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
5.4 Pothole detection (PotNet) – using computer vision approach . . . . 110
5.4.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.5 Experimental results and discussion . . . . . . . . . . . . . . . . . . . 113
5.5.1 Vehicle to pothole distance measurement . . . . . . . . . . . . 115
5.5.2 Intelligent behavior of prototype vehicle against detected pot-
hole objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
5.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
6 Design and development of automated system for detection of marked
and unmarked speed bump in real time scenario using CNN 119
6.1 Speed Breaker Detection Model Using Convolutional Neural Network
(SBDNet) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
6.2 Experimental results and discussion (SBDNet) . . . . . . . . . . . . . 123
6.2.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
6.2.2 Impact of learning rate on SBDNet . . . . . . . . . . . . . . . 124
6.2.3 Performance evaluation of SBDNet . . . . . . . . . . . . . . . 125
6.2.4 Evaluation of SBDNet in the real time and performance as-
sessment with the state-of-the-art methods . . . . . . . . . . . 126
6.3 Speed Bump Detection using Deep learning approach (Hardware) . . 129
6.3.1 Object to camera distance calculation . . . . . . . . . . . . . . 131
6.3.2 Vehicle Prototype configuration . . . . . . . . . . . . . . . . . 132
6.4 Experimental results and discussion (Hardware) . . . . . . . . . . . . 133
6.4.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
xiv
6.4.2 Performance of speed bump detection model . . . . . . . . . . 134
6.4.3 Intelligent behavior of vehicle using Raspberry pi and Arduino
communication . . . . . . . . . . . . . . . . . . . . . . . . . . 137
6.4.4 Evaluation of proposed model in real time and comparison
with state-of-art methods . . . . . . . . . . . . . . . . . . . . 139
6.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
7 Conclusions and Future Work 142
7.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
7.2 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
References R-1
Author’s Bio-data A-1
