Advanced Robotics: Autonomous Navigation of Vehicles

Presented by:

Dr. Gaurav Pandey

Assistant Professor

IIT Kanpur

Autonomous Driving: History

*Video Credit: Ford

2

Autonomous Driving: Today

3

*Video Credit: Ford

Motivation: Safety & Fuel Economy

• Worldwide 1.2 million people die each year in car accidents

• In US alone 33,963 traffic deaths were reported last year.

• In India 2,00,000 traffic deaths were reported last year.

4

Worldwide that’s 3287

deaths every DAY

That’s 2 deaths

every minute

Unsafe Drivers

5

Autonomous Navigation System

6

Localization Planning Control

3D Map (Prior)

Obstacle Dtection/ Classification

Sensing: Lasers, Cameras, Radars

IMU: DGPS, Gyroscope, Wheel Encoders

Sensing: Lasers, Cameras, Radars

Where am I in the map ?

7

MAP Current

Observation

?

8

Research Approach

X21

Xij

XN(N-1)

X0

Fusion of sensor

modalities / extrinsic

calibration of sensors

Alignment of two

instances of fused data

Place Recognition within

prior 3D map

? Generate 3D map

9

Research Approach: Step 1

Fusion of sensor

modalities / extrinsic

calibration of sensors

Velodyne Laser Scanner

10

Ladybug3 Omnidirectional Camera

11

Extrinsic Calibration of Perception Sensors

12

Related Work

Target based Targetless

Requires special targets for

calibration

Q. Zhang & R. Pless [2004]

R. Unnikrishnan & M. Hebert [2005]

C. Mei & P. Rives [2006]

P. Nunez et. al. [2009]

G. Pandey et. al. [2010]

F. M. Mirzaei et al. [2012]

Utilizes the correlation between the

sensor data for calibration.

Bougharbal et. al. [2000]

Williams et. al. [2004]

D. Scaramuzza et. al. [2007]

G. Pandey et. al. [2012]

13

Data from the Perception Sensors

14

Correlation Between Sensor Measured Intensities

15

Joint Histogram

Reflectivity

Gra

yscale

Mu

tual

Info

rmat

ion

Mathematical Formulation • {Pi ; i = 1, 2, … n} = Set of 3D points

• {Xi ; i = 1, 2, … n} = Reflectivity values for these points

• {pi ; i = 1, 2, … n} = Projection of 3D points on image

pi = K[R | t] Pi

• {Yi ; i = 1, 2, … n} = Intensity values for projected points

16

Image Plane

Pi, Xi

pi, Yi

[R, t]

Mutual Information based Calibration

),(maxarg

),(log),(),(

)(log)()(

)(log)()(

),()()(),(

YXMI

yxpyxpYXH

ypypYH

xpxpXH

YXHYHXHYXMI

Yy

XYXY

Xx

Yy

YY

X

Xx

X

17

•X = Reflectivity of 3D points; Y = Intensity of the pixels where 3D points are projected;

• = Extrinsic calibration parameters that allows projection of 3D points onto the image

Sensor data fusion: Lidar and Camera

Crosswalk in front of RIC

19

Research Approach: Step 2

X21

Xij

XN(N-1)

X0

Fusion of sensor

modalities / extrinsic

calibration of sensors

Alignment of two

instances of fused

data

Scan Alignment

20

Before Scan Matching After Scan Matching

Direction of Motion

MI-based Scan Registration Algorithm

Calibration Registration

Random:

Reflectivity &

Intensity values

Random:

Extracted

Features

21

MI-based Registration

22

Joint Histogram of Codewords Scans to be aligned (red & blue)

-40 -30 -20 -10 0 10 20 30 Angle (degree)

Mu

tual

In

form

atio

n

Mathematical Formulation • X = {ic

P ; i = 1, 2, … n} = Codewords extracted from scan P

• Y = {icQ ; i = 1, 2, …m} = Codewords extracted from scan Q

• {pi ; i = 1, 2, … n} = 3D points corresponding to codewords

• {qi ; i = 1, 2, … m} = 3D points corresponding to codewords

qi = R pi + t

23

0 0 4 0 0 1

0 3 0 0 0 0

2 0 0 2 4 0

0 0 1 0 3 0

0 0 0 5 1 0

0 4 0 2 0 0

Codewords (P)

Codew

ord

s (Q

)

Sparse Joint Histogram (ML Estimate)

• Typically K x K >> n

– E.g. If the vocabulary size K = 250 and the number of features extracted from the scan is ~1000

– 250*250 = 62500 >> 1000

24

James Stein Estimate

• This method was proposed by Hausser & Strimmer for entropy and MI estimation and is based on shrinking the ML estimator of the distribution of a random variable Z towards a target distribution T = [T1 T2 …. TK]:

25

Hausser, J., and K. Strimmer. 2009. Entropy inference and the James-Stein estimator, with application to nonlinear gene association networks. J. Mach. Learn. Res. 10:1469-1484.

0 0 .33 0

0 .25 0 0

.17 0 0 .17

0 0 .08 0

.0625 .0625 .0625 .0625

.0625 .0625 .0625 .0625

.0625 .0625 .0625 .0625

.0625 .0625 .0625 .0625

.0312 .0312 .1963 .0312

.0312 .1562 .0312 .0312

.1163 .0312 .0312 .1163

.0312 .0312 .0713 .0312

ML Estimate = n/N Prior (e.g. Uniform)

JS Estimate

Dictionary of Codewords and Target Distribution

• We learn a dictionary of codewords (codebook) from a training dataset by hierarchical K-means clustering.

26

D. Nistér and H. Stewénius. Scalable recognition with a vocabulary tree. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, pages 2161-2168, June 2006.

Tra

inin

g

Testi

ng

Comparison of Cost Function

27

MI cost with ML estimator MI cost with JS estimator

28

Scan Alignment

Not-Aligned Aligned

29

Research Approach

X21

Xij

XN(N-1)

X0

Fusion of sensor

modalities / extrinsic

calibration of sensors

Alignment of two

instances of fused data

Place Recognition within

prior 3D map

? Generate 3D map

Prior Mapping

30

Video Credit: Ford

Prior Map

X0 X1 X2 X3

X8 X7 X6 X5

X9 X4

32

Accurate 3D Map of Environment

Constraints from scan alignment (Z)

Odometry Constraints (U)

Prior Map Data

33

Satellite Imagery Lidar Reflectivity

3D point cloud Z-height data

Localization: Where am I in the map ?

34

MAP

Current Observation

?

• GPS • Lidar Data

• Reflectivity • 3D Point cloud

• Camera Imagery

Intensity localization

?

36

Challenges: Changing Weather, Dynamic Objects,

Lighting Conditions…

Map

Current

37

Voxelization of 3D world

Maximum range of Velodyne laser scanner = 100m

Number of Voxels = 200 x 200 x 50

Vertical FOV of Velodyne = 26.8 degrees

Regularized MI Estimation

• Goods-Turing correction to account for missing words:

• Chao-Shen entropy estimate:

38

Chao, A., and T. J. Shen (2003), Nonparametric estimation of Shannons index of diversity when there are unseen species in sample, Environmental and Ecological Statistics, 10(4), 429–443.

= Estimate of probability of observing a new word

= probability of observing word K

39

Experiment & Results

40

PR Curve: 2010 data localized on 2009 3D map

Nister, D., and H. Stewenius (2006), Scalable recognition with a vocabulary tree, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 2161–2168, New York, NY, USA.

Proposed method with different features Comparison with image-only method

41

Localization Results Q

uery

Scan

D

ata

base (3D

Map)

42

PR Curve: 2011 data localized on 2009 3D map

Nister, D., and H. Stewenius (2006), Scalable recognition with a vocabulary tree, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 2161–2168, New York, NY, USA.

Proposed method with different features Comparison with image-only method

43

Localization Results Snow

Without Snow

Qu

ery

Scan

D

ata

base (3D

Map)

Summary • In our work we exploit the statistical dependence

between the data obtained from different modalities in an information theoretic framework to enhance the robustness of algorithms required for autonomous navigation of vehicles.

44

Thank You !

45