detection and tracking of the vanishing point on a...

Detection and Tracking of the Vanishing

Point on a Horizon for Automotive

Applications

Young-Woo Seo and Ragunathan (Raj) Rajkumar

GM-CMU Autonomous Driving Collaborative Research Lab

Carnegie Mellon University

Motivation

- Instantaneous driving direction of road

- Image sub-regions about drivable regions

- Search direction/region about road-occupants such as

vehicles, pedestrians

- Geometric relation between image plane and road plane

Knowledge of a horizon line and the vanishing point on

the horizon line provides us with the the important

information about driving environments

Motivation

The location of the vanishing point on a horizon line provides important information

about driving environments


- Image sub-regions about drivable Regions

- Search direction of moving objects such as vehicles, pedestrians


Motivation







[Rasmussen, 2004] Grouping dominant orientations for ill-structured road following

Motivation







[Moghadam and Dong, 2012] Road region detection from unpaved road images

Motivation







[Kong et al., 2009] Vanishing point detection for road detection

Motivation

The location of the vanishing point on

a horizon line provides important

information about driving

environments

- Instantaneous driving direction of

road

- Image sub-regions about drivable

Regions

- Search direction of moving objects

such as vehicles, pedestrians

- Geometric relation between image

plane and road plane

[Miksik et al., 2011] Road-detection based on vanishing point detection

Motivation







However, the location of the vanishing point detected by frame-

by-frame basis may be inconsistent over frames, due to,

primarily, 1) overfitted image features and 2) absence of relevant

image features

Motivation

Knowledge of a horizon line and the vanishing point on the horizon line

provides us with the information about the important information about

driving environments

Vanishing Point Detection - Line extraction

- Line classification: Vertical and Horizontal

- Vanishing Point Detection through RANSAC

Vanishing Point Tracking using EKF - Motion model

- Observation model

Vanishing Point Detection and Tracking Applications

Experiments

Summary and Future Work

Contents

Knowledge of a horizon line and the vanishing point on the

horizon line provides us with the information about the

important information about driving environments

Fact: Two parallel lines appearing on a perspective image

meet at a point, vanishing point

- Line extraction

- Line classification based on prior, [0, 0, 1] (horizontal), [0, 1, 0] (vertical)

- Find vanishing points through RANSAC

- Find one vanishing point from vertical line class and more than one

vanishing point from horizontal line class

Vanishing Point Detection: Overview

Algorithm: Line Extraction

1. Execute Histogram Equalization to normalize an input image’s intensity

2. Smooth the image w/ a Gaussian kernel to suppress noises

3. Compute the gradients of the image, and magnitudes and orientations of the

gradient

4. Execute a bilateral filtering to preserve natural edges

5. Compute Canny edges to collect pixel groups

6. Remove those pixel groups of which extents are too small or too large

7. Fit a line segment to each of the pixel groups

Vanishing Point Detection: Line Extraction

Vanishing Point Detection: Line Extraction

Vanishing Point Detection: Line Classification

Given a line segment,

1) Compute the angle between the line

and a vanishing point prior

2) Group the line into a vertical group

if



Vertical lines in red and horizontal lines in blue

- Line extraction

- Initial line classification based on prior, [0, 0, 1] (horizontal), [0, 1, 0] (vertical)

- Find vanishing points through RANSAC

- Find the vanishing point from horizontal and vertical line groups

- Choose a pair of lines to generate a hypothesis of vanishing point

- Count the number of outliers based on orientation difference (e.g., 5 degrees)

- Claim the vp hypothesis that has the smallest number of outliers

- Find one vanishing point from vertical line class and more than one vanishing point

from horizontal line class

Vanishing Point Detection: An Example

Estimated Horizon line

A vanishing point on horizon line

Vanishing Point Detection: Detection Results

Vanishing Point Tracking: Overview

Extended Kalman Filter for tracking the vanishing point on the horizon: - The locations of the vanishing point detected frame-by-frame basis may be

inconsistent over the frames

- Track the image coordinates of a vanishing point using the extracted lines, which

are used for detecting the vanishing point

- Smooth the detected locations of the vanishing point appearing on the horizon line,

even with absence of relevant image features


Initialization?

State?

Process Model?

Measurement

Model?

Vanishing Point Tracking: State Definition and Initialization

The coordinates of the vanishing point are

represented in the (normalized) camera

coordinates

Re-Initialization: Re-initialize the state

when the coordinates of the estimated

vanishing point are projected out of the

image coordinate

Vanishing Point Tracking: Process Model

Predict the coordinates of the vanishing point at the next frame

No motion model (for now)

Vanishing Point Tracking: Measurement Model

^x k = [ x k ; y k ] T

Predict the expected line from the predicted state

Measurement update based on a line’ fidelity to the current vanishing point: The

longer a line the lower chance it is an outlier

Vanishing Point Tracking: Measurement Model

Vanishing Point Tracking: Summary

Vanishing Point Detection and Tracking: Applications

Estimation of road driving direction: To improve the performance of lane-marking

detection [Seo and Rajkumar, 2014a] (IV-2014)

Estimation of pitch angle: To compute metric information of interesting objects on

ground plane [Seo and Rajkumar, 2014b] (ITSC-14)

Metric Measurement: Homography

Estimation of pitch angle: To compute metric information of interesting objects on

ground plane [Seo and Rajkumar, 2014b]

Metric Measurement: Homography

The underlying idea is to compute the pitch (or yaw) angles from the

computation of the difference of coordinates between the camera center and

the vanishing point on a horizon line

Metric Measurement: Pitch Angle Estimation

A house foundation, Robot City,

Estimated Pitch=0.0283 (1.6215 degree)

A: ~5m

E: 5.35m

A: ~10m

E: 10.16m

Actual distance (A): ~15m

Estimated distance (E): 14.88m

Vanishing point location

Camera center

Metric Measurement: Model Verification

Actual distance: ~3m

Estimated distance: 2.74 m

A: ~ 3m

E: 3.25 m

A: ~5 m

E: 5.6 m

Gesling Stadium, CMU

Estimated Pitch=-0.0161 (0.9225 degree)

Metric Measurement: Model Verification

Metric Measurement: Example

Experiments

Experimental Settings - The developed algorithms were implemented in C++ and OpenCV and ran on a

self-driving car at 10Hz.

- Sensors and System:

- Monocular vision sensor

- Flea3 (FL3-GE-50S5C-C), CCD 2/3”, 2448x2024 (1224x1024), 8fps

- 8mm, HFOV=57.6, VFOV=44.8

- Mounting height: 1.46m from the ground

- Navigation solution

- Applanix POS-LV w/ RTK corrections

- RMS, 0.02 (0.06) degree pitch angle measurement with RTK corrections

(GPS outage)

- Testing roads

- Mostly inter-city highways, i.e., I-376, I-279, I-76

- Some urban streets in Pittsburgh

MSE=2.0847 degree

Experimental Results: Pitch Angle Comparison

Compare the pitch angles measured by IMU with that measured by the developed

algorithm

Green circle is

the vanishing

point tracked

over the frames.

Red circle is the

one detected

from each

frame.

Yellow

horizontal line is

a detected

horizon line.

Vanishing Point Detection and Tracking: Video

Summary and Future Work

Developed a computer vision algorithm - Detected vanishing points using the extracted lines

- Tracked, using EKF, the vanishing point on a horizon over frames

Through testing with inter-city highways videos, we

demonstrated that the developed algorithms produced stable

and reliable performance in tracking the vanishing point on a

horizon line

Developed methods are used for 1) approximating road driving

direction and 2) estimating the pitch angle between image and

road plane

More field testing: To determine the limits of our algorithms,

continue testing it against various driving environments.

Thank You

Questions or Comments?

[email protected]

Acknowledgements I would like to thank Dr. Myung Hwangbo for fruitful discussion about 3D

vision, Junsung Kim for data collection, and Prof. Raj Rajkumar for his

support on this work.

detection and tracking of the vanishing point on a...

Documents