vision-based 3d bicycle tracking using deformable part model and interacting multiple model filter...

Vision-based 3D Bicycle Tracking using Deformable Part Model

and Interacting Multiple Model Filter

May 11, 2011

Hyunggi Cho1, Paul Rybski1,2, and Wende Zhang3

1Electrical and Computer Engineering

School of Engineering

Carnegie Mellon University

3The Electrical and Controls

Integration Lab.

General Motors R&D

2Robotics Institute

School of Computer Science

Carnegie Mellon University

Outline

Motivation and Overview

Bicycle Detection

Bicycle Tracking

Experimental Results

Conclusion and Future work

Motivation

Motivation

- In 2009, 630 bicyclists were killed and 51,000 were injured in traffic accidents in the United States*.

- Bicyclists and pedestrians are the most vulnerable traffic participants.

- There is less research on bicyclist detection and tracking compared to that of pedestrians.

*http://www.nhtsa.gov

Movie clip: Bicycle messengers in New York City (Youtube)

Sensors on Cars

Source : http://www.tartanracing.org

System Overview

Input : video

Track bicycles using a single video camera mounted on a vehicle

System

output : position & velocity

System block diagram

BicycleDetector

BicycleTracker

Bicycle’sposition & velocity

Bicycle Detection – Deformable Part Model HOG Detector Eight view-based bicycle detection

root filters

coarse resolution

part filters

finer resolution

deformation

models

P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan, Object Detection with Discriminatively Trained Part Based Models, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010

DPM HOG Detector – Object hypothesis

Bicycle detection process

P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan, Object Detection with Discriminatively Trained Part Based Models, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010

DPM HOG Detector – Performance Analysis Examples of bicycle detection

Test images from Google image

DPM HOG Detector – Performance Analysis

Terminology

Eight view

Precision-Recall Curve (VOC2009 + Ours)

Recall (True Positive Rate)

Precision

)( FNTP

TP

P

TPrecall

)( FPTP

TPprecision

True Positive

False Negative

False Positive

Total No. of Positive

Average Precision( Area Under Curve )

TP

FNFPP

AP

Training Set : 350 positive / 3300 negative Test Set : : VOC2009 ‘val’ dataset

Overview of our single bicycle tracking system

Prediction stage Update stage

y

x

Kalman filter-based tracking

][ yxyxx state space :

2D image space


Model

Dynamic system model

v

y

x

y

x

y

x

y

x

~

10

01

00

00

0000

0000

1000

0100

kkk

kkk

vxhz

wFxx

)(1

),0(~

),0(~

),0(~ 00

kk

kk

RNv

QNw

Nx

Motion model : constant velocity

kkk wFxx 1

y

x

: Dynamic equation

: Measurement equation

: Initial state

: Process noise

: Measurement noise


Measurement model : perspective projection

rotation matrix

translation vector

focal length

optical center

Rtf

cc vu ,

kkk vxhz )(

343331

1413111 ayaxa

ayaxa

w

uh

kk

kk

343331

2423212 ayaxa

ayaxa

w

vh

kk

kk

1

0

34333231

24232221

14131211

k

k

y

x

aaaa

aaaa

aaaa

1

0|

100

0

0

k

k

c

c

y

x

tRvf

uf

w

v

u

Horizon

Image plane

height

Image

IMM - Choosing a model set

Constant Velocity Coordinated Turn

Constant Velocity Simplified Bicycle with

CV and CY angle

Model Set I Model Set II Model Set IIIGM

GM

CV

CA

Constant Velocity Constant Acceleration

CV

CT

CV

SB

IMM - Choosing a model set

Constant Velocity model :

Simplified Bicycle model :

v

y

x

y

x

y

x

y

x

~

10

01

00

00

0000

0000

1000

0100

][ yxyxx

IMM - Performance analysis

We tested the IMM method on the GM bicycle dataset

Test Set : 6 sequences with a stationary GM test vehicle

Data statistics : Size : 320x240 , FPS : 10~12 , No. of bicycles : 1

IMM Tracking performance

Details of six bicycle sequences ( SM vs. IMM )

Seq. ego-vehicle bicycle RMSE(SM) RMSE(IMM)

‘seq1’ stationary laterally 0.0183 0.0216

‘seq2’ stationary longitudinally 6.6207 6.6196

‘seq3’ stationary randomly 0.1515 0.1443

‘seq4’ moving laterally 2.3493 2.3860

‘seq5’ moving longitudinally 7.0884 6.860

‘seq6’ moving randomly 11.0929 10.6281

IMM - Performance analysis

Sequence 3 case

Multiple bicycle tracking using a Rao-Blackwellized particle filter

Single bicycle tracking : We solved this problem.

Data association : Given a measurement, which target produced it, if any ?

Unknown number of targets : How many bicycles are there ?

Multiple bicycle tracking problem

In our multiple bicycle tracking case

)|(),|()|,( :1:0:0:1:0:1:0:0 tttttttt yspsyrpysrp Particle filterKalman filter

},{ ttt ces

TTttt rrr ][ ,1, Joint state vector

: Data association indicator

:Target visibility indicator

tc

te

Simo Särkkä, Aki Vehtari, and Jouko Lampinen (2007). Rao-Blackwellized Particle Filter for Multiple Target Tracking. Information Fusion Journal, Volume 8, Issue 1, Pages 2-15

Multiple bicycle tracking using a Rao-Blackwellized particle filter

Only one target can die

is associated with : (a) Clutter(b) One of the existing targets(c) A newborn target

ky

All possible events between two measurements and 1ky ky

Example

t-2

t-1

t

t-2

t-1

t

y1

y2

y3

: Target

: Measurement

: Trajectory

Particle filter for data association problem

Experimental Results

We tested our detection/tracking system on our bicycle dataset

Test Set : A challenging sequence from a moving Boss (so called ‘Free for all’)

Data statistics : Size : 320 x 240 , Frame rate : 13~15 frame per second

Sensor coverage area

Tracking performance

15 m

5 m

0 m

4 m

Minimum pixel sizeHOG Detector : 32x64

4 m

Experimental Results - data collection

Rank & Rate Description Illustration

10(3.9%)

Motorist Overtaking-OtherThe motorist was overtaking a

bicyclists.

9(4.3%)

Bicyclist Left Turn in front of trafficThe bicyclist made a left turn in front

of traffic travelling in the same direction.

8(4.4%)

Ride Out At MidblockThe bicyclist entered the roadway at a

shoulder or curb midblock location.

US Bicyclists Crash Types – Top 10covering 61% database samples

W.H. Hunter, W.E. Pein, and J.C. Stutts, Bicycle Crash Types: A 1990's Informational Guide, Publication No. FHWA-RD-96-104, Federal Highway Administration, Washington, DC, April, 1997



7(4.7%)

Motorist Right TurnThe motorist was making a right turn and the bicyclist was riding in either

the same or opposing direction.

6(5.1%)

Ride Out At Residential DrivewayThe bicyclist entered the roadway from

a residential driveway or alley.

5(5.9%)

Motorist Left Turn– Facing BicyclistThe motorist made a left turn while

facing the approaching bicyclist.

4(6.9%)

Ride Out At MidblockThe motorist was entering the roadway

from a driveway or alley




3(7.1%)

Ride Out At Intersection - OtherThe crash occurred at an intersection, signalized or uncontrolled, at which the

bicyclist failed to yield.

2(9.3%)

Drive Out At Stop SignThe crash occurred at an intersection

at which the motorist was facing a stop sign.

1(9.7%)

Ride Out At Stop SignThe crash occurred at an intersection

at which the bicyclist was facing a stop sign or flashing red light.

We categorized the upper scenarios into 4 different classes in terms of bicycle motion patterns !!!


Experimental Results – Performance analysis

Scenario – Random moving case (‘Free for all’)

2D Bounding box {view (x coordinate, y coordinate)} Uncertainty level

Experimental Results – Performance analysis

Scenario – Random moving case with 3D visualization

2D Bounding box {view (x coordinate, y coordinate)} Uncertainty level

Summary and Future Work

Summary Data collection

- Based on bicycle accident statistics

Detection part- Applied DPM HOG detector into a multiple bicycle tracking system

Tracking part- Incorporate Interacting Multiple Model (IMM) algorithm into our multiple bicycle tracking system to exploit several types of motion models- RBPF data association algorithm

Future work Real-time C++ implementation ( > 10fps)

Integration the system into the perception system of our autonomous vehicles at CMU

Single bicycle tracking using an IMM

True motion of a bicycle cannot be exactly modeled by just one model, only be sufficiently approximated by using several motion models for representing dynamic driving behaviors of a target (i.e., maneuverings of a bicycle).

The IMM filter runs several motion models in parallel and estimates a state by computinga weighted sum of several filter results which are based on different motion models.

Main idea of Interacting Multiple Model (IMM)

Integral HOG Detector - Performance Analysis II

Related works

Vision-based Bicycle Detector

Publication Sensors Features Attention Focusing Stage Classification stage

Gavrila IV2004 Stereo Edge map

Stereo-based depthChamfer matching

Texture classification

Papageorgiou

IJCV2000Monocular Haar wavelet

Can add motion/stereo modules for preprocessing

SVM classifier on Haar wavelet features

Viola & Jones

CVPR2001Monocular Haar-like wavelet NA AdaBoost

Dalal & Trigg

CVPR2005Monocular HOG NA Linear SVM on HOG

Zhu & Avidan

CVPR2006Monocular Integral HOG NA

AdaBoost with linear SVM as a weak

classifier

Miko.

ECCV2004Monocular

SIFT-like orientation feature

NA AdaBoost

Wu & Nevatia

ICCV2005Monocular Edgelets NA

AdaBoost with hard-coded mid-level features

Felzenszwalb

CVPR2008Monocular HOG NA

Deformable part model with LSVM

vision-based 3d bicycle tracking using deformable part model and interacting multiple model filter...

Documents

simplified bicycle model

d bicycle tracking

bicycle messengers

gm bicycle datasettest

bicycle sequences sm

object detection

multiple model filtermay

negativetest set