kalman ip cv-without motion

8/13/2019 Kalman IP CV-Without Motion

1/49

KALMAN FILTER

APPLICATION IN COMPUTER VISION &

IMAGE PROCESSING

Hamid Bazargani

School of Electrical & Computer Eng.


2/49

Outline:

A brief introduction to Kalman filter

Classical Kalman Filter

Kalman & Stereo vision localization

Different extensions of Kalman Filter

Image and Video De-noising (Kalman approach)

Kalman Tracking for Robust pose estimation


3/49

Introduction:

Kalman is a recurrent algorithm relying on series of uncertain

measurements to optimally estimate an unknown variable.

The Kalman filter is 50 years old but is still a popular method for

sensory information analysis .

The Kalman applications span area of navigation systems,aerospace, robotics, information processing , computer vision, etc

Main properties:

Requires small computational cost

State space representation and ease of implementationRecursive algorithm which only uses last state and current

observation


4/49

Formulation:

Where F is transition matrix, G is a control input matrix . H is output

matrix.

Random variables w and v are process and measurement noiserespectively.


5/49

Predict or time update Correct or measurement update

Kalman stages


6/49

Multivariate Gaussians:

posterior

prior

Measurement

evidence

Markov model:

)()|()|( xpxzpzxp


7/49


posterior

prior

Measurement

evidence

)()(2

1exp)( 1 xxxp T

)()(2

1exp)|( 1 zxzxxzp T

)()(

11111new

z

++=

( ) 111new +=


8/49


CS373 - Programming a Robotic Car - Unit 2

Kalman Filter Prediciton - CS373 Unit 2 - Udacity

By Sebastian Thrun


9/49

Kalman & Stereo Vision Localization:

Stereo localization requires both cameras information at each time

step.


10/49

Kalman & Stereo Vision Localization:

There is a small time delay between the two cameras imagecapturing. This time lag causes an error in estimation of object

position in Cartesian coordinates.


11/49

Non-linearity & Kalman Filter:

[Refer to Video1]

Python and Blender used for data acquisition which brings Physics engine togenerate exact trajectory.

Blender communicate with matlab over UDP protocol.


12/49


Extended Kalman Filter

Figures from CS-223 (Stanford)


13/49





14/49





15/49

Adaptive Kalman Filter:

Adaptive Fading Kalman Filter

Innovation based Adaptive Kalman Filter


16/49

Adaption

Non-linearityKalman EKF

AEKF

AKF

Kalman Filter Categories


17/49

SV Results

Noise level 05.0=n Noise level 5.0=n


18/49

Image and Video de-noising:

noise is assumed to be signal-independent at each pixel


19/49


Wiener Filter

Bilateral Filter

Median Filter

Non-Local Means Filtering


20/49


Wiener Filter

Assume Stationary signalSignal and noise are independent with constant PSD


21/49


Wiener Filter


22/49


Wiener Filter 15=n


23/49


Wiener FilterFiltered


24/49


Non-Local Means Filter

Each pixel is replaced by weighted average of its neighborhoods.

Larger patch size leads to expensive computation. while Smaller patch

size results in poor de-noising effect.

All weights should be normalized .


25/49


Non-Local Means Filter 15=n


26/49


Non-Local Means Filter


27/49


Comparison

15=noriginal

Wiener filter NLM filter

Noisy


28/49


Comparison

0

2

4

68

10

12

14

PSNR (db) PSNR (db)

Vid d i i


29/49

Video de-noising:

Each pixel is replaced by 3D spatio-temporal weighted average of its

neighborhoods.

M i i i & K l Fil


30/49

Motion estimation & Kalman Filter

Optical Flow Algorithms

Block Matching Algorithms

Full Search Algorithm (FSA)

Three Step Algorithm (TSA)

Four Step Algorithm (FSA)

Diamond Search Algorithm (DSA)

Adaptive Rood Pattern Search

M ti ti ti & K l Filt


31/49


Why Kalman Filter?

Motion estimation inaccuracy when illumination is changedInaccuracy based on noise

The size of moving object may differ from searching region size

Fails in presence of occlusion

M ti ti ti & K l Filt


32/49


2D Causal AR model:

B(m,n)

B(m,n-1)

B(m-1,n)

),(),1()1,(),(

),(),1()1,(),(

,,10,01,

,,10,01,

nmwnmvbnmvbnmv

nmwnmvanmvanmv

yiyiyiyi

xixixixi

++=

++=



33/49


State space representation :

+

=

0

0

)1,(

),1(

)1,(

),1(

*

0100

00

0001

00

),1(

),(

),1(

),(

,

,

,

,

1101

1101

,

,

,

,

q

q

nmv

nmv

nmv

nmv

bb

aa

nmv

nmv

nmv

nmv

xi

xi

xi

xi

xi

xi

xi

xi

+

=

=

0

0

),1(

),(

),1(

),(

*01

00

00

01

),(

),(

,

,

,

,

,

,

r

r

nmv

nmv

nmv

nmv

nmv

nmvz

xi

xi

xi

xi

yi

xi

=

0000

00

0000

00

22

22

qq

qq

Q

=

0000

00

0000

00

22

22

rr

rr

R



34/49




35/49




36/49




37/49




38/49




39/49


Green arrows show motion vectors

Red arrows show corrected motion vectors

Kalam uses corrected version of neighboring vectors if available

Motion estimation Results


40/49

[Refer to Video2]

Results are generated using the following approaches:

2D 3x3 spatial domain NLM

3D 3x3x3 spatio-temporal domain NLM

3D 5x3x3 spatio-temporal domain NLM

Motion estimation Results


41/49

[Refer to Video2]



42/49

g p

Object Recognition

Object recognition requires set of 2D correspondences between

target image and video frame.

Robust RANSAC (RANdom SAmple Consensus) is used to robustly

estimate the pose against wrong matches.

Robust pose estimation brings an exhaustive computational cost.



43/49

g p

Tracking

Feature points tracking

Tracking the region of search for matches.

Keep tracking of estimated pose after pose estimation process.

Keep tracking of inliers (best matches) found from RANSAC loop.

(My Contribution)



44/49

g p

Pose estimation:

Point

Matching

RANSAC Loop

VideoFrame points Target Points

M-estimator

(LSM)Kalman tracker



45/49

Pose estimation:

[Refer to Video3]

Implementation


46/49

Programs are written with C++ using OpenCV library

C++ implementation of generic Kalman Filter algorithm

C++ implementation of Non-Local Means algorithm

C++ implementation of FSA Block Matching algorithm

C++ implementation of Wiener filter for image denoising

Matlab full implementation of Conventional, Extended , Adaptive Fading and

Innovation-based Adaptive Kalman filter

Will be available for download soon onhttps://github.com/hamidb

Conclusion:


47/49

We studied Kalman Filter as an adaptive filter which can be applied for

Smoothing, prediction , filtering and tracking

Three main extensions of Kalman filter including conventional, extended

and adaptive

Kalman filter are reviewed.

Applying Kalman Filter in Video de-noising and compared the result with

other filters like Wiener filter

Applying Kalman filter in computer vision applications such as multiple view

geometry and pose estimation

References:


48/49

1. Dan Simon. Optimal State Estimation: Kalman, H Infinity, and Nonlinear Approaches. Wiley-Interscience, 1

edition, August 2006.

2. S. Sangsuk-Iam and T. E. Bullock. Analysis of discrete time kalman filtering under incorrect noise

covariances. Automatic Control, IEEE Transactions

3. Qijun Xia, Ming Rao, Yiqun Ying, and Xuemin Shen. Adaptive fading kalman filter with an application.

Automatica

4. Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: IEEE Conference on

Computer Vision and Pattern Recognition (CVPR). (2005)

5. Buades, A., Coll, B., Morel, J.M.: Denoising image sequences does not require motion estimation. In: IEEE

International Conference on Advanced Video and Signal Based Surveillance. (2005)

6. Han, Yu Bing and Chen, Rushan: Efficient video denoising based on dynamic nonlocal means . Journal of

Image Vision Comput. (2012)7. Chung-Ming Kuo; Chaur-Heh Hsieh; Yue-Dar Jou; Hsieh-Cheng Lin; Po-Chiang Lu, "Motion estimation for

video compression using Kalman filtering," Broadcasting, IEEE Transactions on, Jun 1996

8. Bazargani, H.; Omidi, E.; Talebi, H.A., "Adaptive Extended Kalman Filter for asynchronous shuttering error of

stereo vision localization," Robotics and Biomimetics (ROBIO), 2012 IEEE International Conference on, vol.,

no., pp.2254,2259, 11-14 Dec. 2012

9. Ruiz, V.; Fotopoulos, V.; Skodras, A.N.; Constantinides, A.G., "An 88-block based motion estimation using

Kalman filter," Image Processing, 1997.10. Antoni Buades and Bartomeu Coll and Jean Michel Morel, On image denoising methods,Technical Note,

CMLA (Centre de Mathematiques et de Leurs Applications,(2004)

Th k f Att ti


49/49

Thank you for your Attention.

http://www.slideshare.net/hamidbazargani/venn-diagrams-29181371

kalman ip cv-without motion

Documents