iterative image registration: lucas & kanade revisited kentaro toyama vision technology group...

Iterative Image Registration:

Lucas & Kanade Revisited

Kentaro Toyama

Vision Technology Group

Microsoft Research

Every writer creates his own precursors. His work modifies our conception of the past, as it will modify the future.

Jorge Luis Borges

History

• Lucas & Kanade (IUW 1981)

LK BAHH ST S BJ HB BL G SI CETSC

• Bergen, Anandan, Hanna, Hingorani (ECCV 1992)

• Shi & Tomasi (CVPR 1994)

• Szeliski & Coughlan (CVPR 1994)

• Szeliski (WACV 1994)

• Black & Jepson (ECCV 1996)

• Hager & Belhumeur (CVPR 1996)

• Bainbridge-Smith & Lane (IVC 1997)

• Gleicher (CVPR 1997)

• Sclaroff & Isidoro (ICCV 1998)

• Cootes, Edwards, & Taylor (ECCV 1998)

Image Registration

Applications

Applications

• Stereo


Applications

• Stereo

• Dense optic flow


Applications

• Stereo


• Image mosaics


Applications

• Stereo


• Image mosaics

• Tracking


Applications

• Stereo


• Image mosaics

• Tracking

• Recognition


?

Lucas & Kanade

#1

Derivation

L&K Derivation 1

I0(x)

)('0 xI

h

xIhxIh

)()(lim 00

0

)('0 xI

L&K Derivation 1

)('0 xI

h

xIhxI )()( 00

h I0(x)

I0(x+h)

L&K Derivation 1

h I0(x)

)('0 xI

h

xIxI )()( 0

I(x)

L&K Derivation 1

h I0(x)

h)(

)()('0

0

xI

xIxI

I(x)

L&K Derivation 1

I0(x)

h

Rx xI

xIxI

R )(

)()(

||

1'0

0

RI(x)

L&K Derivation 1

I0(x)

h

RxxxI

xIxIxw

xw )(

)]()()[(

)(

1'0

0

I(x)

L&K Derivation 1

h0 I0(x)

0h

I(x)

RxxxI

xIxIxw

xw )(

)]()()[(

)(

1'0

0

L&K Derivation 1

1h

Rxx

hxI

hxIxIxw

xwh

)(

)]()()[(

)(

1

0'0

000

I0(x+h0)

I(x)

L&K Derivation 1

2h

Rxx

hxI

hxIxIxw

xwh

)(

)]()()[(

)(

1

1'0

101

I0(x+h1)

I(x)

L&K Derivation 1

1kh

Rx k

k

x

k hxI

hxIxIxw

xwh

)(

)]()()[(

)(

1'0

0

I0(x+hk)

I(x)

L&K Derivation 1

1kh

Rx k

k

x

k hxI

hxIxIxw

xwh

)(

)]()()[(

)(

1'0

0

I0(x+hf)

I(x)

Lucas & KanadeDerivation

#2

L&K Derivation 2

• Sum-of-squared-difference (SSD) error

E(h) = [ I(x) - I0(x+h) ]2x R

E(h) [ I(x) - I0(x) - hI0’(x) ]2x R

L&K Derivation 2

2[I0’(x)(I(x) - I0(x) ) - hI0’(x)2] x Rh

E

I0’(x)(I(x) - I0(x))x R h I0’(x)2

x R

= 0

Comparison

I0’(x)[I(x) - I0(x)] h I0’(x)2

x

x

h

w(x)[I(x) - I0(x)]

w(x)x

x I0’(x)

Comparison

I0’(x)[I(x) - I0(x)] h I0’(x)2

x

h

x

w(x)[I(x) - I0(x)]

w(x)x

x I0’(x)

Generalizations

Original

h ) = x R

(E [I( x ) - (x ]2)+ h I

Original

• Dimension of image

h ) = x R

(E [I( x ) - (x ]2)+ h

1-dimensional

I


Generalization 1a


h ) = x R

(E [I( x ) - (x ]2)+ h

y

xx2D:

I


Generalization 1b


h ) = x R

(E [I( x ) - (x ]2)+ h

1

y

x

xHomogeneous 2D:

I


Problem A


Does the iteration converge?

Problem A

Local minima:

Problem B

- I0’(x)(I(x) - I0(x))x R h I0’(x)2

x R

h is undefined if I0’(x)2 is zerox R


Zero gradient:

Problem B

Zero gradient:

?

Problem B’

- (x)(I(x) - I0(x))x R

hy 2

x R

y

I )(0 xy

I

)(0 x

Aperture problem:


Problem B’

No gradient along one direction:

?

Solutions to A & B

• Possible solutions:– Manual intervention


• Possible solutions:– Manual intervention– Zero motion default


Solutions to A & B

• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”


Solutions to A & B

• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”– Reliance on good features


Solutions to A & B

• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”– Reliance on good features– Temporal filtering


Solutions to A & B

• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”– Reliance on good features– Temporal filtering– Spatial interpolation / hierarchical estimation


Solutions to A & B

• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”– Reliance on good features– Temporal filtering– Spatial interpolation / hierarchical estimation– Higher-order terms


Solutions to A & B

Original

h ) = x R

(E [I( x ) - (x ]2)+ h I

Original

• Transformations/warping of image

h ) = x R

(E [I( x ) -I(x ]2)+ h

Translations:

y

x

h


Problem C

What about other types of motion?

Generalization 2a


A, h) = x R

(E [I(Ax ) - (x ]2)+h

Affine:

dc

baA

y

x

h

I


Generalization 2a

Affine:

dc

baA

y

x

h

Generalization 2b


A ) = x R

(E [I( A x ) - (x ]2)

Planar perspective:

187

654

321

aa

aaa

aaa

A

I


Generalization 2b

Planar perspective:

187

654

321

aa

aaa

aaa

A

Affine +

Generalization 2c


h ) = x R

(E [I( f(x, h) ) - (x ]2)

Other parametrized transformations

I


Generalization 2c

Other parametrized transformations

Problem B”

-(JTJ)-1 J (I(f(x,h)) - I0(x)) h ~

Generalized aperture problem:


- I0’(x)(I(x) - I0(x))x R h I0’(x)2

x R

Problem B”

?

Generalizedaperture problem:

Original

h ) = x R

(E [I( x ) - (x ]2)+ h I

Original

• Image type

h ) = x R

(E [I( x ) - (x ]2)+ h

Grayscale images

I


Generalization 3

• Image type

h ) = x R

(E ||I( x ) -I(x ||2)+ h

Color images


Original

h ) = x R

(E [I( x ) - (x ]2)+ h I

Original

• Constancy assumption

h ) = x R

(E [I( x ) -I(x ]2)+ h

Brightness constancy


Problem C

What if illumination changes?

Generalization 4a


h, )=x R

(E [I( x ) - I(x ]2)++ h

Linear brightness constancy


Generalization 4a

Generalization 4b


h,) = x R

(E [I( x ) - B(x]2)+ h

Illumination subspace constancy


Problem C’

What if the texture changes?

Generalization 4c


h,) = x R

(E [I( x ) - ]2+ h

Texture subspace constancy

B(x)


Problem D

Convergence is slower as #parameters increases.

• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc.


Solutions to D

• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization

Solutions to D


• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization – Offline precomputation

Solutions to D



• Difference decomposition

LK BAHH ST S BJ HB G SI CETSC

Solutions to D

BL

Solutions to D




– Improvements in gradient descent


Solutions to D

BL

• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization– Offline precomputation


– Improvements in gradient descent• Multiple estimates of spatial derivatives


Solutions to D

BL

Solutions to D

• Multiple estimates / state-space sampling

Generalizations

x R

[I( x ) - (x ]2)+ h I

Modifications made so far:

Original

• Error norm

h ) = x R

(E [I( x ) -I(x ]2)+ h

Squared difference:


Problem E

What about outliers?

Generalization 5a

• Error norm

h ) = x R

(E (I( x ) -I(x ))+ h

Robust error norm:

22

2

)(uk

uuρ


Original

h ) = x R

(E [I( x ) - (x ]2)+ h I

Original

• Image region / pixel weighting

h ) = x R

(E [I( x ) -I(x ]2)+ h

Rectangular:


Problem E’

What about background clutter?

Generalization 6a


h ) = x R

(E [I( x ) -I(x ]2)+ h

Irregular:


Problem E”

What about foreground occlusion?

Generalization 6b


h ) = x R

(E [I( x ) -I(x ]2)+ h

Weighted sum:

w(x)


Generalizations

x R

[I( x ) - (x ]2)+ h I

Modifications made so far:

Generalizations: Summary

= x R

(I( ) - w(x) (x ))h )(E f(x, h)

h ) = x R

(E [I( x ) - (x ]2)+ h I

Foresight













Summary

• Generalizations– Dimension of image– Image transformations / motion models– Pixel type– Constancy assumption– Error norm– Image mask

L&K ?Y

Y

n

Y

n

Y

Summary

• Common problems:– Local minima– Aperture effect– Illumination changes– Convergence issues– Outliers and occlusions

L&K ?Y

maybe

Y

Y

n

• Mitigation of aperture effect:– Manual intervention– Zero motion default– Coefficient “dampening”– Elimination of poor textures– Temporal filtering– Spatial interpolation / hierarchical – Higher-order terms

Summary

L&K ?n

n

n

n

Y

Y

n

Summary

• Better convergence:– Coarse-to-fine, filtering, etc.– Selective parametrization – Offline precomputation


– Improvements in gradient descent• Multiple estimates of spatial derivatives

L&K ?Y

nmaybe

maybe

maybe

maybe

Hindsight












iterative image registration: lucas & kanade revisited kentaro toyama vision technology group...

Documents

wx x x

derivation slide

b slide

i0xi0x slide

x h wxix

h x wxix

e rx e r slide

h i0xi0x ixix slide