ulas bagci bagci@ucfbagci/teaching/computervision17/lec8.pdfpiaalls & alternaves • brightness...

CAP5415-ComputerVisionLecture8-Mo8onModels,Feature

Tracking,andAlignment

[email protected]

1

Readings

•  Szeliski,R.Ch.7•  Bergenetal.ECCV92,pp.237-252.•  Shi,J.andTomasi,C.CVPR94,pp.593-600.•  Baker,S.andMaPhews,I.IJCV2004,pp.221-255.

•  SlideCredits:Szeliski,ShahandB.Freeman

2

Recap:Es8ma8ngOp8calFlow

3

•  Assumetheimageintensityisconstant

( )tyxI ,, ( )dttdyydxxI +++ ,,=

I

Time=t Time=t+dt

FirstAssump8on:BrightnessConstraint

4

( )tyxI ,, ( )dttdyydxxI +++ ,, !I(x(t) + u.�t, y(t) + v.�t)� I(x(t), y(t), t) ⇡ 0

AssumingIisdifferen8ablefunc8on,andexpandthefirsttermusingTaylor’sseries:

@I

@x

dx

dt

+@I

@y

dy

dt

+@I

@t

= 0

Ix

u+ Iy

v + It

= 0Compactrepresenta8on

Brightnessconstancyconstraint

SecondAssump8on:GradientConstraint

5

Lecture10:Mo8onModels,FeatureTracking,andAlignment

Velocityvectorisconstantwithinasmallneighborhood(LUCASANDKANADE)

E(u, v) =

Z

x,y

(Ix

u+ I

y

v + I

t

)2dxdy

@E(u, v)

@u=

@E(u, v)

@v= 0

2(Ix

u+ Iy

v + It

)Ix

= 0

2(Ix

u+ Iy

v + It

)Iy

= 0

Recap:Lucas-Kanade

6


PI2x

PIx

IyP

Ix

Iy

PI2y

� uv

�= �

PIx

ItP

Iy

It

�

Txx

Txy

Txy

Tyy

� uv

�= �

Txt

Tyt

�StructuralTensorrepresenta8on

u =Tyt

Txy

� Txt

Tyy

Txx

Tyy

� T 2xy

and v =Txt

Txy

� Tyt

Txx

Txx

Tyy

� T 2xy

Piaalls&Alterna8ves

•  Brightnessconstancyisnotsa8sfied– Correla8onbasedmethodcouldbeused

•  Apointmaynotmovelikeitsneighbors– Regulariza8onbasedmethods

•  Themo8onmaynotbesmall(Taylordoesnothold!)– Mul8-scalees8ma8oncouldbeused

7


Mul8-ScaleFlowEs8ma8on

8


imageIt-1 imageI

GaussianpyramidofimageIt GaussianpyramidofimageIt+1

imageIt+1imageItu=10pixels

u=5pixels

u=2.5pixels

u=1.25pixels

Recap:Horn&Schunck

•  Globalmethodwithsmoothnessconstrainttosolveapertureproblem

•  Minimizeaglobalenergyfunc8on

•  Takepar8alderiva8vesw.r.t.uandv:

9


E(u, v) =

Z

x,y

[(Ix

u+ I

y

v + I

t

)2 + ↵

2(|ru|2 + |rv|2)]dxdy

(Ix

u+ Iy

v + It

)Ix

� ↵2ru = 0

(Ix

u+ Iy

v + It

)Iy

� ↵2rv = 0

GlobalMo8onModels(Parametric)Allpixelsareconsideredtosummarizeglobalmo8on!•  2DModels

– Affine– Quadra8c–  Planarprojec8ve(homography)

•  3DModels–  Inst.Cameramo8onmodels–  Homography+epipole–  Plane+parallax

10

Mo8onModels

11


Translation

2 unknowns

Affine

6 unknowns

Perspective

8 unknowns

3D rotation

3 unknowns

GlobalMo8on

12

EsOmatemoOonusingallpixelsintheimage

GlobalMo8on

13


GlobalMoOoncanbeusedto•  Removecamera

mo8on•  Object-based

segmenta8on•  generatemosaics

GlobalMo8on

14


GlobalMoOoncanbeusedto•  Removecamera

mo8on•  Object-based

segmenta8on•  generatemosaics

Recap:ObjectTracking

•  Trackanobjectoverasequenceofimages

15

ChallengesinObjectTracking

16


•  Whichfeaturestotrack?

17


•  Whichfeaturestotrack?•  Efficienttracking

18


•  Whichfeaturestotrack?•  Efficienttracking•  Appearanceconstraintviola8on•  …

19


•  Whichfeaturestotrack?•  Efficienttracking•  Appearanceconstraintviola8on•  …

20

Shi-TomasiFeatureTracker

•  GoodFeaturestoTrack

21


•  GoodFeaturestoTrack– FindgoodfeaturesusingeigenvaluesofHessianmatrix(thresholdonthesmallesteigenvaluewhencompu8ngHarriscornerdetec8on)

22



– TrackfromframetoframewithLK

23



– TrackfromframetoframewithLK– Checkconsistencyoftracksby“affineregistra8on”tothefirstobservedinstanceofthefeature

24


25

ShiandTomasiCVPR1994GoodFeaturesToTrack.

KLTTracking

•  KLT:Kanade-Lucas-Tomasi

26

KLTTracking

•  KLT:Kanade-Lucas-Tomasi•  Trackingdealswithes8ma8ngthetrajectoryofanobjectintheimageplaneasitmovesaroundascene

27

KLTTracking


•  Objecttracking(car,airplane,person)

28

KLTTracking


•  Objecttracking(car,airplane,person)•  Featuretracking(Harriscorners)

29

KLTTracking


•  Objecttracking(car,airplane,person)•  Featuretracking(Harriscorners)•  Mul8pleobjecttracking

30

KLTTracking


•  Objecttracking(car,airplane,person)•  Featuretracking(Harriscorners)•  Mul8pleobjecttracking•  Trackinginsingle/mul8plecamera(s)

31

KLTTracking


•  Objecttracking(car,airplane,person)•  Featuretracking(Harriscorners)•  Mul8pleobjecttracking•  Trackinginsingle/mul8plecamera(s)•  Trackinginfixed/movingcamera 32

KLTTrackingAlgorithm•  FindGoodFeaturesToTrack

– HarrisCorners(thresholdedonsmallesteigenvalues)

•  UseLKalgorithmtofindop8calflows•  UseCoarse-to-Finestrategytodealwithlargemovements

•  Whencrea8nglongtracks,checkappearanceofregisteredpatchagainstappearanceofini8alpatchtofindpointsthathavedrited

33

RecentDevelopmentsatOp8calFlow

•  StartwithLKorsimilarmethods+ Gradientconsistency+ Energyminimiza8onwithsmoothingterm+ Regionmatching+ KeyPointmatching

34Large displacement optical flow, Brox et al., CVPR 2009

Region-based +Pixel-based +Keypoint-based

RecentDevelopmentsatOp8calFlow•  UseofMachineLearning

– DeepLearning(ICCV2015,Fischeretal.,FlowNet)

35

DeepFlow(LargeDisplacementOp8calFlow)•  Basicallyitisamatchingalgorithmwithvaria8onalapproach

[Weinzaepfeletal.,ICCV2013].

•  Densecorrespondence(matching)•  Self-smoothmatching•  Largedisplacementop8calflow

–  hPps://www.youtube.com/watch?v=k_wkDLJ8lJE

9/22/16

36

•  CanweuseSIFTfeaturesfortracking?

9/22/16

37


Ex:SIFTTracking

38

à Frame 0 Frame 100

Howtoevaluatecorrectnessofop8calflows?

9/22/16

39


Op8calFlow-Quan8ta8veEvalua8on

•  Whereu=(u,v)iscomputed,u=(u*,v*)groundtruthvelocityvectors.

40

Eep2 =p

(u� u⇤)2 + (v � v⇤)2

Eep1 = |u� u⇤|+ |v � v⇤|

Eang = arccos (

uTu⇤

|u||u⇤| )

Interpreta8onofOp8calFlowFields

41


42

ObjectfeaturesallhaveZerovelocity.


43


44

ObjectismovingtotheRight.


45


46

ObjectismovingDirectlytowardthecamerathatissta8onary


47


48

Cameraismovingintothescene,andanobjectmovingpassedthecamera


49


50

Objectisrota8ngaboutthelineofsighttothecamera


51


52

Objectisrota8ngaboutanaxisperpendiculartothelineofsight.

Applica8oninImageAlignment

•  Mo8oncanbeusedforimagealignment

53⎥⎦

⎤⎢⎣

⎡⎥⎦

⎤⎢⎣

⎡=⎥

⎦

⎤⎢⎣

⎡

yx

dcba

yx''

Pixelloca8onsat8metPixelloca8onsat8met+1

Prac8ce:HomogenousCoordinates

54

y

x

tyytxx

+=

+=

''

Q:HowcanwerepresenttranslaOonasa3x3matrix?


55

y

x

tyytxx

+=

+=

''


⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=

1001001

y

x

tt

ranslationT


56

y

x

tyytxx

+=

+=

''


⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=

1001001

y

x

tt

ranslationT⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

+

+

=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

111001001

1''

y

x

y

x

tytx

yx

tt

yx

tx=2ty=1

Prac8ce:Basic2DTransforma8ons

57

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

ΘΘ

Θ−Θ

=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

11000cossin0sincos

1''

yx

yx

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

11001001

1''

yx

tt

yx

y

x

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

11000101

1''

yx

shsh

yx

y

x

Translate

Rotate Shear

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

11000000

1''

yx

ss

yx

y

x

Scale

AffineTransforma8on

58

•  Affinetransforma8onsarecombina8onsof…–  Lineartransforma8ons,and–  Transla8ons

•  Proper8esofaffinetransforma8ons:–  Origindoesnotnecessarilymaptoorigin–  Linesmaptolines–  Parallellinesremainparallel–  Ra8osarepreserved–  Closedundercomposi8on–  Modelschangeofbasis

⎥⎥

⎦

⎤

⎢⎢

⎣

⎡

⎥⎥

⎦

⎤

⎢⎢

⎣

⎡=

⎥⎥

⎦

⎤

⎢⎢

⎣

⎡

wyx

fedcba

wyx

100''

⎥⎥

⎦

⎤

⎢⎢

⎣

⎡

⎥⎥

⎦

⎤

⎢⎢

⎣

⎡=

⎥⎥

⎦

⎤

⎢⎢

⎣

⎡

wyx

ihgfedcba

wyx

'''

projec8ve

AffineTransforma8on

59

⎥⎥

⎦

⎤

⎢⎢

⎣

⎡

⎥⎥

⎦

⎤

⎢⎢

⎣

⎡=

⎥⎥

⎦

⎤

⎢⎢

⎣

⎡

wyx

fedcba

wyx

100''

⎥⎥

⎦

⎤

⎢⎢

⎣

⎡

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⎥⎥

⎦

⎤

⎢⎢

⎣

⎡

⎥⎥⎦

⎤

⎢⎢⎣

⎡ΘΘΘ−Θ

⎥⎥

⎦

⎤

⎢⎢

⎣

⎡=

⎥⎥

⎦

⎤

⎢⎢

⎣

⎡

wyx

sysx

tytx

wyx

1000000

1000cossin0sincos

1001001

'''

p’ = T(tx,ty) R(Θ) S(sx,sy) p

Affinematrixdecomposi8onTranslaOon+rotaOon+scaling

Ques8ons?

•  PA2willbeincludingop8calflowes8ma8ons.

9/22/16

60


ulas bagci bagci@ucfbagci/teaching/computervision17/lec8.pdfpiaalls & alternaves • brightness...

Documents