cs376 computer vision lecture 6: optical flowhuangqx/cs376_lecture_6.pdf · 2019. 2. 11. · cs376...

CS376 Computer VisionLecture 6: Optical Flow

Qixing Huang

Feb. 11th 2019

Slides Credit: Kristen Grauman and Sebastian Thrun, Michael Black, Marc Pollefeys

2

Optical Flow

Image sequence(single camera)

Tracked sequence 3D structure+

3D trajectory

Image tracking3D computation

3

)1( +tI

What is Optical Flow?

Optical Flow

}{),( iptI

1p

2p

3p

4p

1v

2v

3v

4v

}{ iv

Velocity vectors

A common assumption is brightness constancy:

)1,(),( ++= tvpItpI iii

Optical flow is the 2D projection of the physical movement of points relative to the observer

When does Brightness Assumption Break down?

• TV is based on illusory motion – the set is stationary yet things seem to move

• A uniform rotating sphere – nothing seems to move, yet it is rotating

• Changing directions or intensities of lighting can make things seem to move – for example, if the specular highlight on a rotating sphere moves

• Muscle movement can make some spots on a cheetah move opposite direction of motion

5

Optical Flow Assumptions:Brightness Constancy

* Slide from Michael Black, CS143 2003

6

Optical Flow Assumptions:

* Slide from Michael Black, CS143 2003

Neighboring pixels tend to have similar motions

When does this break down?

7

Optical Flow Assumptions:

* Slide from Michael Black, CS143 2003

• The image motion of a surface path changes gradually over time

1D Optical Flow

9

Optical Flow: 1D CaseBrightness Constancy Assumption:

)),(()),(()( dttdttxIttxItf ++=

0)(

=

+

txt t

I

t

x

x

I

Ix v It

x

t

I

Iv =

{0

)(=

t

xfBecause no change in brightness with time

2D Optical Flow

11

From 1D to 2D tracking

0)(

=

+

txt t

I

t

x

x

I1D:

0)(

=

+

+

txtt t

I

t

y

y

I

t

x

x

I2D:

0)(

=

+

+

txtt t

Iv

y

Iu

x

I

One equation but two velocity (u,v) unknowns…

12

How does this show up visually?Known as the “Aperture Problem”

13

Aperture Problem in Real Life

14

From 1D to 2D tracking

The Math is very similar:

v

x

),( txI )1,( +txI

1v

2v

3v

4v

x

y)1,,( +tyxI

),,( tyxI

x

t

I

Iv −

Aperture problem

bGv 1−−

=

p yyx

yxx

III

IIIG

aroundwindow 2

2

=

p ty

tx

II

IIb

aroundwindow

Window size here ~ 11x11

15

More Detail:Solving the aperture problem

• How to get more equations for a pixel? -- impose additional constraints

• most common is to assume that the flow field is smooth locally

• one method: pretend the pixel’s neighbors have the same (u,v)

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

Suppose a 5x5 window

16

Lukas-Kanade flow

• Prob: we have more equations than unknowns

– The summations are over all pixels in the K x K window

– This technique was first proposed by Lukas & Kanade (1981)

• described in Trucco & Verri reading

• Solution: solve least squares problem

– minimum least squares solution given by solution (in d) of:

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

17

Conditions for solvability

– Optimal (u, v) satisfies Lucas-Kanade equation

When is This Solvable?• ATA should be invertible

• ATA should not be too small due to noise

– eigenvalues l1 and l2 of ATA should not be too small

• ATA should be well-conditioned

– l1/ l2 should not be too large (l1 = larger eigenvalue)

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

18

Eigenvectors of ATA

– gradients along edge all point the same direction

– gradients away from edge have small magnitude

– is an eigenvector with eigenvalue

– What’s the other eigenvector of ATA?

• let N be perpendicular to

• N is the second eigenvector with eigenvalue 0

• The eigenvectors of ATA relate to edge direction and magnitude

• Suppose (x,y) is on an edge. What is ATA?

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

19

Edge

– large gradients, all the same

– large l1, small l2

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

20

Low texture region

– gradients have small magnitude

– small l1, small l2

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

21

High textured region

– gradients are different, large magnitudes

– large l1, large l2

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

22

Observation

• This is a two image problem BUT– Can measure sensitivity by just looking at one of the

images!

– This tells us which pixels are easy to track, which are hard• very useful later on when we do feature tracking...

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

23

Errors in Lukas-KanadeWhat are the potential causes of errors in this procedure?

– Suppose ATA is easily invertible

– Suppose there is not much noise in the image

• When our assumptions are violated

– Brightness constancy is not satisfied

– The motion is not small

– A point does not move like its neighbors

• window size is too large

• what is the ideal window size?

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

24

– Can solve using Newton’s method

• Also known as Newton-Raphson method

– Lukas-Kanade method does one iteration of Newton’s method

• Better results are obtained via more iterations

Improving accuracy• Recall our small motion assumption

• This is not exact

– To do better, we need to add higher order terms back in:

• This is a polynomial root finding problem

It-1(x,y)

It-1(x,y)

It-1(x,y)

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

25

Iterative Refinement

• Iterative Lukas-Kanade Algorithm1. Estimate velocity at each pixel by solving Lucas-

Kanade equations

2. Warp I(t-1) towards I(t) using the estimated flow field

- use image warping techniques

3. Repeat until convergence

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

26

Revisiting the small motion assumption

• Is this motion small enough?

– Probably not—it’s much larger than one pixel (2nd order terms dominate)

– How might we solve this problem?

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

27

Reduce the resolution!

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

28

Optical Flow Results

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

29

Optical Flow Results

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

30

31

* From Marc Pollefeys COMP 256 2003

32

Affine Flow

* Slide from Michael Black, CS143 2003

33

Horn & Schunck algorithm

Additional smoothness constraint :

,))()(( 2222 dxdyvvuue yxyxs +++=

besides Opt. Flow constraint equation term

,)( 2dxdyIvIuIe tyxc ++=

minimize es+aec

➔ * From Marc Pollefeys COMP 256 2003

34

Horn & Schunck algorithm

In simpler terms: If we want dense flow, we need to regularize what happensin ill conditioned (rank deficient) areas of the image. We take the old cost function:

( )

++−=Nx

d tdxItxId2

)1,(),(minarg

And add a regularization term to the cost:

( ) dtdxItxIdNx

d a+++−=

2)1,(),(minarg

Convex Program!

We will see a lot of such formulations in in robust regression!

Discussion: What are the other methods to improve optical flows?