Brightness Constancy16-385 Computer Vision (Kris Kitani)

Carnegie Mellon University

Optical FlowProblem Definition

Assumptions

Brightness constancy

Small motion

Given two consecutive image frames, estimate the motion of each pixel

Brightness constancy

· · ·

(x(1), y(1))

(x(2), y(2))

(x(k), y(k))

Scene point moving through image sequence

Assumption 1

Brightness constancy

I(x, y, 1) I(x, y, 2)

· · ·

I(x, y, k)

(x(1), y(1))

(x(2), y(2))

(x(k), y(k))

Scene point moving through image sequence

Assumption 1

Brightness constancy

I(x, y, 1) I(x, y, 2)

· · ·

I(x, y, k)

(x(1), y(1))

(x(2), y(2))

Assumption:Brightness of the point will remain the same

(x(k), y(k))

Scene point moving through image sequence

Assumption 1

Brightness constancy

I(x, y, 1) I(x, y, 2)

· · ·

I(x, y, k)

(x(1), y(1))

(x(2), y(2))

Assumption:Brightness of the point will remain the same

I(x(t), y(t), t) = C

constant

(x(k), y(k))

Scene point moving through image sequence

Assumption 1

Small motion

I(x, y, t) I(x, y, t+ �t)

Assumption 2

Small motion

I(x, y, t)

(x, y)

(x+ u�t, y + v�t)

(x, y)

I(x, y, t+ �t)

Assumption 2

Small motion

I(x, y, t)

(x, y)

(x+ u�t, y + v�t)

(x, y)

(u, v) (�x, �y) = (u�t, v�t)Optical flow (velocities): Displacement:

I(x, y, t+ �t)

Assumption 2

Small motion

I(x, y, t)

(x, y)

(x+ u�t, y + v�t)

(x, y)

(u, v) (�x, �y) = (u�t, v�t)Optical flow (velocities): Displacement:

I(x+ u�t, y + v�t, t+ �t) = I(x, y, t)

… the brightness between two consecutive image frames is the same

I(x, y, t+ �t)

For a really small space-time step…

Assumption 2

dI

dt

=@I

@x

dx

dt

+@I

@y

dy

dt

+@I

@t

= 0

total derivative partial derivative

Equality is not obvious. Where does this come from?

These assumptions yield the …

Brightness Constancy Equation

dI

dt

=@I

@x

dx

dt

+@I

@y

dy

dt

+@I

@t

= 0

total derivative partial derivative

Where does this come from?

These assumptions yield the …

Brightness Constancy Equation

proof!

I(x+ u�t, y + v�t, t+ �t) = I(x, y, t)For small space-time step, brightness of a point is the same

I(x+ u�t, y + v�t, t+ �t) = I(x, y, t)For small space-time step, brightness of a point is the same

Insight:If the time step is really small,

we can linearize the intensity function

I(x+ u�t, y + v�t, t+ �t) = I(x, y, t)

Multivariable Taylor Series Expansion (First order approximation, two variables)

f(x, y) ⇡ f(a, b) + f

x

(a, b)(x� a)� f

y

(a, b)(y � b)

I(x+ u�t, y + v�t, t+ �t) = I(x, y, t)

Multivariable Taylor Series Expansion (First order approximation, two variables)

f(x, y) ⇡ f(a, b) + f

x

(a, b)(x� a)� f

y

(a, b)(y � b)

assuming small motionI(x, y, t) +

@I

@x

�x+@I

@y

�y +@I

@t

�t = I(x, y, t)

I(x+ u�t, y + v�t, t+ �t) = I(x, y, t)

assuming small motionI(x, y, t) +

@I

@x

�x+@I

@y

�y +@I

@t

�t = I(x, y, t)

cancel terms

Multivariable Taylor Series Expansion (First order approximation, two variables)

f(x, y) ⇡ f(a, b) + f

x

(a, b)(x� a)� f

y

(a, b)(y � b)

fixed point

partial derivative

I(x+ u�t, y + v�t, t+ �t) = I(x, y, t)

assuming small motionI(x, y, t) +

@I

@x

�x+@I

@y

�y +@I

@t

�t = I(x, y, t)

@I

@x

�x+@I

@y

�y +@I

@t

�t = 0 cancel terms

Multivariable Taylor Series Expansion (First order approximation, two variables)

f(x, y) ⇡ f(a, b) + f

x

(a, b)(x� a)� f

y

(a, b)(y � b)

I(x+ u�t, y + v�t, t+ �t) = I(x, y, t)

assuming small motion

divide by

take limit

�t

�t ! 0

I(x, y, t) +@I

@x

�x+@I

@y

�y +@I

@t

�t = I(x, y, t)

@I

@x

�x+@I

@y

�y +@I

@t

�t = 0

Multivariable Taylor Series Expansion (First order approximation, two variables)

f(x, y) ⇡ f(a, b) + f

x

(a, b)(x� a)� f

y

(a, b)(y � b)

I(x+ u�t, y + v�t, t+ �t) = I(x, y, t)

assuming small motion

@I

@x

dx

dt

+@I

@y

dy

dt

+@I

@t

= 0

divide by

take limit

�t

�t ! 0

I(x, y, t) +@I

@x

�x+@I

@y

�y +@I

@t

�t = I(x, y, t)

@I

@x

�x+@I

@y

�y +@I

@t

�t = 0

Multivariable Taylor Series Expansion (First order approximation, two variables)

f(x, y) ⇡ f(a, b) + f

x

(a, b)(x� a)� f

y

(a, b)(y � b)

I(x+ u�t, y + v�t, t+ �t) = I(x, y, t)

assuming small motion

@I

@x

dx

dt

+@I

@y

dy

dt

+@I

@t

= 0

divide by

take limit

�t

�t ! 0

I(x, y, t) +@I

@x

�x+@I

@y

�y +@I

@t

�t = I(x, y, t)

@I

@x

�x+@I

@y

�y +@I

@t

�t = 0

Brightness Constancy Equation

Multivariable Taylor Series Expansion (First order approximation, two variables)

f(x, y) ⇡ f(a, b) + f

x

(a, b)(x� a)� f

y

(a, b)(y � b)

@I

@x

dx

dt

+@I

@y

dy

dt

+@I

@t

= 0

Ix

u+ Iy

v + It

= 0 shorthand notation

rI>v + It = 0 vector form

Brightness Constancy Equation

(x-flow) (y-flow)

(1 x 2) (2 x 1)

Ix

u+ Iy

v + It

= 0

(putting the math aside for a second…)

What do the term of the brightness constancy equation represent?

Ix

u+ Iy

v + It

= 0

(putting the math aside for a second…)

What do the term of the brightness constancy equation represent?

Image gradients (at a point p)

Ix

u+ Iy

v + It

= 0

(putting the math aside for a second…)

What do the term of the brightness constancy equation represent?

flow velocities

Image gradients (at a point p)

Ix

u+ Iy

v + It

= 0

(putting the math aside for a second…)

What do the term of the brightness constancy equation represent?

flow velocities

temporal gradient

How do you compute these terms?

Image gradients (at a point p)

spatial derivative

Ix

u+ Iy

v + It

= 0

I

x

=@I

@x

Iy =@I

@y

How do you compute …

spatial derivative

Ix

u+ Iy

v + It

= 0

I

x

=@I

@x

Iy =@I

@y

Forward difference Sobel filter Scharr filter

…

How do you compute …

spatial derivative temporal derivative

Ix

u+ Iy

v + It

= 0

I

x

=@I

@x

Iy =@I

@yIt =

@I

@t

Forward difference Sobel filter Scharr filter

…

How do you compute …

spatial derivative temporal derivative

Ix

u+ Iy

v + It

= 0

I

x

=@I

@x

Iy =@I

@yIt =

@I

@t

Forward difference Sobel filter Scharr filter

…

How do you compute …

frame differencing

Frame differencing

1 1 1 1 11 1 1 1 11 10 10 10 101 10 10 10 101 10 10 10 101 10 10 10 10

1 1 1 1 11 1 1 1 11 1 1 1 11 1 10 10 101 1 10 10 101 1 10 10 10

0 0 0 0 00 0 0 0 00 9 9 9 90 9 0 0 00 9 0 0 00 9 0 0 0

It =@I

@tt t+ 1

(example of a forward difference)

- =

Example:1 1 1 1 11 1 1 1 11 10 10 10 101 10 10 10 101 10 10 10 101 10 10 10 10

1 1 1 1 11 1 1 1 11 1 1 1 11 1 10 10 101 1 10 10 101 1 10 10 10

y

x

y

x

- 0 0 0 -- 0 0 0 -- 9 0 0 -- 9 0 0 -- 9 0 0 -- 9 0 0 -

y

x

- - - - -0 0 0 0 00 9 9 9 9- 0 0 0 00 0 0 0 0- - - - -

y

x

I

x

=@I

@x

Iy =@I

@yIt =

@I

@t

t t+ 1

0 0 0 0 00 0 0 0 00 9 9 9 90 9 0 0 00 9 0 0 00 9 0 0 0

y

x

3 x 3 patch

-1 0 1

-1 0 1

spatial derivative optical flow

Ix

u+ Iy

v + It

= 0

I

x

=@I

@x

Iy =@I

@y u =dx

dt

v =dy

dt

Forward difference Sobel filter Scharr filter

…

How do you compute …

temporal derivative

It =@I

@t

frame differencingHow do you compute this?

spatial derivative optical flow

Ix

u+ Iy

v + It

= 0

I

x

=@I

@x

Iy =@I

@y u =dx

dt

v =dy

dt

Forward difference Sobel filter Scharr filter

…

How do you compute …

temporal derivative

It =@I

@t

frame differencingWe need to solve for this!(this is the unknown in the

optical flow problem)

spatial derivative optical flow

Ix

u+ Iy

v + It

= 0

I

x

=@I

@x

Iy =@I

@y u =dx

dt

v =dy

dt

Forward difference Sobel filter Scharr filter

…

(u, v)Solution lies on a line

Cannot be found uniquely with a single constraint

How do you compute …

temporal derivative

It =@I

@t

frame differencing

Ix

u+ Iy

v + It

= 0

known

unknown

We need at least ____ equations to solve for 2 unknowns.

Ix

u+ Iy

v + It

= 0

known

unknown

Where do we get more equations (constraints)?

Ix

u+ Iy

v + It

= 0

u

v

Solution lies on a straight line

The solution cannot be determined uniquely with a single constraint (a single pixel)

many combinations of u and v will satisfy the equality

spatial derivative optical flow temporal derivative

Ix

u+ Iy

v + It

= 0

I

x

=@I

@x

Iy =@I

@y u =dx

dt

v =dy

dtIt =

@I

@t

How can we use the brightness constancy equation to estimate the optical flow?