MSU Fall 2014 1

Computing Motion from Images

Chapter 9 of S&S plus otherwork.

MSU Fall 2014 2

General topics

Low level change detection Region tracking or matching over

time Interpretation of motion MPEG compression Interpretation of scene changes in

video Understanding human activites

MSU Fall 2014 3

Motion important to human vision

MSU Fall 2014 4

What’s moving: different cases

MSU Fall 2014 5

Image subtraction

Simple method to remove unchanging background from

moving regions.

MSU Fall 2014 6

Change detection for surveillance

MSU Fall 2014 7

Change detection by image subtraction

Closing = dilation+erosion

MSU Fall 2014 8

http://homepages.inf.ed.ac.uk/rbf/HIPR2/close.htm

MSU Fall 2014 9

What to do with regions of change?

Discard small regions Discard regions of non interesting

features Keep track of regions with

interesting features Track in future frames from motion

plus component features

MSU Fall 2014 10

Some effects of camera motion that can cause problems

MSU Fall 2014 11

Motion field

MSU Fall 2014 12

FOE and FOC

Will return to use the FOE or FOC or detection of panning to determine what the camera is doing in video tapes.

MSU Fall 2014 13

Gaming using a camera to recognize the player’s motion

Decathlete game

MSU Fall 2014 14

Decathlete game

Cheap camera replaces usual mouse for input

Running speed and jumping of the avatar is controlled by detected motion of the player’s hands.

MSU Fall 2014 15

Motion detection input device

Running (hands)Jumping (hands)

MSU Fall 2014 16

Motion analysis controls hurdling event (console)

• Top left shows video frame of player

• Middle left shows motion vectors from multiple frames

• Center shows jumping patterns

MSU Fall 2014 17

Related work

Motion sensed by crude cameras Person dances/gestures in space Kinect/Leap motion sensors System maps movement into

music Creative environment? Good exercise room?

MSU Fall 2014 18

Computing motion vectors from corresponding “points”

High energy neighborhoods are used to define points for

matching

MSU Fall 2014 19

Match points between frames

Such large motions are unusual. Most systems track small motions.

MSU Fall 2014 20

Requirements for interest points

Match small neighborhood to small neighborhood. The previous “scene” contains several highly textured neighborhoods.

MSU Fall 2014 21

Interest = minimum directional variance

Used by Hans Moravec in his robot stereo vision system.

Interest points were used for stereo matching.

MSU Fall 2014 22

Detecting interest points in I1

MSU Fall 2014 23

Match points from I1 in I2

MSU Fall 2014 24

Search for best match of point P1 in nearby window of I2

For both motion and stereo, we have some constraints on where to search for a matching interest point.

MSU Fall 2014 25

Motion vectors clustered to show 3 coherent regions

All motion vectors are clustered into 3 groups of similar vectors showing motion of 3 independent objects. (Dina Eldin)

Motion coherence: points of same object tend to move in the same way

MSU Fall 2014 26

Two frames of aerial imagery

Video frame N and N+1 shows slight movement: most pixels are same, just in different locations.

MSU Fall 2014 27

Can code frame N+d with displacments relative to frame N

for each 16 x 16 block in the 2nd image

find a closely matching block in the 1st image

replace the 16x16 intensities by the location in the 1st image (dX, dY)

256 bytes replaced by 2 bytes! (If blocks differ too much, encode

the differences to be added.)

MSU Fall 2014 28

Frame approximation

Left is original video frame N+1. Right is set of best image blocks taken from frame N. (Work of Dina Eldin)

MSU Fall 2014 29

Best matching blocks between video frames N+1 to N (motion vectors)

The bulk of the vectors show the true motion of the airplane taking the pictures. The long vectors are incorrect motion vectors, but they do work well for compression of image I2!

Best matches from 2nd to first image shown as vectors overlaid on the 2nd image. (Work by Dina Eldin.)

MSU Fall 2014 30

Motion coherence provides redundancy for compression

MPEG “motion compensation” represents motion of 16x16 pixels blocks, NOT objects

MSU Fall 2014 31

MPEG represents blocks that move by the motion vector

MSU Fall 2014 32

MPEG has ‘I’, ‘P’, and ‘B’ frames

MSU Fall 2014 33

Computing Image Flow

MSU Fall 2014 34

Motion Field & Optical Flow Field

Motion Field = Real world 3D motion

Optical Flow Field = Projection of the motion field onto the 2d image

3D motion vector

2D optical flow vector

vu,u

CCD

Slides from Lihi Zelnik-Manor

MSU Fall 2014 36

Assumptions

When does it break?

The screen is stationary yet displays motion

Homogeneous objects generate zero optical flow.

Fixed sphere. Changing light source.

Non-rigid texture motion

Slides from Lihi Zelnik-Manor

MSU Fall 2014 38

Image flow equation 1 of 2

MSU Fall 2014 39

Image flow equation 2 of 2

Estimating Optical Flow Assume the image intensity is

constant

tyxI ,, dttdyydxxI ,,

ITime = t Time = t+dt

dyydxx ,

yx,

Slides from Lihi Zelnik-Manor

MSU Fall 2014 40

Brightness Constancy Equation dttdyydxxItyxI ,,,,

dttI

dyyI

dxxI

tyxI

,,

First order Taylor Expansion

0 dtIdyIdxI tyx

Simplify notations:

Divide by dt and denote:

dtdx

u dtdy

v

tyx IvIuI Problem I: One equation, two unknowns

Slides from Lihi Zelnik-Manor MSU Fall 2014 41

Time t+dt

Problem II: “The Aperture Problem”

Time t

?Time t+dt

Where did the yellow point move to?

We need additional constraints

For points on a line of fixed intensity we can only recover the normal flow

Slides from Lihi Zelnik-Manor

MSU Fall 2014 42

Use Local InformationSometimes enlarging the aperture can help

Slides from Lihi Zelnik-Manor

MSU Fall 2014 43

Local smoothnessLucas Kanade (1984)

Assume constant (u,v) in small neighborhood

tyx IvIuI tyx Iv

uII

2

1

22

11

t

t

yx

yx

I

I

v

uII

II

bA u

Slides from Lihi Zelnik-Manor

MSU Fall 2014 44

Lucas Kanade (1984)

bA u

bAAA TT 1u

Goal: Minimize2u bA

bAAA TT u

2x2 2x1 2x1

Method: Least-Squares

Slides from Lihi Zelnik-Manor

MSU Fall 2014 45

Lucas-Kanade Solution

2

2

yyx

yxxT

III

IIIAA

We want this matrix to be invertible.i.e., no zero eigenvalues

bAAA TT 1u

Slides from Lihi Zelnik-Manor

MSU Fall 2014 46

Break-downs Brightness constancy is not satisfied

A point does not move like its neighbors what is the ideal window size?

The motion is not small (Taylor expansion doesn’t hold)

Correlation based methods

Regularization based methods

Use multi-scale estimation

Slides from Lihi Zelnik-Manor

MSU Fall 2014 47

Multi-Scale Flow Estimation

image It-1 image I

Gaussian pyramid of image It Gaussian pyramid of image It+1

image It+1image Itu=10 pixels

u=5 pixels

u=2.5 pixels

u=1.25 pixels

Slides from Lihi Zelnik-Manor

MSU Fall 2014 48

MSU Fall 2014 49

Tracking several objects

Use assumptions of physics to compute multiple smooth

paths.(work of Sethi and R. Jain)

MSU Fall 2014 50

MSU Fall 2014 51

Tracking in images over time

MSU Fall 2014 52

General constraints from physics

MSU Fall 2014 53

Other possible constraints

Background statistics stable Object color/texture/shape might

change slowly over frames Might have knowledge of objects

under survielance Objects appear/disappear at

boundary of the frame

MSU Fall 2014 54

MSU Fall 2014 55

Sethi-Jain algorithm

MSU Fall 2014 56

MSU Fall 2014 57

Total smoothness of m paths

MSU Fall 2014 58

Greedy exchange algorithm

MSU Fall 2014 59

Example data structure

Total smoothness for trajectories of Figure 9.14

MSU Fall 2014 60

Example of domain specific tracking (Vera Bakic)

Tracking eyes and nose of PC user. System presents menu (top). User moves face to position cursor to a particular box (choice). System tracks face movement and moves cursor accordingly: user gets into feedback-control loop.

MSU Fall 2014 61

Segmentation of videos/movies

Segment into scenes, shots, specific actions, etc.

MSU Fall 2014 62

Types of changes in videos

MSU Fall 2014 63

Anchor person scene at left

Street scene for news story

Scene break

From Zhang et al 1993

How do we compute the scene change?

MSU Fall 2014 64

Histograms of frames across the scene change

Histograms at left are from anchor person frames, while histogram at bottom right is from the street frame.

MSU Fall 2014 66

American sign language example

MSU Fall 2014 67

Example from Yang and Ahuja

MSU Fall 2014 68

MSU Fall 2014 69