fundamental matrixhays/compvision/lectures/11.pdfdeep learning neural networks optical flow hough...

Fundamental matrix

Let p be a point in left image, p’ in right image

Epipolar relation

• p maps to epipolar line l’

• p’ maps to epipolar line l

Epipolar mapping described by a 3x3 matrix F

p p’

Fundamental matrix

This matrix F is called

• the “Essential Matrix”

– when image intrinsic parameters are known

• the “Fundamental Matrix”

– more generally (uncalibrated case)

Can solve for F from point correspondences

• Each (p, p’) pair gives one linear equation in entries of F

• F has 9 entries, but really only 7 or 8 degrees of freedom.

• With 8 points it is simple to solve for F, but it is also possible

with 7. See Marc Pollefey’s notes for a nice tutorial

The scale of algorithm name quality

better

RANSAC

Dynamic Programming

Essential and Fundamental Matrix

Deep Learning

Neural Networks

Optical Flow

Hough Transform

Today’s lecture

• Stereo Matching (Spare correspondence to Dense Correspondence)

• Optical Flow (Dense motion estimation)

Motion and Optic FlowCS 4495 Computer Vision – A. Bobick

Stereo Matching

Stereo image rectification

• Reproject image planes onto a common plane parallel to the line between camera centers

• Pixel motion is horizontal after this transformation

• Two homographies (3x3 transform), one for each input image reprojection

C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.

Rectification example

The correspondence problem

• Epipolar geometry constrains our search, but we still have a difficult correspondence problem.

Fundamental Matrix + Sparse correspondence

Fundamental Matrix + Dense correspondence

SIFT + Fundamental Matrix + RANSAC

Building Rome in a Day

By Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M. Seitz, Richard Szeliski

Communications of the ACM, Vol. 54 No. 10, Pages 105-112

Sparse to Dense Correspodence

Building Rome in a Day

By Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M. Seitz, Richard Szeliski

Communications of the ACM, Vol. 54 No. 10, Pages 105-112

Structure from motion (or SLAM)

• Given a set of corresponding points in two or more images, compute the camera parameters and the 3D point coordinates

Camera 1Camera 2 Camera 3

R1,t1 R2,t2R3,t3

? ? ? Slide credit:

Noah Snavely

Structure from motion ambiguity

• If we scale the entire scene by some factor k and, at the same time, scale the camera matrices by the factor of 1/k, the projections of the scene points in the image remain exactly the same:

It is impossible to recover the absolute scale of the scene!

XPPXx kk

How do we know the scale of image content?

Bundle adjustment

• Non-linear method for refining structure and motion

• Minimizing reprojection error

jiijDE XPxXP

Correspondence problem

Multiple match

hypotheses

satisfy epipolar

constraint, but

which is correct?

Figure from Gee & Cipolla 1999

• Beyond the hard constraint of epipolar geometry, there

are “soft” constraints to help identify corresponding points

• Similarity

• Uniqueness

• Ordering

• Disparity gradient

• To find matches in the image pair, we will assume

• Most scene points visible from both views

• Image regions for the matches are similar in appearance

Dense correspondence search

For each epipolar line

For each pixel / window in the left image

• compare with every pixel / window on same epipolar line

in right image• pick position with minimum match cost (e.g., SSD,

normalized correlation)

Adapted from Li Zhang

Matching cost

disparity

Left Right

scanline

Correspondence search with similarity constraint

• Slide a window along the right scanline and compare

contents of that window with the reference window in

the left image

• Matching cost: SSD or normalized correlation

Left Right

scanline

Left Right

scanline

Norm. corr

Source: Andrew Zisserman

Intensity

profiles

Neighborhoods of corresponding points are

similar in intensity patterns.

Correlation-based window matching

Textureless regions are non-distinct; high ambiguity for matches.

Effect of window size

W = 3 W = 20

Figures from Li Zhang

Want window large enough to have sufficient intensity

variation, yet small enough to contain only pixels with

about the same disparity.

Effect of window size

Left image Right image

Results with window search

Window-based matching

(best window size)

Ground truth

Better solutions

• Beyond individual correspondences to estimate

disparities:

• Optimize correspondence assignments jointly

• Scanline at a time (DP)

• Full 2D grid (graph cuts)

Stereo as energy minimization

• What defines a good stereo correspondence?

1. Match quality

• Want each pixel to find a good match in the other image

2. Smoothness

• If two pixels are adjacent, they should (usually) move about

the same amount

Stereo matching as energy minimization

I1I2 D

• Energy functions of this form can be minimized using

graph cuts

Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001

W1(i) W2(i+D(i)) D(i)

)(),,( smooth21data DEDIIEE

jDiDE,neighbors

smooth )()( 2

21data ))(()( i

iDiWiWE

Source: Steve Seitz

Better results…

Graph cut methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,

International Conference on Computer Vision, September 1999.

Ground truth

For the latest and greatest: http://www.middlebury.edu/stereo/

Challenges

• Low-contrast ; textureless image regions

• Occlusions

• Violations of brightness constancy (e.g., specular

reflections)

• Really large baselines (foreshortening and appearance

change)

• Camera calibration errors

Active stereo with structured light

• Project “structured” light patterns onto the object

• Simplifies the correspondence problem

• Allows us to use only one camera

camera

projector

L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured

Light and Multi-pass Dynamic Programming. 3DPVT 2002

Kinect: Structured infrared light

http://bbzippo.wordpress.com/2010/11/28/kinect-in-infrared/

iPhone X

3 minute break

Ninio, J. and Stevens, K. A. (2000) Variations on the Hermann grid: an extinction illusion.

Perception, 29, 1209-1217.

Computer Vision

Motion and Optical Flow

Many slides adapted from S. Seitz, R. Szeliski, M. Pollefeys, K. Grauman and others…

• A video is a sequence of frames captured over time

• Now our image data is a function of space

(x, y) and time (t)

Motion and perceptual organization

Gestalt psychology

(Max Wertheimer,

1880-1943)

• Sometimes, motion is the only cue

Gestalt psychology

(Max Wertheimer,

1880-1943)

• Even “impoverished” motion data can evoke a strong

percept

• Even “impoverished” motion data can evoke a strong

percept

Experimental study of apparent behavior.

Fritz Heider & Marianne Simmel. 1944

Motion estimation: Optical flow

Will start by estimating motion of each pixel separately

Then will consider motion of entire image

Optic flow is the apparent motion of objects or surfaces

Problem definition: optical flow

How to estimate pixel motion from image I(x,y,t) to I(x,y,t+1) ?

• Solve pixel correspondence problem

– given a pixel in I(x,y,t), look for nearby pixels of the same color in I(x,y,t+1)

Key assumptions

• color constancy: a point in I(x,y,t) looks the same in I(x,y,t+1)

– For grayscale images, this is brightness constancy

• small motion: points do not move very far

This is called the optical flow problem

( , , )I x y t ( , , 1)I x y t

Optical flow constraints (grayscale images)

• Let’s look at these constraints more closely

• brightness constancy constraint (equation)

• small motion: (u and v are less than 1 pixel, or smooth)

Taylor series expansion of I:

( , , )I x y t ( , , 1)I x y t

( , ) ( , ) [higher order terms]I I

I x u y v I x y u vx y

( , )I I

I x y u vx y

( , , ) ( , , 1)I x y t I x u y v t

0 ( , , 1) ( , , )

( , , 1) ( , , )x y

I x u y v t I x y t

I x y t I u I v I x y t

Optical flow equation• Combining these two equations

(Short hand: 𝐼𝑥 =𝜕𝐼

𝜕𝑥

for t or t+1)

0 ( , , 1) ( , , )

( , , 1) ( , , )

[ ( , , 1) ( , , )]

I x u y v t I x y t

I x y t I x y t I u I v

I I u I v

I I u v

𝜕𝑥

for t or t+1)

0 ( , , 1) ( , , )

( , , 1) ( , , )

[ ( , , 1) ( , , )]

I x u y v t I x y t

I x y t I x y t I u I v

I I u I v

I I u v

In the limit as u and v go to zero, this becomes exact

Brightness constancy constraint equation

0x y tI u I v I

0 ,tI I u v

𝜕𝑥

for t or t+1)

How does this make sense?

• What do the static image gradients have to do with motion

estimation?

Brightness constancy constraint equation

0x y tI u I v I

The brightness constancy constraint

• How many equations and unknowns per pixel?

The component of the motion perpendicular to the

gradient (i.e., parallel to the edge) cannot be measured

(u’,v’)

gradient

(u+u’,v+v’)

If (u, v) satisfies the equation,

so does (u+u’, v+v’ ) if

•One equation (this is a scalar equation!), two unknowns (u,v)

0'v'uIT

Can we use this equation to recover image motion (u,v) at

each pixel?

0x y tI u I v I 0 ,tI I u v or

Aperture problem

The barber pole illusion

http://en.wikipedia.org/wiki/Barberpole_illusion

The barber pole illusion

http://en.wikipedia.org/wiki/Barberpole_illusion

Solving the ambiguity…

• How to get more equations for a pixel?

• Spatial coherence constraint

• Assume the pixel’s neighbors have the same (u,v)• If we use a 5x5 window, that gives us 25 equations per pixel

B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In

Proceedings of the International Joint Conference on Artificial Intelligence, pp. 674–679, 1981.

• Least squares problem:

Solving the ambiguity…

Matching patches across images• Overconstrained linear system

The summations are over all pixels in the K x K window

Least squares solution for d given by

Conditions for solvabilityOptimal (u, v) satisfies Lucas-Kanade equation

Does this remind you of anything?

When is this solvable? I.e., what are good points to track?• ATA should be invertible

• ATA should not be too small due to noise

– eigenvalues 1 and 2 of ATA should not be too small

• ATA should be well-conditioned

– 1/ 2 should not be too large ( 1 = larger eigenvalue)

Criteria for Harris corner detector

Low texture region

– gradients have small magnitude

– small 1, small 2

– large gradients, all the same

– large 1, small 2

High textured region

– gradients are different, large magnitudes

– large 1, large 2

The aperture problem resolved

Actual motion

The aperture problem resolved

Perceived motion

Errors in Lucas-Kanade

• A point does not move like its neighbors

• Motion segmentation

• Brightness constancy does not hold

• Do exhaustive neighborhood search with normalized correlation -

tracking features – maybe SIFT – more later….

• The motion is large (larger than a pixel)

1. Not-linear: Iterative refinement

2. Local minima: coarse-to-fine estimation

Revisiting the small motion assumption

• Is this motion small enough?

• Probably not—it’s much larger than one pixel

• How might we solve this problem?

image 2image 1

Gaussian pyramid of image 1 Gaussian pyramid of image 2

image 2image 1 u=10 pixels

u=5 pixels

u=2.5 pixels

u=1.25 pixels

Coarse-to-fine optical flow estimation

image Iimage J

Gaussian pyramid of image 1 Gaussian pyramid of image 2

image 2image 1

Coarse-to-fine optical flow estimation

run iterative L-K

warp & upsample

Optical Flow Results

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

Optical Flow Results

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

fundamental matrixhays/compvision/lectures/11.pdfdeep learning neural networks optical flow hough...

Documents

seasonalityofblueandﬁnwhalecallsandtheinﬂuenceof...

compvision מצגת אבי ניר- מנכ"ל

electronic correspondence

deep freeze: iceland's economic collapse - mises institute...

moon illusion in virtual reality - susquehanna...

project correspondence management · outgoing...

social correspondence

modeling statistik untuk computer vision -...

employment correspondence

public correspondence public correspondence received as of...

deepwarp: photorealistic image resynthesis for gaze...

oconnor correspondence

correspondence analysis.doc

correspondence table

proving triangles congruent. the congruence postulates sss...

business correspondence

correspondence principle

feature lucas-kanade(lk) image registration & tracking...

planting fields est#4649635sub-series # 1: lowell and...

military correspondence