Computer Vision

Stereo Vision

Bahadir K. Gunturk 2

Pinhole Camera

Bahadir K. Gunturk 3

Review: Perspective Projection

' ' 'x y f

x y z

Bahadir K. Gunturk 4

Stereo scene pointscene point

optical centeroptical center

image planeimage plane

p p’

p p’

Bahadir K. Gunturk 5

Stereo Constraints

X1

Y1

Z1

O1

Image plane

Focal plane

M

p p’

Y2

X2

Z2O2

Epipolar Line

Epipole

Bahadir K. Gunturk 6

A Simple Stereo System

Zw=0

LEFT CAMERA

Left image:reference

Right image:target

RIGHT CAMERA

Elevation Zw

disparity

Depth Z

baseline

Bahadir K. Gunturk 7

Stereo View

Left View Right View

Disparity

Bahadir K. Gunturk 8

Stereo Disparity The separation between two matching objects

is called the stereo disparity.

Bahadir K. Gunturk 9

Parallel Cameras

ZT

fZxxTlr

OOll OOrr

PP

ppll pprr

TT

ZZ

xxll xxrr

ff

T is the stereo baseline

rlxx

TfZ

rlxxd Disparity:

Bahadir K. Gunturk 10

Correlation Approach

For Each point (xl, yl) in the left image, define a window centered at the point

(xl, yl)LEFT IMAGE

Bahadir K. Gunturk 11

Correlation Approach

… search its corresponding point within a search region in the right image

(xl, yl)RIGHT IMAGE

Bahadir K. Gunturk 12

Correlation Approach

… the disparity (dx, dy) is the displacement when the correlation is maximum

(xl, yl)dx(xr, yr)RIGHT IMAGE

Bahadir K. Gunturk 13

Maximize Cross correlation

Minimize Sum of Squared Differences

Comparing Windows ==??

ff gg

Bahadir K. Gunturk 14

Feature-based correspondence Features most commonly used:

Corners Similarity measured in terms of:

surrounding gray values (SSD, Cross-correlation) location

Edges, Lines Similarity measured in terms of:

orientation contrast coordinates of edge or line’s midpoint length of line

Bahadir K. Gunturk 15

Feature-based Approach

For each feature in the left image…

LEFT IMAGE

corner line

structure

Bahadir K. Gunturk 16

Feature-based Approach

Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum

RIGHT IMAGE

corner line

structure

Bahadir K. Gunturk 17

Correspondence Difficulties Why is the correspondence problem difficult?

Some points in each image will have no corresponding points in the other image.(1) the cameras might have different fields of view.

(2) due to occlusion.

A stereo system must be able to determine the image parts that should not be matched.

Bahadir K. Gunturk 18

Structured Light Structured lighting

Feature-based methods are not applicable when the objects have smooth surfaces (i.e., sparse disparity maps make surface reconstruction difficult).

Patterns of light are projected onto the surface of objects, creating interesting points even in regions which would be otherwise smooth.

Finding and matching such points is simplified by knowing the geometry of the projected patterns.

Bahadir K. Gunturk 19

Stereo results

Ground truthScene

Data from University of Tsukuba

(Seitz)

Bahadir K. Gunturk 20

Results with window correlation

Estimated depth of field Ground truth

(Seitz)

Bahadir K. Gunturk 21

Results with better method

A state of the art methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,

International Conference on Computer Vision, September 1999.

Ground truth

(Seitz)

Bahadir K. Gunturk 22

Other constraints

It is possible to put some constraints. For example: smoothness. (Disparity usually doesn’t

change too quickly.)

Bahadir K. Gunturk 23

Parameters of a Stereo System Intrinsic Parameters

Characterize the transformation from camera to pixel coordinate systems of each camera

Focal length, image center, aspect ratio

Extrinsic parameters Describe the relative

position and orientation of the two cameras

Rotation matrix R and translation vector T

pl

pr

P

Ol Or

Xl

Xr

Pl Pr

fl fr

Zl

Yl

Zr

Yr

R, T

Bahadir K. Gunturk 24

Applications

courtesy of Sportvision

First-down line

Bahadir K. Gunturk 25

ApplicationsVirtual advertising

courtesy of Princeton Video Image