ComputerVision

Multiple View Geometry& Stereo

Marc PollefeysCOMP 256

ComputerVision

2

C1

C2

l2

l1

e1

e20p p 1

T2 F

Fundamental matrix (3x3 rank 2

matrix)

1. Computable from corresponding points

2. Simplifies matching3. Allows to detect wrong

matches4. Related to calibration

Underlying structure in set of matches for rigid scenes

l2

C1p1

L1

p2

L2

M

C2

p1

p2

C1

C2

l2

l1

e1

e2

p1

L1

p2

L2

M

l2lT1

Canonical representation:

]λe'|ve'F][[e'P' 0]|[IP T

Last class: epipolar geometry

ComputerVision

3

Jan 16/18 - Introduction

Jan 23/25 Cameras Radiometry

Jan 30/Feb1 Sources & Shadows Color

Feb 6/8 Linear filters & edges Texture

Feb 13/15 Multi-View Geometry Stereo

Feb 20/22 Optical flow Project proposals

Feb27/Mar1 Affine SfM Projective SfM

Mar 6/8 Camera Calibration Silhouettes and Photoconsistency

Mar 13/15 Springbreak Springbreak

Mar 20/22 Segmentation Fitting

Mar 27/29 Prob. Segmentation Project Update

Apr 3/5 Tracking Tracking

Apr 10/12 Object Recognition Object Recognition

Apr 17/19 Range data Range data

Apr 24/26 Final project Final project

Tentative class schedule

ComputerVision

4

Multiple Views (Faugeras and Mourrain, 1995)

ComputerVision

5

Two Views

Epipolar Constraint

ComputerVision

6

Three Views

Trifocal Constraint

ComputerVision

7

Four Views

Quadrifocal Constraint(Triggs, 1995)

ComputerVision

8

Geometrically, the four rays must intersect in P..

ComputerVision

9

Quadrifocal Tensorand Lines

ComputerVision

10

Quadrifocal tensor

• determinant is multilinear

thus linear in coefficients of lines !

• There must exist a tensor with 81 coefficients containing all possible combination of x,y,w coefficients for all 4 images: the quadrifocal tensor

wi

wi

yi

yi

xi

xii

Ti MlMlMl Ml

Twi

yi

xi lll ][

s

r

q

p

pqrs

MMMM

Q

4

3

2

1

det

ComputerVision

11

from perspective to omnidirectional cameras

perspective camera(2 constraints / feature)

radial camera (uncalibrated)(1 constraints / feature)

3 constraints allow to reconstruct 3D point

more constraints also tell something about cameras

multilinear constraints known as epipolar, trifocal and quadrifocal constraints

(0,0)

l=(y,-x)

(x,y)

ComputerVision

12

Radial quadrifocal tensor

• Linearly compute radial quadrifocal tensor Qijkl from 15 pts in 4 views

• Reconstruct 3D scene and use it for calibration

(2x2x2x2 tensor)

(2x2x2 tensor)

Not easy for real data, hard to avoid degenerate cases (e.g. 3 optical axes intersect in single point). However, degenerate case leads to simpler 3 view algorithm for pure rotation• Radial trifocal tensor Tijk from 7 points in 3 views

• Reconstruct 2D panorama and use it for calibration

(x,y)

ComputerVision

13

Non-parametric distortion calibration(Thirthala and Pollefeys, ICCV’05)

normalized radius

angle

• Models fish-eye lenses, cata-dioptric systems, etc.

ComputerVision

14

Non-parametric distortion calibration(Thirthala and Pollefeys, ICCV’05)

normalized radiusangle

90o

• Models fish-eye lenses, cata-dioptric systems, etc.

ComputerVision

15

STEREOPSIS

Reading: Chapter 11.

• The Stereopsis Problem: Fusion and Reconstruction• Human Stereopsis and Random Dot Stereograms• Cooperative Algorithms• Correlation-Based Fusion• Multi-Scale Edge Matching• Dynamic Programming• Using Three or More Cameras

ComputerVision

16

An Application: Mobile Robot Navigation

The Stanford Cart,H. Moravec, 1979.

The INRIA Mobile Robot, 1990.

Courtesy O. Faugeras and H. Moravec.

ComputerVision

17

Reconstruction / Triangulation

ComputerVision

18

(Binocular) Fusion

ComputerVision

19

Reconstruction

• Linear Method: find P such that

• Non-Linear Method: find Q minimizing

ComputerVision

20

Rectification

All epipolar lines are parallel in the rectified image plane.

ComputerVision

21

Image pair rectification

simplify stereo matching by warping the images

Apply projective transformation so that epipolar linescorrespond to horizontal scanlines

e

e

map epipole e to (1,0,0)

try to minimize image distortion

problem when epipole in (or close to) the image

He001

ComputerVision

23

ComputerVision

24

ComputerVision

25

Polar re-parameterization around epipoles

Requires only (oriented) epipolar geometry

Preserve length of epipolar linesChoose so that no pixels are

compressed

original image rectified image

Polar rectification(Pollefeys et al. ICCV’99)

Works for all relative motionsGuarantees minimal image size

ComputerVision

26

polar rectification: example

ComputerVision

27

polar rectification: example

ComputerVision

28

Example: Béguinage of Leuven

Does not work with standard Homography-based approaches

ComputerVision

29

Example: Béguinage of Leuven

ComputerVision

30

Reconstruction from Rectified Images

Disparity: d=u’-u. Depth: z = -B/d.

ComputerVision

31

Stereopsis

Figure from US Navy Manual of Basic Optics and Optical Instruments, prepared by Bureau of Naval Personnel. Reprinted by Dover Publications, Inc., 1969.

ComputerVision

32

Human Stereopsis: Reconstruction

Disparity: d = r-l = D-F.

d=0

d<0

In 3D, the horopter.

ComputerVision

33

Human Stereopsis: experimental horopter…

ComputerVision

34

Iso-disparity curves: planar retinas

X0X1

C1

X∞

C2

Xi Xj

ii

i

i

1

1:

1

01

1

01

11

:

11

10

11

0

the retina act as if it were flat!

ComputerVision

35

What if F is not known?

Human Stereopsis: Reconstruction

Helmoltz (1909):

• There is evidence showing the vergence anglescannot be measured precisely.

• Humans get fooled by bas-relief sculptures.

• There is an analytical explanation for this.

• Relative depth can be judged accurately.

ComputerVision

36

Human Stereopsis: Binocular Fusion

How are the correspondences established?

Julesz (1971): Is the mechanism for binocular fusiona monocular process or a binocular one??• There is anecdotal evidence for the latter (camouflage).

• Random dot stereograms provide an objective answerBP!

ComputerVision

37

A Cooperative Model (Marr and Poggio, 1976)

Excitory connections: continuity

Inhibitory connections: uniqueness

Iterate: C = C - w C + C .e i 0

Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr. 1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.

ComputerVision

38

Correlation Methods (1970--)

Normalized Correlation: minimize instead.

Slide the window along the epipolar line until w.w’ is maximized.

2Minimize |w-w’|.

ComputerVision

39

Correlation Methods: Foreshortening Problems

Solution: add a second pass using disparity estimates to warpthe correlation windows, e.g. Devernay and Faugeras (1994).

Reprinted from “Computing Differential Properties of 3D Shapes from Stereopsis without 3D Models,” by F. Devernay and O. Faugeras, Proc. IEEE Conf. on Computer Vision and Pattern Recognition (1994). 1994 IEEE.

ComputerVision

40

Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979-81)

• Edges are found by repeatedly smoothing the image and detectingthe zero crossings of the second derivative (Laplacian).• Matches at coarse scales are used to offset the search for matchesat fine scales (equivalent to eye movements).

ComputerVision

41

Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979-81)

One of the twoinput images

Image Laplacian

Zeros of the Laplacian

Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr. 1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.

ComputerVision

42

Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979-81)

Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr. 1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.

ComputerVision

43

The Ordering Constraint

But it is not always the case..

In general the pointsare in the same orderon both epipolar lines.

ComputerVision

44

Dynamic Programming (Baker and Binford, 1981)

Find the minimum-cost path going monotonicallydown and right from the top-left corner of thegraph to its bottom-right corner.

• Nodes = matched feature points (e.g., edge points).• Arcs = matched intervals along the epipolar lines.• Arc cost = discrepancy between intervals.

ComputerVision

45

Dynamic Programming (Ohta and Kanade, 1985)

Reprinted from “Stereo by Intra- and Intet-Scanline Search,” by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and MachineIntelligence, 7(2):139-154 (1985). 1985 IEEE.

ComputerVision

46

Three Views

The third eye can be used for verification..

ComputerVision

47

More Views (Okutami and Kanade, 1993)

Pick a reference image, and slide the correspondingwindow along the corresponding epipolar lines of allother images, using inverse depth relative to the firstimage as the search parameter.

Use the sum of correlation scores to rank matches.

Reprinted from “A Multiple-Baseline Stereo System,” by M. Okutami and T. Kanade, IEEE Trans. on PatternAnalysis and Machine Intelligence, 15(4):353-363 (1993). \copyright 1993 IEEE.

ComputerVision

48

Stereo matching

Optimal path(dynamic programming )

Similarity measure(SSD or NCC)

Constraints• epipolar

• ordering

• uniqueness

• disparity limit

• disparity gradient limit

Trade-off

• Matching cost (data)

• Discontinuities (prior)

(Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

ComputerVision

49

Hierarchical stereo matchingD

ow

nsam

plin

g

(Gau

ssia

n p

yra

mid

)

Dis

pari

ty p

rop

ag

ati

on

Allows faster computation

Deals with large disparity ranges

(Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

ComputerVision

50

Disparity map

image I(x,y) image I´(x´,y´)Disparity map D(x,y)

(x´,y´)=(x+D(x,y),y)

ComputerVision

51

Example: reconstruct image from neighboring images

ComputerVision

52

I1 I2 I10

Reprinted from “A Multiple-Baseline Stereo System,” by M. Okutami and T. Kanade, IEEE Trans. on PatternAnalysis and Machine Intelligence, 15(4):353-363 (1993). \copyright 1993 IEEE.

ComputerVision

53

Multi-view depth fusion

• Compute depth for every pixel of reference image– Triangulation– Use multiple views– Up- and down sequence– Use Kalman filter

(Koch, Pollefeys and Van Gool. ECCV‘98)

Allows to compute robust texture

ComputerVision

54

Real-time stereo on graphics hardware

• Computes Sum-of-Square-Differences• Hardware mip-map generation used to aggregate

results over support region• Trade-off between small and large support window

Yang and Pollefeys CVPR03

140M disparity hypothesis/sec on Radeon 9700pro140M disparity hypothesis/sec on Radeon 9700proe.g. 512x512x20disparities at 30Hze.g. 512x512x20disparities at 30Hz

Shape of a kernel for summing up 6 levels

ComputerVision

55

Sample Re-Projections

near far

ComputerVision

56

(1x1)

(1x1+2x2)

(1x1+2x2 +4x4+8x8)

(1x1+2x2 +4x4+8x8 +16x16)

Combine multiple aggregation windows using hardware mipmap and multiple texture units in single pass

video

ComputerVision

57

Cool ideas

• Space-time stereo (varying illumination, not shape)

ComputerVision

58

More on stereo …

The Middleburry Stereo Vision Research Pagehttp://cat.middlebury.edu/stereo/

Recommended reading

D. Scharstein and R. Szeliski. A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms.IJCV 47(1/2/3):7-42, April-June 2002. PDF file (1.15 MB) - includes current evaluation.Microsoft Research Technical Report MSR-TR-2001-81, November 2001. PDF file (1.27 MB).

ComputerVision

59

Next class:Optical Flow: where do pixels move to?