csc83029 3-d computer vision / ioannis stamos 3-d computer vision csc 83029 stereo

CSc83029 3-D Computer Vision / Ioannis StamosCSc83029 3-D Computer Vision / Ioannis Stamos

3-D Computer Vision3-D Computer VisionCSc 83029CSc 83029

StereoStereo


StereopsisStereopsis

Recovering 3D information (depth) from Recovering 3D information (depth) from two images.two images. The correspondence problem.The correspondence problem. The reconstruction problem.The reconstruction problem. Epipolar constraint.Epipolar constraint. The 8-point algorithm.The 8-point algorithm.


The 2 problems of StereoThe 2 problems of Stereo

Correspondence: Correspondence: Which parts of the left and Which parts of the left and right images are projections of the same scene right images are projections of the same scene element?element?

Reconstruction: Reconstruction: Given:Given: A number of corresponding points between the left A number of corresponding points between the left

and right image,and right image, Information on the geometry of the stereo system,Information on the geometry of the stereo system,

Find:Find:

3-D structure of observed objects3-D structure of observed objects..

The setting: Simultaneous acquisition of 2 images(left, right) of a static scene.


Stereo VisionStereo Vision

depth map


A simple stereo systemA simple stereo system

f

T

Z

P

Ol Or

Left Camera Right Camera

X-axis

Z-axis

cl cr

FixationPoint:Infinity.Paralleloptical axes.


TriangulationTriangulation

f

T

Z

P

pl pr

Ol Or


X-axis

Z-axis

cl cr


Calibrated Cameras


f

T

Z

P

pl pr

Ol Or


X-axis

Z-axis

cl cr


Calibrated Cameras

xl xr

rlrl xxd

d

TfZ

Z

T

fZ

xxT

,Similar triangles:

d:disparity (difference in retinal positions).T:baseline.Depth (Z) is inversely proportional to d (fixation at infinity)


f

T

Z

P

pl pr

Ol Or


X-axis

Z-axis

cl cr


Calibrated Cameras

xl xr

rlrl xxd

d

TfZ

Z

T

fZ

xxT

,Similar triangles:

d:disparity (difference in retinal positions).T:baseline.Baseline T: accuracy/robustness of depth calculation.


f

T

Z

P

pl pr

Ol Or


X-axis

Z-axis

cl cr


Calibrated Cameras

xl xr

rlrl xxd

d

TfZ

Z

T

fZ

xxT

,Similar triangles:

d:disparity (difference in retinal positions).T:baseline.Small baselines: less accurate measurements.


f

T

Z

P

pl pr

Ol Or


X-axis

Z-axis

cl cr


Calibrated Cameras

xl xr

rlrl xxd

d

TfZ

Z

T

fZ

xxT

,Similar triangles:

d:disparity (difference in retinal positions).T:baseline.Large baselines: occlusions/foreshortening.


Parameters of Stereo SystemParameters of Stereo System

f

T

Z

P

pl pr

Ol Or


X-axis

Z-axis

cl cr


1) Intrinsic parameters (i.e. f, cl, cr)2) Extrinsic parameters: relative position and orientation of the 2 cameras.

xl xr

STEREO CALIBRATION PROBLEM

• Difficulties – ambiguities, large changes of appearance, due to changeOf viewpoint, non-uniquess

Stereo – Photometric ConstraintStereo – Photometric Constraint• Same world point has same intensity in both images.

• Lambertian fronto-parallel• Issues (noise, specularities, foreshortening) From Jana Kosecka


Correspondence Is DifficultCorrespondence Is Difficult

Ambiguity: there may be many possible 3D reconstructions.



No texture: difficult to find a unique match.



Foreshortening: the projection in each image is different.


Occlusions: there may not be a correspondence.

Assumptions: 1) Most scene points are visible from both views. 2) Corresponding image regions are similar.



Curved surfaces: triangulation produces incorrect position.


Correspondence is difficult: The Correspondence is difficult: The Ordering ConstraintOrdering Constraint

But it is not always the case ...

Points appear in the same order


More Correspondence ProblemsMore Correspondence Problems Regions without textureRegions without texture Highly Specular surfacesHighly Specular surfaces Translucent objectsTranslucent objects


Methods For CorrespondenceMethods For Correspondence

Correlation based (dense correspondences).Correlation based (dense correspondences). Feature based (such as edges/lines/corners).Feature based (such as edges/lines/corners).

Correlation-Based MethodsCorrelation-Based Methods

pl

R(pl )

Left Image Right Image

1) For each pixel pl in the left image search in a region R(pl) in the right image for corresponding pixel pr.2) Use image windows of size (2W+1)x(2W+1).3) Select the pixel pr that maximizes a correlation function.

HAVE TO SPECIFY: Region R, size W, and correlation function ψ.


pl

R(pl )

Left Image Right ImageFor each pixel pl=[i,j] in the left image

For each displacement d=[d1,d2] in R(pl)Compute

The disparity of pl is the d that maximizes c(d)

)),(),,(()( 21

W

Wk

W

Wll dljdkiIrljkiIc d

HAVE TO SPECIFY: Region R, size W, and correlation function ψ.


pl

R(pl )


2)(),(

),(

vuvu

uvvu

SUM OF SQUARED DIFFERENCESSSD

CROSS-CORRELATION

SSD is usually preferred: handles different intensity scales.

Normalized cross-correlation is better (but is more expensive).



Intensities in window may differ.

Normalized cross-correlation may help.


Image NormalizationImage Normalization Even when the cameras are identical models, there Even when the cameras are identical models, there

can be differences in gain and sensitivity.can be differences in gain and sensitivity. The cameras do not see exactly the same surfaces, The cameras do not see exactly the same surfaces,

so their overall light levels can differ.so their overall light levels can differ. For these reasons and more, it is a good idea to For these reasons and more, it is a good idea to

normalize the pixels in each window:normalize the pixels in each window:

pixel Normalized ),(

),(ˆ

magnitude Window )],([

pixel Average ),(

),(

),(),(

2

),(

),(),(),(

1

yxW

yxWvuyxW

yxWvuyxW

m

mm

m

m

II

IyxIyxI

vuII

vuII

From Sebastian Thrun/Jana Kosecka


It is closely related to the SSD:It is closely related to the SSD:

Maximize Cross correlation

Minimize Sum of Squared Differences

Comparing Windows:Comparing Windows:

From Jana Kosecka


• Sum of squared differences

• Normalize cross-correlation

• Sum of absolute differences

Region based Similarity MetricsRegion based Similarity Metrics

From Jana Kosecka


NCC score for two widely NCC score for two widely separated viewsseparated views

NCC score

From Jana Kosecka


Window sizeWindow size

W = 3 W = 20

Better results with adaptive window• T. Kanade and M. Okutomi,

A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991.

• D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998

Effect of window sizeEffect of window size

(S. Seitz)

http://www.cs.cmu.edu/~ph/869/papers/kanade-okutomi.pdf

http://www.cs.washington.edu/education/courses/cse590ss/01wi/notes/papers/ScharsteinSzeliskiIJCV98.pdf


Stereo resultsStereo results

Ground truthScene

Data from University of TsukubaData from University of Tsukuba

(Seitz)


Results with window correlationResults with window correlation

Window-based matching(best window size)

Ground truth

(Seitz)


Results with better methodResults with better method

Boykov et al., Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision, September

1999.(Seitz)

Ground truthState of the art

http://www.cs.cornell.edu/rdz/Papers/BVZ-iccv99.pdf


Feature-Based MethodsFeature-Based Methods


Match sparse sets of extracted features.A feature descriptor for a line could contain:

length l, orientation o, midpoint (x,y), average contrast c

An example similarity measure (w’s are weights):

23

22

2211

20 )()()()(

1

rlrlrl ccwmmwwllwS


Correspondence Using Correspondence Using CorrelationCorrelation

Left Disparity Map

Images courtesy of Point Grey Research


LEFT IMAGE

corner line

structure

Correspondence By FeaturesCorrespondence By Features


Correspondence By FeaturesCorrespondence By FeaturesRIGHT IMAGE

corner line

structure

Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximummaximum



Comparison of Matching MethodsComparison of Matching Methods

Dense depth maps.Dense depth maps. Need textured imagesNeed textured images Sensitive to Sensitive to

foreshorening/illumination foreshorening/illumination changeschanges

Need close viewsNeed close views

Sparse depth maps.Sparse depth maps. Insensitive to Insensitive to

illumination changes.illumination changes.

A-priori info used.A-priori info used. Faster.Faster.

Problems: occlusions/spurious matches:=>Introduce constraints in matching (i.e. left-right consistency constraint)

Correlation-Based Feature-Based


Epipolar Constraint (Geometry)Epipolar Constraint (Geometry)

Center of projection

Image planeπl

Scene point


Epipoles

Ol Or

P

Pl Pr

plpr

EPIPOLARPLANE

el er

Image planeπr

EPIPOLARLINE EPIPOLAR

LINE

Epipolar ConstraintEpipolar Constraint


Image planeπl

Scene point


Epipoles

Ol Or

P

Pl Pr

plpr

EPIPOLARPLANE

el er

Image planeπr


LINE

Extrinsic parameters: Left/Right Camera Frames: Pr=R(Pl-T), T=Or-Ol (1)



Image planeπl

Scene point


Epipoles

Ol Or

P

Pl Pr

plpr

EPIPOLARPLANE

el er

Image planeπr


LINE

Given pl, pr is constrained to lie on the Epipolar Line (E.L.).For each left pixel pl, find the corresponding right E.L.Searching for pr reduces to a 1-D problem.

Ol, Or, pl =>Enough to define right E.L.



Image plane

Scene points


Epipoles

All E.L.s go through epipoles.Parallel image planes => epipoles at infinity.

el er

Essential MatrixEssential Matrix

T

lPTPP lr

coplanar are and T-TPP ll ,

0)( lT

rT PTPR

0 lT

r PTRP

Estimate the epipolar geometry: correspondencebetween points and E.L.s.

0)( lT

l PTTP

(1)

0

0

0

XY

XZ

YZ

ll

TT

TT

TT

S

SPPT

0)( lT

r PRSP0lT

r EPP

Link bw/ epipolar constraint and extrinsic parameters of stereo system.


Essential MatrixEssential Matrix

T

lPrP

lprp

lu

ru

00

0

0

lTT

TT

TTT

r pRpxy

xz

yz

E

rT

l

lr

pEu

Epu

Epipolar lines are found by

Essential matrixRank 2

0lT

r EPPPerspectiveProjection

0lT

r Epp

scoordinate camera in points are and rl pp

erel

Perspective: pl=[xl,yl,fl]T, pr=[xr,yr,zr]T

pl= fl/Zl Pl, pr=fr/Zr Pr

Camera Models (linear versions)Camera Models (linear versions)

World Point(Xw, Yw,Zw)

Measured Pixel(xim, yim)

Elegant decomposition.No distortion!

?

HomogeneousCoordinates


Fundamental MatrixFundamental Matrix

rrr

lll

pMp

pMp

Camera to pixel coordinates:

01 ll

Tr

Tr pEMMp

Essential matrix equation becomes:

F

rT

l

lr

pFu

pFu

Epipolar lines:

T

lPrP

lprp

lu

ru

F: pixel coordinates !E: camera coordinates !

erel

Ml (Mr) matrix of intrinsic parameters for left (right) camera.

Fundamental matrix


ConclusionsConclusions

Encodes information on Encodes information on extrinsic parametersextrinsic parameters..

Has rank 2.Has rank 2. Its 2 non-zero singular Its 2 non-zero singular

values are equal.values are equal.

Encodes information on Encodes information on both the both the extrinsicextrinsic and and intrinsicintrinsic parameters. parameters.

Has rank 2.Has rank 2.

Essential Matrix Fundamental Matrix


Estimating the epipolar geometryEstimating the epipolar geometry

el erel er



Problem: Find the fundamental matrix from a set of image correspondences

0 liT

ri pFp

ir

il pp ,

el erel er



2 min liT

ri

iF

pFpWith the respect to the constraint: Rank(F) = 2.

el er

The 8-point algorithmThe 8-point algorithm

0 liT

ri pFp 0vA

n>=8 correspondences

v: the 9 elements of F.A: n x 9 measurement matrix.Solve using SVD (solution up to a scale factor).Enforce rank(F)=2 =>SVD on the computed F.Be careful: numerical instabilities.


Epipolar Lines – ExampleEpipolar Lines – Example

ExampleExampleTwo views

Point Feature Matching

From Jana Kosecka


ExampleExampleEpipolar Geometry

Camera Pose and

Sparse Structure Recovery

From Jana Kosecka


Locating the Epipoles from E & FLocating the Epipoles from E & F

el er

F => el, er in pixel coordinates.E => el, er in camera coordinates.

Fact: All epipolar lines pass through epipoles.

Accurate epipole localization:1) Refining epipolar lines.2) Checking for consistency.3) Uncalibrated stereo.


Image rectificationImage rectification

Given general displacement how to warp the viewsGiven general displacement how to warp the views Such that epipolar lines are parallel to each other Such that epipolar lines are parallel to each other How to warp it back to canonical configurationHow to warp it back to canonical configuration (more details later)(more details later)

(Seitz)


Epipolar rectificationEpipolar rectification

• Rectified Image Pair • Corresponding epipolar lines are aligned with the scan-lines• Search for dense correspondence is a 1D search


Epipolar rectificationEpipolar rectification

Rectified Image Pair


Rectification (Trucco, Ch. 7)Rectification (Trucco, Ch. 7)

Rotate left camera so that epipole goes to Rotate left camera so that epipole goes to infinity (known R, known epipoles)infinity (known R, known epipoles)

Apply same rotation to right cameraApply same rotation to right camera Rotate right camera by RRotate right camera by R Adjust scale in both camera reference framesAdjust scale in both camera reference frames


RectificationRectification

Problem: Epipolar lines not parallel to scan Problem: Epipolar lines not parallel to scan lineslines

plp

r

P

Ol Orel er

Pl Pr

Epipolar Plane

Epipolar Lines

Epipoles



RectificationRectification

Problem: Epipolar lines not parallel to scan Problem: Epipolar lines not parallel to scan lineslines

plp

r

P

Ol Or

Pl Pr

Epipolar Plane

Epipolar Lines

Epipoles at infinity

Rectified Images



3-D Reconstruction3-D Reconstruction

Reprinted from “Stereo by Intra- and Intet-Scanline Search,” by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and MachineIntelligence, 7(2):139-154 (1985). 1985 IEEE.


3-D Reconstruction3-D Reconstruction

Intrinsic and extrinsicIntrinsic and extrinsic Intrinsic onlyIntrinsic only No informationNo information

Unambiguous (triangulation)Unambiguous (triangulation) Up to unknown scaling factorUp to unknown scaling factor Up to unknown projective Up to unknown projective

transformationtransformation

A Priori Knowledge 3-D Reconstruction from two views


Projective ReconstructionProjective Reconstruction

Euclidean reconstruction Projective reconstruction



Euclidean vs Projective Euclidean vs Projective reconstructionreconstruction

Euclidean reconstructionEuclidean reconstruction – true metric properties of objects – true metric properties of objects lenghts (distances), angles, parallelism are preserved lenghts (distances), angles, parallelism are preserved

Unchanged under rigid body transformationsUnchanged under rigid body transformations => Euclidean Geometry – properties of rigid bodies under => Euclidean Geometry – properties of rigid bodies under

rigid body transformations, similarity transformationrigid body transformations, similarity transformation

Projective reconstructionProjective reconstruction – lengths, angles, parallelism are – lengths, angles, parallelism are NOT NOT preserved – we get distorted images of objects – their preserved – we get distorted images of objects – their distorted 3D counterparts --> 3D projective reconstructiondistorted 3D counterparts --> 3D projective reconstruction

=> Projective Geometry => Projective Geometry



Check this out!Check this out!

http://www.well.com/user/jimg/stereo/stereo_list.html

http://www.well.com/user/jimg/stereo/stereo_plunge.html


How can We Improve Stereo?How can We Improve Stereo?

Space-time stereo scanneruses unstructured light to aidin correspondence

Result: Dense 3D mesh (noisy)



Active Stereo: Adding Texture to SceneActive Stereo: Adding Texture to Scene

By James Davis, By James Davis, Honda Research,Honda Research,

Now UCSCNow UCSC



rect

ified

Active Stereo (Structured Light)Active Stereo (Structured Light)



Range Images (depth images, depth Range Images (depth images, depth maps, surface profiles, 2.5-D images)maps, surface profiles, 2.5-D images)

•Sensors that produce depth directly.•Pixel of a range image is the distance between a known reference frame and a visible point in the scene.•Representations:

•Cloud of Points (x,y,z)•Rij form (spatial information is explicit)


Active Range SensorsActive Range Sensors

Project energy or control sensor’s parameters.Project energy or control sensor’s parameters. Laser, Radars (accurate)/Sonars(inaccurate).Laser, Radars (accurate)/Sonars(inaccurate). Active Focusing/Defocusing.Active Focusing/Defocusing.



Xc

YcZc

Image

Light Stripe System.

Light Plane:AX+BY+CZ+D=0 (in camera frame)Image Point:x=f X/Z, y=f Y/Z (perspective)

Triangulation: Z=-D f/(A x + B y + C f)

Move light stripe or object.


Time of FlightTime of Flight

csc83029 3-d computer vision / ioannis stamos 3-d computer vision csc 83029 stereo

Documents

d fixation

stereo slide

infinity slide

d computer vision csc

baseline t

d information depth

parallel optical axes

depth z