3d computer vision

3D computer vision techniques v.4b2 1

3D computer vision techniques

KH Wong


Seminar Title: 3D computer vision techniques. Abstract

In this talk, the ideas of obtaining 3D information of objects (or called 3D reconstruction) using different techniques are discussed. Currently, the most popular one is the image based method that uses 2D cameras for 3D reconstruction; in particular reconstruction based on one-image, two-image and multiple-image are discussed. Moreover, batch and sequential treatments of input data are studied. I will also talk about novel techniques, such as using multiple cameras and laser based methods to obtain 3D information. And I will discuss how 3D computer vision is used in film and game production. Finally naked-eye 3D display technologies will be mentioned.


Overview (part1) Introduction From 2D to 3D

Camera systems/calibration Feature extraction/correspondence Reconstruction algorithms

2 views, 3 views , N views Real-time algorithms/Kalman filter

Previous projects Virtual viewer/ Projector camera systems Keystone correction

Novel setups Multiple cameras/ Camera array

Obtain 3D directly Structured light Laser approach Kinect approach Photometric stereo


Overview (part 2) Applications

Photos from tourists (photo tourism) http://phototour.cs.washington.edu/

3D displays Possible future research

Classification based on 3D information Content search 3D based on 3D keys Merging with sound information

http://phototour.cs.washington.edu/




Motivation We live in a 3D world We see 2D images but perceive the world in

3D Intelligent robot should have this 3D

reconstruction capability


How to obtain 3D information? Cameras-2D Range sensors-3D


Challenges Obtain 3D information for tasks in a 3D world.

2D-to-3D reconstruction from a camera 3D directly— laser range sensor, kinect sensor

Novel sensors Camera array/ multiple camera One pixel camera light field camera


2D-to-3D reconstruction (feature based method) Camera (perspective projection) Features-extraction and correspondences Methods

One-image method Two-image (Stereo) method Three-image method N-image method

Bundle adjustment Kalman filter


Camera: 3D to 2D projection

Perspective model u=F*X/Z (nonlinear relation) v=F*Y/Z

FZ

Y

v

World center

F

Thin lensor a pin hole

Virtual Screenor CCD sensor

RealScreenOr CCD sensor

Pinhole Camera

http://upload.wikimedia.org/wikipedia/en/8/81/Pinhole-camera.png


Perspective Projective

Model M at t=1

c (Image center, ox,oy)

F=focal length

image

Oc=(0,0,0) (Camera center)

Xc-axis

Zc-axis

Yc-axis

v-axis

u-axis

X,Y,Z

(u,v)

(0,0) of image plane

Camera Coordinates.

WorldCoordinates

YwZw

Xw Rc,Tc

Principal axis


In paintings Western Fresco by Raphael, 1510 -

1511, Stanza della Signatura, Vatican Palace, Rome.

Chinese 《富春山居圖》是元朝畫家黃公望的作品，創作於

1347年至 1350年 Dwelling in the

Fuchun Mountains (富春山居圖 ) by Huang Gongwang (1269–1354)

http://www.es.flinders.edu.au/~mattom/science+society/lectures/illustrations/lecture17/schoolathens.htmlhttp://jsl641124.blog.163.com/blog/static/17702514320115219508530/

http://zh.wikipedia.org/wiki/%E5%85%83%E6%9C%9D

http://zh.wikipedia.org/wiki/%E9%BB%83%E5%85%AC%E6%9C%9B

http://en.wikipedia.org/wiki/Huang_Gongwang

http://en.wikipedia.org/wiki/Huang_Gongwang

http://www.es.flinders.edu.au/~mattom/science+society/lectures/illustrations/lecture17/schoolathens.html


Feature correspondences--Camera moved, find correspondences for neighboring images--We can use feature to identify the motions of projected 3D features in 2D.

Image at t=t0

(or left image)Image at t=t0+dt (or right image)

Area a


Demo Youtube Movie http://www.youtube.com/watch?v

=azl-DGK6e1U

http://www.youtube.com/watch?v=azl-DGK6e1U




One-image 2D-to-3D reconstruction


One image 2D-to-3D reconstruction method Difficult and with ambiguity

http://ai.stanford.edu/~asaxena/reconstruction3d/


One image 2D-to-3D Using prior knowledge (e.g. face)

http://www.wisdom.weizmann.ac.il/~ronen/papers/Hassner Basri - Example Based 3D Reconstruction from Single 2D Images.pdf

http://www.wisdom.weizmann.ac.il/~ronen/papers/


Two-image 2D-to-3D reconstruction


Two-image 2D-to-3D reconstruction method: stereo vision

Objectives: Basic idea of stereo vision Stereo reconstruction by epipolar geometry

Stereo camera pair calibration (find Fundamental matrix F)

Construct the 3D (graphic) model from 2 images

Graphic model

Inside a computer


X’l X’r

FocalLengthf

ObjectPx(x,y,z)

z

Left camera center (reference point) Horizontal

Disparity=xL-xR

b (Baseline)

Left CameraPrincipleaxis

Right CameraPrincipleaxis

LeftImage plane

RightImage plane

,

( )

elimate (

'l

'r

' 'l r

x x=z fx - b x =

z ff bx z =x - x )

By similar triangle,w.r.t left camera lens center

By similar triangle,w.r.t right camera lens center

if camera motion is pure translation : Triangular calculation

One major problem is to locate x’l and x’r The correspondence problem


If camera motion is NOT pure translation : Use Epipolar Geometry

X

O2O1

right Frame Plane-2 2

left Frame Plane-1 1

e1e2

Left epipolar lineRight epipolar line

R,T

(x1,y1)

(x2,y2)

Left side is the reference

Focal length=f1

Focal length=f2

Base line=||T||

Plane-3 3

Perpendicular to TX2 or TX1

Right_image_pointT*E*left_image_point=0


Method: 8-point algorithmhttp://www.cs.manchester.ac.uk/ugt/COMP37111/papers/Hartley.pdf

Find 8 point corresponded ( Map 8 Right_image_points to left_image_point

Solve the epeiolar formula Right_image_pointT*E*left_image_point=0 Find E. From E we can find camera R (rotation) ,T

(translation) From R,T we can find model (3D positions of the

left feature points (using left as reference)


An example of stereo reconstruction An example Short-Baseline

Stereo Systems for Mobile Devices

http://www.lelaps.de/videos.html#SQx5vU8BA-Mhttp://www.lelaps.de/projects/stmobile.html

http://www.lelaps.de/videos.html#SQx5vU8BA-M


Stereo-based Free-space Estimation Another example

http://www.lelaps.de/videos.html#VrKBNtAN03ohttp://www.lelaps.de/projects/freespace.html

http://www.lelaps.de/videos.html#VrKBNtAN03o


Three-image 2D-to-3D reconstruction


Three-image 2D-to-3D reconstruction method More robust using 3 views It contains 3 epipolar

relations Stereo1: view1,2 , Stereo2: view2,3, Stereo3 :view 3,1.

Combine 3 epipolar geometry information together.

Similar to the algorithm in epipolar geometry (apply 3 times)

http://www.cs.unc.edu/~marc/tutorial/node45.html

M=3-D model pointM, m’, m” are image pointsC,C’,C” are camera centers


Example of 3-image reconstruction Example

LIBVISO: Feature Matching for Visual Odometry http://www.youtube.com/watch?v=DPLh6MoxPAk


N-image

2D-to-3D reconstruction(batched method: order of

images can be random )


N-image 2D-to-3D reconstruction method Bundle adjustment approach

Guess iteratively the solution for 3D to explain the measurements of feature points in all images

Math: Q(u,v)=g(X), g is nonlinear (projection) because u=fX/Z v=fY/Z, f=focal length Given Q (image measurement) , we want to find X=(X,Y,Z)i

from image points (u,v)i of all N model points (i=1,,,N), g is the projection formulas

A typical non linear optimization problem, Gauss-Newton for non linear optimization method is used.


Batched method: order of images can be random From measurement [u,v]I find X

…

Camera motion

Imaget=1

Imaget=2 Image

t=3Imaget=m

…

v1

v2v3

vm

X

[u,v]2

[u,v]1

[u,v]3

[u,v]m O1

O2

O3

Om

R2,T2R3,T3

Rm,Tm

R1,T1


Example Bundle adjustment reconstruction

http://www.cse.cuhk.edu.hk/%7Ekhwong/demo/canyon2b2.mpg


N-image

2D-to-3D reconstruction(Sequential method: order of images are used like in a move )


Sequential method: order of images are used like in a move From measurement [u,v]I find X

…

Camera motion

Imaget=1

Imaget=2 Image

t=3Imaget=m

…

v1

v2v3

vm

X

[u,v]2

[u,v]1

[u,v]3

[u,v]m O1

O2

O3

Om

R2,T2R3,T3

Rm,Tm

R1,T1


Kalman Filter

33pictures by Ko Hoi Fung


Kalman filter example

34

t = 0:t = 0:Position = x0Position = x0Velocity = v0Velocity = v0

t’ = 1:t’ = 1:Position = x1’Position = x1’Velocity = v1’Velocity = v1’x1’ = v0 * t + x0x1’ = v0 * t + x0

t = 1:t = 1:Position = x1Position = x1

PredictionPrediction

UpdateUpdate


Example Hernan Badino and Takeo

Kanade:"A Head-Wearable Short-Baseline Stereo System for the Simultaneous Estimation of Structure and Motion".IAPR Conference on Machine Vision Applications (MVA), Nara, Japan, June

2011 http://www.youtube.com/watch?v=SQx5vU8BA-M


Novel sensors : Camera array/ Multiple camera systems Camera array/ multiple camera: High

Performance Imaging - Using Large Camera Array

http://www.youtube.com/watch?v=0W_1Ce2lTBohttp://graphics.stanford.edu/papers/CameraArray/


The Self-Reconfigurable Camera Array

http://chenlab.ece.cornell.edu/projects/MobileCamArray/

Demo moviehttp://chenlab.ece.cornell.edu/projects/MobileCamArray/videos/train.movhttp://chenlab.ece.cornell.edu/projects/MobileCamArray/videos/self_reconfiguration.mov

Each camera

http://chenlab.ece.cornell.edu/projects/MobileCamArray/

http://chenlab.ece.cornell.edu/projects/MobileCamArray/videos/train.mov


Applications


Photo tourism






Projector-camera system

Application of computer vision


A Projector-Camera system


Projector-Camera calibration

http://www.youtube.com/watch?v=YHhQSglmuqY&feature=channel_page


Our setup


Calibration procedure


Quadrangle tracking


Experiments


Projection result


Results


Hand held direct manipulation 3D Display

http://www.youtube.com/watch?v=vVW9QXuKfoQ&feature=relmfu


Keystone correction

Configuration


Aim of this work Desired Results

Keystoned projection Corrected projection


Overview Three major modules

Projector-camera pair calibration Projection region detection and tracking Automatic keystone correction

3D projectionregion recovery

Projector-camera pair calibration

G, K

Corrected projection

Keystoned projection

Pre-warped image3D rectangle

Keystone correctionProjection region tracking

Camera frame

Flow chart


Pre-warp projection image

Pre-warped projection image Display result

http://www.youtube.com/watch?v=y5XYdeh8Bno&list=UUfy2EumiHMeoUorMFR0woZA&index=1&feature=plcp


Keystone correction Some real correction results


Obtain 3D directly Laser range sensor

Time of flight Kinect


Photometric stereo

http://www.taurusstudio.net/research/photex/ps/equation.htm

Lamertian light formula

•Given 3 or more known light source we can find the normal N•From the set of N we can approximate the surface

http://www.wisdom.weizmann.ac.il/~vision/photostereo/


Photometric stereo using multiple cameras and multiple light sources

Demo

Dynamic Shape Capture using Multi-View Photometric Stereo SIGGRAPH 2009 http://www.youtube.com/watch?v=9hgs5zN38lk


Multiple cameras fro human body reconstruction

Homepage://media.au.tsinghua.edu.cn


Experimental Results

59 3D Modeling Using MVML Dome 23/5/3


Multiple camera doom

http://www.mpi-inf.mpg.de/~yliu/


Structured light method Calculate the shape by how the strip is

distorted.

http://www.laserfocusworld.com/articles/2011/01/lasers-bring-gesture-recognition-to-the-home.html


Real time Virtual 3D Scanner - Structured Light Technology Demo

http://www.youtube.com/watch?v=a6pgzNUjh_s


Time of flight laser method Send the IR-laser light to

different directions and sense how each beam is delayed.

Use the delay to calculate the distance of the object point



LIDAR light detection and ranging scanner

http://hodcivil.edublogs.org/2011/11/06/lidar-%E2%80%93-light-detection-and-ranging/http://commons.wikimedia.org/wiki/File:Lidar_P1270901.jpg

Leica terrestrial lidar (light detection and ranging) scanner

http://www.youtube.com/watch?v=MuwQTc8KK44

http://hodcivil.edublogs.org/2011/11/06/lidar-%E2%80%93-light-detection-and-ranging/

http://commons.wikimedia.org/wiki/Leica

http://commons.wikimedia.org/wiki/Lidar


3D Laser Scanning - Underground Mine Mapping Demo

http://www.youtube.com/watch?v=BZbvz8fePeQ


Motion capture for film production (MOCAP)

http://upload.wikimedia.org/wikipedia/commons/7/73/MotionCapture.jpghttp://www.naturalpoint.com/optitrack/products/s250e/indepth.html

IR light emitterand camera

http://www.youtube.com/watch?v=IxJrhnynlN8

http://upload.wikimedia.org/wikipedia/commons/7/73/MotionCapture.jpg


3D body scanner

http://www.cyberware.com/products/scanners/ps.htmlhttp://www.cyberware.com/products/scanners/wbx.html

http://www.youtube.com/watch?v=86hN0x9RycM


3-D Face capture

http://www.captivemotion.com/products/

http://www.youtube.com/watch?v=-TTR0JrocsI&feature=related

http://www.youtube.com/watch?v=-TTR0JrocsI&feature=related


Dimensional Imaging 4D Video Face Capture with Textures

Dimensional Imaging 4D Video Face Capture with Textures

http://www.youtube.com/watch?v=XtTN7tWaXTM&feature=related


Kinect Another structure light

method Use dost rather than

strips



Kinect Hardware


See the IR-dots emitted by KINECT

http://www.youtube.com/watch?v=-gbzXjdHfJA

http://www.youtube.com/watch?v=dTKlNGSH9Po&feature=related

http://www.youtube.com/watch?v=-gbzXjdHfJA


Novel sensors : light field cameraSpin off from Stanford camera array

light field camera : LYTRO camera

Be able to refocus after the picture is taken

https://www.lytro.com/camera

http://www.youtube.com/watch?v=7QV152jc3Ac


light field camera How does it work

http://www.quora.com/Lytro/How-does-the-new-Lytro-camera-work


3D (Volumetric) displayRendering for an Interactive 360º Light Field Display

SIGGRAPH 2007 Papers Proceedings

http://gl.ict.usc.edu/Research/3DDisplay/

http://www.youtube.com/watch?v=h6aUIS44ezE


Occlusion-Capable Volumetric 3D Display by Cossairt,etal.

Actuality Systems, Inc

http://www.3dcgi.com/cooltech/displays/displays.htm

http://www.youtube.com/watch?v=8KaQmn2VTzs


3D display Using a lattice with thin slits, viewer's eyes see different pixels on the screen to create 3d perception

http://www.televisions.com/tv-articles/TV-in-3D/Displaying-3D-Without-Glasses.php


The future Content search in 3D video data bases Shot boundary detection Video data mining Video classification


Appendix


Essential matrix E (a 3x3 matrix) P.110[2] X1 is 3-D X in left camera (reference) system X2 is 3-D X in right camera system

RTEE

RT(i)

TT

TT TTTTT

TTTTiRTT

TTR

T

T

TT

BABxA

where0** hence

0)( , from

0)(X as same ,0)(X so

)( lar toperpendicu is plane, same theon are and0 andmatrix symmetricskew

and ][ Since )(

by sides bothmultiply ,

12

12

2222

222

12

12

BABxA

x

x

TTTT

tttt

tt

ttt

T

][

00

0][

12

13

23

3

2

1

Exercise1:Draw vectors TX2 or TX1 in the diagram


Essential Matrix E

Right_image_pointT*E*left_image_point=0

2 1

2 2 2 1 1 1

2 1

2 1

2 12 2 2 1 1 1

2 1

2 2 1 1 12

X * * X 0

* * 0

_ _ _ _ _ tan _ ,

* * 0

* * 0

since X, ,

T

T

T

T

T

E

X Y Z E X Y Z

f fno harm to prefix some cons t termsZ Z

f fX Y Z E X Y ZZ Z

x y f E x y f

X YX Y Z and x f y fZ Z

3d computer vision

Documents