building a panorama image mosaics and...

1

Image Mosaics and Panoramas

Lots of slides from from Bill Freeman, Alyosha Efros, Steve Seitz and Rick Szeliski. Some slides are from Vaibhav Vaish, Yung-Yu Chuang, Darya Frolova, Denis Simakov, Alex RavAcha, and from Shmuel Peleg.

Building a PanoramaBuilding a Panorama

M. Brown and D. G. Lowe. Recognising Panoramas. ICCV 2003

Why Panoramas ?Why Panoramas ?

• Cartography: stitching aerial images to make maps

Manhattan, 1949 Google Maps

2

Google Street View

Slide from Brown & Lowe

• Virtual wide-angle camera– Consumer camera: 50˚ x 35˚


• Virtual wide-angle camera– Consumer camera: 50˚ x 35˚– Human Vision: 176˚ x 135˚

3


• Virtual wide-angle camera– Consumer camera: 50˚ x 35˚– Human Vision: 176˚ x 135˚– Panoramic mosaics: up to 360˚ x 180˚

Mosaics: stitching images togetherMosaics: stitching images together

virtual wide-angle camera

– convert masked images into a background sprite for content-based coding

+ + +

=

Video CompressionVideo Compression Video SummarizationVideo Summarization

4

• Virtual reality: a sense of being there

http://www.panoramas.dk/

• Video Panorama

The First Panoramas …The First Panoramas The First Panoramas ……

Paris, c. 1845-50, photographer unknown

How to create a panorama?How to create a panorama?

1. Take a sequence of images2. Compute transformation between second image and first3. Transform the second image to overlap with the first4. Blend the two together to create a mosaic5. If there are more images, repeat

Don’t we need the 3D geometry of the scene?

5

Panorama Panorama -- from Rotated Camerafrom Rotated Camera

If the sequence of images are taken from the sameposition (Rotate the camera about its optical center),then the 3D geometry of the scene is NOT needed.

First Panoramic CameraFirst Panoramic Camera

Al-Vista, 1899 ($20)

The CameraThe Camera

• http://www.reallyrightstuff.com/pano/index.html

A pencil of rays contains all viewsA pencil of rays contains all viewsas long as they have the same center of projection!

realcamera

syntheticcamera

6

Increasing the Field of ViewIncreasing the Field of View

Camera Center

P

Q

ExampleExample

Camera Center

Aligning ImagesAligning Images

How can we find the homographies required for stitching ?

• From calibration parameters– Works, but these aren’t always known

What’s the relation between corresponding points?e.g. translation, Euclidean, affine, projective

Translation

2 parameters

Affine

6 parameters

Perspective

8 parameters

7

Trial Trial -- aligning images by translationaligning images by translation

Translations are not enough to align the images

left on top right on top

TT ZfYZfXZYX )/,/(),,( a

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

10100

1ZYX

ff

ZfYfX

ZYX

a

Linear projection in homogeneous coordinates!

Pinhole camera model Pinhole camera model -- Perspective ProjectionPerspective Projection

(X,Y,Z)

Tyx

T )oZ/fY,oZ/fX()Z,Y,X( ++a

principal pointTyx )o,o(

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⎥⎥⎦

⎤

⎢⎢⎣

⎡=

⎟⎟⎠

⎞⎜⎜⎝

⎛++

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

1ZYX

010of0of

ZZofYZofX

1ZYX

y

x

xx

a

Pinhole point offsetPinhole point offset

x

(ox,oy)y

Pinhole camera model Pinhole camera model -- Perspective ProjectionPerspective Projection

Camera Center (0,0,0)

Perspective warps (Homographies)Perspective warps (Perspective warps (HomographiesHomographies))

p1

p1 ≈ K P

P (X, Y, Z)

(x, y)

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡≈⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

+=

+=

ZYX

ofof

yx

oZfYy

oZfXx

y

x

y

x

1000

0

1

8


P

p1

p2

p1 ≈ K Pp2 ≈ K R P

(X, Y, Z)

(x’, y’)



P

p1

p2

3x3 Homography

K-1 p1 ≈ P

p2 ≈ K R K-1 p1


p1 ≈ K Pp2 ≈ K R P

3x3 Homography

p1` ≈ H p1

Calculate HomographyCalculate Calculate HomographyHomography

Given corresponding points in 2 images:

pi p`i

Find H satisfying

Requires 4 matching points.(8 unknowns since H up to Scale)

HomographyHomography

• Projective – mapping between any two projection planes (PPs) with the same center of projection

– rectangle should map to arbitrary quadrilateral – parallelism is not preserved – straight lines are preserved

• Called Homography PP2

PP1⎥⎥⎦

⎤

⎢⎢⎣

⎡

⎥⎥⎦

⎤

⎢⎢⎣

⎡=

⎥⎥⎦

⎤

⎢⎢⎣

⎡

1yx

*********

wwy'wx'

H pp’

p

p’

9

Camera Center

P

p1

p2 ≈ K (R P+T)p1 ≈ K P

Changing Camera CenterChanging Camera CenterWhat about mapping between 2 general images

(NOT from rotating camera)?

p2NOT Homography

p2 p1

Changing camera centerChanging camera center

PP1

PP2

HomographyHomographyHowever: Mapping between any two arbitrary PPs IS possible

if the object is planar.

PP2PP1

⎥⎥⎦

⎤

⎢⎢⎣

⎡

⎥⎥⎦

⎤

⎢⎢⎣

⎡=

⎥⎥⎦

⎤

⎢⎢⎣

⎡

1yx

*********

wwy'wx'

H pp’

p

p’

homography- K1K2-1

3x3 Homography

HomographyHomography -- planar sceneplanar scene

PP1

PP2

Object Planehomography- K1

homography- K2

p2 ≈ K2 Pp1 ≈ K1 P

p1 ≈ K1 K2-1p2

P

p1

p2

10

Julian Julian BeeverBeever: Manual : Manual HomographiesHomographies

http://users.skynet.be/J.Beever/pave.htm

EpipolarEpipolar GeometryGeometry

PP2PP1

[ ]⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

1yx

*********

yx 1''0

Fundamental Matrix

p

For non planar scene, there is no Homography between two generalviews , however there is a geometric constraint:The Epipolar Constraint between any two arbitrary PPs

EpipolarEpipolar GeometryGeometry Image warping with Image warping with homographieshomographies

image plane in front image plane belowblack areawhere no pixelmaps to

11

To unwarp (rectify) an image• Find the homography H given a set of p and p’ pairs• How many pairs are needed?• What if more points are given?

pp’

Finding the Finding the HomographyHomography using Feature Pointsusing Feature Points

p2 ≈ K R K-1 p1

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡≈

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

111''

yx

hgfedcba

yx

Features = e.g. SIFT, Harris etc

1'

1'

++++

=

++++

=

hygxfeydxy

hygxcbyaxx

Two linear equations per matching feature

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

⎥⎦⎤

⎢⎣⎡

−−−−=⎥⎦

⎤⎢⎣⎡

′′

fedcba

'yy'xy1yx000y'xx'x0001yx

yx

Finding the Finding the HomographyHomography using Feature Pointsusing Feature Points Solving for Solving for homographieshomographies

• Set up a system of linear equations:

Ah = b• Where vector of unknowns h = [a,b,c,d,e,f,g,h]T

• Need at least 8 eqs, but the more the better…• Solve for h. If overconstrained, solve using least-squares:

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

1yx

ihgfedcba

wwy'wx'

p’ = Hp

2min bAh −

12

Feature Matching PanoramaFeature Matching Panorama

• Reference image is marked red.• For each blue image the homography was calculated

using matched features.

Homographic Warp Homographic Warp -- ExampleExample

Homographic Warp Homographic Warp -- RevisitedRevisited

Panorama

SceneHomographic Warp Homographic Warp -- ExampleExample

13

Homographic Warp Homographic Warp -- ExampleExample

Ken Chu, 2004

Full PanoramasFull Panoramas

What if you want a 360° field of view?

– Map 3D point (X,Y,Z) onto cylinder

Cylindrical projectionCylindrical projection

XY

Z

unit cylinder

unwrapped cylinder

– Convert to cylindrical coordinates

cylindrical image

– Convert to cylindrical image coordinates

Cylindrical ProjectionCylindrical Projection

Y

X

14

In the cylindrical coordinates rotational warping is translated into translational warping.

Assembling the panoramaAssembling the panorama

• Stitch pairs together, blend, then crop

EndEnd--toto--end alignment and cropend alignment and crop

NOTE: straight lines become conics.

360360°° PanoramaPanorama

15

Inverse Cylindrical ProjectionInverse Cylindrical Projection

X

YZ

(X,Y,Z)

(sinθ,h,cosθ)

x

z

θ( )( )

hyzxfx

zx

syyhsxx

C

C

====

−=−=

ˆˆcosˆsinˆ

~~~~

θθ

θf

x

z

x

QuicktimeQuicktime VRVR

Panoramas: http://www.apple.com/quicktime/technologies/qtvr/gallery/times_square.htmlhttp://www.panoramas.dk/fullscreen5/f45-everest-mars.htmlhttp://www.panoramas.dk

Demo

TimesSquare-Med.mov

MultiMulti--Perspective PanoramaPerspective Panorama

Suppose we do not want a 360° view. We want a view of a “whole street” or even more complicated - a route alongcity streets.

Using homography to align all images,(i.e. creating a single wide angle view)farthest objects will be distorted tovery large regions.

Thus need to simulate a different camera model

PushbroomPushbroom Camera ModelCamera Model

16

The generalized camera model:

X direction - parallel projectionY direction - perspective projection

x

y

from: http://www.wisdom.weizmann.ac.il/~deniss/2002-05_stereo_generalized_cameras/stereo_with_generalized_cameras.ppt

PushbroomPushbroom Camera ModelCamera Model Pushbroom Panorama - Example

The Stanford CityBlock Project: http://graphics.stanford.edu/projects/cityblock/

PushbroomPushbroom Camera ModelCamera Modelt t+1 t+2

PushbroomPushbroom Camera ModelCamera Model

Implementation through cuts of 3D video array.

• Take images while moving a usual camera

• Stack them into 3D array

• Take a cut along the “time” dimension

t

17

Pushbroom Panorama - Example

Gooseneck State Park, Utah

t

u

kk+1

u

t

Stationary Camera Panning Camera

kk+1

Space-Time Volume Slicing - - Video MosaicAlign the frames of the Space-Time Volume

From: Rav-Acha and Peleg

Space-Time Volume Slicing - Panorama

t

x

Panorama (still)


Space-Time Volume Slicing - Dynamosaicing

t

x

Forward Panning camera


18

Original - Panning



Iguazu Falls


t

x

Backward Panning camera


Backward - Panning




t

x

backward panning forward motion


19

Backward Panning - forward motion


Space-Time Volume Slicing - Dynamosaicing Space-Time Volume Slicing - Video Panorama

t

x

video panorama


Backward Panning - forward motion PANORAMA


Space-Time Volume Slicing - Dynamosaicing Another example

20

Space-Time Volume Slicing- Evolving Time Fronts

Still CameraMoving objects

t

kk+1


Space-Time Volume Slicing- Evolving Time Fronts u

t

u

t

xMapping each TF to a new frame





21

Next topic: Stitching and CompositionNext topic: Stitching and Composition

Stitching = alignment + blending

geometricalregistration

photometricregistration

Panorama BlendingPanorama Blending

THE ENDTHE END

building a panorama image mosaics and...

Documents