image mosaics and panoramas building a panorama - idc · 1 1 image mosaics and panoramas lots of...

1

1

Image Mosaics and Panoramas

Lots of slides stolen from Bill Freeman, who stole them from Alyosha Efros, who stole them from Steve Seitz and Rick Szeliski. Some slides are from Vaibhav Vaish, Yung-Yu Chuang, Darya Frolova, Denis Simakov.Links: http://groups.csail.mit.edu/graphics/classes/CompPhoto06/http://robots.stanford.edu/cs223b05/ 2

Building a Panorama

M. Brown and D. G. Lowe. Recognising Panoramas. ICCV 2003

3

Why Panoramas ?• Cartography: stitching aerial

images to make maps

Manhattan, 1949 4

2

5

Slide from Brown & Lowe

• Virtual wide-angle camera– Consumer camera: 50˚ x 35˚

6


• Virtual wide-angle camera– Consumer camera: 50˚ x 35˚– Human Vision: 176˚ x 135˚

7


• Virtual wide-angle camera– Consumer camera: 50˚ x 35˚– Human Vision: 176˚ x 135˚– Panoramic mosaics: up to 360˚ x 180˚

8

Mosaics: stitching images together

virtual wide-angle camera

3

9

• Video Coding: – convert masked images into a background sprite

for content-based coding

+ + +

=

10

• Virtual reality: a sense of being there

11

How to make a panorama?1. Take a sequence of images from the same position

(Rotate the camera about its optical center)

2. Compute transformation between second image and first3. Transform the second image to overlap with the first4. Blend the two together to create a mosaic5. If there are more images, repeat

• Don’t we need the 3D geometry of the scene?

12

The Camera• http://www.reallyrightstuff.com/pano/index.html

4

13

A pencil of rays contains all views

realcamera

syntheticcamera

Can generate any synthetic camera viewas long as it has the same center of projection! 14

Increasing the Field of View

Camera Center

P

Q

15

Example

Camera Center 16

Aligning Images

How can we find the homographies required for stitching ?• From calibration parameters

– Works, but these aren’t always known

What’s the relation between corresponding points?e.g. translation, Euclidean, affine, projective

5

17

Translation

2 parameters

Affine

6 parameters

Perspective

8 parameters18

Trial - aligning images by translation

Translations are not enough to align the images

left on top right on top

19Camera Center (0,0,0)

Perspective warps (Homographies)Perspective warps (Homographies)

p1

p1 ≈ K P

P (X, Y, Z)

(x, y)

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡≈⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

+=

+=

ZYX

ofof

yx

oZfYy

oZfXx

y

x

y

x

1000

0

1


P

p1

p2

p1 ≈ K Pp2 ≈ K R P (X, Y, Z)

(x’, y’)


6


P

p1

p2

3x3 Homography

K-1 p1 ≈ P

p2 ≈ K R K-1 p1


p2 ≈ K R Pp1 ≈ K P

22

Homography• Projective – mapping between any two projection planes (PPs)

with the same center of projection– rectangle should map to arbitrary quadrilateral – parallelism is not preserved – straight lines are preserved

• called Homography PP2

PP1⎥⎥⎦

⎤

⎢⎢⎣

⎡

⎥⎥⎦

⎤

⎢⎢⎣

⎡=

⎥⎥⎦

⎤

⎢⎢⎣

⎡

1yx

*********

wwy'wx'

H pp’

p

p’

23

Homography• Mapping between any two arbitrary

PPs if the object is planar.

PP2PP1

⎥⎥⎦

⎤

⎢⎢⎣

⎡

⎥⎥⎦

⎤

⎢⎢⎣

⎡=

⎥⎥⎦

⎤

⎢⎢⎣

⎡

1yx

*********

wwy'wx'

H pp’

p

p’

24

Epipolar Geometry• The epipolar geometry introduces constraints between any two

arbitrary PPs :

PP2PP1

[ ]⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

1yx

*********

yx 1''0

Fundamental Matrix

p

7

25

Image warping with homographies

image plane in front image plane belowblack areawhere no pixelmaps to 26

To unwarp (rectify) an image• Find the homography H given a set of p and p’ pairs• How many correspondences are needed?• Tricky to write H analytically, but we can solve for it!

• Find such H that “best” transforms points p into p’• Use least-squares!

pp’

27

Finding the Homography by Matching Features

p2 ≈ K R K-1 p1

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡≈

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

111''

yx

hgfedcba

yx

28

Homographies by Feature Matching

p2 ≈ K R K-1 p1

1'

1'

++++

=

++++

=

hygxfeydxy

hygxcbyaxx

Two linear equations per matching feature

8

29

Solving for homographies

• Can set scale factor i=1. So, there are 8 unkowns.• Set up a system of linear equations:

Ah = b• where vector of unknowns h = [a,b,c,d,e,f,g,h]T

• Need at least 8 eqs, but the more the better…• Solve for h. If overconstrained, solve using least-squares:

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

1yx

ihgfedcba

wwy'wx'

p’ = Hp

2min bAh −30

Feature Matching Panorama

• Reference image is marked red.• For each blue image the homography was calculated

using matched features.

31

Full Panoramas

• What if you want a 360° field of view?

mosaic Projection Cylinder

32

– Map 3D point (X,Y,Z) onto cylinder

Cylindrical projection

XY

Z

unit cylinder

unwrapped cylinder

– Convert to cylindrical coordinates

cylindrical image

– Convert to cylindrical image coordinates

9

33

Cylindrical Projection

Y

X34

Assembling the panorama

• Stitch pairs together, blend, then crop

35

End-to-end alignment and crop

36

Inverse Cylindrical projection

X

YZ

(X,Y,Z)

(sinθ,h,cosθ)

Panoramas: http://www.panoramas.dk/fullscreen5/f45-everest-mars.html

10

37

Matching with Features•Detect feature points in both images

38


•Find corresponding pairs

39


•Find corresponding pairs

•Use these pairs to align images

40

• Problem 1: Detect the same point in both images (invariant to possible deformations)

• Problem 2: For each point correctly recognize the corresponding one

• We need a reliable and distinctive feature descriptor

no chance to match!

Matching with Features

11

41

• Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters

D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”, IJCV 2004SIFT = Scale Invariant Feature Transform

SIFT – scale invariant feature transform

42

• Descriptor overview:– Determine scale (by maximizing DoG in scale and in space),

local orientation as the dominant gradient direction.Use this scale and orientation to make all further computations invariant to scale and rotation.

– Compute gradient orientation histograms of several small windows (128 values for each point)

– Normalize the descriptor to make it invariant to intensity change

D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. Accepted to IJCV 2004

SIFT – scale invariant feature transform

• Find local maximum of Difference of Gaussians in space and scale

scale

x

y

← DoG →

←D

oG→

43

Select canonical orientation

• Create histogram of local gradient directions computed at selected scale

• Assign canonical orientation at peak of smoothed histogram

• Each key specifies stable 2D coordinates (x, y, scale, orientation)

0 2π

44

SIFT descriptor vector • Thresholded image gradients are sampled over 16x16

array of locations in scale space• Create array of orientation histograms• 8 orientations x 4x4 histogram array = 128 dimensions

12

45

Matching SIFT Features

[Brown 2003]How do deal with outliers? 46

RANSAC

• RANSAC = Random Sample Consensus• an algorithm for robust fitting of models in the

presence of many data outliers• Compare to robust statistics

• Given N data points xi, assume that mjority of them are generated from a model with parameters Θ, try to recover Θ.

47

RANSAC algorithm

Run k times:(1) draw n samples randomly(2) fit parameters Θ with these n samples(3) for each of other N-n points, calculate

its distance to the fitted model, count the number of inlier points, c

Output Θ with the largest c

How many times?How big? Smaller is better

How to define?Depends on the problem.

48

How to determine kp: probability of real inliersP: probability of success after k trials

knpP )1(1 −−=n samples are all inliers

a failure

failure after k trials

)1log()1log(

npPk

−−

=

2930.56970.66350.53kpn

for P=0.99

13

49

Example: line fitting

50

n=2

51

Model fitting

52

Measure distances

14

53

Count inliersc=3

54

Another trialc=3

55

The best modelc=15

56

Finding SIFT features

15

57

Rejecting outliers using RANSAC

58

Finding the Homography and stitch images

59

Image Stitching• Naïve stitching might seem unnatural due to

parallax, lens distortion, scene motion, exposure difference

• Solution: Multiband blending

60

Multi-band Blending

• Burt & Adelson 1983

16

61

Image Blending

62

Image Blending

63

Blending

64

Left Image Right Image

Left + Right Narrow Transition Wide Transition Multibands blend

The effect of blending window

17

65

Multibands

Bandpass Images

Lowpass Images

66

Multiband Pyramid

expand

-

expand

expand

-

- =

=

=

Gaussian Pyramid

LaplacianPyramid

67

Gaussian pyramid construction

filter mask

• Repeat– Filter– Subsample

• Until minimum resolution reached – can specify desired number of levels (e.g., 3-level pyramid)

• The whole pyramid is only 4/3 the size of the original image! 68

What does blurring take away?

original

18

69

What does blurring take away?

smoothed (5x5 Gaussian) 70

High-Pass filter

smoothed – original

71

Multiband Blending

72

Pyramid Blending

19

73

laplacianlevel

4

laplacianlevel

2

laplacianlevel

0

left pyramid right pyramid blended pyramid74

Pyramid Blending

75

Blending Regions

76

The colored version

© prof. dmartin

20

77

Blending in Gradient Domain

• In Pyramid Blending, we decomposed our image into 2nd derivatives (Laplacian) and a low-res image

• Let us now look at 1st derivatives (gradients):– No need for low-res image

• captures everything (up to a constant)– Idea:

• Differentiate• Blend• Reintegrate

78

Gradient Domain blending (1D)

Twosignals

Regularblending

Blendingderivatives

bright

dark

79

Gradient Domain Blending – 2DPerez et. al. 2003

• Given vector field v (pasted gradient), find the value of f in unknown region that optimize:

Pasted gradient Mask

Background

unknownregion

80

Example

From Perez et al. 2003

21

81

Example

From Perez et al. 2003 82

Example

83

Another example

84

Another example

22

85

Comparisons: Levin et al, 2004

86

Don’t blend, CUT!

• So far we only tried to blend between two images. What about finding an optimal seam?

Moving objects become ghosts

87

Davis, 1998• Segment the mosaic

– Single source image per segment– Avoid artifacts along boundries

• Dijkstra’s algorithm

88

Input texture

B1 B2

Random placement of blocks

block

B1 B2

Neighboring blocksconstrained by overlap

B1 B2

Minimal errorboundary cut

Efros & Freeman, 2001

23

89 min. error boundary

Minimal error boundaryoverlapping blocks vertical boundary

__ ==22

overlap error 90

Video Mosaics

• So far we make panoramas from still images.• Is it possible to make panorama from video?• Dynamosaics (A. Rav-Acha ,Y. Pritch ,D. Liscinski

S. Peleg CVPR 2005)• Slides: courtesy of A. Rav Acha

DynaMosaics

Dynamosaics: A. Rav-Acha ,Y. Pritch ,D. Liscinski S. Peleg , CVPR 2005

Slides: courtesy of A. Rav Acha

92

Dynamic Mosaics•Cancel Camera Motion•Preserve Scene Motion

24

93

Constructing an alignedSpace-Time Volume

dtv

u

aα

t

bMotion Parallax: Align by straightening lines in SP Volume 94

t

u

kk+1

u

t

Stationary Camera Panning Camera

kk+1

Aligned ST Volume: View from Top

95

u

t

Evolving Time Frontu

t

xMaping each TF to a new frame 96

Temporal Video Editing

25

97

Iguazu Falls

u

t

98

wθ

TimeFlow

u

t

w

tk

α

ua ub

tl

Time Front

Output Frame

Reversing Scan Direction

99

Reversing ScanDirection

100

THE END

image mosaics and panoramas building a panorama - idc · 1 1 image mosaics and panoramas lots of...

Documents