homographies and panoramas - brown university
TRANSCRIPT
![Page 1: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/1.jpg)
Homographies and Panoramas
cs129: Computational Photography
James Hays, Brown, Fall 2012
Slides from Steve Seitz, Rick
Szeliski, Alexei Efros, Fredo
Durand, and Kristin Grauman
![Page 2: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/2.jpg)
Why Mosaic?
Are you getting the whole picture?
• Compact Camera FOV = 50 x 35°
Slide from Brown & Lowe
![Page 3: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/3.jpg)
Why Mosaic?
Are you getting the whole picture?
• Compact Camera FOV = 50 x 35°
• Human FOV = 200 x 135°
Slide from Brown & Lowe
![Page 4: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/4.jpg)
Why Mosaic?
Are you getting the whole picture?
• Compact Camera FOV = 50 x 35°
• Human FOV = 200 x 135°
• Panoramic Mosaic = 360 x 180°
Slide from Brown & Lowe
![Page 5: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/5.jpg)
Panoramic Photos are old
Sydney,1875
Beirut, late 1800’s
![Page 6: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/6.jpg)
Panorama Capture
![Page 7: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/7.jpg)
Mosaics: stitching images together
virtual wide-angle camera
![Page 8: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/8.jpg)
Big idea
As with high dynamic range imaging, we
are compensating for the shortcomings of
traditional cameras by capturing and
fusing multiple images
![Page 9: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/9.jpg)
Today’s lecture
Today we will assume that correspondence
between photos is known and we focus on
finding and applying homographies.
Wednesday and Friday we will discuss
automatic correspondence.
![Page 10: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/10.jpg)
How to do it?
Basic Procedure
• Take a sequence of images from the same position
– Rotate the camera about its optical center
• Compute transformation between second image and first
• Transform the second image to overlap with the first
• Blend the two together to create a mosaic
• If there are more images, repeat
…but wait, why should this work at all?
• What about the 3D geometry of the scene?
• Why aren’t we using it?
![Page 11: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/11.jpg)
Aligning images
Translations are not enough to align the images
left on top right on top
![Page 12: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/12.jpg)
A pencil of rays contains all views
real
camera synthetic
camera
Can generate any synthetic camera view
as long as it has the same center of projection!
![Page 13: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/13.jpg)
mosaic PP
Image reprojection
The mosaic has a natural interpretation in 3D • The images are reprojected onto a common plane
• The mosaic is formed on this plane
• Mosaic is a synthetic wide-angle camera
![Page 14: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/14.jpg)
Image reprojection
Basic question • How to relate two images from the same camera center?
– how to map a pixel from PP1 to PP2
PP2
PP1
Answer • Cast a ray through each pixel in PP1
• Draw the pixel where that ray intersects PP2
But don’t we need to know the geometry
of the two planes with respect to the eye?
Observation:
Rather than thinking of this as a 3D reprojection,
think of it as a 2D image warp from one image to another
![Page 15: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/15.jpg)
Back to Image Warping
Translation
2 unknowns
Affine
6 unknowns
Perspective
8 unknowns
Which t-form is the right one for warping PP1 into PP2?
e.g. translation, Euclidean, affine, projective
![Page 16: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/16.jpg)
Homography
A: Projective – mapping between any two PPs with the
same center of projection
• rectangle should map to arbitrary quadrilateral
• parallel lines aren’t
• but must preserve straight lines
• same as: project, rotate, reproject
called Homography
PP2
PP1
1
yx
*********
w
wy'wx'
H p p’
To apply a homography H
• Compute p’ = Hp (regular matrix multiply)
• Convert p’ from homogeneous to image
coordinates
![Page 17: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/17.jpg)
Image warping with homographies
image plane in front image plane below
black area
where no pixel
maps to
![Page 18: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/18.jpg)
Mosaics: main steps • Collect correspondences (manually)
• Solve for homography matrix H
– Least squares solution
• Warp content from one image frame to the other to combine: say im1 into im2 reference frame
– Determine bounds of the new combined image
• Where will the corners of im1 fall in im2’s coordinate frame?
• We will attempt to lookup colors for any of these positions we can get from im1.
– Compute coordinates in im1’s reference frame (via homography) for all points in that range
– Lookup all colors for all these positions from im1
• Inverse warp : interp2 (watch for nans)
• Overlay im2 content onto the warped im1 content.
– Careful about new bounds of the output image: minx, miny
![Page 19: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/19.jpg)
Homography
1
yx
*********
w
wy'wx'
H p p’
wyw
wxw
,
yx ,
yx,
To apply a given homography H
• Compute p’ = Hp (regular matrix multiply)
• Convert p’ from homogeneous to image
coordinates
![Page 20: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/20.jpg)
Homography
11, yx 11, yx
To compute the homography given pairs of corresponding
points in the images, we need to set up an equation where
the parameters of H are the unknowns…
22 , yx22 , yx
…
…
nn yx , nn yx ,
![Page 21: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/21.jpg)
Least Squares Example
Say we have a set of data points (X1,X1’), (X2,X2’),
(X3,X3’), etc. (e.g. person’s height vs. weight)
We want a nice compact formula (a line) to predict X’s
from Xs: Xa + b = X’
We want to find a and b
How many (X,X’) pairs do we need?
What if the data is noisy?
'
22
'
11
XbaX
XbaX'
2
'
1
2
1
1
1
X
X
b
a
X
XAx=B
.........
1
1
1
'
3
'
2
'
1
3
2
1
X
X
X
b
a
X
X
X
overconstrained
2min BAx
![Page 22: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/22.jpg)
Solving for homographies
Can set scale factor i=1. So, there are 8 unknowns.
Set up a system of linear equations:
Ah = b
where vector of unknowns h = [a,b,c,d,e,f,g,h]T
Need at least 8 eqs, but the more the better…
Solve for h. If overconstrained, solve using least-squares:
>> help lmdivide
1
y
x
ihg
fed
cba
w
wy'
wx'
p’ = Hp
2min bAh
![Page 23: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/23.jpg)
Example system for finding homography
Source: Paul Heckbert, Fundamentals of Texture Mapping and Image Warping
…
![Page 24: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/24.jpg)
Mosaics: main steps • Collect correspondences (manually)
• Solve for homography matrix H
– Least squares solution
• Warp content from one image frame to the other to combine: say im1 into im2 reference frame
– Determine bounds of the new combined image
• Where will the corners of im1 fall in im2’s coordinate frame?
• We will attempt to lookup colors for any of these positions we can get from im1. : meshgrid
– Compute coordinates in im1’s reference frame (via homography) for all points in that range: H-1
– Lookup all colors for all these positions from im1
• Inverse warp : interp2 (watch for nans : isnan)
• Overlay im2 content onto the warped im1 content.
– Careful about new bounds of the output image: minx, miny
![Page 25: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/25.jpg)
im2 im1
![Page 26: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/26.jpg)
Mosaics: main steps • Collect correspondences (manually)
• Solve for homography matrix H
– Least squares solution
• Warp content from one image frame to the other to combine: say im1 into im2 reference frame
– Determine bounds of the new combined image
• Where will the corners of im1 fall in im2’s coordinate frame?
• We will attempt to lookup colors for any of these positions we can get from im1. : meshgrid
– Compute coordinates in im1’s reference frame (via homography) for all points in that range: H-1
– Lookup all colors for all these positions from im1
• Inverse warp : interp2 (watch for nans : isnan)
• Overlay im2 content onto the warped im1 content.
– Careful about new bounds of the output image: minx, miny
![Page 27: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/27.jpg)
im2 im1
![Page 28: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/28.jpg)
Mosaics: main steps • Collect correspondences (manually)
• Solve for homography matrix H
– Least squares solution
• Warp content from one image frame to the other to combine: say im1 into im2 reference frame
– Determine bounds of the new combined image
• Where will the corners of im1 fall in im2’s coordinate frame?
• We will attempt to lookup colors for any of these positions we can get from im1. : meshgrid
– Compute coordinates in im1’s reference frame (via homography) for all points in that range: H-1
– Lookup all colors for all these positions from im1
• Inverse warp : interp2 (watch for nans : isnan)
• Overlay im2 content onto the warped im1 content.
– Careful about new bounds of the output image: minx, miny
![Page 29: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/29.jpg)
im1 warped into reference frame of im2. im2
im1
Use interp2 to ask for the colors (possibly interpolated) from im1 at all the positions needed in im2’s reference frame.
![Page 30: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/30.jpg)
Mosaics: main steps • Collect correspondences (manually)
• Solve for homography matrix H
– Least squares solution
• Warp content from one image frame to the other to combine: say im1 into im2 reference frame
– Determine bounds of the new combined image
• Where will the corners of im1 fall in im2’s coordinate frame?
• We will attempt to lookup colors for any of these positions we can get from im1. : meshgrid
– Compute coordinates in im1’s reference frame (via homography) for all points in that range: H-1
– Lookup all colors for all these positions from im1
• Inverse warp : interp2 (watch for nans : isnan)
• Overlay im2 content onto the warped im1 content.
– Careful about new bounds of the output image: minx, miny
![Page 31: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/31.jpg)
![Page 32: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/32.jpg)
Fun with homographies
St.Petersburg
photo by A. Tikhonov
Virtual camera rotations
Original image
![Page 33: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/33.jpg)
Analyzing patterns and shapes
Automatically
rectified floor
The floor (enlarged)
What is the shape of the b/w floor pattern?
Slide from Criminisi
![Page 34: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/34.jpg)
From Martin Kemp The Science of Art
(manual reconstruction)
Au
tom
ati
c r
ec
tifi
ca
tio
n
Analyzing patterns and shapes
2 patterns have been discovered !
Slide from Criminisi
![Page 35: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/35.jpg)
Automatically rectified floor
St. Lucy Altarpiece, D. Veneziano
Analyzing patterns and shapes
What is the (complicated)
shape of the floor pattern?
Slide from Criminisi
![Page 36: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/36.jpg)
From Martin Kemp, The Science of Art
(manual reconstruction)
Automatic
rectification
Analyzing patterns and shapes
Slide from Criminisi
![Page 37: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/37.jpg)
Analyzing patterns and shapes
The Ambassadors by Hans Holbein the Younger, 1533
![Page 38: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/38.jpg)
Julian Beever: Manual Homographies
http://users.skynet.be/J.Beever/pave.htm
![Page 39: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/39.jpg)
changing camera center
Does it still work? synthetic PP
PP1
PP2
![Page 40: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/40.jpg)
Planar scene (or far away)
PP3 is a projection plane of both centers of projection,
so we are OK!
This is how big aerial photographs are made
PP1
PP3
PP2
![Page 41: Homographies and Panoramas - Brown University](https://reader030.vdocuments.site/reader030/viewer/2022012720/61b286ef473d0675f86ee874/html5/thumbnails/41.jpg)
Planar mosaic