image-based renderingdjacobs/cmsc427/imagebasedrendering.pdf · sample the set of light rays in the...
TRANSCRIPT
![Page 1: Image-Based Renderingdjacobs/CMSC427/ImageBasedRendering.pdf · Sample the set of light rays in the world. Then generate an image by selecting the right rays. Mosaicing: simpler,](https://reader034.vdocuments.site/reader034/viewer/2022050409/5f85ccd0f1c55038000aed8c/html5/thumbnails/1.jpg)
1
Image-Based Rendering
Produce a new image from real images.Combining imagesInterpolationMore exotic methods
Why Image-Based Rendering?
What’s the most realistic image? A photograph.But photographs lack flexibility.
Can’t change viewpoint.Can’t change lighting.
![Page 2: Image-Based Renderingdjacobs/CMSC427/ImageBasedRendering.pdf · Sample the set of light rays in the world. Then generate an image by selecting the right rays. Mosaicing: simpler,](https://reader034.vdocuments.site/reader034/viewer/2022050409/5f85ccd0f1c55038000aed8c/html5/thumbnails/2.jpg)
2
The need for correspondence
Image-based rendering is mostly combining images to get a new image.Correspondences needed to sensibly combine images.
If viewpoint has changed this can be hard.If not, it’s trivial.
How to get correspondences
By hand:Works if few correspondences needed
By matching intensitiesThis is really ~ ½ of computer vision.
![Page 3: Image-Based Renderingdjacobs/CMSC427/ImageBasedRendering.pdf · Sample the set of light rays in the world. Then generate an image by selecting the right rays. Mosaicing: simpler,](https://reader034.vdocuments.site/reader034/viewer/2022050409/5f85ccd0f1c55038000aed8c/html5/thumbnails/3.jpg)
3
Matching
Simplest: SSD with windows.Windows needed because pixels not informative enough.
Compare windows ==??
ff gg
• Search for windows that match well
Morphing
What happens if you interpolate images?Need corresponding points.
![Page 4: Image-Based Renderingdjacobs/CMSC427/ImageBasedRendering.pdf · Sample the set of light rays in the world. Then generate an image by selecting the right rays. Mosaicing: simpler,](https://reader034.vdocuments.site/reader034/viewer/2022050409/5f85ccd0f1c55038000aed8c/html5/thumbnails/4.jpg)
4
Morphing
Corresponding points needed.Often done by hand.
Interpolate each point.Positionand intensity.
Also use interpolation for more correspondences.
Linear Interpolationof Position
![Page 5: Image-Based Renderingdjacobs/CMSC427/ImageBasedRendering.pdf · Sample the set of light rays in the world. Then generate an image by selecting the right rays. Mosaicing: simpler,](https://reader034.vdocuments.site/reader034/viewer/2022050409/5f85ccd0f1c55038000aed8c/html5/thumbnails/5.jpg)
5
Other Interpolation
Also interpolate intensities.Interpolate to find other point correspondences.
Light-Field Rendering
Sample the set of light rays in the world.Then generate an image by selecting the right rays.Mosaicing: simpler, just sample rays through one focal point.If one has all rays then camera can also move.
![Page 6: Image-Based Renderingdjacobs/CMSC427/ImageBasedRendering.pdf · Sample the set of light rays in the world. Then generate an image by selecting the right rays. Mosaicing: simpler,](https://reader034.vdocuments.site/reader034/viewer/2022050409/5f85ccd0f1c55038000aed8c/html5/thumbnails/6.jpg)
6
Mosaics
Take multiple images and construct one big image.
Represented as image, cylinder or sphere.
Allows panning and zooming.Simplest kind of motion.
•Fixed focal point.Correspondence needed to align images.
•Image rectification
![Page 7: Image-Based Renderingdjacobs/CMSC427/ImageBasedRendering.pdf · Sample the set of light rays in the world. Then generate an image by selecting the right rays. Mosaicing: simpler,](https://reader034.vdocuments.site/reader034/viewer/2022050409/5f85ccd0f1c55038000aed8c/html5/thumbnails/7.jpg)
7
![Page 8: Image-Based Renderingdjacobs/CMSC427/ImageBasedRendering.pdf · Sample the set of light rays in the world. Then generate an image by selecting the right rays. Mosaicing: simpler,](https://reader034.vdocuments.site/reader034/viewer/2022050409/5f85ccd0f1c55038000aed8c/html5/thumbnails/8.jpg)
8
(slide from Mark Ollila)
![Page 9: Image-Based Renderingdjacobs/CMSC427/ImageBasedRendering.pdf · Sample the set of light rays in the world. Then generate an image by selecting the right rays. Mosaicing: simpler,](https://reader034.vdocuments.site/reader034/viewer/2022050409/5f85ccd0f1c55038000aed8c/html5/thumbnails/9.jpg)
9
Other mosaicing issues
Pixel interpolation needed.Mosaicing can provide more information at each pixel.
Higher resolution images possible.Higher dynamic range.
Interpolating Views
![Page 10: Image-Based Renderingdjacobs/CMSC427/ImageBasedRendering.pdf · Sample the set of light rays in the world. Then generate an image by selecting the right rays. Mosaicing: simpler,](https://reader034.vdocuments.site/reader034/viewer/2022050409/5f85ccd0f1c55038000aed8c/html5/thumbnails/10.jpg)
10
Light Field Rendering (Levoy and Hanrahan; paper and slides)
Light Field Rendering (Levoy and Hanrahan; paper and slides)
![Page 11: Image-Based Renderingdjacobs/CMSC427/ImageBasedRendering.pdf · Sample the set of light rays in the world. Then generate an image by selecting the right rays. Mosaicing: simpler,](https://reader034.vdocuments.site/reader034/viewer/2022050409/5f85ccd0f1c55038000aed8c/html5/thumbnails/11.jpg)
11
Light Field Rendering (Levoy and Hanrahan; paper and slides)
Light Field Rendering (Levoy and Hanrahan; paper and slides)
![Page 12: Image-Based Renderingdjacobs/CMSC427/ImageBasedRendering.pdf · Sample the set of light rays in the world. Then generate an image by selecting the right rays. Mosaicing: simpler,](https://reader034.vdocuments.site/reader034/viewer/2022050409/5f85ccd0f1c55038000aed8c/html5/thumbnails/12.jpg)
12
Light Field Rendering (Levoy and Hanrahan; paper and slides)
Interpolation
We have possibly non-uniform samples of a 4D space. Must interpolate to fill in.Worry about aliasing
![Page 13: Image-Based Renderingdjacobs/CMSC427/ImageBasedRendering.pdf · Sample the set of light rays in the world. Then generate an image by selecting the right rays. Mosaicing: simpler,](https://reader034.vdocuments.site/reader034/viewer/2022050409/5f85ccd0f1c55038000aed8c/html5/thumbnails/13.jpg)
13
Light Field Rendering (Levoy and Hanrahan; paper and slides)
Using Depth in IBR
Example, generating views for teleconferencing, with depth from stereo.
![Page 14: Image-Based Renderingdjacobs/CMSC427/ImageBasedRendering.pdf · Sample the set of light rays in the world. Then generate an image by selecting the right rays. Mosaicing: simpler,](https://reader034.vdocuments.site/reader034/viewer/2022050409/5f85ccd0f1c55038000aed8c/html5/thumbnails/14.jpg)
14
Generating using two views in which speaker isn’t looking at camera.
Segmentation and compositing
![Page 15: Image-Based Renderingdjacobs/CMSC427/ImageBasedRendering.pdf · Sample the set of light rays in the world. Then generate an image by selecting the right rays. Mosaicing: simpler,](https://reader034.vdocuments.site/reader034/viewer/2022050409/5f85ccd0f1c55038000aed8c/html5/thumbnails/15.jpg)
15
Linear basis for lighting
�Z
�Y
�X
Surface normal = (X,Y,Z) albedo = �
Directional source = (Lx,Ly,Lz)
I = �(Lx,Ly,Lz)(X,Y,Z) = Lx*
�X + Ly*
�Y + Lz*
�Z
Take Max of this and 0
Using Linear Basis for Rendering
Render three imagesTake linear combinations.Why can’t we do this with three real images?
![Page 16: Image-Based Renderingdjacobs/CMSC427/ImageBasedRendering.pdf · Sample the set of light rays in the world. Then generate an image by selecting the right rays. Mosaicing: simpler,](https://reader034.vdocuments.site/reader034/viewer/2022050409/5f85ccd0f1c55038000aed8c/html5/thumbnails/16.jpg)
16
0 1 2 30
0.5
1
0 1 2 30
0.5
1
1.5
2
�
r
Reflectance smooths lighting
Basis from diffuse lighting
� �Z
�Y
�X
2 2 22!(Z -X -Y ) �XZ
�YZ
2 2!(X -Y ) �XY
![Page 17: Image-Based Renderingdjacobs/CMSC427/ImageBasedRendering.pdf · Sample the set of light rays in the world. Then generate an image by selecting the right rays. Mosaicing: simpler,](https://reader034.vdocuments.site/reader034/viewer/2022050409/5f85ccd0f1c55038000aed8c/html5/thumbnails/17.jpg)
17
Note, this can also be done with 9 real images, because this is a basis that contains real images
In 3D, real images aren’t in the 3D space, we have to take the max with 0 to get real images.
Non-Photorealistic Rendering
Take a photo and turn it into a different kind of image.
![Page 18: Image-Based Renderingdjacobs/CMSC427/ImageBasedRendering.pdf · Sample the set of light rays in the world. Then generate an image by selecting the right rays. Mosaicing: simpler,](https://reader034.vdocuments.site/reader034/viewer/2022050409/5f85ccd0f1c55038000aed8c/html5/thumbnails/18.jpg)
18
De Carlo and Santella
Video
Image Analogies
Given A, A’ and B, generate B’
A bit like Efros and Leung texture synthesis.