image-based rendering. 3d scene = shape + shading source: leonard mcmillan, unc-ch
TRANSCRIPT
![Page 1: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/1.jpg)
Image-Based Rendering
![Page 2: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/2.jpg)
3D Scene = Shape + Shading
Source: Leonard mcMillan, UNC-CH
![Page 3: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/3.jpg)
Modeling is Hard!
• Good 3D geometric models need a lot of work.
• Getting correct material properties is even harder:– R, G, B, Ka, Kd, Ks, specularity for Phong
model.– BRDF– Subsurface reflectance, diffraction…etc.
![Page 4: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/4.jpg)
How to Create 3D Contents
• AutoCAD: used for architectures (buildings)
• 3D Studio Max, Blender…etc.
• Maya is a major production tool used in movie studios.
• Problems? It takes an artist, and it’s still hard to make it look real!
![Page 5: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/5.jpg)
QuickTime VR
![Page 6: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/6.jpg)
3D Photography
• Can building 3D models be as easy as taking 2D photos?
• How do we digitize the massive assets in various museums?– QuickTime VR object movies– 3D Scans: Cyberware
scanner, Digital MichelangeoSource: www.cyberware.com
![Page 7: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/7.jpg)
Image-Based Rendering
• Can we build 3D contents from photographs directly?– Difference from computer vision?
• Can we make the objects look more real?– Difference from texture mapping?
![Page 8: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/8.jpg)
Top Level Survey
3D Graphics
Image-Based Rendering& Modeling
Geometry orSurface BasedRendering & Modeling
Sample-BasedGraphics
VolumeRendering
![Page 9: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/9.jpg)
Traditional Computer Graphics
• Input: Geometry, Material Properties (Color, Reflectance,…etc.), Lighting.
• Transformation and Rasterization.Transform(& Lighting)
Rasterization3D Graphics Pipeline
![Page 10: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/10.jpg)
Role of Images• Used as textures.
• Or, as input to computer vision methods in order to recover the 3D models.
3D Graphics Pipeline
Vision
![Page 11: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/11.jpg)
Image-Based Rendering
• To bypass the 3D models altogether.
3D Graphics Pipeline
VisionImage-Based Rendering
![Page 12: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/12.jpg)
Image-Based Rendering• Input: Regular Images or “Depth
Images."
• No 3D model is constructed.
• Example: 3D Warping.
![Page 13: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/13.jpg)
3D Warping: Another Example
• Reading room of UNC CS department– Source images contain depths in each
pixel. – The depths are obtained from a laser
range finder.
![Page 14: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/14.jpg)
Why IBR?
• Problems of triangle-based graphics:– Always starts from scratch.– Millions of sub-pixel triangles.
Geometry IBR
Modeling Difficult Easy
Complexity #triangles #pixels
Fidelity Synthetic Acquired
![Page 15: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/15.jpg)
Why is It Possible?
• 5D Plenoptic Function.– Color = f(x, y, z, , )– (x, y, z) defines the viewpoint.– (, ) defines the view
direction.
• 4D Light Field/Lumigraph– Color = f(u, v, s, t)– (u, v) defines the viewpoint.– (s, t) defines the pixel coord.
Picture source: Leonard McMillan
![Page 16: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/16.jpg)
– Each pixel in the source images has coordinates (u1, v1), depth info 1, and color.
– Warping Equation is applied to each pixel(u2, v2) = f(u1, v1, 1)
= ( au1+bv1+c+d1 , eu1+fv1+g+h1 ) iu1+jv1+k+l1 iu1+jv1+k+l1
where variables a to l are fixed for the same view.
– Rendering Time = O(#pixels)
3D Image Warping
![Page 17: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/17.jpg)
![Page 18: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/18.jpg)
u1
v1
u2
v2
![Page 19: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/19.jpg)
Artifacts of 3D Image Warping
• Surfaces that were occluded in source images.
• Non-uniform sampling (an example in the next slide).
![Page 20: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/20.jpg)
Reconstruction
![Page 21: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/21.jpg)
Using Multiple Source Images
![Page 22: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/22.jpg)
IBR Survey
Image-Based Rendering& Modeling
Light Field Rendering(or Lumigraph)Stanford/Microsoft
3D Warping(or Plenoptic Modeling)UNC-Chapel Hill
![Page 23: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/23.jpg)
Light Field & Lumigraph
![Page 24: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/24.jpg)
Images as 4D Samples
• Consider each image pixel a sample of 4D Light Field.
![Page 25: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/25.jpg)
Does it Matter Where We Place the Planes?
• Yes!
• Depth correction in Lumigraphs:
![Page 26: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/26.jpg)
Concentric Mosaic
• Hold a camera on a stick, then sweep a circle.
• Viewpoint is constrained on a 2D plane.
• Reducing the 4D light field to a 3D subspace.
![Page 27: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/27.jpg)
Surface Light Field
• May be considered a compression scheme for light field data.
• 3D geometry required.
![Page 28: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/28.jpg)
LYTRO Light Field Camera• See https://www.lytro.com/science_inside
![Page 29: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/29.jpg)
Further Reading on Light Field
• See Marc Levoy’s IEEE Computer 2006 article at: http://graphics.stanford.edu/papers/lfphoto/levoy-lfphoto-ieee06.pdf
![Page 30: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/30.jpg)
Camera Array
Source: http://sabia.tic.udc.es/gc/Contenidos%20adicionales/trabajos/Peliculas/FX/ej3.html
![Page 31: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/31.jpg)
Image-Based Lighting -- Light Stage
• Take photos under single point light at various positions.
• Trivial questions: how to produce new images at:– Two point lights?– Area light?– Environment light (captured by light
probe)?
![Page 32: Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH](https://reader036.vdocuments.site/reader036/viewer/2022062408/56649e575503460f94b50794/html5/thumbnails/32.jpg)