illumination capture and rendering for augmented reality · illumination capture and rendering for...
TRANSCRIPT
![Page 1: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/1.jpg)
1
Illumination Capture and Rendering for Augmented Reality
RFIA 2004Toulouse, France, January 27th 2004.
University of Manchester, UK
Simon Gibson
![Page 2: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/2.jpg)
2
Augmented Reality
• The process of augmenting a view of a real scene with additional objects or information
• Some applications require visually realistic synthetic objects– Interior design– Broadcast– Computer Games/Entertainment– …
![Page 3: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/3.jpg)
3
Visual realism
• Geometric– Register object correctly with the
camera– Resolve occlusions between real and
virtual objects• Radiometric
– Shade the object consistently with the rest of the scene
– Cast shadows between real and virtual objects
![Page 4: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/4.jpg)
4
Tutorial overview
• Modelling and rendering for Augmented Reality• Geometric camera calibration• Modelling scene geometry• HDR Illumination reconstruction• Rendering and compositing
– Shading and shadows
• Examples and Videos
![Page 5: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/5.jpg)
5
Modelling and rendering for Augmented Reality
• Basic requirements to augment images– Camera position for current frame– 3D scene model to resolve occlusions and collisions– HDR representation of the scene lighting – Radiometric camera response
![Page 6: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/6.jpg)
6
• Calibrate the camera position and model scene geometry using interactive techniques
• Capture illumination using a light-probe and HDR imaging, storing data with scene model
Modelling and rendering for Augmented Reality
![Page 7: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/7.jpg)
7
• Rendering objects using OpenGL– Clear colour and depth buffers– Draw background image into colour
buffer– Set OpenGL camera state to match
calibrated camera position– Draw 3D scene model into depth
buffer– Draw virtual objects into colour and
depth buffers
Modelling and rendering for Augmented Reality
![Page 8: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/8.jpg)
8
• “Illumination mesh” used to shade virtual objects– Diffuse & specular reflection– Multi-pass rendering– GPU shadow mapping
• Virtual objects rendered at interactive rates (5-15fps)
Modelling and rendering for Augmented Reality
![Page 9: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/9.jpg)
9
Tutorial overview
• Modelling and rendering for Augmented Reality• Geometric camera calibration• Modelling scene geometry• HDR Illumination reconstruction• Rendering and compositing
– Shading and shadows
• Examples and Videos
![Page 10: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/10.jpg)
10
• To register objects correctly with the real camera, we need to know the camera projection matrix, P:
P = K.[R|t]where
R = camera rotation (3x3 matrix)t = camera translation (3x1 vector)K = camera calibration parameters (3x3 matrix)
Geometric camera calibration
![Page 11: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/11.jpg)
11
Geometric camera calibration
• Moving camera– Must be tracked for each frame– Intrinsic parameters can generally be calculated off-line– Extrinsic parameters updated at each frame using marker-based
or marker-less tracking algorithms
• Static camera– Can be calibrated off-line– 2 or more orthogonal “vanishing points” provide enough
information to determine both intrinsic and extrinsic parameters
![Page 12: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/12.jpg)
12
Static camera calibration
User draws two or more orthogonal vanishing points in the image
![Page 13: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/13.jpg)
13
Tutorial overview
• Modelling and rendering for Augmented Reality• Geometric camera calibration• Modelling scene geometry• HDR Illumination reconstruction• Rendering and compositing
– Shading and shadows
• Examples and Videos
![Page 14: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/14.jpg)
14
Reconstructing scene geometry
• 3D geometric model needed for:– Resolving occlusions between real and virtual objects– Collision detection with background scene– Constructing an illumination environment to shade objects– “Catching” shadows cast by virtual objects
![Page 15: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/15.jpg)
15
Reconstructing scene geometry
• Many different approaches…• Automatic
– Dense stereo matching (e.g. Pollefeys ‘01)– Laser scanning (e.g. Levoy ‘00)– Structured light projection (e.g. Scharstein ‘03)
• Semi-automatic– Primitive-based modelling (e.g. Debevec ‘96, Cipolla ‘98, El
Hakim ‘00, Gibson ‘03)
![Page 16: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/16.jpg)
16
Automatic scene reconstruction2D feature
2D feature
Dense stereo matching requires multiple images and scene texture
![Page 17: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/17.jpg)
17
Automatic vs. semi-automatic reconstruction
Automatic
Accurate geometryRelatively fast and easy-to-useUnstructured geometryHole and occlusion problemsRequires scene “texture”Requires more than one image
Semi-automatic
Less accuracy & simple shapesMore labour-intensiveGood scene structureCan apply “user-knowledge”Works without scene “texture”Works with a single image
![Page 18: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/18.jpg)
18
Automatic
Accurate geometryRelatively fast and easy-to-useUnstructured geometryHole and occlusion problemsRequires scene “texture”Requires more than one image
Semi-automatic
Less accuracy & simple shapesMore labour-intensiveGood scene structureCan apply “user-knowledge”Works without scene “texture”Works with a single image
For Augmented Reality, we don’t need a “pixel perfect” scene model!
Automatic vs. semi-automatic reconstruction
![Page 19: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/19.jpg)
19
Semi-automatic geometry reconstruction
• Manipulate geometric primitives to match features in the calibrated background image– Use a library of pre-defined geometric shapes– User-specified constraints assist in the modelling process– Non-linear optimisation algorithm adjusts primitives so constraints are
satisfied
(see Gibson et al. Computers and Graphics, April 2003)
![Page 20: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/20.jpg)
20
Semi-automatic geometry reconstruction
![Page 21: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/21.jpg)
21
Semi-automatic geometry reconstruction
~20 mins construction time
![Page 22: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/22.jpg)
22
Example scene reconstruction
![Page 23: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/23.jpg)
23
Tutorial overview
• Modelling and rendering for Augmented Reality• Geometric camera calibration• Modelling scene geometry• HDR Illumination reconstruction• Rendering and compositing
– Shading and shadows
• Examples and Videos
![Page 24: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/24.jpg)
24
Illumination reconstruction
• In order to illuminate synthetic objects, we must build a representation of the real light in the scene
• Capture an image-based representation of light using a “light-probe” and calibration grid
![Page 25: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/25.jpg)
25
High dynamic range imaging
• Standard cameras are not able to capture the full dynamic range of light in a single image
Bright areas are washed-out
Loss of detail in dark areas
![Page 26: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/26.jpg)
26
High dynamic range imaging
• By combining multiple images at different exposure times, all detail can be recovered, as well as the camera response function
Short exposure times give detail in bright areas
Long exposure times give detail in dark areas
![Page 27: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/27.jpg)
27
Recovering camera response
See Debevec and Malik, SIGGRAPH 1997 for further detailsSee Debevec and Malik, SIGGRAPH 1997 for further details
![Page 28: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/28.jpg)
28
Calibrating the probe image
• Known spot locations can be used to find the camera projection matrix.
• Known light-probe dimensions used to locate probe area in HDR image• Probe always positioned over most
distant spot• Probe size and mount height
known
![Page 29: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/29.jpg)
29
Calibrating the probe image
• Light-probe is not a perfect reflector
• Can approximate light-probe reflectivity using pixel ratios in HDR image
(4.91, 5.01, 5.11)
(9.17,9.97,10.67)
Reflectivity ≈~ (0.53,0.50,0.48)
![Page 30: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/30.jpg)
30
Orienting the light probe within the scene
• In order to map radiance onto scene, light-probe must be correctly positioned and oriented within the scene model
• Reconstruct geometric representation of calibration grid, and use to position and orient light-probe coordinatesystem
![Page 31: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/31.jpg)
31
Orienting the light probe within the scene
![Page 32: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/32.jpg)
32
Building the illuminated scene model
• Radiance values projected outwards from the light-probe location onto the scene model
• Lighting information stored in a triangular “illumination mesh”
• Radiosity transfer used to estimate missing values
![Page 33: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/33.jpg)
33
Building the illuminated scene model
• Diffuse reflectance for each visible patch estimated using the ratio of radiance to irradiance
reflectance ≈ Li / Ei
• Radiance obtained from HDR image
• Irradiance gathered from environment using geometric form-factors
Ei
Li
i
![Page 34: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/34.jpg)
34
Tutorial overview
• Modelling and rendering for Augmented Reality• Geometric camera calibration• Modelling scene geometry• HDR Illumination reconstruction• Rendering and compositing
– Shading and shadows• Examples and Videos
![Page 35: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/35.jpg)
35
Illuminating synthetic objects
• Traditionally done using raytracing (e.g. Debevec ’98)High accuracySupports a wide range of surface reflectance functionsNot interactive
• Recent advances in GPU functionality allow interactive rendering with image-based lighting (e.g. Sloan ’03)
Fast renderingSupports a wide range of surface reflectance functionsOnly supports distant illumination
![Page 36: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/36.jpg)
36
Illuminating synthetic objects
• Render diffuse and specular reflectance separately• Diffuse shading uses an Irradiance Volume (Greger ’98)• Specular reflections approximated using dynamically
generated environment maps
Fast and easy to useSupports non-distant illuminationLimited support for complex reflectance functions
![Page 37: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/37.jpg)
37
Irradiance volume• Volumetric representation of
irradiance at every point and direction in space
• Irradiance sampled at every grid vertex and store in the volume
• Spherical Harmonics used to reduce storage and computation requirements
• Vertex shaded by querying irradiance using table lookups and interpolation
See Greger et al, IEEE CG&A 1998, and Ramamoorthi and Hanrahan, SIGGRAPH 2001See Greger et al, IEEE CG&A 1998, and Ramamoorthi and Hanrahan, SIGGRAPH 2001
![Page 38: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/38.jpg)
38
Querying the irradiance volume
• To shade each object vertex, we query the irradiancevolume using vertex position and normal direction
• Irradiance at voxel corners evaluated by summing spherical harmonic coefficients
• Irradiance evaluated at vertex position using tri-linear interpolation Irradiance voxel
Vertex position
N
![Page 39: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/39.jpg)
39
Diffuse shading
• Irradiance at each vertex is multiplied by the object’s diffuse reflectance
• Radiosity is tone-mapped for display using the camera response function
• Object is drawn using tone-mapped vertex colours, and GPU is used to interpolate shading across each triangle
![Page 40: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/40.jpg)
40
Example of diffuse shading
• Can shade approximately 350K vertices per second on a 2.5GHz Pentium4
• Equivalent to 11K vertices at 30 frames-per-second
![Page 41: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/41.jpg)
41
Specular reflections
• Specular reflections generated using dynamically generated cubic environment maps
• We assume that the diffuse and specular components can be tone-mapped independently and combined:
T(D+S) ≈ T(D) + T(S)
![Page 42: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/42.jpg)
42
Specular reflections
• Multiply 3D scene radiance by object’s specular reflection coefficient
• Tone-map scene using table lookup into camera response function
• Render cubic environment maps• Render virtual object again, with
environment mapping enabled
![Page 43: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/43.jpg)
43
Specular reflections
• Diffuse and specular components combined using OpenGL additive blending
• We assume:T(D+S) ≈ T(D) + T(S)
![Page 44: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/44.jpg)
44
Rendering shadows
• We must account for four types of shadow:
– Virtual-to-real (shadow cast by synthetic object)– Virtual-to-virtual (self-shadowing on synthetic object)
– Real-to-virtual (shadow cast onto synthetic object)– Real-to-real (existing shadow in the real scene)
![Page 45: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/45.jpg)
45
Rendering shadows
• We must account for four types of shadow:
– Virtual-to-real (shadow cast by synthetic object)– Virtual-to-virtual (self-shadowing on synthetic object)
– Real-to-virtual (shadow cast onto synthetic object)– Real-to-real (existing shadow in the real scene)
Irradiance volume
Background image
![Page 46: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/46.jpg)
46
Overview of shadow rendering
1. Build a shaft hierarchy to encode light transport between all patches in the scene model
2. Identify the significant sources of shadow for each virtual object3. Generate a shadow map from each shadow source4. Remove contributions of light emitted by each source from the
background imageRepeat for every frame
![Page 47: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/47.jpg)
47
Related work
• Soft-shadow approximated using multiple overlapping hard shadows– Brotman and Badler `84, Herf and Heckbert `97
• Shadow generation for Augmented Reality– Fournier et al. `93, Drettakis et al. `97, Loscos et al. `99
• Line-space hierarchy to encode light transport– Drettakis and Sillion `97
• HDR Image-based lighting and differential rendering– Debevec `98, Sato `99
![Page 48: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/48.jpg)
48
Shaft hierarchy
• Shaft hierarchy stores a coarse representation of the light transport from groups of source patches to groups of receiver patches
• Assume source and receiver patches emit and reflect light diffusely
• Radiance transfer is pre-computed between each source and receiver pair
Source Hierarchy
Source Patches
Receiver Patches
Shaft
Receiver Hierarchy
…………Shaft
![Page 49: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/49.jpg)
49
Radiance transfer pre-computation
• Calculate “form-factor” between each source patch and each receiver
• Visibility evaluated using ray-casting
• Radiance transfer summed and stored within shaft hierarchy
![Page 50: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/50.jpg)
50
Shadow source identification• Given the bounding box of an object, a list of potentially occluded shafts
can be found very quickly before rendering each frame
• Define source set:– All source patches associated
with all potentially occluded shafts
• Define receiver set:– All receiver patches associated
with all potentially occluded shafts
![Page 51: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/51.jpg)
51
Differential rendering
• Given a rendered image, Iobj, of scene and virtual objects and a second image Inoobj without virtual objects, subtract their difference, Ie from the background photograph:
Ifinal = Ib – Ie = Ib – (Inoobj – Iobj)
-+ =
![Page 52: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/52.jpg)
52
Shadow rendering• Assuming we can operate entirely with HDR frame buffer data,
repeat for each patch in the source set:1. Generate a shadow map that encloses
the synthetic object2. Enable OpenGL subtractive blending
and initialise projection/modelviewmatrices to the calibrated camera position
3. Render receiver set with shadow mapping enabled, with vertex colours set to the pre-computed radiance transfer from the source patch
![Page 53: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/53.jpg)
53
HDR image operations• Ideally, we want to perform all operations on HDR image
data and tone-map afterwards
Ifinal = T(Lb – ΣLjMj)
– Lb is the HDR background image– Lj is the radiance transfer from source patch j– Mj is the shadow mask (1 occluded, 0 unoccluded)– T() is the camera response function
![Page 54: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/54.jpg)
54
Mapping to LDR image data
• If T() was linear, we could separate the terms:
Ifinal = T(Lb) – T(ΣLjMj)
= Ib - Σ Mj T(Lj)
![Page 55: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/55.jpg)
55
Mapping to LDR image data
• If T() was linear, we could separate the terms:
Ifinal = T(Lb) – T(ΣLjMj)
= Ib - Σ Mj T(Lj)
• Camera response is non-linear, so T(A-B) ≠ T(A)– T(B)
Ifinal = Ib - Σ Sj Mj
Sj is LDR “equivalent” of Lj
![Page 56: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/56.jpg)
56
Mapping to LDR image data
• Non-linearity of T() means that we can only approximate Sj– Dependent on order of evaluation of source patches– Requires knowing all Mj before evaluation
![Page 57: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/57.jpg)
57
Mapping to LDR image data
• Non-linearity of T() means that we can only approximate Sj– Dependent on order of evaluation of source patches– Requires knowing all Mj before evaluation
• Fix order of source patch evaluation (brightest to darkest)• Assign approximate Mj (e.g. 50% occluded, 50%
unoccluded)
(see Gibson et al. Eurogaphics Rendering Symposium, 2003)
![Page 58: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/58.jpg)
58
Examples of shadow rendering
![Page 59: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/59.jpg)
59
Examples of shadow rendering
![Page 60: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/60.jpg)
60
Self-shadows
• Virtual objects cast shadows onto themselves!• Standard shadow-mapping can’t be used, because we can’t pre-
compute radiance transfer to the virtual object• Pre-compute an approximate per-vertex visibility term to account for
self-occlusions
** ==
![Page 61: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/61.jpg)
61
Ambient Occlusion
• Commonly used in film post-production to approximate global-illumination effects
• Assumes that object is illuminated by a uniform sphere of light
See: H. Landis, “Production Ready Global Illumination”, SIGGRAPH 2002 Course NotesSee: H. Landis, “Production Ready Global Illumination”, SIGGRAPH 2002 Course Notes
Ambient occlusion= 4/6
![Page 62: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/62.jpg)
62
![Page 63: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/63.jpg)
63
Visual comparison (1)
14 frames-per-second Ray-traced Photograph
Interior, natural illumination
![Page 64: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/64.jpg)
64
Visual comparison (2)
12 frames-per-second Ray-traced Photograph
Interior, artificial illumination
![Page 65: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/65.jpg)
65
Visual comparison (3)
16 frames-per-second Ray-traced Photograph
Exterior, natural illumination
![Page 66: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/66.jpg)
66
Visual Comparison (4): Self-Shadows
No self-shadows (12fps)No self-shadows (12fps) With self-shadows (12fps)With self-shadows (12fps) Ray-tracedRay-traced
![Page 67: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/67.jpg)
67
Summary
• Illumination capture achieved using HDR imaging and a light-probe
• Interactive rendering achieved using a combination of software and hardware techniques– Diffuse and specular shading– Soft shadows
• Rendering rates of between 5 and 15 frames-per-second on a GeForce4 Ti4600
![Page 68: Illumination Capture and Rendering for Augmented Reality · Illumination Capture and Rendering for Augmented Reality RFIA 2004 Toulouse, France, January 27th 2004. University of Manchester,](https://reader030.vdocuments.site/reader030/viewer/2022040806/5e463354f9dda01c483a4fee/html5/thumbnails/68.jpg)
68
Thank You!
Questions?