csce 641 computer graphics: image-based rendering (cont.) jinxiang chai

CSCE 641 Computer Graphics: Image-based Rendering (cont.)

Jinxiang Chai

Outline

Light field rendering

Plenoptic sampling (light field sampling)

Layered depth image/Post-Rendering 3D Warping

View-dependent texture mapping

Unstructured lumigraph

Layered depth image [Shade et al, SIGGRAPH98]

Layered depth image:

- image with depths

Layered depth image [Shade et al, SIGGRAPH98]

Layered depth image:

- rays with colors and depths

Layered depth image [Shade et al, SIGGRAPH98]

Layered depth image: (r,g,b,depth)

- image with depths

- rays with colors and depths

Layered depth image [Shade et al, SIGGRAPH98]

Rendering from layered depth image

Layered depth image [Shade et al, SIGGRAPH98]

Rendering from layered depth image

- Incremental in X and Y

- Forward warping one pixel with depth

Layered depth image [Shade et al, SIGGRAPH98]

Rendering from layered depth image

- Incremental in X and Y

- Forward warping one pixel with depth

Layered depth image [Shade et al, SIGGRAPH98]

Rendering from layered depth image

- Incremental in X and Y

- Forward warping one pixel with depth

How to deal with occlusion/visibility problem?

How to form LDIs

Synthetic world with known geometry and texture - from multiple depth images

- modified ray tracer

Real images - reconstruct geometry from multiple images (e.g., voxel coloring, stereo reconstruction) - form LDIs using multiple images and reconstructed geometry

Kinect sensors - record both image data and depth data

Image-based Rendering Using Kinect Sensors

Capture both video/depth data using kinect sensors

Using 3D warping to render a video from a novel view point [e.g., Post-Rendering 3D Warping]

Demo: click here

Outline

Light field rendering

Plenoptic sampling (light field sampling)

Layered depth image/Post-Rendering 3D Warping

View-dependent texture mapping

Unstructured lumigraph

View-dependent surface representation

From multiple input image

- reconstruct the geometry

- view-dependent texture

View-dependent surface representation

From multiple input image

- reconstruct the geometry

- view-dependent texture

View-dependent surface representation

From multiple input image

- reconstruct the geometry

- view-dependent texture

View-dependent surface representation

From multiple input image

- reconstruct the geometry

- view-dependent texture

View-dependent texture mapping

[Debevec et al 98]

View-dependent texture mapping

Subject's 3D proxy points

V

C0 C2 C3C1

0

1

D

2

3

- Virtual camera at point D

- Textures from camera Ci mapped onto triangle faces

- Blending weights in vertex V

- Angle θi is used to compute the weight values:

wi = exp(-θi2/2σ2)

Videos: view-dependent texture mapping

Outline

Light field rendering

Plenoptic sampling (light field sampling)

Layered depth image/Post-Rendering 3D Warping

View-dependent texture mapping

Unstructured lumigraph

The Image-Based Rendering Problem

Synthesize novel views from reference images• Static scenes, fixed lighting

• Flexible geometry and camera configurations

The ULR Algorithm [Siggraph01]

Designed to work over a range of image and geometry configurations

Geometric Fidelity

# o

f Im

ag

es

VDTMVDTM

LFLF

The ULR Algorithm [Siggraph01]

Designed to work over a range of image and geometry configurations

Geometric Fidelity

# o

f Im

ag

es

VDTMVDTM

LFLFULRULR

The ULR Algorithm [Siggraph01]

Designed to work over a range of image and geometry configurations

Designed to satisfy desirable properties

Geometric Fidelity

# o

f Im

ag

es

VDTMVDTM

LFLFULRULR

Desired Camera

“Light Field Rendering,” SIGGRAPH ‘96

u0

s0

u

s

Desired color interpolated from “nearest cameras”

Desired Camera

“Light Field Rendering,” SIGGRAPH ‘96

u

s

Desired Camera

“The Scene”

“The Lumigraph,” SIGGRAPH ‘96

u

PotentialArtifact

“The Scene”

“The Lumigraph,” SIGGRAPH ‘96

Desired Property #2: Use of geometric proxy

Desired Camera

“The Lumigraph,” SIGGRAPH ‘96

“The Scene”

Desired Camera

“The Lumigraph,” SIGGRAPH ‘96

“The Scene”

Rebinning

Note: all images are resampled.

Desired Camera

Desired Property #3: Unstructured input images

“The Lumigraph,” SIGGRAPH ‘96

“The Scene”

Desired Property #4: Real-time implementation

Desired Camera

Desired Camera

View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98

“The Scene”

Occluded

Out of view

Desired Camera

“The Scene”

Desired Property #5: Continuous reconstruction

View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98

Desired Camera

“The Scene”

θ1

θ2

θ3

View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98

Desired Camera

“The Scene”

θ1

θ2

θ3Desired Property #6: Angles measured w.r.t. proxy

View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98

“The Scene”

Desired Camera

Desired Camera

“The Scene”

Desired Property #7: Resolution sensitivity

Demo