Image-Based Rendering

• Geometry and light interaction may be difficult and expensive to model– Think of how hard radiosity is– Imagine the complexity of modeling the exact

geometry of carpet (as just one example)

• Image based rendering seeks to replace geometry and surface properties with images

Texture Mapping• Replace small scale geometry and surface changes with

an image attached to the surface• Define a mapping from surface points into a texture

image• When the properties of that point are required, look up

the pixel in the texture image• A research field in itself

– Various methods of sampling the texture– Various ways of combining the samples with direct lighting

calculations– Ways of arranging and accessing textures in memory

Environment Mapping

• Replace the world around an object with images• Generally used for fast reflection computations• Typically, define a cube around the object and

associate images with the inside of each face• When seeking reflection color, shot ray from surface

point onto cube and take corresponding image point• Images represent a view of the world from the center

of the object– Assumes that the world wouldn’t look much different if

viewed from points on the object’s surface

Plenoptic Function• Describes the intensity of light:

– passing through a given point, x– in a given direction, (,)– with given wavelength, – at a given time, t

• Many image-based rendering approaches can be cast as sampling from and reconstructing the plenoptic function

• Note, function is generally constant along segments of a line (assuming vacuum)

Image-Based Rendering

• Aim: Generate an image from a desired view using existing images from other views– May or may not know the viewing parameters for

the existing images– Existing images may be photographs or computer

generated renderings

A Plethora of Approaches

• Methods differ in many ways:– The range of new viewpoints allowed– The density of input images– The representation for samples (known images)– The amount of user help required– The amount of additional information required

(such as intrinsic camera parameters)– The method for gathering the input images

Movie-Map Approaches

• Film views from fixed locations, closely spaced, and store on video-disc (would be DVD now)

• Allow the user to jump from location to location, and pan

• Appropriate images are retrieved from disk and displayed

• No reprojection – just uses nearest existing sample• Still used in video games today, but with computer

generated movies

Quicktime VR (Chen, 1995)

• Movie-maps in software• Construct panoramic images by stitching

together a series of photographs– Semi automatic process, based on correlation

• Scale/shift images so that they look most alike

– Works best with >50% overlap

• Finite set of panoramas – user jumps from one to the other

View Interpolation (Chen and Williams, 1993)

• Input: A set of synthetic images with known depth and camera parameters (location, focal length, etc)

• Computes optical flow maps related each pair of images– Optical flow map is the set of vectors describing where

each point in the first image moves to in the second image

• Morphs between images by moving points along flow vectors

• Intermediate views are “real” views only in special cases

View Morphing (Seitz and Dyer, 1997)

• Uses interpolation to generate new views such that the intermediate views represent real camera motion

• Observation: Interpolation gives correct intermediate views if the initial and final images are parallel views

View Morphing Process• Basic algorithm:

– Pre-warp input images to get them into parallel views– Interpolate for intermediate view– Post-warp to get final result

• User specifies a camera path that rotates and translates initial camera onto final camera

• Requires knowledge of projection matrices for the input images– Found with vision algorithms. User may supply

correspondences

• Intermediate motion can be specified by giving trajectories of four points

Plenoptic Modeling (McMillan and Bishop, 1995)

• Input: Set of panoramic images gathered by panning a video camera about its vertical optical axis on a tripod

• The algorithm:– Determines the properties of each camera and registers

the images associated with each pan– Determines the relative locations of each camera– Determines the depth of each point seen by multiple

cameras– Generates new views by reconstructing the plenoptic

function from the available samples

Fitting Each Camera• Problem:

– Given several images all taken with the same camera rotated about its optical center

– Determine the camera parameters– Determine the angle between each image

• Approach:– Multi-stage optimization– First stage estimates angle between images and focal length– Second stage refines and gets remaining parameters

• When done, can map a pixel in one image to its correct position in any other images

Locating the Cameras• Given cylindrical projections (panoramic images)

for two cameras• User identifies 100-500 tie-points (points seen in

both images)• Each tie-point defines two rays – one from the

center of each camera through the tie-point– These rays should intersect at the world location for the

point

• Minimization step finds the camera locations and some other parameters that minimize the perpendicular distance between all the rays

Determining Disparity• The minimization algorithm gives us the world

location of the tie-points, but what about the rest of the points in the image?

• Use standard computer vision techniques to find the remaining disparities– Disparity is the angular difference between the

locations of a point in two images

– Directly related to the depth of the point

• Makes heavy use of the epipolar constraint

The Epipolar Constraint• The location of a point in one image constrains it

to lie on a ray in space passing through the camera and image point

• This ray projects to a curve in the second image– Line for planar projection– Sine curve for cylindrical projection

• Since the point must lie on the ray in world space, it must lie on the curve in the second image

• Reduces the search for correspondences to a linear one along the line

Reconstructing a New View• The disparity from a known cylinder pair can be

transferred to a new cylinder, and then reprojected onto a plane (the image)– Can all be done in one step

• Have to resolve occlusion problems– Two points from the reference image could map to the

same point in the output image– Solution: Define a fill ordering that guarantees correct

occlusion (an important contribution of this paper)

• Also have to fill holes

Summary

• Image-based rendering obviously relies heavily on computer vision techniques– Particularly: Depth from stereo

– The problem is very hard with real images

– These techniques are not perfect!

• Sampling remains a problem– Images tend to appear blurry

– Relatively little work on reconstruction algorithms