Surface Light Fieldsfor 3D PhotographyDaniel Wood Daniel Azuma Wyvern AldingerBrian Curless Tom DuchampDavid Salesin Werner Stuetzle

3D PhotographyGoalsRendering and editing

InputsPhotographs and geometry

RequirementsEstimation and compression

View-dependent texture mappingDebevec et al. 1996, 1998Pulli et al. 1997

View-dependent texture mappingDebevec et al. 1996, 1998Pulli et al. 1997

Two-plane light fieldLevoy and Hanrahan 1996Gortler et al. 1996

Surface light fieldsWalter et al. 1997Miller et al. 1998Nishino et al. 1999

Lumisphere-valued texture mapsLumisphere

OverviewDataacquisitionEstimationandcompressionRenderingEditing

OverviewDataacquisitionEstimationandcompressionRenderingEditing

Scan and reconstruct geometryReconstructed geometryRange scans(only a few shown . . .)

Take photographsCamera positionsPhotographs

Register photographs to geometryGeometryPhotographs

Register photographs to geometryUser selected correspondences (rays)

Parameterizing the geometryBase meshScanned geometryMap

Sample base mesh facesBase meshDetailed geometry

Assembling data lumispheresData lumisphere

OverviewDataacquisitionEstimationandcompressionRenderingEditing

Pointwise fairingFaired lumisphereData lumisphere

Pointwise fairing resultsInput photographPointwise faired(177 MB)

Pointwise fairingMany input data lumispheresMany faired lumispheres

CompressionSmall set of prototypes

Compression / EstimationSmall set of prototypesMany input data lumispheres

Reflected reparameterization

Reflected reparameterization

Reflected reparameterization

Reflected reparameterizationBeforeAfter

Median removal+ReflectedMedian(diffuse)Median-removed(specular)+

Median removalMedian valuesSpecularResult

Function quantizationCodebook of lumispheresInput data lumisphere

Lloyd iterationInput data lumispheres

Lloyd iterationCodeword

Lloyd iterationPerturb codewords to create larger codebook

Lloyd iterationForm clusters around each codeword

Lloyd iterationOptimize codewords based on clusters

Lloyd iterationCreate new clusters

Function quantization resultsInput photographFunction quantized(1010 codewords, 2.6 MB)

Principal function analysisSubspace of lumispheresInput data lumispherePrototype lumisphere

Principal function analysisApproximating subspacePrototype lumisphere

Principal function analysis

Principal function analysis

Principal function analysis resultsInput photographPFA compressed(Order 5 - 2.5 MB)

Compression comparisonPointwise fairing(177 MB)Function quantization(2.6 MB)Principal functionanalysis (2.5 MB)

Comparison with 2-plane light field(uncompressed)Pointwise-fairedsurface light field (177 MB)Uncompressedlumigraph / light field (177 MB)

Comparison with 2-plane light field(compressed)Compressed (PFA)surface light field (2.5 MB)Vector-quantizedlumigraph / light field (8.1 MB)

OverviewDataacquisitionEstimationandcompressionRenderingEditing

View-dependent level-of-detail

Render texture domain and coordinates in false color

Evaluate surface light field

Interactive rendererscreen capture

OverviewDataacquisitionEstimationandcompressionRenderingEditing

Lumisphere filteringOriginal surface light fieldGlossier coat

Lumisphere filtering

Rotating the environmentOriginal surface light fieldRotated environment

DeformationOriginalDeformed

Deformation

SummaryEstimation and compressionFunction quantizationPrincipal function analysis

RenderingFrom compressed representationWith view-dependent level-of-detail

EditingLumisphere filteringGeometric deformations and transformations

Future workBetter geometry-to-image registration

More complex surfaces (mirrored, refractive, fuzzy) under more complex illumination

Derive geometry from images

Combining FQ and PFA

AcknowledgementsMarc Levoy and Pat Hanrahan(Thanks for the use of the Stanford Spherical Gantry)

Michael Cohen and Richard Szeliski

National Science Foundation

The end

Geometry (fish)Reconstruction: 129,000 facesMemory for reconstruction: 2.5 MBBase mesh: 199 facesRe-mesh (4x subdivided): 51,000 facesMemory for re-mesh: 1 MBMemory with view-dependence: 7.5 MB

Light field data and rep (fish)Time to acquire: 1 hourInput images: 661Raw data size: ~500 MBLumisphere representation: 3 times subdivided octehedronLumisphere size: 258 directions

Lumigraph (fish)Lumigraph images: 400x400Lumigraph viewpoints per slab: 8x8Number of slabs: 6Lumigraph size (w/o geom): 184 MBLumigraph VQ dimension: 16384Lumigraph VQ codewords: 2x2x2x2x3Compressed size (w/o geom): 8.1 MB

Compression (fish)Pointwise faired: Memory = 177 MBRMS error = 9FQ (2000 codewords)Memory = 3.4 MBRMS error = 23PFA (dimension 3)Memory = 2.5 MBRMS error = 24PFA (dimension 5)Memory = 2.9 MBRMS error = ?

Pre-processing times (fish)Compute times on ~450 MHz P-IIIRange scanning time: 3 hoursGeometry registration: 2 hoursImage to geometry alignment: 6 hoursMAPS (sub-optimal): 5 hoursAssembling data lumispheres: 24 hoursPointwise fairing: 30 minutesFQ codebook construction (10%): 30 hoursFQ encoding: 4 hoursPFA codebook construction (0.1%): 20 hoursPFA encoding: 2 hours

Breakdown and rendering (fish)For PFA dimension 3Direction mesh: 11 KBNormal maps: 680 KBMedian maps: 680 KBIndex maps: 455 KBWeight maps: 680 KBCodebook: 3 KBGeometry w/o view dependence:

Construct codebook using Lloyd iterationIterate until convergence:

Assign all data lumispheres to closest codeword, forming clusters.

Compute new codeword for each cluster by cluster-wise fairing.

Then split all codewords and start over.

Data extrapolationPhotographSurface light field

Comparison with 2-plane light field(uncompressed)Pointwise-fairedsurface light field (177 MB)Uncompressed2-plane light field (177 MB)

Comparison with 2-plane light field(compressed)Principal function analysissurface light field (2.5 MB)Vector-quantized2-plane light field (8.1 MB)

DetailsInput photographPointwise fairing(177 MB)Function quantization (3.4 MB)Principal function analysis (2.5 MB)