Shape and Appearance Models from Multiple Images
Richard Szeliski Richard Szeliski Microsoft ResearchMicrosoft Research
Workshop on Image-Based Modeling and RenderingWorkshop on Image-Based Modeling and RenderingStanfordUniversity, March 24, 1998.StanfordUniversity, March 24, 1998.
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 2
Image-Based ModelingImage-Based Modeling
Create 3D models from one or more imagesCreate 3D models from one or more images– calibration and camera pose recoverycalibration and camera pose recovery– correspondence (matching, tracking, stereo)correspondence (matching, tracking, stereo)– 3D model construction3D model construction– appearance extractionappearance extraction
Mix with graphics & re-render (new views)Mix with graphics & re-render (new views)
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 3
Image-Based RenderingImage-Based Rendering
Render 3D graphics from imagesRender 3D graphics from images– sprite-basedsprite-based 3D rendering (Talisman) 3D rendering (Talisman)– view interpolationview interpolation: warp and blend (: warp and blend (morphmorph) )
between several imagesbetween several images– LumigraphLumigraph: full 2D manifold of images: full 2D manifold of images– layered depth images layered depth images ((LDILDIs): voxel-based s): voxel-based
representationrepresentation
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 4
Applications
““Desktop scanning” for 3D world and Desktop scanning” for 3D world and object building (“3D home page”)object building (“3D home page”)
Collaborative design (“3D fax”)Collaborative design (“3D fax”) Virtual environment construction Virtual environment construction
(virtual tourism, home sales/redesign)(virtual tourism, home sales/redesign) Video editing and special effects:Video editing and special effects:
uncalibrated, uncontrolled videouncalibrated, uncontrolled video
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 5
Shape and Appearance Representations
Depth mapsDepth maps
Volumetric modelsVolumetric models
Surface modelsSurface models
View-based representationsView-based representations
Scene decompositions: layers/spritesScene decompositions: layers/sprites
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 6
Outline
Volumes from silhouettesVolumes from silhouettes Surface meshes from matched curvesSurface meshes from matched curves Depth maps from stereoDepth maps from stereo Range data merging and surface modelingRange data merging and surface modeling Appearance recovery (texture maps)Appearance recovery (texture maps) (Sub-) pixel-accurate, multi-view stereo(Sub-) pixel-accurate, multi-view stereo Discussion and summaryDiscussion and summary
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 7
Volumes from Silhouettes
Start with collection of “calibrated” imagesStart with collection of “calibrated” images
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 8
Volumes from Silhouettes
Convert images into binary silhouettesConvert images into binary silhouettes
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 9
Volumes from Silhouettes
Intersect generalized cones (using octree)Intersect generalized cones (using octree)
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 10
Volumes from Silhouettes
Cup on turntable exampleCup on turntable example
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 11
Volumes from Silhouettes
Advantages:Advantages:– simple to implement, fairly robustsimple to implement, fairly robust– fast executionfast execution– complete (closed) surfacecomplete (closed) surface
Limitations:Limitations:– only produces only produces line hullline hull– limited resolutionlimited resolution– sensitive to classification (thresholding)sensitive to classification (thresholding)
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 12
3D Curves from Edges
““Feature-based” stereo matchingFeature-based” stereo matching
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 13
3D Curves from Edges
Extract extremal and internal edgesExtract extremal and internal edges
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 14
3D Curves from Edges
Match curves along epipolar linesMatch curves along epipolar lines
viewing rayviewing rayepipolar planeepipolar plane
epipolar lineepipolar line
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 15
3D Curves from Edges
Reconstruct 3D curves…Reconstruct 3D curves…
… is there a problem?
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 16
3D Curves from Edges
Silhouette curves don’t match in 3DSilhouette curves don’t match in 3D
Solution: fit circular arcs in epipolar planeSolution: fit circular arcs in epipolar plane
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 17
3D Curves from Edges
Coffee jar exampleCoffee jar example
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 18
3D Curves from Edges Advantages:Advantages:
– correct estimates at occluding contourscorrect estimates at occluding contours– good for smoothly curved objectsgood for smoothly curved objects– provides intrinsic surface estimatesprovides intrinsic surface estimates– works on interior surface markingsworks on interior surface markings
Limitations:Limitations:– fails in highly textured regionsfails in highly textured regions– fails in textureless fails in textureless interiorinterior areas areas– incomplete surface (not closed)incomplete surface (not closed)
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 19
Dense Stereo Matching
Compute Compute depth mapdepth map using correlation using correlation
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 20
Dense Stereo Matching
Move correlation windows along epipolar Move correlation windows along epipolar lineslines
– projected window shape depends on surface orientationprojected window shape depends on surface orientation
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 21
Dense Stereo Matching
View extrapolation resultsView extrapolation results
inputinput depth image depth image novel view novel view [Matthies,Szeliski,Kanade’88][Matthies,Szeliski,Kanade’88]
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 22
Dense Stereo Matching
Newer view extrapolation resultsNewer view extrapolation results
inputinput depth imagedepth image novel viewnovel view
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 23
Dense Stereo Matching
Compute certainty map from correlationsCompute certainty map from correlations
inputinput depth map certainty map depth map certainty map
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 24
Range Data Merging
Convert sparse depths to 3D points Convert sparse depths to 3D points
Aggregate with certainty weightingAggregate with certainty weighting[Soucy [Soucy et al., et al., Curless & Levoy, Puli, …]Curless & Levoy, Puli, …]
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 25
Dense Stereo Matching
Advantages:Advantages:– gives detailed surface estimatesgives detailed surface estimates– multi-view aggregation improves accuracymulti-view aggregation improves accuracy
Limitations:Limitations:– narrow baseline narrow baseline noisy estimates noisy estimates– fails in textureless areasfails in textureless areas– sparse, incomplete surfacesparse, incomplete surface– sensitive to non-Lambertian effectssensitive to non-Lambertian effects
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 26
3D Surface Fitting
Convert 3D points into smooth surfaceConvert 3D points into smooth surface– physically-based oriented particlesphysically-based oriented particles
[Szeliski, Tonnesen & Terzopoulos][Szeliski, Tonnesen & Terzopoulos]– triangulation and mesh simplificationtriangulation and mesh simplification
[Hoppe [Hoppe et al., ...et al., ...]]– distance functions and isosurface extractiondistance functions and isosurface extraction
[Curless & Levoy][Curless & Levoy]
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 27
Oriented Particles
Use a collection of small surface elements Use a collection of small surface elements – local coordinates: position, normal, curvaturelocal coordinates: position, normal, curvature
– interaction potentials enforce smoothness interaction potentials enforce smoothness – simulate motions using dynamics simulate motions using dynamics – local triangulation/interpolation scheme local triangulation/interpolation scheme – topology changes occur automaticallytopology changes occur automatically
nn
ee11
nnee22
ee22
ee11
interactioninteraction
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 28
Oriented Particles
Interactive particle-based surface modelingInteractive particle-based surface modeling
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 30
Texture Map Recovery
For each model patch:For each model patch:– determine visibility (item buffer)determine visibility (item buffer)
– blend together textures (weight by view)blend together textures (weight by view)
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 31
Texture Map Recovery
3D model building example3D model building example
octree octree 3D curves 3D curves texture- texture-mappedmapped
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 32
Beyond Texture-Mapped Models
Capture view-dependent appearanceCapture view-dependent appearance– recovering BRDF recovering BRDF [Sato [Sato et al.et al., Yu & Malik], Yu & Malik]– view-dependent texture maps view-dependent texture maps [Debevec [Debevec et al.et al.]]– view interpolation view interpolation [Chen & Williams, …,[Chen & Williams, …,
Seitz & Dyer]Seitz & Dyer]– lightfield and Lumigraph lightfield and Lumigraph [Levoy & Hanrahan, [Levoy & Hanrahan,
Gortler Gortler et al.et al.]]
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 33
Lumigraph Example
acquisition stageacquisition stage volumetric modelvolumetric model novel novel viewview
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 34
Multi-Image Scene Recovery
Problems with “classical” approachProblems with “classical” approach– narrow baseline narrow baseline noisy results noisy results– single depth map misses informationsingle depth map misses information– ignores (or improperly treats) occlusionsignores (or improperly treats) occlusions– ignores mixed (partially transparent) pixelsignores mixed (partially transparent) pixels
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 35
Multi-Image Scene Recovery
Goals of new stereo algorithmGoals of new stereo algorithm
– simultaneously recover simultaneously recover disparitiesdisparities, , colorscolors, and , and
opacitiesopacities (c.f. blue screen matting) (c.f. blue screen matting)
– explicitly handle occlusionsexplicitly handle occlusions
– true multi-frame setting [Collins]true multi-frame setting [Collins]
– details in [Szeliski & Golland, ICCV’98]details in [Szeliski & Golland, ICCV’98]
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 36
Plane Sweep Stereo
Sweep family of planes through volumeSweep family of planes through volume
– each plane defines an image each plane defines an image composite homography composite homography
virtual cameravirtual camera
compositecompositeinput imageinput image
projectiveprojective re-sampling of ( re-sampling of (X,Y,ZX,Y,Z))
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 37
Plane Sweep Stereo
For each depth planeFor each depth plane– compute composite (mosaic) image — compute composite (mosaic) image — meanmean
– compute error image — compute error image — variancevariance– convert to confidence and aggregate spatiallyconvert to confidence and aggregate spatially
Select winning depth at each pixelSelect winning depth at each pixel
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 38
Plane Sweep Stereo
““Stack of acetates” model (related to LDI...)Stack of acetates” model (related to LDI...)
– warp and composite (warp and composite (overover) back-to-front) back-to-front
layers (“sprites”)layers (“sprites”)
synthesizedsynthesized imageimage
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 39
Plane Sweep Stereo
Compute Compute visibilityvisibility each input/layer pair each input/layer pair
Recompute means, confidences, and opacitiesRecompute means, confidences, and opacities
input imageinput image
layer compositelayer composite
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 40
Voxel Coloring
Generalizes plane sweep camera geometryGeneralizes plane sweep camera geometry– replace plane sweep with surface sweepreplace plane sweep with surface sweep
[Seitz & Dyer][Kutulakos & Seitz][Seitz & Dyer][Kutulakos & Seitz]
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 41
Voxel Coloring
Results for dinosaur and roseResults for dinosaur and rose
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 42
Stereo with Matting
Estimate fractional opacities for pixelsEstimate fractional opacities for pixels– adjust layer “sprites” (colors and opacities) to adjust layer “sprites” (colors and opacities) to
best match input imagesbest match input images– optimization criteria:optimization criteria:
re-synthesis errorre-synthesis error color and opacity smoothnesscolor and opacity smoothness prior distribution on opacitiesprior distribution on opacities
– corresponds to MAP Bayesian estimatorcorresponds to MAP Bayesian estimator
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 43
Stereo with Matting
SRI Trees sequence exampleSRI Trees sequence example
input images input images stereo layers stereo layers
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 44
Stereo with Matting
Advantages:Advantages:– true multi-image matchingtrue multi-image matching– deals with occlusions and mixed pixelsdeals with occlusions and mixed pixels
Limitations:Limitations:– too many degrees of freedom (volume)too many degrees of freedom (volume)– breaks up surfaces into “voxels”breaks up surfaces into “voxels”– no “sub-pixel” depthsno “sub-pixel” depths
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 45
Layered Stereo
Use arbitrarily oriented sprites Use arbitrarily oriented sprites [Baker,Szeliski,Anandan’98][Baker,Szeliski,Anandan’98]
Estimate 3D plane equation for each spriteEstimate 3D plane equation for each sprite
layers (“sprites”)layers (“sprites”)
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 46
Layered Stereo Demo
SpriteViewerSpriteViewer: renders sprites with depth: renders sprites with depth
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 47
Layered Stereo
Assign pixel to different “layers” (objects, Assign pixel to different “layers” (objects, sprites)sprites)
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 48
Layered Stereo Track each layer from frame to frame, Track each layer from frame to frame,
compute plane eqn. and composite mosaiccompute plane eqn. and composite mosaic
Re-compute pixel assignment by comparing Re-compute pixel assignment by comparing original images to spritesoriginal images to sprites
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 49
Layered Stereo
Resulting sprite collectionResulting sprite collection
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 50
Layered Stereo
Estimated depth mapEstimated depth map
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 51
Layered Stereo
Re-synthesize original or novel images Re-synthesize original or novel images from collection of spritesfrom collection of sprites
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 52
Layered Stereo Per-pixel residual depth estimationPer-pixel residual depth estimation
– plane plus parallaxplane plus parallax [Anandan [Anandan et al.et al.]]– model-based stereomodel-based stereo [Debevec [Debevec et al.et al.]]
– better accuracy / fidelitybetter accuracy / fidelity– makes makes forward warpingforward warping more difficult more difficult
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 53
Layered Stereo
Advantages:Advantages:– can represent occluded regionscan represent occluded regions– can represent transparent and border (mixed) can represent transparent and border (mixed)
pixels (sprites have pixels (sprites have alphaalpha value per pixel) value per pixel)– works on texture-less interior regionsworks on texture-less interior regions
Limitations:Limitations:– fails for high depth-complexity scenesfails for high depth-complexity scenes– may need manual initialization / controlmay need manual initialization / control
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 54
Image-Based Modeling & Rendering
Grand Unified Theory of Image-Based Grand Unified Theory of Image-Based Modeling and RenderingModeling and Rendering
Design continuumDesign continuum
3D models3D models imagesimages
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 55
Modeling & Rendering
Silhouettes Silhouettes volume volume Curves Curves 3D mesh 3D mesh Stereo Stereo depth map depth map Range data mergingRange data merging 3D surface modeling3D surface modeling Texture recoveryTexture recovery Multi-view stereoMulti-view stereo
3D texture-mapped 3D texture-mapped modelmodel
View-dependent View-dependent texture mapstexture maps
Sprites with depthSprites with depth Layered Depth ImagesLayered Depth Images Colored depth mapsColored depth maps LumigraphLumigraph LightfieldLightfield
viewsviews
objectsobjects
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 56
Open Problems
Automatic scene segmentationAutomatic scene segmentation Complex scenes: forests… Complex scenes: forests… Non-static scenesNon-static scenes Non-rigid motionNon-rigid motion Moving illumination, specularities, … Moving illumination, specularities, … … … … … but potential of IBMR looks greatbut potential of IBMR looks great
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 57
Acknowledgements
ColleaguesColleagues– CMUCMU: Takeo Kanade, Geoffrey Hinton,: Takeo Kanade, Geoffrey Hinton,
Larry MatthiesLarry Matthies
– DEC CRLDEC CRL: Demetri Terzopoulos, David Tonnesen, : Demetri Terzopoulos, David Tonnesen, Sing Bing Kang, James Coughlan, Richard WeissSing Bing Kang, James Coughlan, Richard Weiss
– MicrosoftMicrosoft: Michael Cohen, Steven Gortler, Radek : Michael Cohen, Steven Gortler, Radek Grzeszczuk, Polina Golland, Grzeszczuk, Polina Golland, Heung-Yeung Shum, Simon Baker, Anandan, Mei HanHeung-Yeung Shum, Simon Baker, Anandan, Mei Han
BibliographyBibliography– see http://www.research.microsoft.com/research/vision/szeliski/IBMRsee http://www.research.microsoft.com/research/vision/szeliski/IBMR
IBMR, March 24, 1998 Richard Szeliski, Microsoft Research 58
© Microsoft Corp., 1998© Microsoft Corp., 1998