Shape from Contours and Multiple Stereo

A Hierarchical, Mesh-Based Approach

Hendrik Kück, Wolfgang Heidrich, Christian Vogelgsang

The goalThe goal

The goalThe goal

Our approachOur approach

• Perform reconstruction using• Color• Object’s silhouettes

1. Create initial approximation based on silhouettes (Visual Hull)

2. Improve mesh using color information within an optimization approach

• Perform reconstruction using• Color• Object’s silhouettes

1. Create initial approximation based on silhouettes (Visual Hull)

2. Improve mesh using color information within an optimization approach

Talk outlineTalk outline

• Image data & preprocessing• Visual Hull

• Definition• Computation as a triangle mesh

• Image based mesh optimization• Results

• Image data & preprocessing• Visual Hull

• Definition• Computation as a triangle mesh

• Image based mesh optimization• Results

Image data & preprocessingImage data & preprocessing

• Calibrated images

• Foreground/ background segmentation

• Calibrated images

• Foreground/ background segmentation

Shape from SilhouettesShape from Silhouettes

• Silhouette + camera information: Silhouette Cone

• Completely contains real object

• Silhouette + camera information: Silhouette Cone

• Completely contains real object

Shape from SilhouettesShape from Silhouettes

• Silhouette + camera information: Silhouette Cone

• Completely contains real object

• Silhouettes can have holes

• Silhouette + camera information: Silhouette Cone

• Completely contains real object

• Silhouettes can have holes

Visual HullVisual Hull

• Definition: Largest volume that produces the same silhouettes as the object

• Construction:Intersection of the silhouette cones

• Definition: Largest volume that produces the same silhouettes as the object

• Construction:Intersection of the silhouette cones

Visual HullVisual Hull

Computing the Visual HullComputing the Visual Hull

• Extract Visual Hull using Extended Marching Cubes algorithm(Kobbelt, Botsch, Schwanecke, Seidel, 2001)

• Vertices lie exactly on isosurface• Can preserve sharp discontinuities

• Requires signed directed distance functionD(VH,x,d)• Distance from x to VH surface along direction d• Positive, if x outside VH, negative if inside

• Extract Visual Hull using Extended Marching Cubes algorithm(Kobbelt, Botsch, Schwanecke, Seidel, 2001)

• Vertices lie exactly on isosurface• Can preserve sharp discontinuities

• Requires signed directed distance functionD(VH,x,d)• Distance from x to VH surface along direction d• Positive, if x outside VH, negative if inside

Computing the Visual HullComputing the Visual Hull

• D(VH,x,d) = D( SCi ,x,d) = maxi D(SCi ,x,d)

• D(SCi ,x,d) can be efficiently computed in image space

• D(VH,x,d) = D( SCi ,x,d) = maxi D(SCi ,x,d)

• D(SCi ,x,d) can be efficiently computed in image space

Computing the Visual HullComputing the Visual Hull

14000 triangles 1400 triangles

The Optimization StageThe Optimization Stage

• Evolve triangle mesh into a shape that is• Consistent with color in the images• Consistent with silhouettes in images• Free from self-intersections• Smooth (low curvature)• Composed of well shaped triangles

• Evolve triangle mesh into a shape that is• Consistent with color in the images• Consistent with silhouettes in images• Free from self-intersections• Smooth (low curvature)• Composed of well shaped triangles

OptimizationOptimization

• Only geometry is optimized, not topology

• 3 Nv degrees of freedom

• Global optimization hopeless Use local per-vertex optimizations

• Locally minimize energy function E(vi) using 3D Simplex Method

• Iterate over vertices

• Only geometry is optimized, not topology

• 3 Nv degrees of freedom

• Global optimization hopeless Use local per-vertex optimizations

• Locally minimize energy function E(vi) using 3D Simplex Method

• Iterate over vertices

Per Vertex Energy FunctionPer Vertex Energy Function

color

silhouette

triangle shape

local curvature

self penetration

Color consistencyColor consistency

• Assumption: Lambertian reflectance• Surface points appear the same from all

viewing directions• Points on the real surface will project

onto pixels of the same color in all images that see them

• Projecting images onto the mesh• If surface is consistent, the color from

different images will match• Color cost term color mismatch (L2 norm)

• Assumption: Lambertian reflectance• Surface points appear the same from all

viewing directions• Points on the real surface will project

onto pixels of the same color in all images that see them

• Projecting images onto the mesh• If surface is consistent, the color from

different images will match• Color cost term color mismatch (L2 norm)

Color consistencyColor consistency

• Use OpenGL for color projection• Projective texture

mapping

• And for determining visibility• Shadow mapping

• Use OpenGL for color projection• Projective texture

mapping

• And for determining visibility• Shadow mapping

v v

v v

Color consistencyColor consistency

• Set up orthographic view of triangle fan around vertex v• Choose scale

according to sampling rate in the images

• Render fan to get samples of color and occlusion, once for each (relevant) image

• Set up orthographic view of triangle fan around vertex v• Choose scale

according to sampling rate in the images

• Render fan to get samples of color and occlusion, once for each (relevant) image

Silhouette consistencySilhouette consistency

• No part of the geometry may project outside any silhouette (must stay inside the visual hull)

Strongly penalize distance outside Visual Hull¼ maxi ( distance outside silhouettes in image i )

• Geometry may be smaller than Visual Hull• Where color does not provide enough

information, use Visual Hull as fallback solution Slightly penalize distance inside Visual Hull

¼ mini ( distance inside silhouettes in image i )

• No part of the geometry may project outside any silhouette (must stay inside the visual hull)

Strongly penalize distance outside Visual Hull¼ maxi ( distance outside silhouettes in image i )

• Geometry may be smaller than Visual Hull• Where color does not provide enough

information, use Visual Hull as fallback solution Slightly penalize distance inside Visual Hull

¼ mini ( distance inside silhouettes in image i )

Silhouette consistencySilhouette consistency

• Encode distance from silhouette in alpha channel of OpenGL textures• Project onto triangle fan along with

color & visibility

• Encode distance from silhouette in alpha channel of OpenGL textures• Project onto triangle fan along with

color & visibility

Multi-Resolution OptimizationMulti-Resolution Optimization

• Local minima are a problem,especially when• Triangle size is small compared to

geometric error• Texture frequencies are high

compared to geometric error

• Solution: Perform optimization at multiple resolutions

• Local minima are a problem,especially when• Triangle size is small compared to

geometric error• Texture frequencies are high

compared to geometric error

• Solution: Perform optimization at multiple resolutions

Multi-Resolution OptimizationMulti-Resolution Optimization

• Start with low resolution Visual Hull mesh

• Start with low resolution Visual Hull mesh

Multi-Resolution OptimizationMulti-Resolution Optimization

• Start with low resolution Visual Hull mesh

• Optimize until convergence

• Start with low resolution Visual Hull mesh

• Optimize until convergence

Multi-Resolution OptimizationMulti-Resolution Optimization

• Start with low resolution Visual Hull mesh

• Optimize until convergence

• Subdivide & optimize more

• Start with low resolution Visual Hull mesh

• Optimize until convergence

• Subdivide & optimize more

Multi-Resolution OptimizationMulti-Resolution Optimization

• Start with low resolution Visual Hull mesh

• Optimize until convergence

• Subdivide & optimize more

• Do it again, …

• Start with low resolution Visual Hull mesh

• Optimize until convergence

• Subdivide & optimize more

• Do it again, …

Final ResultsFinal Results

before

Final ResultsFinal Results

before after

Final ResultsFinal Results

before after

Thank you