state of the art in surface reconstruction from point clouds · state of the art in surface...

State of the Art in Surface Reconstruction from Point Clouds

1

Matthew Berger Andrea Tagliasacchi

Lee M. Seversky

Joshua A. Levine

Pierre Alliez

Claudio T. SilvaAndrei Sharf

State of the Art in Surface Reconstruction from Point Clouds

2

Matthew Berger Andrea Tagliasacchi

Lee M. Seversky

Joshua A. Levine

Pierre Alliez

Claudio T. SilvaAndrei Sharf

Berger, Tagliasacchi, Seversky, Alliez, Levine, Sharf, SilvaState of the Art in Surface Reconstruction from Point Clouds

Introduction

3

Longstanding Goal in Computer Graphics

Model The World Around Us

Need to first acquire the world


Acquisition

• Single object

• Fine-grained control

• Controlled setting

4

Desktop Scanners

[Lanman & Taubin 09]


Acquisition

• Large scenes

• Low control

• Occlusion

5

Terrestrial LiDAR


Acquisition

• Large environments

• Top-down perspective

• Lower resolution

6

Aerial LiDAR


Acquisition

• Fine-grained detail

• Small scenes

• Easy and cheap to use

7

Compact Real-time Scanning

[Kim et al. SIGA’12]


General Pipeline

8


General Pipeline

8

?


General Pipeline

8

?

Surface Reconstruction from Point Clouds!


Why Reconstruction?• Captured point cloud unsuitable for many geometry processing tasks

• Noisy, incomplete

• Topology not well-defined

• Does not define continuous representation

9


Why Reconstruction?

10

Raw Point Cloud Not Enough!


Why Reconstruction?

10

Raw Point Cloud Not Enough!Mobius Voting For Surface Correspondence

Yaron Lipman Thomas FunkhouserPrinceton University

Abstract

The goal of our work is to develop an efficient, automatic algo-rithm for discovering point correspondences between surfaces thatare approximately and/or partially isometric.

Our approach is based on three observations. First, isometries area subset of the Mobius group, which has low-dimensionality – sixdegrees of freedom for topological spheres, and three for topolog-ical discs. Second, computing the Mobius transformation that in-terpolates any three points can be computed in closed-form after amid-edge flattening to the complex plane. Third, deviations fromisometry can be modeled by a transportation-type distance betweencorresponding points in that plane.

Motivated by these observations, we have developed a Mobius Vot-ing algorithm that iteratively: 1) samples a triplet of three randompoints from each of two point sets, 2) uses the Mobius transforma-tions defined by those triplets to map both point sets into a canoni-cal coordinate frame on the complex plane, and 3) produces “votes”for predicted correspondences between the mutually closest pointswith magnitude representing their estimated deviation from isom-etry. The result of this process is a fuzzy correspondence matrix,which is converted to a permutation matrix with simple matrix op-erations and output as a discrete set of point correspondences withconfidence values.

The main advantage of this algorithm is that it can find intrinsicpoint correspondences in cases of extreme deformation. Duringexperiments with a variety of data sets, we find that it is able to finddozens of point correspondences between different object types indifferent poses fully automatically.

1 IntroductionFinding correspondences between a discrete set of points on twodifferent surface meshes is a fundamental problem in computergraphics, geometric processing, and medical imaging. Among itsmany applications are shape interpolation, attribute transfer, surfacecompletion, statistical shape modeling, symmetry analysis, shapematching, and deformable surface tracking.

For many of these applications, the input meshes represent differentobjects in different poses, and thus alignment of extrinsic shapes isinsufficient. However, the intrinsic shapes for objects of the sameclass are often approximately isometric, and sometimes they arecomposed of large parts that are nearly isometric. For example,this is the case for the dog and the wolf shown in Figure 1, as wellas for cloth deformations (perfect isometries), faces, and surfaces

Figure 1: Correspondences found automatically between a dog anda wolf. Corresponding points are shown as spheres of the samecolor with a radius proportional to the confidence of the correspon-dence.

of brains and other anatomical organs. The goal of our work isto develop efficient algorithms for discovering dense sets of pointcorrespondences for these shapes automatically.

This problem is challenging for two reasons. First, the space ofpossible correspondences is very large (O(N!) for N input points),and thus strategies for searching this space with the hope of find-ing the correspondence with minimal deformation error are eithervery slow or resort to heuristics. Second, even if it were possibleto consider all potential correspondence sets, it would be difficultto compute an appropriate deformation error (deviation from isom-etry) for each one. Therefore, previous methods have relied uponhumans to provide initial landmark correspondences, work only onexamples with similar local shape features, compute approximatedeformation errors, and/or take long computation times.

In this paper, we propose an algorithm that we call Mobius Voting.The key observation is that the space of isometries between simply-connected surfaces is contained within the Mobius group, whichhas low dimensionality. For genus zero surfaces (sphere topology),the Mobius group has six degrees of freedom; and for patches withboundaries (disc topology), it has only three. Thus, defining the en-tire mapping between two isometric surfaces is possible with justthree point correspondences. Moreover, there is a Conformal Flat-tening (Uniformization) operator (based on Pinkall and Polthier’snon-conforming conjugate harmonic maps [1993]) that takes a mid-edge representation of the mesh onto a 2D canonical domain de-fined by three points, where: 1) the discrete conformal error of themapping is zero, 2) the Mobius transformation defined by the threepoint correspondences can be computed in a closed form, as a ratio-nal linear function in the complex plane, 3) the Mobius transforma-tion can be applied to any point in the complex plane by computinga simple rational function that is fast to compute, and 4) deviationsfrom an isometric mapping can be modeled with a simple functionbased on transportation-type distances.

The main implication of these observations is that it is possible todesign an algorithm that finds a dense set of correspondences be-tween nearly isometric meshes in polynomial-time. At the core ofour algorithm is a Hough-style voting scheme where three randompoints are repeatedly sampled from each of two meshes and used todefine Mobius transformations that map them to a shared canonical

[Lipman & Funkhouser SIG’09]


Why Reconstruction?

10

Raw Point Cloud Not Enough!Mobius Voting For Surface Correspondence

Yaron Lipman Thomas FunkhouserPrinceton University

Abstract

The goal of our work is to develop an efficient, automatic algo-rithm for discovering point correspondences between surfaces thatare approximately and/or partially isometric.

Our approach is based on three observations. First, isometries area subset of the Mobius group, which has low-dimensionality – sixdegrees of freedom for topological spheres, and three for topolog-ical discs. Second, computing the Mobius transformation that in-terpolates any three points can be computed in closed-form after amid-edge flattening to the complex plane. Third, deviations fromisometry can be modeled by a transportation-type distance betweencorresponding points in that plane.

Motivated by these observations, we have developed a Mobius Vot-ing algorithm that iteratively: 1) samples a triplet of three randompoints from each of two point sets, 2) uses the Mobius transforma-tions defined by those triplets to map both point sets into a canoni-cal coordinate frame on the complex plane, and 3) produces “votes”for predicted correspondences between the mutually closest pointswith magnitude representing their estimated deviation from isom-etry. The result of this process is a fuzzy correspondence matrix,which is converted to a permutation matrix with simple matrix op-erations and output as a discrete set of point correspondences withconfidence values.

The main advantage of this algorithm is that it can find intrinsicpoint correspondences in cases of extreme deformation. Duringexperiments with a variety of data sets, we find that it is able to finddozens of point correspondences between different object types indifferent poses fully automatically.

1 IntroductionFinding correspondences between a discrete set of points on twodifferent surface meshes is a fundamental problem in computergraphics, geometric processing, and medical imaging. Among itsmany applications are shape interpolation, attribute transfer, surfacecompletion, statistical shape modeling, symmetry analysis, shapematching, and deformable surface tracking.

For many of these applications, the input meshes represent differentobjects in different poses, and thus alignment of extrinsic shapes isinsufficient. However, the intrinsic shapes for objects of the sameclass are often approximately isometric, and sometimes they arecomposed of large parts that are nearly isometric. For example,this is the case for the dog and the wolf shown in Figure 1, as wellas for cloth deformations (perfect isometries), faces, and surfaces

Figure 1: Correspondences found automatically between a dog anda wolf. Corresponding points are shown as spheres of the samecolor with a radius proportional to the confidence of the correspon-dence.

of brains and other anatomical organs. The goal of our work isto develop efficient algorithms for discovering dense sets of pointcorrespondences for these shapes automatically.

This problem is challenging for two reasons. First, the space ofpossible correspondences is very large (O(N!) for N input points),and thus strategies for searching this space with the hope of find-ing the correspondence with minimal deformation error are eithervery slow or resort to heuristics. Second, even if it were possibleto consider all potential correspondence sets, it would be difficultto compute an appropriate deformation error (deviation from isom-etry) for each one. Therefore, previous methods have relied uponhumans to provide initial landmark correspondences, work only onexamples with similar local shape features, compute approximatedeformation errors, and/or take long computation times.

In this paper, we propose an algorithm that we call Mobius Voting.The key observation is that the space of isometries between simply-connected surfaces is contained within the Mobius group, whichhas low dimensionality. For genus zero surfaces (sphere topology),the Mobius group has six degrees of freedom; and for patches withboundaries (disc topology), it has only three. Thus, defining the en-tire mapping between two isometric surfaces is possible with justthree point correspondences. Moreover, there is a Conformal Flat-tening (Uniformization) operator (based on Pinkall and Polthier’snon-conforming conjugate harmonic maps [1993]) that takes a mid-edge representation of the mesh onto a 2D canonical domain de-fined by three points, where: 1) the discrete conformal error of themapping is zero, 2) the Mobius transformation defined by the threepoint correspondences can be computed in a closed form, as a ratio-nal linear function in the complex plane, 3) the Mobius transforma-tion can be applied to any point in the complex plane by computinga simple rational function that is fast to compute, and 4) deviationsfrom an isometric mapping can be modeled with a simple functionbased on transportation-type distances.

The main implication of these observations is that it is possible todesign an algorithm that finds a dense set of correspondences be-tween nearly isometric meshes in polynomial-time. At the core ofour algorithm is a Hough-style voting scheme where three randompoints are repeatedly sampled from each of two meshes and used todefine Mobius transformations that map them to a shared canonical

[Lipman & Funkhouser SIG’09]

EUROGRAPHICS 2012 / P. Cignoni, T. Ertl(Guest Editors)

Volume 31 (2012), Number 2

Explicit Mesh Surfaces for Particle Based Fluids

Jihun Yu1, Chris Wojtan2, Greg Turk3 and Chee Yap4

1Industrial Light and Magic, 2IST Austria3Georgia Institute of Technology, 4New York University

Figure 1: A drop falling into a shallow pool creates a water crown.

Abstract

We introduce the idea of using an explicit triangle mesh to track the air/fluid interface in a smoothed particlehydrodynamics (SPH) simulator. Once an initial surface mesh is created, this mesh is carried forward in time usingnearby particle velocities to advect the mesh vertices. The mesh connectivity remains mostly unchanged acrosstime-steps; it is only modified locally for topology change events or for the improvement of triangle quality. In orderto ensure that the surface mesh does not diverge from the underlying particle simulation, we periodically projectthe mesh surface onto an implicit surface defined by the physics simulation. The mesh surface gives us severaladvantages over previous SPH surface tracking techniques. We demonstrate a new method for surface tensioncalculations that clearly outperforms the state of the art in SPH surface tension for computer graphics. We alsodemonstrate a method for tracking detailed surface information (like colors) that is less susceptible to numericaldiffusion than competing techniques. Finally, our temporally-coherent surface mesh allows us to simulate high-resolution surface wave dynamics without being limited by the particle resolution of the SPH simulation.

Categories and Subject Descriptors (according to ACM CCS): I.3.5 [Computer Graphics]: Computational Geometryand Object Modeling—Physically based modeling I.3.7 [Computer Graphics]: Three-Dimensional Graphics andRealism—Animation

1. Introduction

Within the field of computer graphics, there is a diverseset of different techniques for animating liquids. Each tech-nique has its own merits and drawbacks: Eulerian techniquescan produce very realistic animations, but they are compu-tationally expensive and unsuitable for simulating certaineffects like spray and foam. Shallow water discretizationsrun at real-time rates, but they are restricted to height fields.Particle-based simulations, like smoothed particle hydrody-namics (SPH), easily retain volume and momentum, andthey have been used in a wide range of applications like liq-uids, deformable solids, multi-phase fluids, viscoelastic ma-terials, controlled liquids, and porous flow. However, these

particle-based methods traditionally produce noisy and un-realistic surfaces.

The standard technique for producing a surface for a par-ticle simulation involves the creation of an implicit surfacethat essentially wraps around all of the particles in the simu-lation. While recent research has greatly improved the visualquality of the surface that results from such an implicit for-mulation, this implicit surface strategy generally has diffi-culty producing and retaining high resolution surface detailslike waves, textures, and ripples.

We introduce a new explicit method for tracking the sur-face of a particle-based fluid simulation, which overcomesmany of these difficulties. Our surface tracking approach be-

c� 2012 The Author(s)Computer Graphics Forum c� 2012 The Eurographics Association and Blackwell Publish-ing Ltd. Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ,UK and 350 Main Street, Malden, MA 02148, USA.

[Yu et al. EG’12]


Traditional Reconstruction

• Take point cloud as input

• Output: continuous representation

• Assumption: output should be smooth

11






11

M. Kazhdan / Reconstruction of Solid Models

Figure 5: Reconstructed surfaces of a human face using samplesfrom a 3D model. Views of the original model are shown in the toprow. Views of the reconstructed, water-tight model are shown in thebottom row.

method returns a solid model that accurately fits the inputsamples while providing a reasonable reconstruction of thesurface in the regions where no samples could be provided.

Figure 6 shows the reconstructions for a point set that wasuniformly sampled from the surface of the toes of Michelan-gelo’s David model (N = 100,000, b = 128). The initialmodel is shown on the left, with a crack between the first twotoes resulting from the scanner’s inability to see the region.Using our method to reconstruct the surface of the model byassigning uniform weights to each sample point gives rise tothe surface shown in the middle column. While this surfaceaccurately approximates the data and results in a water-tightreconstruction, it introduces a topological handle connect-ing the first two toes. By assigning weights to the samplesthat are inversely proportional to the regional sampling den-sity, as described in Section 3.5, we obtain a new reconstruc-tion (right column) that gives more weight to the points nearthe boundary of the crack. This forces the reconstruction tomaintain the surface orientation near the missing data andresults in a reconstruction that does not have the topologicalartifact introduced when uniform weights are used.

To evaluate the performance of our method in the pres-ence of non-uniform sampling, we generated an orientedpoint set by randomly sampling 100,000 points from the sur-face of the “Happy Buddha” model, Figure 7(a), where theprobability of choosing a point was a function of the sur-face curvature. An image of the point set is shown in Fig-ure 7(b), with sparse sampling in low curvature regions (e.g.the stomach and the base of the pedestal) and dense sam-pling in high curvature regions. Figure 7(c) shows the re-construction obtained using uniform weighting which over-integrates the high curvature areas, resulting in a poor recon-struction in planar regions. In contrast, Figure 7(d) shows thereconstruction obtained when we use our weighting methodto assign weights to the samples. As the figure indicates,

Figure 6: The toes of Michelangelo’s David model (left) and the re-constructions obtained using uniform weights (middle) and weightsthat are inversely proportional to the sampling density (right).

Figure 7: Reconstructions from a non-uniform point set. The ini-tial Buddha model (a), the point set sampled as a function of sur-face curvature (b), the reconstructed surface obtained using uniformpoint weighting (c), and the surface reconstructed using our weightassignment method (d).

the reconstructions closely approximates the initial surface(Figure 7(a)), indicating that though our weight assignmentmethod is a heuristic, it gives a good approximation to thetrue sampling density and results in robust reconstructions.

Finally, to test how well our method performs in the pres-ence of noise, we sampled a cow model at 100,000 pointsand added noise to both the position and normal of eachsample. Noise was added to each sample by randomly dis-placing the position by a fixed distance and randomly chang-ing the direction of the normal by a fixed angle. The recon-structed cow models are shown in Figure 8. The rows showthe change in the reconstructed model as the positional noiseis increased and the columns show the change as the angularnoise is increased. The results in the image indicate that ourreconstruction method is robust in the presence of both po-sitional and angular error. In particular, the figure shows thatour method reconstructs all but the features of the model thatare smaller than the displacement size. For example, whena displacement value of 1/32-nd of the bounding radius isused (right column), the body of the cow is reconstructed,

c⃝ The Eurographics Association 2005.






11





















[Kazhdan SGP’05]


Departure from Traditional

How do we handle substantial artifacts in the point cloud?

12

Priors!

• Priors on the scanned shape

• Priors on the acquisition

• Priors on the interaction


Priors

13

Volume Smoothness

[Tagliasacchi et al. SIG’09]


Priors

14

Global Regularity

[Zheng et al. SIG’10]


Priors

15

Data-Driven

[Shen et al. SIGA’12]


Outline

• Characterization of surface reconstruction

• Surface smoothness methods

• Specialized priors

• visibility, volume smoothness, primitives, global regularity, data-driven, interactive

• Where surface reconstruction is headed

16

Present survey of surface reconstruction from the perspective of priors


Characterization• Point Cloud Artifacts

• Point Cloud Input

• Shape Class

17




• Shape Class

18


Artifacts

• Distribution of sampled points

• Useful in computing many surface quantities

• Challenge: nonuniform

19

Sampling Density


Algebraic Point Set Surfaces

Gael Guennebaud Markus GrossETH Zurich

Figure 1: Illustration of the central features of our algebraic MLS framework. From left to right: efficient handling of very complex pointsets, fast mean curvature evaluation and shading, significantly increased stability in regions of high curvature, sharp features with controlledsharpness. Sample positions are partly highlighted.

Abstract

In this paper we present a new Point Set Surface (PSS) definitionbased on moving least squares (MLS) fitting of algebraic spheres.Our surface representation can be expressed by either a projectionprocedure or in implicit form. The central advantages of our ap-proach compared to existing planar MLS include significantly im-proved stability of the projection under low sampling rates and inthe presence of high curvature. The method can approximate orinterpolate the input point set and naturally handles planar pointclouds. In addition, our approach provides a reliable estimate of themean curvature of the surface at no additional cost and allows forthe robust handling of sharp features and boundaries. It processesa simple point set as input, but can also take significant advantageof surface normals to improve robustness, quality and performance.We also present an novel normal estimation procedure which ex-ploits the properties of the spherical fit for both direction estima-tion and orientation propagation. Very efficient computational pro-cedures enable us to compute the algebraic sphere fitting with up to40 million points per second on latest generation GPUs.

CR Categories: I.3.5 [Computer Graphics]: Computational Ge-ometry and Object Modeling—Curve and surface representations

Keywords: point based graphics, surface representation, movingleast square surfaces, sharp features.

1 IntroductionA key ingredient of most methods in point based graphics is the un-derlying meshless surface representation which computes a contin-uous approximation or interpolation of the input point set. The byfar most important and successful class of such meshless represen-tations are point set surfaces (PSS) [Alexa et al. 2003] combining

high flexibility with ease of implementation. PSS generally definea smooth surface using local moving least-squares (MLS) approxi-mations of the data [Levin 2003]. The degree of the approximationcan easily be controlled, making the approach naturally well suitedto filter noisy input data. In addition, the semi-implicit nature ofthe representation makes PSS an excellent compromise combiningadvantages both of explicit representations, such as parametric sur-faces, and of implicit surfaces [Ohtake et al. 2003].

Since its inception, significant progress has been made to betterunderstand the properties and limitations of MLS [Amenta andKil 2004a,2004b] and to develop efficient computational schemes[Adamson and Alexa 2004]. A central limitation of the robustnessof PSS, however, comes from the plane fit operation that is highlyunstable in regions of high curvature if the sampling rate drops be-low a threshold. Such instabilities include erroneous fits or the lim-ited ability to perform tight approximations of the data. This be-havior sets tight limits to the minimum admissible sampling ratesfor PSS [Amenta and Kil 2004b; Dey et al. 2005].

In this paper we present a novel definition of moving least squaressurfaces called algebraic point set surfaces (APSS). The key ideais to directly fit a higher order algebraic surface [Pratt 1987] ratherthan a plane. For computational efficiency all methods in this paperfocus on algebraic sphere fitting, but the general concept could beapplied to higher order surfaces as well. The main advantage ofthe sphere fitting is its significantly improved stability in situationswhere planar MLS fails. For instance, tight data approximation isaccomplished, spheres perform much better in the correct handlingof sheet separation (figure 3) and exhibit a high degree of stabilityboth in cases of undersampling (figure 2) and for very large weightfunctions. The specific properties of algebraic spheres make APSSsuperior to simple geometric sphere fitting. It allows us to elegantlyhandle planar areas or regions around inflection points as limits inwhich the algebraic sphere naturally degenerates to a plane.

Furthermore, the spherical fitting enables us to design interpolatoryweighting schemes by using weight functions with singularities atzero while overcoming the fairness issue of previous MLS surfaces.The sphere radius naturally serves as a for-free and reliable esti-mate of the mean curvature of the surface. This enables us, forinstance, to compute realtime accessibility shading on large inputobjects (figure 1).

Central to our framework are the numerical procedures to efficientlyperform the sphere fit. For point sets with normals we designed

[Guennebaud et al. SIG’07]

Artifacts

• Distribution of sampled points

• Useful in computing many surface quantities

• Challenge: nonuniform

19

Sampling Density


Artifacts

20

Noise

• Points randomly distributed near the surface

• Basic assumption: noise distribution is zero mean

• Due to numerous factors

• Sensor noise, depth quantization, surface material properties

• May also be spatially-varying


Artifacts

20

Noise

[Avron et al. TOG’10]







Artifacts

20

Noise

[Avron et al. TOG’10]

Structure Recovery by Part Assembly

Chao-Hui Shen1 Hongbo Fu2 Kang Chen1 Shi-Min Hu11TNList, Tsinghua University, Beijing 2City University of Hong Kong

Figure 1: Given single-view scans by the Kinect system, containing highly noisy and incomplete 3D scans (upper left) and correspondingRGB images (lower left), our approach is able to faithfully recover their underlying structures (yellow) by assembling suitable parts (red) inthe repository models (blue).

Abstract

This paper presents a technique that allows quick conversion ofacquired low-quality data from consumer-level scanning devices tohigh-quality 3D models with labeled semantic parts and meanwhiletheir assembly reasonably close to the underlying geometry. This isachieved by a novel structure recovery approach that is essentiallylocal to global and bottom up, enabling the creation of newstructures by assembling existing labeled parts with respect to theacquired data. We demonstrate that using only a small-scale shaperepository, our part assembly approach is able to faithfully recovera variety of high-level structures from only a single-view scan ofman-made objects acquired by the Kinect system, containing ahighly noisy, incomplete 3D point cloud and a corresponding RGBimage.

Links: DL PDF WEB

1 Introduction

3D scanning devices provide a quick way to acquire 3D modelsof real-world objects or environment, which benefit a variety ofapplications. However, the acquired models, typically representedas unorganized point clouds, are often corrupted with noise andoutlier. Worse, large regions or even entire parts might remainmissing (see an example in Figure 1), possibly due to occlusions,grazing angle views, or scanner-unfriendly lighting/materials (e.g.,highly reflective materials). These problems further deteriorate

for consumer-level scanning devices like the Kinect system of Mi-crosoft, which provide an economical solution to 3D capturing butat the cost of low-quality acquisition of geometry and appearance.

It is challenging to faithfully recover the underlying geometryor structure from such highly incomplete and noisy scan data.Most of the existing works (e.g., [Sharf et al. 2004; Shalomet al. 2010]) focus on geometry completion or reconstruction, andtackle inputs with small deficiencies or simple missing geometryonly. Still, it is unclear how to effectively recover the underlyingstructure even if the geometry gets completed. The template-basedapproaches [Pauly et al. 2005; Kraevoy and Sheffer 2005] havegreat potential in completing larger, more complex holes. It ispossible to transfer the structural information from the templates tothe scan data. However, the existing approaches largely operate in aglobal-to-local manner, and thus heavily rely on the availability oftemplate models that are globally similar to the underlying object.Although there exist a few online shape repositories like Google3D Warehouse, the available models are still far from capturingreal-world objects exhibiting complex structures, causing the mainbottleneck for the existing template-based approaches.

The recent advance in mesh segmentation greatly simplifies the seg-mentation and labeling of parts in a set of 3D models [Kalogerakiset al. 2010; Huang et al. 2011; Sidi et al. 2011]. The recent worksdemonstrate how to significantly enlarge the existing database of3D models via shape synthesis by part composition [Kalogerakiset al. 2012; Jain et al. 2012; Xu et al. 2012]. However, in practicethis would result in a 3D model database that grows exponentially,making both the storage and the retrieval challenging to manage.We show that it is unnecessary to explicitly prepare such largerdatabase by part composition and it is possible to retrieve andassemble suitable parts on the fly for structure recovery.

We propose a part assembly approach for structure recovery froma highly incomplete, noisy 3D scan of a man-made object togetherwith the corresponding RGB image acquired by the Kinect system(Figure 1). Our approach is based on the key fact that many classesof man-made objects (e.g., chairs, bicycles etc.) lie in a low-dimensional shape space defined with respect to the relative sizesand positions of shape parts [Ovsjanikov et al. 2011]. This allows us

Structure Recovery by Part Assembly

Chao-Hui Shen1 Hongbo Fu2 Kang Chen1 Shi-Min Hu11TNList, Tsinghua University, Beijing 2City University of Hong Kong

Figure 1: Given single-view scans by the Kinect system, containing highly noisy and incomplete 3D scans (upper left) and correspondingRGB images (lower left), our approach is able to faithfully recover their underlying structures (yellow) by assembling suitable parts (red) inthe repository models (blue).

Abstract

This paper presents a technique that allows quick conversion ofacquired low-quality data from consumer-level scanning devices tohigh-quality 3D models with labeled semantic parts and meanwhiletheir assembly reasonably close to the underlying geometry. This isachieved by a novel structure recovery approach that is essentiallylocal to global and bottom up, enabling the creation of newstructures by assembling existing labeled parts with respect to theacquired data. We demonstrate that using only a small-scale shaperepository, our part assembly approach is able to faithfully recovera variety of high-level structures from only a single-view scan ofman-made objects acquired by the Kinect system, containing ahighly noisy, incomplete 3D point cloud and a corresponding RGBimage.

Links: DL PDF WEB

1 Introduction

3D scanning devices provide a quick way to acquire 3D modelsof real-world objects or environment, which benefit a variety ofapplications. However, the acquired models, typically representedas unorganized point clouds, are often corrupted with noise andoutlier. Worse, large regions or even entire parts might remainmissing (see an example in Figure 1), possibly due to occlusions,grazing angle views, or scanner-unfriendly lighting/materials (e.g.,highly reflective materials). These problems further deteriorate

for consumer-level scanning devices like the Kinect system of Mi-crosoft, which provide an economical solution to 3D capturing butat the cost of low-quality acquisition of geometry and appearance.

It is challenging to faithfully recover the underlying geometryor structure from such highly incomplete and noisy scan data.Most of the existing works (e.g., [Sharf et al. 2004; Shalomet al. 2010]) focus on geometry completion or reconstruction, andtackle inputs with small deficiencies or simple missing geometryonly. Still, it is unclear how to effectively recover the underlyingstructure even if the geometry gets completed. The template-basedapproaches [Pauly et al. 2005; Kraevoy and Sheffer 2005] havegreat potential in completing larger, more complex holes. It ispossible to transfer the structural information from the templates tothe scan data. However, the existing approaches largely operate in aglobal-to-local manner, and thus heavily rely on the availability oftemplate models that are globally similar to the underlying object.Although there exist a few online shape repositories like Google3D Warehouse, the available models are still far from capturingreal-world objects exhibiting complex structures, causing the mainbottleneck for the existing template-based approaches.

The recent advance in mesh segmentation greatly simplifies the seg-mentation and labeling of parts in a set of 3D models [Kalogerakiset al. 2010; Huang et al. 2011; Sidi et al. 2011]. The recent worksdemonstrate how to significantly enlarge the existing database of3D models via shape synthesis by part composition [Kalogerakiset al. 2012; Jain et al. 2012; Xu et al. 2012]. However, in practicethis would result in a 3D model database that grows exponentially,making both the storage and the retrieval challenging to manage.We show that it is unnecessary to explicitly prepare such largerdatabase by part composition and it is possible to retrieve andassemble suitable parts on the fly for structure recovery.

We propose a part assembly approach for structure recovery froma highly incomplete, noisy 3D scan of a man-made object togetherwith the corresponding RGB image acquired by the Kinect system(Figure 1). Our approach is based on the key fact that many classesof man-made objects (e.g., chairs, bicycles etc.) lie in a low-dimensional shape space defined with respect to the relative sizesand positions of shape parts [Ovsjanikov et al. 2011]. This allows us








Artifacts

21

Outliers

• Points far from the surface

• Structural artifacts in acquisition

• Can be randomly distributed in the volume

• But can also be highly structured


Artifacts

21

Outliers

S. Giraudot, D. Cohen-Steiner & P. Alliez / Noise-Adaptive Shape Reconstruction from Raw Point Sets 9

Figure 16: Two levels of noise (generated). Left: raw pointset containing a noise-free and a noisy torus. Middle: pointset & reconstruction. Right: reconstruction only.

Figure 17: Two levels of noise. Left: raw point set with ad-ditional noise on the top half part. Middle: point set & re-construction. Right: reconstruction only.

Figure 18: Noise and structured outliers. Left: raw pointset. Middle: closeup on point set. Right: closeup on point set& reconstruction.

Figure 19: Noise almost beyond dimension assumption.Left: raw point set with high noise on the top half part. Mid-dle: point set & reconstruction. Right: reconstruction only.

Figure 20 illustrates a failure case where the noise trulyexceeds the dimension assumption.

Figure 20: Noise beyond dimension assumption (gener-ated). Left: raw point set. Middle: point set & reconstruc-tion. Right: reconstruction only.

Finally, our framework is resilient to noise and outliers,but not to widely variable sampling density as low densityareas are considered outliers (Figure 21).

Figure 21: Variable density (generated). Left: raw point set.Middle: point set & reconstruction. Right: reconstructiononly. Our method fails in capturing the correct dimensionin the low density area.

Figure 22 gives an overview of how our algorithm per-forms in terms of time and memory use.

4.1. Limitations

Using a uniform grid for the nodes of the graph requires alarge number of nodes to capture the correct topology ofshapes with small feature size due to small separation orthickness. This obviously leads to scalability issues. We ex-perimented with a non-uniform graph where the nodes arereusing the vertices of the adaptive triangulation used instep 1, and where appropriate weights per edge are devisedto compensate for the non-uniformity. None of these experi-ments led to satisfactory results.

For computing the sign attribute of an edge of the graph,we use an exhaustive combinatorial search in the numberof (retained) local minima. In our experiments the numberof retained minima is on average below 6, but for complexshapes with many sheets this can also lead to scalability is-sues.

Finally, we are using a two-step approach for guessing thesign, then solving for the signed implicit function. Our ap-proach is scalable as involves only linear solves, but it wouldbe more consistent to do everything in one step without ham-pering the scalability.

submitted to Eurographics Symposium on Geometry Processing (2013)

[Giraudot et al. SGP’13]

• Points far from the surface

• Structural artifacts in acquisition

• Can be randomly distributed in the volume

• But can also be highly structured


Artifacts

22

Misalignment

• Imperfect registration of range scans

• Introduces highly structured noise


Artifacts

22

Misalignment

[Li et al. SIG’11]

• Imperfect registration of range scans

• Introduces highly structured noise


Artifacts

23

Missing Data

• Regions of zero sampling density

• Different ways to handle missing data

• Watertightness

• Reconstruct higher-level information in lieu of the original shape


Artifacts

23

Missing Data

Figure 12: The smoothness penalty discards outliers and completes smoothly missing parts, as can be seen in the tiger’s head. From left toright: the original statue, the raw data points (the missing parts are in black), the FEM field with two close-ups on outliers and large missingparts in the head, and our interactive reconstruction of the model with 12 scribbles.

Figure 13: Due to the statue’s base (left), the camel’s legs cannot be covered (center-left). Scribble constraints drawn on automaticallygenerated 2D tablets (center-right) can be used to generate a coherent geometry (right).

Figure 14: The topological analysis detects weak regions (center-right) in the presence of noise. Weak regions are typically located wheretwo parts of the shape are close and the topology is ambiguous, such as the arm-leg contacts in the sitting woman. Missing parts withunambiguous topology, such as her right shoulder, are automatically smoothly completed.

[Sharf et al. SIG’07]



• Watertightness



Artifacts

23

Missing Data

Figure 12: The smoothness penalty discards outliers and completes smoothly missing parts, as can be seen in the tiger’s head. From left toright: the original statue, the raw data points (the missing parts are in black), the FEM field with two close-ups on outliers and large missingparts in the head, and our interactive reconstruction of the model with 12 scribbles.

Figure 13: Due to the statue’s base (left), the camel’s legs cannot be covered (center-left). Scribble constraints drawn on automaticallygenerated 2D tablets (center-right) can be used to generate a coherent geometry (right).

Figure 14: The topological analysis detects weak regions (center-right) in the presence of noise. Weak regions are typically located wheretwo parts of the shape are close and the topology is ambiguous, such as the arm-leg contacts in the sitting woman. Missing parts withunambiguous topology, such as her right shoulder, are automatically smoothly completed.

[Sharf et al. SIG’07] Segmentation Model Matching ReconstructionRGBD Image

#2

#3

#1

Figure 2: Semantic modeling system pipeline. The input images are first segmented into semantical regions, and each segmented out regionis replaced by its similar 3D models in our database. The system progressively reconstructs and registers the whole scene using consecutivelycaptured RGBD-images.

framework in [Furukawa and Ponce 2010] is able to reconstruct adense 3D points, it is still difficult to apply it for indoor scene mod-eling to generate dense depth maps since there exist a large amountof textureless areas (e.g., walls) in indoor scenes. Based on theManhattan world assumption, Furukawa et al. [2009a] reconstructdepth maps from 2D images of indoor scenes. Their method is lim-ited to axis-aligned planes. Furukawa et al. [2009b] also propose animage-based rendering method for indoor scenes by building axis-aligned plane geometry proxies.

With the fast development of consumer-level depth cameras, it ispossible to capture dense depth maps of an indoor scene. Researchworks on 3D modeling with consumer-level depth cameras focus onthe fusion of depth maps or the combination of color and depth in-formation. A prominent example is the Kinect Fusion system [Izadiet al. 2011]. It uses a volumetric representation to fuse the depthmap. Henry et al. [2012] explore the registration of depth maps us-ing both color and depth information. To solve the technical chal-lenges in automatic depth map fusion algorithms, Du et al. [2011]propose to integrate on-line user interaction in indoor scene model-ing.

Our work aims to reconstruct not only the geometry information ofan indoor scene but also its semantic representation from sparselycaptured depth images. The reconstruction result of our algorithmconsists of semantic geometry objects, such as chair, sofa, wall-s and so on. It facilitates the usage of reconstructed indoor scenemodels in high-level applications, such as furniture layout and so-cial gaming. Two concurrent efforts have also been made to solvethis challenging indoor scene modeling problem [Nan et al. 2012;Kim et al. 2012].

3D shape matching. Much research has been conducted on find-ing models similar to an input 3D model in a database [Funkhouseret al. 2003; Tangelder and Veltkamp 2008; Bronstein et al. 2011].Unfortunately, these methods cannot be directly applied in our set-ting, where only a single view of 3D models is present in the cap-tured RGBD image. Therefore, we pose the problem of retriev-ing similar models of a segmented object in the image as an objectinstance recognition problem. Spin images [Johnson and Hebert1999] has been widely used in 3D shape recognition problem-

s [Golovinskiy et al. 2009]. Instead of using hand-designed fea-tures, Bo et al. [2011] proposed depth kernel descriptors for objectrecognition, and Blum et al. [2011] proposed unsupervised featureextraction based on k-means clustering for object recognition on theRGBD object dataset in [Lai et al. 2011].

Our problem is unique: we do not require a one-to-one mappingbetween the 3D models in the database and the segmented object-s. Since the texture information in our model database usuallydoes not match the texture in the captured RGBD images, texturebased features are not used in our application. We thus use the ran-dom regression forest in [Fanelli et al. 2011] since it can achievehigh recognition accuracy only using depth information. We ex-tend the algorithm to multi-instance object recognition for indoorscene modeling.

3 Interactive Context-aware Image Segmen-

tation and Labeling

Given RGBD indoor images captured by the user, our goal is tosegment the images into regions with a semantic object label, suchas floor, chair and monitor. To this end, we develop an interactivecontext-aware image segmentation algorithm to fulfill this task. Ithas two important features: (a) It integrates the benefits of both au-tomatic and interactive approaches to achieve a high-quality seg-mentation result. User interaction, which is implemented via astroke-based interface, is minimized since it is necessary only whenthe automatic segmentation result is not satisfactory. (b) It reflectsthe context of the current scene according to the segmentation resultthrough dynamically updating the appearance and geometry mod-el learned from the NYU indoor scene image database [Silbermanand Fergus 2011]. Precisely, the statistical information from thesegmentation result is integrated to make the classifier adapt to thespecific scene and thus improve labeling accuracy.

In image segmentation, each pixel is assigned a semantic label. Theten class labels we support for the purpose of indoor scene modelingare sofa, table, monitor, wall, chair, floor, bed, cabinet, ceiling andbackground. They are all common objects in indoor scenes, except

[Shao et al. SIGA’12]



• Watertightness





• Shape Class

24


Inputs

• The direction perpendicular to the tangent space at each point

• Represents a localized approximation to the surface

• Orientation: all normals are consistently pointing inside/outside of the surface

25

Normals


Inputs

26

Normal Orientation• Provides useful cues about the surface

• Distinguish thin sheets from a single sheet

• Challenging research problem in its own right: [Hoppe et al. SIG’92, Huang et al. SIGA’09, Liu & Wang SMI’10]


Inputs

27

Sensitivity to Normal Orientation

[Hoppe et al. SIG’92] [Kazhdan et al. SGP’06] [Kazhdan et al. SGP’06][Liu & Wang SMI’10]


Inputs

28

Scanner Information• Provides a variety of useful information

• 2D lattice structure

• Estimate sampling density

• Outliers

• Confidence of a point

• Line of sight


Inputs

28

Scanner Information

noise?

• Provides a variety of useful information

• 2D lattice structure

• Estimate sampling density

• Outliers

• Confidence of a point

• Line of sight


Inputs

29

RGB Imagery• Complements depth capture, particularly when data is missing

• Fuse features between the depth and RGB image


Inputs

29

RGB Imagery

Figure 12: The result gallery generated using our part assembly approach for structure recovery from a single-view scan of man-madeobjects acquired by the Kinect system. Parts borrowed from different repository models are shown in different colors.

• Complements depth capture, particularly when data is missing



Inputs

29

RGB Imagery




• Complements depth capture, particularly when data is missing





• Shape Class

30


Shape Class

31

CAD Models


Shape Class

32

Man-made shapes


Shape Class

33

Organic shapes


Shape Class

34

Architectural shapes


Shape Class

35

Indoor environments


Questions?

36


Questions?

36

Point Cloud Artifacts

• Nonuniform Sampling

• Noise

• Outliers

• Misalignment

• Missing Data


Questions?

36

Point Cloud Input

• Normals

• Oriented Normals

• Scanner Information

• RGB Imagery



• Noise

• Outliers

• Misalignment

• Missing Data


Questions?

36

Point Cloud Input

• Normals



• RGB Imagery

Shape Class

• CAD Models

• Man-made Shapes

• Organic Shapes

• Architectural Shapes

• Indoor Environments



• Noise

• Outliers

• Misalignment

• Missing Data


Questions?

36

Point Cloud Input

• Normals



• RGB Imagery

Shape Class

• CAD Models

• Man-made Shapes

• Organic Shapes

• Architectural Shapes

• Indoor Environments

These factors inform prior development in surface reconstruction



• Noise

• Outliers

• Misalignment

• Missing Data

state of the art in surface reconstruction from point clouds · state of the art in surface...

Documents