an interactive approach for functional prototype recovery ... · annotation of high-level relations...

Computational Visual MediaDOI 10.1007/s41095-016-0032-x Vol. 2, No. 1, March 2016, 87–96

Research Article

An interactive approach for functional prototype recovery froma single RGBD image

Yuliang Rong1, Youyi Zheng2, Tianjia Shao1(�), Yin Yang3, and Kun Zhou1

c© The Author(s) 2016. This article is published with open access at Springerlink.com

Abstract Inferring the functionality of an objectfrom a single RGBD image is difficult for two reasons:lack of semantic information about the object, andmissing data due to occlusion. In this paper, wepresent an interactive framework to recover a 3Dfunctional prototype from a single RGBD image.Instead of precisely reconstructing the object geometryfor the prototype, we mainly focus on recoveringthe object’s functionality along with its geometry.Our system allows users to scribble on the image tocreate initial rough proxies for the parts. After userannotation of high-level relations between parts, oursystem automatically jointly optimizes detailed jointparameters (axis and position) and part geometryparameters (size, orientation, and position). Suchprototype recovery enables a better understanding ofthe underlying image geometry and allows for furtherphysically plausible manipulation. We demonstrate ourframework on various indoor objects with simple orhybrid functions.

Keywords functionality; cuboid proxy; prototype;part relations; shape analysis

1 Introduction

That form ever follows function. This is thelaw.

Louis Sullivan

1 State Key Lab of CAD&CG, Zhejiang University,Hangzhou 310058, China. E-mail: Y. Rong, [email protected]; T. Shao, [email protected] (�); K.Zhou, [email protected].

2 ShanghaiTech University, Shanghai 200031, China. E-mail: [email protected].

3 The University of New Mexico, Albuquerque, NM 87131,USA. E-mail: [email protected].

Manuscript received: 2015-12-01; accepted: 2015-12-09

With the popularization of commercial RGBDcameras such as Microsoft’s Kinect, people caneasily acquire 3D geometry information with anRGB image. However, due to occlusion and noise,recovering meaningful 3D contents from singleRGBD images remains one of the most challengingproblems in computer vision and computer graphics.Over the past years, much research effort has beendevoted to recovering high-quality 3D informationfrom RGBD images [1, 2]. Most of these approaches,starting either from a single image or multipleimages, are dedicated to faithfully recovering the3D geometry of image objects, while their semanticrelations, underlying physical settings, or evenfunctionality are overlooked. More recently, researchhas explored use of high-level structural informationto facilitate 3D reconstruction [3–5]. For example,Shao et al. [3] leverage physical stability to suggestpossible interactions between image objects andobtain a physically plausible reconstruction ofobjects in RGBD images. Such high-level semanticinformation plays an important role in constrainingthe underlying geometric structure.

Functionality is the center of object design andunderstanding. Objects in man-made environmentsare often designed for single or multiple intendedfunctionalities (see Fig. 1). That form ever follows

Fig. 1 Objects in man-made environments are often designed forsingle or multiple intended functionalities.

87

88 Y. Rong, Y. Zheng, T. Shao, et al.

function is the law of physical manufacturing [6].In this paper, we develop an interactive systemto recover functional prototypes from a singleRGBD image. Our goal is to allow a novice userto be able to quickly lift image objects into 3Dusing 3D prototypes, using just a few high-levelannotations of joint types and geometric/functionalrelations; the user can meanwhile explicitly exploreand manipulate an object’s function. We focuson prototypes with simple proxies (e.g., cuboids)representing their parts as a means to alleviate thedifficulties of precise 3D reconstruction which is aharder problem. By taking physical functionalityinto consideration, we may gain a much morefaithful interpretation of the underlying objects.The functional properties can then be used forapplications like in-context design and manipulation.

It is a challenging problem to infer objectfunction just from user annotated joint types andgeometric/functional relations. Our system shouldautomatically optimize the detailed joint parameters(axis and position) in order to make the partsmove correctly, whereas this task is typically donein CAD software by the user repeatedly adjustingparameters. Furthermore, initial proxies from user-segmented depths are often rough with incorrectorientation and position, and may be incompletebecause of occlusion. Hence initial proxies often failto satisfy the functional relations such as A covers B.Therefore our system should also optimize the proxyparameters (size, orientation, and position), in orderto make parts satisfy functional relations.

Our method starts with a single RGBD image.The user segments the image object into parts byscribbling on the image using simple strokes orpolygons. Then each segmented part is assembledusing a 3D proxy. We use simple cuboids in thispaper [7]. Given the initial proxies, our systemthen allows the user to annotate the joint types andfunctional/geometric relations between parts. In akey stage, our algorithm simultaneously optimizesthe detailed joint parameters (axis and position)and the proxy parameters (size, orientation, andposition). Finally, a functional prototype is producedwith moving parts satisfying the user annotatedrelations.

We have tested our system on a variety of man-made hybrid functional objects taken from various

sources. Our results show that even using onlya few user annotations, the proposed algorithmis capable of faithfully inferring geometry alongwith appropriate functional relations for the objectparts. In summary, this paper makes the followingcontributions:• Identifying and characterizing the problem

of integrating functionality into image-basedreconstruction;• Simultaneous optimization of detailed joint

and geometry parameters from user’shigh-level annotations of joint types andfunctional/geometric relations;• An interactive tool for functional annotation

which has been tested on a variety of indoorscene images and physical designs.

2 Related work

Proxy-based analysis. A significant amount ofwork has leveraged proxies to understand objects orscenes. Li et al. [8] and Lafarge et al. [9] considerglobal relationships as constraints to optimizeinitial RANSAC-based proxies to produce structuredoutputs; similarly, Arikan et al. [10] use priorrelations plus user annotations to create abstractedgeometry. For scene analysis, many approachesencode input scenes as collections of planes, boxes,cylinders, etc., and study their spatial layout [11–16]. Recently, proxies have frequently been usedin functionality analysis of a design. Umetani etal. [17] use physical stability and torque limitsfor guided furniture design in a modeling andsynthesis setting. Shao et al. [18] create 3D proxymodels from a set of concept sketches that depicta product from different viewpoints, with differentconfigurations of moving parts. Our work is inspiredby Koo et al. [7], who annotate cuboids with high-level functional relationships to fabricate physicalwork-alike prototypes. Unlike their algorithm, ourframework does not require explicitly creating jointpositions, as we consider a larger search space toautomatically infer the joint and part parameters.To our knowledge, we are the first to focus onproxy-based functionality recovery from a singleRGBD image, in particular recovering how the objectworks by jointly optimizing both part geometry andfunctional relationships based on user annotations.

An interactive approach for functional prototype recovery from a single RGBD image 89

Constraint-based modeling. Our work isrelated to constraint-based modeling research inthe graphics and CAD communities. Similar workto ours involves the automatic determination ofrelevant geometric relationships between parts forhigh-level editing and synthesis of 3D models [5, 19–21]. Previous mechanical engineering research hasused declarative methods for specifying relevantgeometric constraints for a mechanical design [22,23]. Some professional CAD software like AutoCADand SolidWorks contains constraint-based modelingmodules, but users are required to manuallyadjust the low-level part/joint parameters tospecify relationships. In contrast, our system canautomatically interpret user annotated high-levelfunctionality to give specific geometric constraints.

3D modeling from a single RGBD image.Much effort has been devoted to obtaining high-quality geometry information from a single RGBDimage [1, 24]. To recover structural information,Shen et al. [4] extract suitable model parts froma database, and compose them to form high-quality models from a single RGBD image. Shaoet al. [3] use physical stability to recover unseenstructures from a single RGBD image using cuboids.However, their techniques focus on creating static3D geometry and structure, whereas our goal is toproduce models with correctly moving parts.

3 Overview

As illustrated in Fig. 2, given a single RGBD image,we first let the user scribble strokes over an imageobject to separate out its functional parts. Theseparts, being either a semantic component or anadded object, are used during function recovery.

To segment the parts, we use a depth-augmentedversion of the GrabCut segmentation method [25],as Ref. [3] does. Optionally, if the color and depthare too similar, making it difficult to separate theparts with GrabCut, the user may use a polygontool as in PhotoShop to perform segmentation(see the accompanying video in the ElectronicSupplementary Material (ESM)). We assemble a setof cuboid proxies, one fitted to each individual part.The user then annotates high-level relations betweenthese cuboids. The relations are of three kinds:joint type relations (e.g., hinge, sliding), functionalrelations (e.g., covers, fits inside, supports, flush,connects with) [7], and geometric relations (e.g.,equal size, symmetry).

Given the user annotated relations, in a keystep, our method recovers each cuboid’s orientation,position, and size along with the joint parametersusing a combined optimization approach. We use acombined optimization strategy because the cuboidparameters are always coupled with the jointparameters: given a set of joints, the cuboidgeometry must be adjusted to satisfy the functionalconstraints.

Optimization is done using a two-stage samplingstrategy. In the first stage, our algorithm samplespossible cuboid edges as joint candidates [18] forthe specified joint type. Given one set of possiblejoint candidates, the cuboid orientation is alignedand the cuboid position is refined, by adjusting thecorresponding joint edges. We assume that the jointmust be snapped to the nearest cuboid face and beparallel to the nearest cuboid edge (as in Ref. [7]).

Having found one set of adjusted joints withcuboid orientations and positions, our methodfurther samples a set of possible candidate rest

Hinge junction

Equal size

Sliding junctionSliding junction

Hinge junction

Cover

Equal size

Just fit inside

Fig. 2 Algorithmic pipeline. Given the input RGBD image (left), our system generates initial proxy cuboids (mid-left) from parts segmentedby the user with strokes or polygon tools. The user then annotates a set of high-level relations between the proxies including joint typesand geometric/functional relations (mid-right). Finally our system simultaneously optimizes joint parameters (axis and position) and partparameters (orientation, position, and size) to obtain a functional prototype with parts moving as the user expects (right).


configurations for the cuboids. A rest configurationis a state where the object is in a closedstate [7]. Because the cuboid size has not yetbeen definitively determined, the system does notknow which state is the closed state. Thus we samplepossible candidates for the rest configurations, asshown in Fig. 6. For each possible rest configuration,we optimize the cuboid size parameters according tothe user annotated functional/geometric relationsas in Ref. [7]. Finally, the optimized cuboids whichbest match the initial point cloud are selected,and the best prototype with best joint and cuboidparameters is produced. We next describe thealgorithm in detail.

4 Algorithm

Our method takes as input an RGBD image of afunctional object. By functional we mean objectshaving particular moving parts, such as a door whichopens by rotating, a sliding drawer, etc. Such objectsare very commonly seen in our daily life, for instance,rolling chairs, folding tables, printers, seesaws, etc.

Initial cuboid generation. Given the inputRGBD image, our first task is to anchor theobject’s functional parts. Automatically identifyingimage objects and object parts in RGBD imageshas been explored in recent works, but withoutprior knowledge, such methods do not yet workwell enough for our purposes. We resort to aninteractive solution. As in Ref. [3], we let theuser scribble on the image object to specify objectparts. In particular, we allow the user to drawfree strokes over parts to indicate a segment (part).We apply a depth-augmented GrabCut algorithm [3]to the underlying point cloud along with its pixeland adjacency information. Optionally, if the colorand depth are too similar, making it difficult toseparate the parts with GrabCut, the user mayuse a polygon tool as in PhotoShop to performsegmentation. We then run the efficient RANSACalgorithm [26] on the seegmented points to generatecandidate planes. The largest plane is selected as theprimary plane, and the second largest plane is madeorthogonal to the primary one. We extract initialcuboids determined by these orthogonal directions(the third direction is the cross product of thetwo plane normals). Figure 3 illustrates the process

Fig. 3 Initial proxy generation. The user is allowed to scribblestrokes on the image (left); based on the scribbles, depth-augmentedGrabCut is applied to segment the input object into different parts(middle). Initial cuboids are then fitted to the corresponding points(right).

of generating the initial cuboids. Note that thegenerated cuboid may have erroneous orientation,position, and size. In subsequent steps, our goal is tosimultaneously optimize these parameters along withthe joint parameters so that the extracted cuboidsform a prototype whose functionality closely followsthe image object.

Relation annotation. Let the set of initialcuboids be (B1, . . . , BN ). An important first stepis for the user to annotate the high-level relationsbetween the cuboids. To this end, we define threecategories of relations. Category I comprises jointrelations (e.g., A has a hinge relation w.r.t. B).Category II comprises functional relations (e.g., Acovers B), while Category III comprises geometricrelations (e.g., symmetry, equal size). In each case,the user selects a pair of cuboids and then indicatesthe relationship.

To further classify the relations, we define twomain types of joint relations: hinges and slidingjoints. For functional relations, following Ref. [7],we define the following function types: A covers B,A fits inside B, A supports B, A is flush with B, andA is connected to B. The main geometric relationsare symmetry and equal size. These relationsimpose different geometric constraints during thefollowing optimization stage. Some relations mightbe dependent on each other; for example, if both Aand C cover B, A is geometrically constrained w.r.t.B and C. Figure 4 shows the joint types and some

Fig. 4 User annotated joint types and some typical functionalrelations. From left to right: hinge joint, sliding joint, exactly covers,just fits, and supports.


typical types of functional relations. Note that unlikeRef. [7], we do not need to explicitly specify jointpositions and axes as well as cuboid orientations andpositions. Instead, we optimize these parameters ina combined manner.

Combined optimization of cuboids andjoints. We now detail our cuboid optimizationalgorithm. Our goal is to simultaneously optimizethe cuboids’ orientations and shape parameters (i.e.,positions and sizes), as well as the detailed jointparameters, in accordance with the user annotatedrelations. The optimized cuboid configuration shoulddeviate little from the input point cloud and move asthe user expects. Given the input point cloud I andinitial cuboids B = (B1, . . . , BN ), along with the userannotated joint types J = (J1, . . . , JM ), functionalrelations F = (F1, . . . , FP ), and geometric relationsG = (G1, . . . , GQ), we want to obtain the best jointparameters ΘΘΘ∗ = (Θ∗1 , . . . , Θ∗M ) for the joint typesJ along with the best cuboids B∗ = (B∗1 , . . . , B∗N ),satisfying the functional relations F and geometricrelations G. We do so using the formulation:argminB,ΘΘΘ

E(B,ΘΘΘ, I) s.t. B,ΘΘΘ satisfy J ,F ,G (1)

Here E(B,ΘΘΘ, I) measures the deviation of theoptimized cuboid configuration from the input pointcloud, defined as

E(B,ΘΘΘ, I) =∑j

∑k

dist(Bj − pkj ) (2)

where∑k dist(Bj − pkj ) measures the deviation of

cuboid Bj from its associated points pkj .The challenge is how to enforce the annotated

relations as geometric constraints while ensuring thecuboids respect the input point cloud. Since theannotated relations are high-level specifications, thisleads to a large optimization search space due tothe potential ambiguities arising from the rathergeneral annotations. Another challenge is that thecuboid parameters are highly coupled with the jointparameters. That is, given a set of joints, the cuboidgeometry should change accordingly to satisfy thefunctional constraints. Thus we cannot optimizethe parameters locally and separately, but mustinstead do so in a global manner. To solve theabove challenges, we use a multi-stage optimizationparadigm which first populates the solution spacewith a two-step sampling algorithm, and thenjointly optimize the cuboid parameters and jointparameters.

In the first stage, we sample possible jointparameters, i.e., axial position and orientation. Letus denote the set of joint types as (J1, . . . , JM ), andthe parameters we wish to estimate as (Θ1, . . . , ΘM ).We start by building a joint configuration graph. Foreach cuboid we create a graph node and for eachjoint type Ji, we create multiple graph connections,with each connection associated with a candidateparameter Θli for Ji. If A forms a hinge relation withB, each cuboid edge of A can be a candidate hingeaxis. We choose only those cuboid edges which areclose to B. More specially, we only choose the edgesparallel to the face if there is also a cover relation,and only choose the edges perpendicular to the faceif there is a fit inside or support relation. This leadsto a configuration graph where any traversal pathof the graph represents a possible configuration ofjoints. Figure 5 shows such a graph. Algorithm 1gives the pseudo-code for building the graph.

Given the joint configuration graph, for eachjoint configuration we optimize each cuboid’sorientation, position, and size based on annotatedfunctional/geometric relations. The cuboid’sorientation and position are firstly adjusted basedon the current candidate joint configuration, byadjusting the corresponding joint edges. We assumethat the joint must be snapped to the nearestcuboid face and be parallel to the nearest cuboidedge (following Ref. [7]). Then we optimize thecuboid’s size to satisfy the functional/geometricrelations given by the current joint configuration.Note that the functional relations typically indicatethe geometry of the cuboids in a closed configuration(i.e., a rest configuration [7]. For instance, if Acovers B, this typically means that one face ofA is rotated about the hinge joint to be in close

Hinge

Slide

n

qp

m

Fig. 5 Junction configuration graph. Each cuboid showncorresponds to a node with the same color. Each annotated joint typecorresponds to multiple connections between nodes. One connectionis associated with one candidate joint parameter.


Algorithm 1: Build the junction configuration graphInput: N initial cuboids (B1, . . . , BN ); M junctions (J1, . . . , JM )

with unknown parameters (Θ1, . . . , ΘM ).Output: Multi-connection junction graph G := (V,E) in which

each edge eji corresponds to a parameter Θj

i for Ji.G← ∅for i = 1 to N doVi ← Bi

end for/* Build multi-connections between nodes */for i = 1 to M doBc ← child cuboid of Ji

Bp ← parent cuboid of Ji

l← 1/* Test each edge of the child cuboid */for j = 1 to 12 doEj ← j-th edge of BcDj ← direction of Ej

Cj ← center of Ej

for k = 1 to 6 doFk ← k-th face of BpNk ← normal of Fk

if dist(Ej , Fk) < εd and (abs(Dj ·Nk) < εa orabs((Dj ·Nk)− 1) < εa) thenΘl

i ← (Cj , Dj) // set candidate parameter for Ji

eli ← Θl

i // add a connection eli

l← l + 1end if

end forend for

end for

Fig. 6 Possible rest poses for the hinge joints. As we do notknow which face of the cabinet door covers the cabinet, we rotate thehinge joints to sample a set of rest configurations to suggest possiblecovering faces.

agreement with a face of B (Fig. 6). Since we do notknow which face covers B, we enumerate the possiblecuboid faces to sample a set of rest configurations(see Fig. 6) and for each rest configuration weoptimize the cuboid parameters. Specifically, givena rest configuration of cuboids, we employ theoptimization strategy in Ref. [7] to optimize thecuboid parameters (B∗1 , . . . , B∗N ). We then computethe optimization cost from Eq. (2). Finally, theconfiguration which best matches the point cloudis selected as the chosen configuration and theoptimized cuboids are then computed. The overallalgorithm is detailed in Algorithm 2.

5 Results

We show use of our system to recover functionality

Algorithm 2: Optimize cuboids and junctionsInput: Input point cloud I; N initial cuboids B = (B1, . . . , BN );

junction configuration graph G; functional relations F ;geometric relations G.

Output: N optimized cuboids B∗ = (B∗1 , . . . , B∗N ); M optimizedjunction parameters ΘΘΘ∗ = (Θ∗1 , . . . , Θ∗M ).

/* Sample candidate junction parameters from G and accordinglyoptimize cuboid orientation, position and size */err ← INF // deviation of cuboids from input point cloudwhile true do

Gather a connection combination (ek1 , . . . , e

lM ) from G

if no more connection combinations thenbreak

end ifCreate junctions with parameters ΘΘΘ′ = (Θk

1 , . . . , ΘlM ) from

(eki , . . . , e

lM )

adjust the cuboids positions and orientations by snapping thejunction edges/* Calculate possible angles for rest configurations */for i = 1 to M do

calculate candidate angles (α1i , . . . , α

wi ) to parallelize parent

and childend for/* Sample possible rest configurations */while true do

Gather an angle combination (αu1 , . . . , α

vM )

if no more angle combinations thenbreak

end ifTransform to rest configuration with (αu

1 , . . . , αvM )

Optimize the cuboids sizes to satisfy F and G, giving asolution B′ = (B′1, . . . , B′N )if E(B′,ΘΘΘ′, I) < err then

err ← E(B′,ΘΘΘ′, I)(B∗1 , . . . , B∗N )← (B′1, . . . , B′N )(Θ∗1 , . . . , Θ∗M )← (Θk

1 , . . . , ΘlM )

end ifend while

end while

prototypes for 6 different objects (see Fig. 7). Thefirst 4 examples (cabinet, drawer, firebox, and chair)are real RGBD images captured with MicrosoftKinect, while the last 2 examples (toolbox anddining table) are synthetic depth data generatedfrom existing 3D designs. See the accompanyingvideo in the ESM for how the various parts move andfit together. Creating a single functional prototypetakes 30 to 300 s for these examples. The timefor user interaction (segmenting points with strokesand specifying part relationships, plus the waitingtime for plane detection for initial cuboid generation)ranges from 27 to 108 s, while the optimization timevaries greatly, from 1 to 241 s, depending on thesize of the sampling space for joint parameters andrest poses. Experimental statistics (including thenumbers of annotated hinge joints, sliding joints,functional relations, and geometric relations) arelisted in Table 1.


(1) C

abin

et(2

) Dra

wer

(3) F

ireb

ox(4

) Cha

ir(5

) Too

lbox

(6) D

inin

g ta

ble

Fig. 7 Experimental results. From left to right: the input RGBD image, initial cuboids, optimized cuboids and joints, and how parts moveand fit after the optimization (3 configurations).

Table 1 Statistics for examples: numbers of hinge and sliding joints,numbers of functional and geometric constraints, and time taken forinteraction and optimization

Model Hinge Slide Funct. Geom. Interact OptimizeCabinet 2 2 4 2 62 s 18 sDrawer 0 2 2 1 29 s 1 sFirebox 2 0 2 0 27 s 16 sChair 2 0 3 0 90 s 2 s

Toolbox 1 3 6 0 108 s 3 sDining table 4 0 6 2 55 s 241 s

As shown in Fig. 2, although the geometry ofour prototypes may appear simple, the relationshipsbetween the moving parts are often complex.Manually adjusting the geometry and relation

parameters would be very time and labor consuming.Our system automatically infers the joint parameters(position and axis) and the geometry parameters(size, position, and orientation) by optimizing themsimultaneously under the user annotated high-levelconstraints. Satisfactory part parameters and jointparameters were obtained in all our experiments. Forexample, in Fig. 7(1), our algorithm automaticallyplaces the hinge joints on the correct edgesof the cabinet doors, adjusts their orientationsappropriately by aligning the hinge joints to thenearest cabinet face, and makes them parallel tothe nearest cabinet edges. The sizes of the doorsare also optimized to be of equal size and cover the


cabinet. The drawers in Figs. 7(1) and 7(2) have thedesired orientations with their sliding joints alignedwith the cabinet, and have sizes which properly fitinside the cabinet and are equal. In Fig. 7(3), thelid and the front door are both optimized to justcover the firebox. In the chair example (Fig. 7(4)),due to occlusion, the initial cuboids for the leg andthe armrest have smaller sizes than they should, butour algorithm successfully extends the leg to supportthe seat, and extends the armrests to connect withthe back. Similarly, the occluded leg in Fig. 7(5) isextended to support the box and has the same sizeas the other legs. In Fig. 7(6), the orientation andsize of the two doors are optimized to support thetable top, and the orientation of the top is optimizedto be horizontal.

User study. To further evaluate whether ourapproach can recover correct functional prototypes,we showed our system to 20 students. 5 werecomputer science undergraduates; 4 students weremaster candidates in industrial design. The otherswere 8 master candidates and 3 Ph.D. candidatesin computer science. We showed them the capturedRGBD images and asked them to imagine how theobjects work. These students then used our systemto add annotations to the pre-generated initialcuboids based on their imagination. All studentsreported that our system successfully recoveredfunctional prototypes in which the parts moved asthey expected. Furthermore, the optimized part

geometry also met their expectations. One exceptionwas that 6 students said they imagined the hingejoint on the cabinet door in Fig. 7(1) to be exactlyat the edge of the cabinet, while our optimizationdid not consider it to be the best location.

Comparison with real objects and 3D designmodels. We also checked the recovered prototypesagainst the captured real objects and 3D designmodels. As illustrated in the top 2 rows in Fig. 8,the generated prototypes have similar functionalityto the real objects, and have parts which aftermovement give almost the same configurations asthe real ones. Furthermore, the optimized simplecuboids approximate the real geometry well, withalmost the same size, orientation, and position. Wealso compared our recovered prototypes with 3Ddesign models whose joints were added and adjustedmanually in Autodesk 3DS Max (bottom row inFig. 8). We can see the prototypes recovered fromthe user’s high-level annotations have very similarfunctionality to the manually designed models.

6 Conclusions

In this work, we have presented a novel approach torecovering a functional prototype from a user’s high-level annotations of relationships. After indicatingjoint types and other functional/geometric relations,the joint parameters and part geometry parametersare simultaneously optimized. This interface allows

Fig. 8 Top two rows: comparison with input data; bottom row: comparison with the 3D design model. Our system faithfully recovers thefunctionality the user expects.


users to focus on the functionality of the targetobject rather than working with low-level geometryand joint parameters. The results demonstrate thatour system can generate functional models froma small number of user annotations. In the userstudy, the recovered prototypes worked correctly, asusers expected. The comparison between the realobjects and 3D design models further demonstratesthe feasibility of our system.

Limitations and future work. The mainlimitation of our approach is that we usecuboids as proxies to approximate part geometry.While compositions of cuboids are sufficient forrepresenting the functionality of many products,users often prefer higher fidelity geometry to betterunderstand the geometry and relationships. Roughproxies may also cause inaccurate reconstruction.Similarly, the restricted set of joint types is anotherlimitation. In the future, we will add other primitivesfor part proxies, such as cylinders and spheres. Wealso plan to integrate more joint types betweenparts, such as ball joints and simple mechanicallinks. The current optimization framework mayneed to be modified to handle more geometry andjoint types. Another future direction is to considerother high-level functional constraints betweenparts. Exploring further high-level relationshipscould allow more sophisticated functional models.

Electronic Supplementary Material Supplementarymaterial is available in the online version of this article athttp://dx.doi.org/10.1007/s41095-016-0032-x.

References

[1] Han, Y.; Lee, J.-Y.; Kweon, I. S. High quality shapefrom a single RGB-D image under uncalibrated naturalillumination. In: Proceedings of IEEE InternationalConference on Computer Vision, 1617–1624, 2013.

[2] Izadi, S.; Kim, D.; Hilliges, O.; Molyneaux, D.;Newcombe, R.; Kohli, P.; Shotton, J.; Hodges,S.; Freeman, D.; Davison, A.; Fitzgibbon, A.KinectFusion: Real-time 3D reconstruction andinteraction using a moving depth camera. In:Proceedings of the 24th Annual ACM Symposiumon User Interface Software and Technology, 559–568,2011.

[3] Shao, T.; Monszpart, A.; Zheng, Y.; Koo, B.; Xu, W.;Zhou, K.; Mitra, N. J. Imagining the unseen: Stability-based cuboid arrangements for scene understanding.ACM Transactions on Graphics Vol. 33, No. 6, Article

No. 209, 2014.[4] Shen, C.-H.; Fu, H.; Chen, K.; Hu, S.-M. Structure

recovery by part assembly. ACM Transactions onGraphics Vol. 31, No. 6, Article No. 180, 2012.

[5] Zheng, Y.; Chen, X.; Cheng, M.-M.; Zhou, K.; Hu,S.-M.; Mitra, N. J. Interactive images: Cuboid proxiesfor smart image manipulation. ACM Transactions onGraphics Vol. 31, No. 4, Article No. 99, 2012.

[6] Sullivan, L. H. The tall office building artisticallyconsidered. Lippincott’s Magazine 57, 1896.

[7] Koo, B.; Li, W.; Yao, J.; Agrawala, M.; Mitra, N. J.Creating works-like prototypes of mechanical objects.ACM Transactions on Graphics Vol. 33, No. 6, ArticleNo. 217, 2014.

[8] Li, Y.; Wu, X.; Chrysanthou, Y.; Sharf, A.;Cohen-Or, D.; Mitra, N. J. GlobFit: Consistentlyfitting primitives by discovering global relations. ACMTransactions on Graphics Vol. 30, No. 4, Article No.52, 2011.

[9] Lafarge, F.; Alliez, P. Surface reconstruction throughpoint set structuring. Computer Graphics Forum Vol.32, No. 2pt2, 225–234, 2013.

[10] Arikan, M.; Schwarzler, M.; Flory, S.; Wimmer, M.;Maierhofer, S. O-snap: Optimization-based snappingfor modeling architecture. ACM Transactions onGraphics Vol. 32, No. 1, Article No. 6, 2013.

[11] Gupta, A.; Efros, A. A.; Hebert, M. Blocks worldrevisited: Image understanding using qualitativegeometry and mechanics. In: Lecture Notes inComputer Science, Vol. 6314. Daniilidis, K.; Maragos,P.; Paragios, N. Eds. Springer Berlin Heidelberg, 482–496, 2010.

[12] Gupta, A.; Hebert, M.; Kanade, T.; Blei, D. M.Estimating spatial layout of rooms using volumetricreasoning about objects and surfaces. In: Advances inNeural Information Processing Systems 23. Lafferty,J.; Williams, C.; Shawe-Taylor, J.; Zemel, R.; Culotta,A. Eds. Curran Associates, Inc., 1288–1296, 2010.

[13] Del Pero, L.; Bowdish, J.; Fried, D.; Kermgard,B.; Hartley, E.; Barnard, K. Bayesian geometricmodeling of indoor scenes. In: Proceedings ofIEEE Conference on Computer Vision and PatternRecognition, 2719–2726, 2012.

[14] Jia, Z.; Gallagher, A.; Saxena, A.; Chen, T. 3D-based reasoning with blocks, support, and stability. In:Proceedings of IEEE Conference on Computer Visionand Pattern Recognition, 1–8, 2013.

[15] Jiang, H.; Xiao, J. A linear approach to matchingcuboids in RGBD images. In: Proceedings ofIEEE Conference on Computer Vision and PatternRecognition, 2171–2178, 2013.

[16] Zheng, B.; Zhao, Y.; Yu, J. C.; Ikeuchi, K.; Zhu,S.-C. Beyond point clouds: Scene understanding byreasoning geometry and physics. In: Proceedings ofIEEE Conference on Computer Vision and PatternRecognition, 3127–3134, 2013.

[17] Umetani, N.; Igarashi, T.; Mitra, N. J. Guidedexploration of physically valid shapes for furniture


design. ACM Transactions on Graphics Vol. 31, No. 4,Article No. 86, 2012.

[18] Shao, T.; Li, W.; Zhou, K.; Xu, W.; Guo, B.; Mitra,N. J. Interpreting concept sketches. ACM Transactionson Graphics Vol. 32, No. 4, Article No. 56, 2013.

[19] Bokeloh, M.; Wand, M.; Seidel, H.-P.; Koltun, V. Analgebraic model for parameterized shape editing. ACMTransactions on Graphics Vol. 31, No. 4, Article No.78, 2012.

[20] Gal, R.; Sorkine, O.; Mitra, N. J.; Cohen-Or,D. iWIRES: An analyze-and-edit approach to shapemanipulation. ACM Transactions on Graphics Vol. 28,No. 3, Article No. 33, 2009.

[21] Xu, W.; Wang, J.; Yin, K.; Zhou, K.; van de Panne,M.; Chen, F.; Guo, B. Joint-aware manipulation ofdeformable models. ACM Transactions on GraphicVol. 28, No. 3, Article No. 35, 2009.

[22] Daniel, M.; Lucas, M. Towards declarative geometricmodelling in mechanics. In: Integrated Design andManufacturing in Mechanical Engineering. Chedmail,P.; Bocquet, J.-C.; Dornfeld, D. Eds. SpringerNetherlands, 427–436, 1997.

[23] Yvars, P.-A. Using constraint satisfaction fordesigning mechanical systems. International Journalon Interactive Design and Manufacturing Vol. 2, No.3, 161–167, 2008.

[24] Zhang, Q.; Ye, M.; Yang, R.; Matsushita, Y.; Wilburn,B.; Yu, H. Edge-preserving photometric stereo viadepth fusion. In: Proceedings of IEEE Conference onComputer Vision and Pattern Recognition, 2472–2479,2012.

[25] Rother, C.; Kolmogorov, V.; Blake, A. “GrabCut”:Interactive foreground extraction using iterated graphcuts. ACM Transactions on Graphics Vol. 23, No. 3,309–314, 2004.

[26] Schnabel, R.; Wahl, R.; Klein, R. Efficient RANSACfor point-cloud shape detection. Computer GraphicsForum Vol. 26, No. 2, 214–226, 2007.

Yuliang Rong is currently a seniorundergraduate student majoringin computer science at ZhejiangUniversity. He will work towards amaster degree at the State Key Labof CAD&CG in Zhejiang Universityafter graduation. His research interestsinclude image analysis and geometric

modeling.

Youyi Zheng is currently an assistantprofessor at the School of InformationScience and Technology, ShanghaiTechUniversity. He obtained his Ph.D.degree from the Department ofComputer Science and Engineeringat Hong Kong University of Science& Technology, and his M.S. and B.S.

degrees from the Department of Mathematics, ZhejiangUniversity. His research interests include geometricmodeling, imaging, and human–computer interaction.

Tianjia Shao is currently an assistantresearcher at the State Key Lab ofCAD&CG, Zhejiang University. Hereceived his Ph.D. degree in computerscience from the Institute for AdvancedStudy, and his B.S. degree fromthe Department of Automation, bothat Tsinghua University. His research

interests include RGBD image processing, indoor scenemodeling, structure analysis, and 3D model retrieval.

Yin Yang received his Ph.D.degree in computer science fromthe University of Texas at Dallas in2013. He is an assistant professorin the Electrical CommunicationEngineering Department, University ofNew Mexico, Albuquerque, USA. Hisresearch interests include physics-based

animation and simulation, visualization, and medicalimaging analysis.

Kun Zhou is a Cheung Kong professorin the Computer Science Department ofZhejiang University, and the director ofthe State Key Lab of CAD&CG. Priorto joining Zhejiang University in 2008,he was a lead researcher in the InternetGraphics Group at Microsoft ResearchAsia. He received his B.S. and Ph.D.

degrees in computer science from Zhejiang University in1997 and 2002, respectively. His research interests arevisual computing, parallel computing, human–computerinteraction, and virtual reality. He currently serves on theeditorial/advisory boards of ACM Transactions on Graphicsand IEEE Spectrum. He is a Fellow of the IEEE.

Open Access The articles published in this journalare distributed under the terms of the CreativeCommons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permitsunrestricted use, distribution, and reproduction in anymedium, provided you give appropriate credit to theoriginal author(s) and the source, provide a link to theCreative Commons license, and indicate if changes weremade.

Other papers from this open access journal are available freeof charge from http://www.springer.com/journal/41095.To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.

an interactive approach for functional prototype recovery ... · annotation of high-level relations...

Documents