shape context preserving deformation of 2d anatomical ...€¦ · volume 0 (1981), number 0 pp....

Volume 0(1981), Number 0 pp. 1–12 COMPUTER GRAPHICS forum

Shape Context Preserving Deformation of 2D AnatomicalIllustrations

Wei Chen†1 and Xiao Liang‡2 and Ross Maciejewski§3 and David S. Ebert3¶

1State Key Lab of CAD&CG, Zhejiang University2Microsoft Research Asia3 Purdue University

AbstractIn this paper we present a novel 2D shape context preserving image manipulation approach which constructsand manipulates a 2D mesh with a new differential mesh editing algorithm. We introduce a novel shape con-text descriptor and integrate it into the deformation framework, facilitating shape-preserving deformation for 2Danatomical illustrations. Our new scheme utilizes an analogy based shape transfer technique in order to learnshape styles from reference images. Experimental results show that visually plausible deformation can be quicklygenerated from an existing example at interactive frame rates. An experienced artist has evaluated our approachand his feedback is quite encouraging.

Categories and Subject Descriptors(according to ACM CCS): I.3.3 [Computer Graphics]: Picture/Image Generation

1. Motivation

In illustration, the proper depiction of shape is fundamentalto the human vision system’s ability to recognize an objector process. The expressiveness of a drawing is greatly influ-enced by the manner in which shape is rendered and variedamongst objects within a scene. In order to efficiently en-hance human perception, reduce occlusion, or even expresscomplex dynamic procedures, hand-drawn illustrations haveexploited various methods for shape and shape style depic-tion [Cla99,Hod88,ST90].

Such illustrative techniques have recently been introducedinto the computer graphics community. Various approachesfor illustrative rendering [GG01,ER02] have been proposedfor simulating artistic styles and concepts, including thevari-ation, emphasis, and subordination of colors, patterns, tex-tures, media and shading [GG01, ONOI04, BG05]. Unfor-tunately, many computer-generated illustration techniquesconcentrate solely on illustrative shading effects, whilene-glecting the influence of the shape of the illustrated objects.

† [email protected]. This work was done while Wei Chen wasvisiting Purdue university.‡ [email protected]§ [email protected]¶ [email protected]

Mathematically, shape is defined as an equivalence classunder a group of transformations. However, this defi-nition only judges whether two shapes are exactly thesame [BMP02]. In the context of computer-generated illus-tration, an appropriate definition of shape perception wouldbe necessary to create faithful and expressive effects. This isanalogous toshape coordinatesin morphometrics [Boo92],which is used to investigate biological forms by comparingshape and shape change. For instance, the two objects shownin Figure1 (a-b) have similar biological forms because theirshapes could be related by certain mathematical transforma-tions [Tho17]. To distinguish various biological forms, a setof shape descriptors have been derived based on the anal-ysis of these transformations [Boo92, BMP02, WAA∗05].However, little attention has been paid to how to identifythe shape dissimilarity between two objects. The shape de-scription that can be used to depict the shape difference iscalled theshape context, which abstracts the geometric de-tails and measures the shape styles. Until now, it has re-mained challenging to capture and learn shape styles forthe purpose of depiction, though there are many attempts toachieve the automatic or semi-automatic learning of render-ing styles [HS99,HJO∗01,HOCS02].

As such, consideration of an object’s shape is key tothe proper rendering of not only still objects, but also de-formable objects. For instance, in medical education there

c© 2008 The Author(s)Journal compilationc© 2008 The Eurographics Association and Blackwell Publishing Ltd.Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ, UK and350 Main Street, Malden, MA 02148, USA.

W. Chen & X. Liang & R. Maciejewski & D. Ebert / Shape Context Preserving Deformation of 2D Anatomical Illustrations

(b)(a)

(d)(c)

Figure 1: The fishes (Pomacanthus and Scarus) shown in(a) and (b) can be linked with region-based linear or non-linear coordinate transformations [Tho17]. Our approachcaptures the shape context difference between (a) and (b),and performs shape context based analogy from (a) and (b)onto (c), another illustration of Pomacanthus [Ito], yieldingthe result (d).

are various terms describing the movements of the limbsand other parts of the human body. These movements nor-mally take place at joints where two or more bones andcartilage articulate with one another [MD99]. Currently, toillustrate deformations, artists typically acquire the modelsin key frames by using measurement instruments [Hod88],which is a laborious process. In contrast, a common methodin the graphics community is to display a sequence of indi-vidual model deformations to form an animation sequence.This is quite challenging when the scene complexity and therequirements of visual reality are high, as shown in Figure2(a) and Figure3 (a).

In this paper, we present a novel deformation algorithmthat uses a 2D input image. One immediate benefit of thisscheme is that a visually pleasing simulation of deformationscan easily be created by applying simple 2D image manip-ulations. Our solution is a tool for an illustrator to quicklygenerate deformations for 2D anatomical illustrations. Itcanalso be used by non-illustrators to freely deform existingimages and check the validity of the results. Figure3 (b-d)show our results.

Our solution reformulates the 2D image deformation asa user-guided differential mesh manipulation that operateson the differential to absolute geometry and distributes de-tail distortions across the entire domain by means of a least-square minimization reconstruction scheme. One main dif-ference between our approach and previous 2D image defor-mation techniques is that our approach pays special attentionto the expressiveness of the deformation and the preserva-

(c)(a) (b)

Figure 2: (a) A volume rendering of a foot dataset [SDS05].(b) A hand-drawn image showing the bending of a foot. (c)Our result that simulates the shape style of (b), by perform-ing shape deformation to (a). The total user time is 1 minute.

tion of shape styles. Specifically, we integrate shape contextinto the interactive deformation framework. We also intro-duce an analogy-based shape transfer technique to efficientlymimic the shape styles from reference images. One exampleis shown in Figure2.

The rest of this paper is organized as follows. We brieflysummarize the related work in Section 2. After describingour deformation approach in Section 3, we introduce theanalogy-based shape style transfer technique in Section 4.The experimental results are reported in Section 5. Finally,we conclude this paper in Section 6.

2. Background and Related Work

Image Deformation and Manipulation Previous image de-formation and manipulation approaches typically deformthe shape space in which the image is embedded. Forinstance, Barrettet al. used space-warp deformation forobject-based image editing [BC02]. Bookstein et al. em-ployed thin-plate splines to find a space deformation that isdefined by several feature points [Boo89]. A potential draw-back of these approaches is that they model deformations assmooth global transformations. Thus, the results undergo lo-cal non-uniform scaling and shearing, which is undesirablein many applications. Although the moving least squares al-gorithm [SMW06] provides a point-based closed-form solu-tion using affine, similarity, and rigid transformations, it isstill an image warping technique.

By triangulating the image into a 2D mesh and build-ing deformations dependent on the specified topology,Igarashiet al. presented an interactive, as-rigid-as-possible,shape manipulation system [IMH05]. Because this algorithmemploys a two-step approximation optimization to achieveinteractive performance, it may create unnatural effects bysolely considering rigid transformations. To overcome thisproblem, Wenget al. introduced an algorithm based ona non-linear least squares optimization [WXW∗06]. Thepreservation of the Laplacian coordinates of the shape

c© 2008 The Author(s)Journal compilationc© 2008 The Eurographics Association and Blackwell Publishing Ltd.


(b)(a)

(c) (d)

Figure 3: (a) The input illustration; (b) The constructed mesh; (c) The head is rotated; (d) The lower arm is moved. Imagesource: http://www.lib.uiowa.edu/hardin/rbr/imaging/mascagni/zoom/zfront1.htm. This example was selected from the exhibit"So Divinely Built a Mansion: Six Centuries of Human Anatomical Illustration," by Paolo Mascagni (1755-1815).

boundary and the region area yields visually plausible defor-mation. However, this approach concentrates on the preser-vation of the shape outline and is probably not effective forhandling the deformations of internal structures. More recentwork by Fang and Hart [FH07] describes an image editingsystem that decouples the feature position from pixel colorgeneration by resynthesizing textures from the source image

in order to preserve its detail and orientation around a newfeature curve location.

Computer-Generated Illustration Illustrators [Cla99,ST90] strive to create perceptually effective images by am-plifying features and exploiting artistic abstractions. Exceptfor pen or paintbrush [Hod88], modern illustrators oftenemploy commercial software (e.g., Adobe Illustrator, Corel



PhotoPaint, Corel Painter, JASC PaintShop Pro ) to manu-ally create still 2D illustrations. For animated 2D images,il-lustrators commonly use Adobe PhotoShop, After Effects,Macromedia Flash, Adobe PhotoShop, Image Ready andCorel Painter [And].

In the computer graphics and visualization community,scenes are modelled as 3D surfaces or volumes. Com-prehensive approaches have been introduced for point,line, surface and volume drawings [BKR∗05, ER02, GG01,LEM∗02].Furthermore, by applying traditional illustrativetechniques, new algorithms have been developed that workdirectly on 2D images [BSM∗07,KML99].

Example based Modeling and RenderingExample-based approaches create new effects by simulating the pat-terns or styles of specified examples. One representativework is texture synthesis, which has been applied in volumeillustration [ONOI04]. Another category of example-basedapproaches is the analogy-based scheme, including imageanalogy [HJO∗01] and rendition analogy [HS99]. The curveanalogy algorithm [HOCS02] generalizes the idea of imageanalogy by transferring the position offset from one curve toanother.

Differential Coordinates Based Mesh ManipulationInspired by the success of PDEs (Partial Differential Equa-tions) in the digital image processing community [PGB03],differential mesh manipulation techniques [SLCO∗04,YZX∗04] transform differential surface properties directlyand reconstruct the results with a global optimization. Thisscheme not only preserves the geometric feature, but alsoavoids artifacts due to per-vertex editing by distributinger-rors globally. Since the differential coordinates are relatedto geometric details, or features, the global optimizationscheme favors detail-preserving results.

3. Our Approach

We construct triangulations to outline the boundary of objectregions and use this topological information to create defor-mations by separating parts of the images.

We deform the constructed 2D triangular mesh within thedifferential mesh editing framework. Poisson-based meshmanipulation is briefly described in Section 3.1. Our ap-proach improves previous image deformation approaches intwo aspects. First, we integrate the properties of differentregions into the deformation to produce an effective illustra-tion. We will explain the enhanced Poisson mesh deforma-tion technique in Section 3.2. Second, we introduce a shapecontext descriptor that offers a globally discriminative char-acterization of the objects in the illustrations. The preserva-tion of the shape context during deformation imposes an ad-ditional deformation constraint. The issues involving shapecontext are discussed in Section 3.3. Finally, we summarizethe user interface and deformation pipeline in Section 3.4.Figure4 shows the conceptual pipeline of our approach.

Inputing Images

Hand-drawn

Images

Surface or Volume

Renderings

Images or Pictures

Building Mesh

Constructing Mesh

Constructing

Region Contours

Specifying

Rigidities

Shape Deformation

Specifying ROI

Interactive

Manipulation

Shape Transfer

Figure 4: The three stages of our approach. The items withdashed frames are optional.

3.1. Preliminaries

Let us consider an arbitrary non-degenerate planar triangularmeshS= (K,V), whereV is the set of two-dimensional ver-tex coordinates, andK describes its vertex connectivity. Ascalar fieldf onScan be defined as a piecewise linear func-tion f (v) = ∑i fiφi(v), where fi is a scalar,φi(·) is a piece-wise linear basis function with value 1 at vertexvi and 0 atother vertices. The discrete gradient operator of an arbitraryscalar fieldf onS is defined as [TLHD03]:

∇ f (v) := ∑i

fi∇φi(v) (1)

It yields the discrete Laplacian operator at each vertexvi :

∆ f (vi) := ∑v j∈Nv(vi)

12A j

(cotB j +cotCj )( fi − f j) (2)

Here,Nv(vi) denotes the set of 1-ring neighboring verticesof vi . A j is the area of thej th triangle, andB j andCj are twoangles opposite to the edge(vi ,v j).

If we set fi for two vertex coordinates,vmi (m= x,y), sep-

arately,Scan be viewed as two discrete scalar fields(Sx,Sy).By applying the discrete gradient operator toSm(m = x,y)separately, we get two gradient vector fields:

∇Sm(v) = ∑i

vmi ∇φi(v) (m= x,y) (3)

Each∇Sm is a piecewise constant vector field which canbe regarded as the guidance vector fieldw involved in solv-ing the discrete Poisson equation:

∆ f = div ·w, f |∂Ω = f ∗|∂Ω (4)

where f is the scalar field to be computed andf ∗ providesthe desired value on the boundary∂Ω. Manipulating thisguidance vector fieldw will result in varied reconstructedvertex coordinates.

By specifying a set of boundary vertices∂Ω and f ∗|∂Ωto construct the boundary condition, we have two equationsfrom Equation4 for solving the vertex coordinates (x andy)in the form of f . The user can modify the position and orien-tation of a set of vertices, resulting in two altered guidancevector fields∇Sx and∇Sy. The new vertex positions(Sx,Sy)



can be reconstructed by solving Equation4 with modified∇Sx and∇Sy.

3.2. Property-Related Deformation

The basic idea for achieving expressive deformation is to de-fine appropriate properties and use them to guide the meshmanipulation. An intuitive choice is the rigidity or elastic-ity that describes the stretching degree under various forces.With the triangulation representation, we regard the rigiditydistribution of the illustrated objects as two piecewise con-stant scalar fields and assign each triangle,T, two constantrigidities km

T (m = x,y) for (x,y). This definition yields twonew piecewise linear basis functionskm

T φi(·)(m= x,y). Intu-itively, km

T (m= x,y) scales the discrete gradient vector alongthex andy axes, and can be represented with a 2D transfor-

mation

(

kxT 00 ky

T

)

. If we consider the deformation inten-

tion imposed on the triangleT as an affine transformation,km

T (m = x,y) defines the response coefficients ofT to thedeformation intention along thex and y axes respectively.Accordingly, we denote the property-related discrete diver-gence operatordivkm as:

divkm ·∇ f (vi) := ∑T∈NT(vi)

kmT AT∇φi |T ·∇ f (vi) (m= x,y)

(5)

It also induces the new discrete Laplacian operator:

∆km f (vi) := ∑v j∈Nv(vi )

12A j

((kmB j )

2 cotB j +(kmCj

)2 cotCj)( fi − f j)

(6)wherekm

B jandkm

Cjare the rigidities of two triangles that share

the edge(vi ,v j).

Finally, we have the rigidity-related Poisson equation:

∆km f = divkm ·w, f |∂Ω = f ∗|∂Ω (7)

In practice, the rigidities are determined by the user andare attached to each region of the illustrated objects. A typi-cal range of the rigidities are from 1.0 to 10.0. The smallestvalue, 1.0, corresponds to softly deformed regions, such asthe fat in the human body. For fully rigid objects, such asthe bones, the rigidities are set to be the largest value, i.e.,kx = 10.0,ky = 10.0. Beyond distinguishing between elas-tic and rigid, this technique simulates the resiliency, i.e., thedegree to which an elastic object resists deformation. Thus,at one end of the spectrum we could have an elastic mate-rial and at the other a rigid material, and somewhere in themiddle would be user-selectable resilient materials. In thebiomedical arena, these resiliency factors could be based onactual tissue properties. Our approach simulates this effect.In addition, non-uniform deformation in the 2D plane canbe simulated by setting different rigidities for any direction(x,y). One example is the muscle on the legs which exhibits

a directional deformation. Figure5 depicts different behav-iors under three types of rigidity configurations of the se-lected region.

(a) (b)

(c) (d)

Figure 5: The deformation results with different rigidity set-tings for a bone with a red star. (a) The input image; (b)Uniform strong elasticities:(kx,ky) = (1.0,1.0); (c) Non-uniform rigidities:(kx,ky) = (10.0,1.0); (d) Uniform strongrigidities: (kx,ky) = (10.0,10.0). The rigidities for other twobones are set as(kx,ky) = (10.0,10.0).

3.3. Shape Context Preserving Deformation

By shape context, we mean any measurement of the con-figuration of shape features that does not change when theglobal shape stretches or shrinks. Figure6 (a-b) shows twofaces with 16 landmarks. In each example, the set of featurepoints constructs a recognizable identification and a specialshape context that can differentiate itself from others.

Although there have been several definitions for shapedescriptors, we find that most of them are unsuitable forthe purpose of shape-preserving image manipulation. Forinstance, the shape descriptor introduced in [BMP02] de-scribes the coarse distribution of the rest of the shape withre-spect to a given point on the shape. It is basically a statisticalproperty and cannot be quantized and manipulated. A less



statistical characterization of geometrical shape would be theproperties of a figure that are not changed by translation, ro-tation or scaling [Boo92]. However, the shape coordinatesrepresentation proposed in [Boo92] only considers the dis-tance ratios and measures the triangular shape that maintainsthe distinctions among three vertices. In our approach, wedefine the shape context as an inter-dependency of a groupof feature points, i.e., a 2D mesh structureC= (K,V) strictlybuilt on these points. All vertices ofC are selected fromV,meaningV ⊂V. To make each triangle ofC as canonical aspossible, the mesh connectivityK is constructed with con-strained Delaunay triangulation algorithm [She05] and is notnecessarily a subset ofK.

We perform shape context preserving deformation by im-posing an additional constraint on the discrete Poisson equa-tion. The goal is to transform the discrete guidance field onC in a uniform way. Thus, we reformulate Equation7 as:

∆km f |S+λ∆km f |C = divkm ·w+λdivkm ·u∗, f |∂Ω = f ∗|∂Ω(8)

In Equation8, λ is an adjustable parameter that weightsthe preservation of the shape context. The definitions of∇Cm and u∗ are analogous to those of∇Sm and w∗ (seeSection 3.1). During mesh manipulation,w∗ is interactivelyaltered by users, i.e., each triangle ofS may be modifiedwith different rotation, scale or translation transformations.Whereas, all triangles inC are transformed with the samerotation and scale transformations for the purpose of pre-serving the shape context. Therefore,u∗ = R(θ)S(s)u with

R(θ) =

(

cosθ −sinθsinθ cosθ

)

and S(s) =

(

s 00 s

)

. The

parametersθ ands are unknown and have to be determinedin each deformation step. To find an optimal solution, wecalculate the centroidvc of C and seek to keep the relativedistances and angles between each vertexvi in C andvc un-changed. This yields the following minimization problem:

argminθ,s ∑vi∈C

(v∗i − v∗c −R(θ)S(s)(vi − vc))2 (9)

wherev∗i denotes theith transformed vertex.

With the assumption that the deformation is adequatelystable, we employ an iterative technique to progressively ap-proximate the optimized solution. In the first step, we solveEquation7 to compute the modified vertex positions, with-out considering the influences of shape context. Based onthese vertices, we calculate an average scales and use itto estimate an optimizedθ in Equation9. Subsequently, wesolve Equation8 with the newR(θ) andS(s) to get an ini-tial estimation of the new vertices. In the next iteration, thecomputed vertices can be used to recover a better estimationof θ ands, and so on. The iteration ends when either the it-eration number exceeds a given maximum, or the differencebetweenθ andsbetween two successive steps is smaller thana given threshold.

It is worth mentioning that the weightλ is set to be a small

value at the first iteration. This is automatically enlargedinthe next iterations because the influence of the shape con-text would gradually become dominant when approachingthe optimized solutions. Figure6 (d) shows the deformationresult by entirely replacing the shape context of Figure6 (c)with that of Figure6 (b).

(b)

(d)

(a)

(c)

Figure 6: (a-b) Two line-drawing images and their shapecontext points [Boo92]; (c) The face of an old man; (d) Ourresult by replacing the shape context of (c) with that of (b).

3.4. Processing Stages

We have built an interactive 2D deformation system for 2Danatomical illustrations. It allows the user to employ a setoftoolkits, including a region constructor, a mesh constructor, aregion of interest (ROI) selector and a mesh manipulator, tointeractively deform the illustration in an intuitive fashion. Inthe context of solving the property-related Poisson equation,all interactions are intended to alter the guidance gradientvector field and specify the boundary conditions. As a re-sult, the efficiency of our system is greatly influenced by themodes of how to capture the user intentions and then changethe guidance vector field and boundary conditions for thespecified manipulation tasks. Below, we describe each oper-ation step that is sequentially demonstrated in Figure7.

Region Construction Beginning from an input illustra-tion, the user manually constructs a polygonal contour for



each region and specifies its rigidities. This process is analo-gous to the segmentation of images and could be acceleratedby directly adjusting the boundaries formed from interac-tive image segmentation. In general, the mesh constructionmay be not accurate. The average case can be constructed inseveral minutes. Figure7 (b) shows the constructed regioncontours on an input illustration (Figure7 (a)). For the sakeof demonstration, the constructed contours are rough.

(b)(a)

(d)

(c)

(f)(e)

Figure 7: The pipeline of our system. (a) The input im-age (MS-0101, Muscular System @1989 John M. Daugh-erty, Highlight Studios); (b) The constructed region contoursdone in 2 minutes. For the sake of demonstration, they arerough. (c) The constructed mesh and the sets of control-ling vertices, constrained vertices and free vertices, whichare shown in yellow, red and other colors; (d) After rotat-ing the chest, we specify a new set of controlling vertices,constrained vertices and free vertices, for the purpose of ro-tating the head; (e) The final result without distance weight-ing; (f) With the Gaussian weighting scheme, a more naturaleffect is achieved. We enlarge the head parts to show the dif-ference between (e) and (f).

Mesh Construction We apply a constrained conformingDelaunay triangulation algorithm [She05] to the constructed

region contours. Note that this algorithm does not result inatrue Delaunay triangulation, i.e., some triangles might not beDelaunay, but all triangles are ensured to be constrained De-launay. To achieve optimized mesh structures, two additionalrequirements for triangulation are set. First, the vertices andedges of the region contours are completely preserved in thefinal mesh. Second, a minimum on the triangle inner anglesand a maximum on the triangle area are set. The constructedmesh based on Figure7 (b) is shown in Figure7 (c).

ROI Selection By ROI (region of interest), we meanthe region where the user’s intentions are imposed. It con-sists of two parts: the operation handles and the free region.The former denotes the actual user-controlled vertices andis used to determine the modification of the discrete guid-ance vector field in Equation8. The free region includes thevertices whose positions are freely changed during deforma-tion. Other vertices that do not belong to the ROI are keptunchanged and are called constrained vertices. Our systemallows the user to manually select a sequence of vertices orsimply choose the vertices of one region as the controllingvertices or the constrained vertices. The remaining verticesare by default free vertices. Additionally, the user can deter-mine a group of shape context vertices and construct a shapecontext mesh. Figure7 (c) depicts the determined control-ling vertices (in yellow), constrained vertices (in red) andfree vertices in other colors. After deforming the illustrationby rotating the chest, we have the deformed mesh shown inFigure7 (d). To further modify the illustration, we fix thebody trunk and let the head be the controlling region (seeFigure7 (d)).

Mesh Manipulation During mesh manipulation, theuser can either freely drag one or several controlling ver-tices, or modify the controlling vertices with specified 2Dtransformations. Another commonly used interaction modeis to simultaneously translate, rotate or scale all vertices of aselected region. Thereafter, the modifications on the control-ling vertices are gradually propagated to the remaining ver-tices. The propagation is dependent on the distance betweenvertex pairs. Our system supports three distance-dependentfunctions which correspond to the nearest, linear and Gaus-sian weighting schemes. After the propagation of the trans-formations to all vertices, a new constant gradient vectoris computed in each triangle, yielding two altered guidancevector fields. Finally, the discrete Poisson equation is for-mulated as a sparse linear system and solved with a di-rect solver [Tol03]. In Figure 7 (e-f), we show two resultswith the same manipulation settings and different weightingschemes.

4. Analogy Based Shape Style Transfer

Transferring shape styles between illustrations requiresap-propriate representations of shape styles. In our system, weconsider two types of shape styles.



4.1. Shape Transfer with Region Contours

Object contours are important features for shape recogni-tion [BMP02]. Every region boundary of the underlying im-age can be regarded as a formulation of its shape style. Thetransfer of region contours can be fulfilled by a simple curveanalogy driven shape deformation scheme.

For a selected region, we first deform its boundary with adifferential curve deformation technique. Suppose we havetwo illustrations whose meshes areM0 andM1. The bound-ary R0 of one region inM0 is modified to mimic anotherregion boundaryR1 in M1. R0 andR1 are represented withtwo polygons, which can be regarded as two piece-wise lin-ear curves. Analogous to the definitions of two-dimensionalscalar fields and the discrete differential operators in Section3.1, we construct a one-dimensional differential representa-tion for a linear curveR= (K,V). For the sake of simplicity,the discrete gradient operator on an arbitrary scalar fieldfdefined inR can be rewritten as:

∇ f (v) = vi − vi−1 s.t. v ∈ −−−→vi−1vi (10)

The Laplacian operator on each vertexvi is defined as:

∆(vi) = vi − (vi−1 +vi+1)/2 (11)

The shape transfer fromR0 to R1 is accomplished by re-placing the gradient vectors ofR1 to those ofR0 and spec-ifying constrained vertices in Equation4. Note that,R0 andR1 have to be aligned and re-sampled with arc-length param-eterization before shape transfer, so that their vertices are aone-to-one correspondence.

In the second step, the modification of the region contouris integrated into the Poisson equation, Equation7 or 8, toalter the guidance vector field and the boundary condition.Figure10 (d) and Figure12 provide examples.

4.2. Shape Transfer with Shape Context

In terms of shape context, its transfer should be performedbetween two illustrations that share a similar context. Twosets of shape context vertices with identical vertex numbersare first specified in the source illustration and the destina-tion. The pairwise correspondences between vertices are de-termined by the user. We regard the transfer of shape contextas solving the following Poisson equation(m= x,y):

∆km f |S+λ∆km f |C = divkm · w+λdivkm ·u∗, f |∂Ω = f ∗|∂Ω(12)

Here,SandC are the mesh and shape context mesh of thedestination illustration.wdenotes the altered guidance vec-tor field after introducing the new shape context.u∗ is theguidance vector field of the shape context mesh in the sourceillustration. Similar to the shape context preserving deforma-tion, a global rotation and a global scale transformation haveto be determined before the shape transfer in order to elim-inate the problem caused by the rotation-variant property ofthe discrete Laplacian operator.

5. Experimental Results and Discussions

We have tested our approach on a PC with an Intel P4 3.2GHZ CPU. For a typical image at the resolution of 380×380,the constructed mesh normally contains 300∼ 1500 verticesand 500∼ 2000 triangles. The user’s time spent on meshconstruction and ROI specification is typically 60−120 and15− 30 seconds, respectively. With a non-optimized Pois-son solver, the average times for the transformation propa-gation and mesh reconstructions are 10 milliseconds and 15milliseconds. In each example we tested, the deformationand shape transfer is performed in real-time. For a fine meshwith more than 50000 vertices, our system can still performat interactive frame rates.

In our implementation, we regard the 2D illustration as a2D texture and map it to a 2D triangular mesh. The imagedeformation is driven by the mesh manipulation, in a sim-ilar mode to previous mesh-based image deformation tech-niques (e.g., [IMH05] and [WXW∗06]). We use OpenGLto implement the texture mapping for high efficiency. Allinput illustrations are rescaled to a fixed window resolution(e.g, 380×380). Due to the rescaling of the input illustra-tions, there are some differences between the input illustra-tions and the deformed results. In addition, some fuzzinessmay appear in the boundary of the shape, due to the defor-mation of the mesh and the use of texture mapping (such asthe line drawings shown in Figure12 (c) and (e)).

5.1. Results

We tested our approach on the rendering results shown inFigure2 (a) and Figure10 (a). By setting appropriate rigidi-ties to the skin, bone and swimming bladder, satisfying de-formations are easily achieved (see Figure2 (c) and Fig-ure 10 (b) and (d)). Specifically, shape styles of Figure2(b) and Figure10 (c) are transferred to Figure2 (c) and Fig-ure 10 (d). The user interaction time for each example isapproximately 1 minute.

There are various terms describing the movements of thelimbs and other parts of the human body. They can be clas-sified into about eleven categories, namely, flexion, exten-sion, abduction, adduction, rotation, circumduction, oppo-sition, retrusion and protrusion, elevation, eversion, prona-tion [MD99]. We tested our system on several 2D hand-drawn illustrations of these anatomical movement types.Figure8 (a-c) shows three hand-drawn illustrations that de-pict the hyperflexion and hyperextension of the cervical tis-sues of a human head. Our approach requires 10 minutes tointeractively illustrate both deformations (see Figure8 (e-f))based on Figure8 (d). Figure9 (a-c) demonstrates our re-sults that simulate the depression and elevation movementsof a human body.

Anatomical shape and its variation, play important roles inmedical research. A particular disorder or aging may causeanatomical changes or differences. In medical illustration,



(b)

(d) (e) (f)

(a) (c)

Figure 8: Three hand-drawn images (a-c) that depict the hyperflexion and hyperextension of the cervical tissues of a humanhead. With the input image (d), our approach allows a user to interactively simulate this dynamic procedure in 10 minutes,resulting in (e) and (f).

the abnormality of certain human anatomy can be illustratedusing a set of comparison images. By referring to two setsof 2D hand-drawn illustrations, we performed two experi-ments. The first one illustrates the systolic (Figure9 (e)) anddiastolic (Figure9 (f)) dysfunctions of a human heart. Com-pared to the normal status of filling blood (Figure9 (d)), thesystolic case presents an over-enlarged ventricle becausetoomuch blood is filled and the diastolic one fills with less bloodbecause the material of the ventricle is too stiff.

Introducing properties into different regions results in ex-pressive deformation effects. Figure5 and Figure7 showtwo such examples. Note that our solution is not physicallybased, and its results do not strictly conform to results gen-erated by 3D deformation approaches as 3D transformationsand deformations may be distorted after perspective projec-tion. Nevertheless, satisfying results can still be achieved

by interactively adjusting the engaged 2D transformations.Shape context preserving deformation and shape transfer arevery useful operations for content re-creation in computer-generated illustration. The examples shown in Figure1, Fig-ure6, Figure12, and Figure13demonstrate the efficiency ofour approach.

5.2. Failure Cases

In terms of limitations, our approach cannot handle the casewhen self-collision occurs (see Figure11 (a) and (b)). Onepossible solution is to add an additional self-collision detec-tion constraint on the discrete Poisson equation7, in a simi-lar way as the shape context preserving Poisson Equation8.

In addition, our approach is suitable for simulating con-tinuous deformations because the topology information has



(b) (c)(a)

(e) (f)(d)

Figure 9: Two results with our approach. (a) An input ab-domen image (Image Courtesy: Deutscher Infografik Dienst:http://www.infografikdienst.de); (b-c) The images showingthe depression and elevation movements; (d) An input heartimage; (e-f) Our results depicting the systolic and diastolicdysfunctions when the ventricles fill with blood.

(a) (b)

(d)(c)

Figure 10: (a) A volume rendering by Stefan Bruck-ner [BG05]; (b) Our result; (c) Another 2D image; (d) Ourresult by simulating the shape styles of (c).

to be preserved during manipulation. In the case of objectsplitting or fractures, as shown in Figure11 (c), an object-based image editing mechanism [BC02] may be introducedfor illustrating the separations of individual objects.

Another failure case is caused by object occlusion duringobject animation or deformation (see Figure11 (d-e)). Al-though it is a common problem for 2D image editing tech-niques, we hope that our approach can overcome this obsta-cle through combinations with 3D algorithms. For example,we can represent the results by a 3D approach with a set ofdepth layers and manipulate these layers on a per-object ba-sis.

5.3. Discussion and Evaluation

Our approach is basically a 2D solution and is not physi-cally accurate. This disadvantage is mainly caused by the in-

(b)

(d)

(a) (c)

(e)

Figure 11: Failure cases. (a) A knee; (b) Our approach maycause self-collision when moving one bone; (c) The com-pound fracture of the bone; (d) A heart with a saphenousvein graft; (e) A heart with the left internal thoracic arterygrafted to the anterior descending coronary artery.

feasibility of image manipulation for 3D transformation andlighting. However, in many cases, we do believe that our ap-proach is more efficient than existing 3D methods due to thesimplicity of 2D image manipulation. Indeed, commercial2D deformation toolkits, such as Avid “Elastic Reality” pro-gram [AVI ], are available for artists to use. In any case, ourapproach could be regarded as a reliable addition, or com-plementary technique, to existing 3D approaches.

Our scheme works in these situations because it reformu-lates complicated 3D transformations as a set of 2D affine,similarity and rigid transformations. The degree of 2D trans-formations can be interactively modulated by users. Addi-tionally, our approach would be a very useful tool to post-process rendered images under selected viewpoint, cutting-plane and lighting conditions.

We have received encouraging feedback from an experi-enced medical illustrator [And]. His evaluation states that“You have come up with a remarkable and useful program.I have used Avid’s “Elastic Reality” program some in thepast, but your program appears much superior. My first andforemost concern was bending soft, plastic tissues withoutdistorting rigid ones. You have allayed my concerns com-pletely, and in a very solid manner.”

6. Conclusions and Future Work

In this paper, we address the problem of 2D shape deforma-tion by means of a differential mesh manipulation approach.Our system allows the users to flexibly design a sequence of



(a)

(d)

(b) (c)

(e)

Figure 12: Shape styles of (a) are transferred to (b) and (d), yielding (c) and (e) respectively.

(b)(a) (d)(c)

Figure 13: (a) An input image; (b) A reference image with 16 landmarks; (c) The constructed mesh based on (a); (d) Our resultby transferring the shape context from (b) to (a).

dynamic effects in an intuitive fashion. We also introduce anew description to shape context that is amenable for shapecontext preserving deformation and shape transfer.

Shape depiction depends on not only the shape styles, butalso the lighting and rendering configurations. Our currentapproach does not consider the influences from perspectiveprojection, shadow and lighting change during deformation.Exploring lighting with regard to deformation is important,and could greatly enhance the value of the proposed method.We plan to exploit a more efficient strategy based on previ-ous image-based rendering approaches to simulate the shad-ing variations during deformation. Another potential issueis to extend the property-related differential mesh manipu-lation algorithm to 3D tetrahedral meshes. More vivid and

complicated 3D deformation can be simulated in a shape-preserving mode. The main challenge for this is the highcomputational cost caused by the large number of tetrahe-dron primitives. Moving the computations of the linear sys-tem solution and volume rendering onto graphics processingunits is a promising solution.

7. Acknowledgements

The authors would like to thank Stefan Bruckner and NikolaiSvakhine for providing images, and thank Nvidia for equip-ment donations. This work is partially supported by 863 pro-gram of China (No. 2006AA01Z314), NSF Grants 0081581,0121288, 0328984, and the U.S. Department of HomelandSecurity.



References

[And] A NDREWSW. M.: Personal Communication. William M.Andrews, http://www.mcg.edu/medart/MI-Faculty.html.

[AVI] http://www.avid.com/. Elastic reality. Software release.

[BC02] BARRETT W. A., CHENEY A. S.: Object-based imageediting. In Proceedings of ACM SIGGRAPH(August 2002),pp. 777–784.

[BG05] BRUCKNER S., GRÖLLER M. E.: VolumeShop: An in-teractive system for direct volume illustration. InProceedings ofIEEE Visualization(October 2005), pp. 671–678.

[BKR∗05] BURNS M., KLAWE J., RUSINKIEWICZ S., FINKEL -STEINA., DECARLO D.: Line drawings from volume data.ACMTransactions on Graphics 24, 3 (2005), 512–518.

[BMP02] BELONGIE S., MALIK J., PUZICHA J.: Shape match-ing and object recognition using shape contexts.IEEE Trans-actions on Pattern Analysis Machine Intelligence 24, 4 (2002),509–522.

[Boo89] BOOKSTEIN F. L.: Principal warps: Thin-plate splinesand the decomposition of deformations.IEEE Transactions onPattern Analysis and Machine Intelligence 11, 6 (1989), 567–585.

[Boo92] BOOKSTEIN F. L.: Morphometric tools for landmarkdata. Cambridge University Press, 1992.

[BSM∗07] BRESLAV S., SZERSZEN K., MARKOSIAN L.,BARLA P., THOLLOT J.: Dynamic 2D patterns for shading 3Dscenes.ACM Transactions on Graphics 26, 3 (2007), 366–373.

[Cla99] CLARK J. O.:A Visual Guide to the Human Body. Barnesand Noble Books, New York, USA, 1999.

[ER02] EBERT D., RHEINGANS P.: Volume illustration: Non-photorealistic rendering of volume models. InProceedings ofIEEE Visualization(October 2002), pp. 253–264.

[FH07] FANG H., HART J. C.: Detail preserving shape defor-mation in image editing.ACM Transactions on Graphics 26, 3(2007), 12–16.

[GG01] GOOCH B., GOOCH A.: Non-photorealistic Rendering.A.K.Peters, USA, 2001.

[HJO∗01] HERTZMANN A., JACOBS C. E., OLIVER N., CUR-LESS B., SALESIN D. H.: Image analogies. InProceedings ofACM SIGGRAPH(August 2001), pp. 327–340.

[HOCS02] HERTZMANN A., OLIVER N., CURLESS B., SEITZ

S. M.: Curve analogies. InProceedings of Eurographics Work-shop on Rendering(June 2002), pp. 233–245.

[Hod88] HODGESE.: The Guild Handbook of Scientific Illustra-tion. John Wiley and Sons Inc, 1988.

[HS99] HAMEL J., STROTHOTTET.: Capturing and re-using ren-dition styles for non-photorealistic rendering.Computer Graph-ics Forum 18, 3 (1999), 173–182.

[IMH05] I GARASHI T., MOSCOVICH T., HUGHES J. F.: As-rigid-as-possible shape manipulation. InProceedings of ACMSIGGRAPH(August 2005), pp. 327–340.

[Ito] I TO K.: Annotated list of fish illustrations.http://www.hrw.com, latest visiting date: March 2007.

[KML99] K IRBY R., MARMANIS H., LAIDLAW D.: Visualizing

multivalued data from 2D incompressible flows using conceptsfrom painting. InProceedings of IEEE Visualization(October1999), pp. 333–340.

[LEM∗02] LU A., EBERT D., MORRIS C., RHEINGANS P.,HANSEN C.: Non-photorealistic volume rendering using stip-pling techniques. InProceedings of IEEE Visualization(October2002), pp. 211–218.

[MD99] M ORE K. L., DALLEY A. F.: Clinically OrientedAnatomy. Lippincott Williams and Wilkins Corp, 1999.

[ONOI04] OWADA S., NIELSEN F., OKABE M., IGARASHI T.:Volume illustration: Designing 3D models with internal textures.In Proceedings of ACM SIGGRAPH(August 2004), pp. 322–328.

[PGB03] PEREZ P., GANGNET M., BLAKE A.: Poisson imageediting. In Proceedings of ACM SIGGRAPH(August 2003),pp. 313–318.

[SDS05] SVAKHINE N., D.EBERT, STREDNEY D.: Illustrationmotifs for effective medical volume illustration.IEEE ComputerGraphics and Applications 25, 3 (2005).

[She05] SHEWCHUK J. R.: A two-dimensionalquality mesh generator and delaunay triangulator.http://www.cs.cmu.edu/∼quake/triangle.html, July 2005.

[SLCO∗04] SORKINE O., LIPMAN Y., COHEN-OR D., ALEXA

M., RÖSSL C., SEIDEL H.-P.: Laplacian surface editing. InProceedings of the Eurographics/ACM SIGGRAPH symposiumon Geometry processing(2004), pp. 179–188.

[SMW06] SCHAEFERS., MCPHAIL T., WARREN J.: Image de-formation using moving least squares.ACM Transactions onGraphics 25, 3 (2006), 533–540.

[ST90] STAUBESAND J., TAYLOR A. N.: Sobotta Atlas of humananatomy. Urban and Schwarzenberg Baltimore-Munich, 1990.

[Tho17] THOMPSOND.: On Growth and Form. Cambridge Uni-versity Press, 1917.

[TLHD03] TONG Y., LOMBEYDA S., HIRANI A. N., DESBRUN

M.: Discrete multiscale vector field decomposition.ACM Trans-actions on Graphics 22, 3 (2003), 445–452.

[Tol03] TOLEDO S.: TAUCS: A Library of SparseLinear Solvers, version 2.2. Tel-Aviv University,http://www.tau.ac.il/∼stoledo/taucs/, 2003.

[WAA ∗05] WILEY D. F., AMENTA N., ALCANTARA D. A.,GHOSH D., KIL Y. J., DELSON E., HARCOURT-SMITH W.,ROHLF F. J., JOHN K. S., HAMANN B.: Evolutionary morph-ing. In Proceedings of IEEE Visualization 2005(October 2005),pp. 431– 438.

[WXW∗06] WENG Y., XU W., WU Y., ZHOU K., GUO B.: 2Dshape deformation using nonlinear least squares optimization.The Visual Computer 22, 9 (2006), 653–660.

[YZX ∗04] YU Y., ZHOU K., XU D., SHI X., BAO H., GUO B.,SHUM H.-Y.: Mesh editing with poisson-based gradient fieldmanipulation.ACM Transactions on Graphics 23, 3 (2004), 644–651.


shape context preserving deformation of 2d anatomical ...€¦ · volume 0 (1981), number 0 pp....

Documents