detail preserving shape deformation in image editing

Detail Preserving Shape Deformation in Image Edit-

ingSIGGRAPH 2007

Hui Fang and John C. Hart

We propose an image editing system

◦ Preserve its detail and orientation by resynthesiz-ing texture from the source

◦ Patch-based texture synthesis that aligns texture features with image features

Abstract

A novel image editing system that allows a user to select and move one or more image feature curves◦ Replacing any texture stretched by the deforma-

tion with texture resynthesized Anisotropic feature-aligned texture synthesis step to

preserve texture detail Distortion to the texture coordinates for each patch to

align the target image features GraphCut textures [Kwatra et al. 2003]

Introduction

A new method that distorts the coordinates of patch◦ Image Analogies [Hertzmann et al. 2001] can synthe-

size a texture to adhere to a given feature line Yields more high-frequency noise unlike modern patch-

based synthesis◦ Image Quilting [Efros and Freeman 2001] could fill dif -

ferent silhouettes with a texture Boundary patches appeared to repeat

◦ Feature matching and deformation for texture synthe-sis [Wu and Yu 2004] distorted neighboring patches to connect their feature lines Not as global as what us did

Introduction

Deformation◦ Draw feature curves in the source image, and then

move them to their desired destination positions Curvilinear Coordinates

◦ Define curvilinear coordinates using curve tangent vectors & Euler integration

Textured Patch Generation◦ A pair of curvilinear coordinate is generated◦ Texture synthesis over the destination grid from source

Image Synthesis◦ Finalize the synthesis via GraphCut

Overview

Deformation

pi(t)

p'i(t)

D(p'f) = pf – p'f

D(∂I’) = 0

Deformation

Original Deformed

Curvilinear Coordinates

p'i(t)

T'

Since the parametrization of each feature curve is arbitrary, one can encounter global orientation inconsistencies◦ Calculate separate tangent field for each curve

then use only the field which is the closest We integrate these diffused tangents to

construct a local curvilinear coordinate sys-tem extending from any chosen “origin” pixel

Curvilinear Coordinates

Curvilinear Coordinates

p'i(t)

jk

Time-step ɛ = 1◦ 30 ~ 40 pixels along spines (j direction)◦ 15 ~ 30 pixels wide ribs (k direction)◦ Two pixels short of nearby feature curve to prevent overlap-

ping

Smooth the coordinates with several Laplacian itera-tions

◦ λ = 0.7◦ Removes singularities and self-intersections that can occur ◦ Does not completely solve the problem (Not very noticeable)

Curvilinear Coordinates

Curvilinear Coordinates

Source origin q0,0 = D(q'0,0)

Bilinear filter to find the color at the source image

Unit-radius cone filter centered at each desti-nation to accumulate the synthesized texture◦ Small reduction in the resolution of the resynthe-

sized texture detail

Textured Patch Generation

Use GraphCut [Kwatra et al. 2003]◦ Generate patches individually, using a priority

queue to generate first patches whose origin pixel is closest to the feature curve and adjacent to a previously synthesized patch

◦ Generate a pool of candidate textured patches synthesized from source patches grown from origins randomly chosen from an 11×11 pixel region surrounding the point D(q'0,0)

◦ Choose one with the least overlapping difference with previously synthesized neighboring patches

Image Synthesis

Selected patch merges into destination via GraphCut

Use Poission Image Editing when the seam produces by GraphCut is unsatisfactory

Image Synthesis

The deformation field D can potentially compress a large source area into a small target area◦ Cause blocky artifacts and seams◦ Occur when the origin pixels of neighboring

patches in the target map to positions in the source with different texture characteristics

Can be overcome by altering the texture synthesis sampling

Scale Adaptive Retexturing

Scale Adaptive Retexturing

We detect these potential problems with a (real) compression field C'

◦ Clamp the compression field to values in [1,3] to limit its effect

◦ The “spine” length and “rib” breadth of patches are reduced by C'(x,y)

Scale Adaptive Retexturing

Scale Adaptive Retexturing

Accelerated the construction of source feature curves by using portions of the segmentation boundary produced by Lazy Snapping [Li et al. 2004]◦ Feature curves do not need to match feature con-

tours exactly, as deformed features were often aligned by the texture search

Used the ordinary Laplacian deformation for in-teractive preview◦ Denoted some feature curves as “passive” to aid tex-

ture orientation

Results

Filtering used for curvilinear grid resampling removes some of the high frequency detail◦ Could be recovered by sharpening with his-

togram interpolation and matching [Matusik et al. 2005]

Results

Results

Results

Results

Failure case

Sharp image changes (like shading changes) should identified by feature curves◦ Lack of feature curves will cause unrealistic dis-

continuities in the result

Poisson image editing hides some of these artifacts◦ by softly blending the misaligned features

Results

ResultsMeasured on a 3.40GHz

Pentium 4 CPU(31 x 31 search domain for beach)

Stretched texture details can be adequately recovered by a local retexturing around user-defined feature curves

Assumes that the orientation of texture detail of an image is related to the orientation of nearby feature curves

Matting can be used to eliminate unwanted artifacts (Fig. 5)

In practice the success of this approach depends pri-marily on the selection of the feature curves◦ The most promising direction of future work in this topic

would be to add the automatic detection and organization of image feature curves

Conclusion