VARIATIONAL RESTORATION OF NON-FLAT IMAGEFEATURES: MODELS AND ALGORITHMSTONY CHAN AND JIANHONG SHENCOMPUTATIONAL AND APPLIED MATHEMATICSDEPARTMENT OF MATHEMATICSLOS ANGELES, CA 90095-1555chan, jhshen@math.ucla.eduAbstract. We develop both mathematical models and computationalalgorithms for variational denoising and restoration of non- at imagefeatures. Non- at image features are those that live on Riemannianmanifolds, instead of Euclidean spaces. Familiar examples include theorientation feature (from optical ows or gradient ows) that lives onthe unit circle S1, the alignment feature (from �ngerprint waves or cer-tain texture images) that lives on the real projective line RP1, and thechromaticity feature (from color images) that lives on the unit sphereS2. In this paper, we apply the variational method to the restoration ofnon- at image features. General mathematical models for both contin-uous image domains and discrete domains (or graphs) are constructed.Riemannian objects such as metric, distance and Levi-Civita connectionplay important roles. We also discuss e�cient computational algorithmsfor the non-linear restoration equations. Numerical results support boththe models and algorithms developed in this paper. The mathematicalframework can also be applied to the restoration of general non- at dataoutside the scope of image processing and computer vision.Date: June 10, 1999. Find typos? Why wait? Email \jhshen@math.ucla.edu."1991 Mathematics Subject Classi�cation. Primary: 9412; Secondary: 65H10, 49M37.Key words and phrases. Variational model, total variation, denoising and restora-tion, non- at features, Riemannian manifold, metric and distance, orientation, alignment,chromaticity.Research supported by grants from NSF under grant number DMS-9626755 and fromONR under N00014-96-1-0277. 1

2 CHAN AND SHEN1. IntroductionIn image analysis, restoration and denoising by variational methods aretypically applied to gray level image functions or vector-valued (color) im-age functions (see Mumford and Shah [10], Rudin, Osher and Fatemi [15],and the monographs by Morel and Solimini [9], and Weickert [21]). Suchfunctions are \ at" or a�ne. By atness or a�nity, we mean a functionthat takes values in a linear space, such as the real line, or the linear RGBspace.There are some important image features that do not live in at spaces,however. One example is the orientation feature, which comes from op-tical ows or gradient ows (of gray images). One can easily see that ina video that records the rigid motion of an object against a static back-ground, the orientation of the motion stores one half of the whole valuableinformation (and speed is the other half). The second example is thealignment feature which is commonly found in many texture images. Forexample, for �ngerprints, the alignment pattern of �ngerprint \waves" iscrucial for identi�cation tasks (see Perona [12]). The chromaticity featureof color images is another example of non- at features (see Tang, Sapiroand Caselles [19, 18]). For a linear RGB color image, the pixel value I is avector in R3. The magnitude kIk measures the brightness, while its traceon the unit sphere f = I=kIk contains the color saturation information, andis called the chromaticity feature in Tang, Sapiro and Caselles [18]. It liveson the unit sphere S2 in the linear RGB space.The mathematical object for modeling non- at features is di�erentialmanifolds. Perceptual sensitivity to a particular type of feature is measuredby Riemannian metrics. Therefore, our models shall be inevitably basedon Riemannian manifolds. The complete picture has been summarized inFigure 1. The �rst step is taken by image processing or video processing( gray level or color )

Interesting feature f

and Riemannian structure

Feature manifold M

( orientation , optical flow, e.t.c. )

Image u or image flowFigure 1. The ow chart of model setup.researchers. Meaningful features are often de�ned according to our visualsystem or the type of task we attempt to carry out. The second step leads tomathematical analysis. Once the feature is described, the manifold stands

RESTORATION OF NON-FLAT IMAGE FEATURES 3there. But a truly perceptually meaningful Riemannian metric depends onour visual system. For example, orientations in 2-D images live on the unitcircle. It is quite reasonable to assume that generally our visual systemdoes not prefer any single angle (�=4, say). Thus we can use the standardRiemannian structure (rotationally invariant) for orientations. Sometimesthe converse is true: familiar manifolds usually have their natural (relatedto certain transform groups on symmetry) Riemannian metrics, and one candirectly start with them for convenience.The starting point of this paper is the last box. That is, we assume thata perceptually meaningful Riemannian structure is already de�ned.As for at features of an image, variational method is one of the moste�cient tools for analyzing non- at features. This is primarily due to itsdenoising and restoration e�ects. The non at features obtained initially areoften noisy due to common factors like(1) the noise contaminating original images, or left there even after a de-noising pre-processing;(2) regional destruction of raw images (due to the aging of a �lm, say), orinsu�ciency of resolution in raw images;(3) ine�ciency of the algorithm that extracts the non- at features.As in the classical literature, properly designed variational models can (i)attenuate noise, (ii) preserve intrinsic singularities, (iii) enhance the \edges"of non- at features. For general features, an \edge" means a segment wherethe feature distribution undergoes abrupt changes. For the orientation fea-ture from optical ows, for example, the edges may be identical to the realphysical boundaries of moving bodies.Perona has taken the pioneering step in [12] by studying a variationalmodel for orientation di�usion with applications to �ngerprint analysis. Hiswork de�nes the right framework for further deeper mathematical and com-putational study on the subject. Our present work is its followup and ex-tends it both theoretically and computationally. (Note: By the time wecompleted the initial version of this paper, we found surprisingly (and hap-pily) that Sapiro and his colleagues were also developing the di�usion the-ory for directions (corresponding to the feature manifold Sm) independently(see [18, 19]), almost at the same time. We shamelessly admit that someideas on chromaticity presented in this �nal version are impossible withoutbene�ting from this inspiring team.)

4 CHAN AND SHENThere are two key mathematical questions upon studying variationalrestoration of non- at features. First, it is not as obvious as in Euclideanspaces to come up with a feasible energy functional for non- at features.Perona [12] o�ered some bright physical intuitions for the orientation fea-ture, but a general mathematical framework is still missing, which maydelay many potentially important applications for other non- at features.Secondly, as one can easily expect, the associated Euler-Lagrange equationsor steepest-descent (di�usion) equations are generally non-linear because ofthe non- atness of feature manifolds and the intrinsic non-linearity of theirclassical counterparts (the TV model, for instance). To get our mathemat-ical models truly work for applications, it is important to develop e�cient(simple, stable, or fast) numerical algorithms for these non-linear equations.Answering these two fundamental questions is the main task of our presentpaper. We present a systematic way for constructing the energy functionsfor general non- at data or features. We interpret Perona's original modelin this general framework and discuss its generalizations. The mathemati-cal framework also extends the classical total variational (TV) model. Nu-merical evidence shows that the TV model is more successful than the L2model for restoring non- at features which have intrinsic singularities. Wealso design e�cient numerical algorithms to solve the non-linear restorationequations. Numerical results highlight the success of both our mathematicalframework and computational algorithms.The organization of the paper is as follows. Section 2 constructs the math-ematical foundation for the work of both Perona and ours. We study therestoration of non- at features in a general and abstract way. The startingpoint is the construction of the energy functional on a Riemannian featuremanifold. In section 3, we discuss restoration models speci�cally for orienta-tions and alignments, two non- at image features frequently encountered ingray level image analysis. Construction of energy functionals is through thenew concept|Locally Riemannian Distance. The latter half of the sectionis devoted to developing numerical algorithms that can solve e�ciently thenonlinear equations associated with the orientation and alignment features.Section 4 studies the computational issues for the chromaticity restorationon S2. A simple two-step �ltering algorithm on the sphere is established forthe discrete models. Numerical examples are gathered in section 5, where wediscuss applications in �ngerprint images, optical ows, and color images.

RESTORATION OF NON-FLAT IMAGE FEATURES 5Whereas our work was initially motivated by image processing, we believethat the mathematical framework and algorithms can �nd their applicationsin the restoration of general non- at data in other �elds as well.2. Variational Models and Restoration EquationsSince non- at features do not live in a at space, the energy function-als and associated restoration equations depend on the \geometry" of thenon- atness. The geometry is described by Riemannian manifolds and Rie-mannian metrics. These are the basic objects that we work with in thissection. We shall establish the most general models for the restoration ofnon- at features.In the two subsections, we study models for both continuous image do-mains and discrete image domains (or graphs, more generally). The latterhas the advantages of easier implementation in computers and friendly ac-cessibility to the widest readers.2.1. Models for continuous image domains. Let be an image domainin R2. For the purpose of image analysis, can be a square, a disk, or otherregular domains.Suppose that the non- at feature of interest lives on an m-dimensionalRiemannian manifold M . We shall call M the feature manifold. For ex-amples, the unit circle S1 is the feature manifold for orientation (or arrowswith unit length), while the real projective line RP1 is the feature manifoldfor alignment (or line segments with unit length).An image is a real function u : ! Rn (n = 1 or 3, practically). Thecollection of \all" images is denoted by the symbol (! Rn). One can putthe class labels such as Lp or BV (bounded variation) in front of it. Butsince we do not study the theoretical sides (i.e., existence and uniqueness) ofthe model equations in the present paper, the symbol allows more freedom.Moreover, the regularity of most images from natural scenes is usually poorsince they belong to the statistical category in a large degree.A map f : ! M is called a feature distribution and ( ! M) denotesthe collection of all feature distributions of interest. AnM -feature of imagesis a map F : (! Rn) �! (!M);such that for any image u, f = Fu is a feature distribution. That is, for anypixel p = (x; y) 2 , f(p) = Fu(p) is a (feature) point on M .

6 CHAN AND SHENIt is often true in applications, as outlined in the introduction section,that theM -feature distribution f = Fu associated to a given image u is oftennoisy. Our major goal is to construct denoising and restoration models usingthe classical tool of variational method.In practice, a feature distribution f 2 (!M) is typically a map withoutoverall smoothness. However, when we constructing a restortation model,as well practiced in the case of a�ne features, f(p) is always assumed tohave certain degree of regularities. For instance, we can assume that f(p)belongs to the Sobolev space W 1;2(;M). In numerical computation, it isfortunately true that models based on the good regularity assumption oftenwork e�ectively for irregular images as well. Next section discusses discretemodels, for which the regularity of data causes no problem in both theoryand computation.Under this setup, @xf(p) and @yf(p) are two tangent vectors in the tangentspace Tf(p)M . Let h �; � if(p) denote the Riemannian inner product at f(p),and k�kf(p) the induced norm in Tf(p)M . We shall omit the location subscriptif it is clear or insigni�cant in the context.We now de�ne the strength functione(f;p) =qk@xf(p)k2 + k@yf(p)k2:(1)To have a better understanding of its meaning, we introduce the correlationmatrix Cf(p) = "h @xf; @xf i h @xf; @yf ih @yf; @xf i h @yf; @yf i# :Then e(f;p) =qtrace(Cf(p)) =q�21 + �22 ;where �1 and �2 are the singular values of the m (the dimension ofM) by 2\matrix" [@xf; @yf] (which is well-de�ned under any local orthonormal frameonM). A similar formulation has also been proposed in Sapiro [17, 16] whenf is a vector valued distribution.The strength function de�ned in this way is rotationally invariant. Thatis, for any orthogonal transform in the image domainR : R2! R2;we have e(f(R�1�);Rp) = e(f(�);p):

RESTORATION OF NON-FLAT IMAGE FEATURES 7The reason is that the corresponding correlation matrices are orthogonallysimilar to each other. This property is meaningful in real life since we do notwant the results to depend on the direction in which an image is displayedbefore us.The L2 total energy of a feature distribution f isE(f) = 12 Z e2(f;p) dp:Inspired by the a�ne case where Rudin and Osher [14], Rudin, Osher andFatemi [15] �rst introduced the total variational (TV) model for imagerestoration, we de�ne the total variation to be the L1 total energyETV(f) = Z e(f;p) dp:Let d denote the metric onM induced by its Riemannian structure. Thatis, for any two points f and g 2M ,d(f; g) = inf� length(�);allowing � to go over all piecewise smooth paths that link f and g. (We shallalso allow other kind of distances later. See the discrete model in the nextsubsection.)Given a noisy feature distribution f(0)(p), the classical restoration modelassociated with the L2 energy is to solveminf E(f; �);where the �tted L2 energyE(f;�) = E(f) + �2 Z d2(f(0); f) dp= 12 Z e2(f;p) dp+ �2 Z d2(f(0); f) dp:As in the a�ne case, the �rst term regularizes the restoration solution andthe second one is a �tting constraint.For the TV restoration, the associated �tted TV energy isETV(f;�) = Z e(f;p) dp + �2 Z d2(f(0); f) dp:The constant � above is related to the Lagrange multiplier for the asso-ciated constrained optimization problem|Minimize E(f) (or ETV(f)) subject to 1jj Z d2(f(0); f)dp = �2;where jj is the area of and � the standard deviation of the noise. Inthe at case, the connection between the constrained and unconstrained

8 CHAN AND SHENformulations was discussed theoretically by Chambolle and Lions [2] andnumerically by Blomgren and Chan [1]. In image analysis, � is often con-veniently �xed or chosen a priori, or estimated by the gradient-projectionmethod (Rudin, Osher and Fatemi [15]).To solve the above variational problems, we study the associated Euler-Lagrange equations. For the �tted L2 energy E(f;�), the Euler-Lagrangeequation can be shown to be�@�x(@xf)� @�y (@yf)) + �2 gradfd2(f(0); f) = 0;where, at any pixel p = (x; y) 2 , gradfd2(f(0); f) denotes the gradientvector of the scalar function d2(f(0)(p); �) on M ; and @�x and @�y denote co-variant derivatives acting on vector �elds de�ned on f(). Speci�cally, let rbe the Levi-Civita connection (see Helgason [7]) on the feature Riemannianmanifold M . Then the covariant derivatives @�x and @�y are given explicitlyby @�x = r@xf; @�y = r@yf:They can legally act on any vector �eld X that is de�ned along f() due tothe locality property of covariant derivatives. For instance, ifM is a domainin Rm, then @�xX = @@xX(f(x; y)):Generally, when M is isometrically embedded in Euclidean space RN (N �m), a vector �eld X onM is also one in RN. Since the at derivative in RN@@xX(f(x; y))usually overshoots from the tangent space of M , one needs an extra step ofprojection for the covariant derivative|@�xX = �f @@xX(f(x; y));(2)where �f is the orthogonal projection from TfRN onto the tangent spaceTfM .The Euler-Lagrange equation can be solved by the in�nitesimal steepestdescent method (or time-marching method). This leads to the �tted di�usionequation @f@t = @�x(@xf) + @�y (@yf))� �2gradfd2(f(0); f);

RESTORATION OF NON-FLAT IMAGE FEATURES 9with the Neumann adiabatic boundary condition plus the initial conditionf ��t=0 = f0;where, f0 can be the given raw feature distribution f(0), for convenience. Allthe terms in the di�usion equation are in the tangent space Tf(p)M . Asthe marching time increases, f evolves to the equilibrium distribution of thefunctional.Similarly, for the TV model, the Euler-Lagrange equation becomes�@�x � @xfe(f;p)�� @�y � @yfe(f;p)�+ �2gradfd2(f(0); f) = 0:Steepest descent leads to a forced anisotropic di�usion equation@f@t = @�x � @xfe(f;p)� + @�y � @yfe(f;p)�� �2gradfd2(f(0); f):Remark 1. Let us understand the �rst two terms on the right-hand side ofthe last equation. Assume the image domain is 1-dimensional (i.e. a realinterval [a; b]) and parameterized by x. Then f(x) is a path on the featuremanifold, and minimization of Z ba kfxkdxis to reduce the total path length. This viewpoint connects our problemto the much familiar object | geodesic. A geodesic is self-parallel in thewell-known sense that if � is the unit tangent vector, thenr�� = 0:This condition also means that it is locally a \straight" line on the manifold.Now it is easy to see that � = fxkfxk = fxe(f;p) ;and the linearity of the Levi-Civita connection leads to� ddx��� fxe(f;p)� = kfxkr�� :This gives us a better geometric sense about the two \TV-terms" appearingin the last two equations.EXAMPLE 1. Feature manifold M embedded in RN.When the feature manifoldM can be isometrically embedded in Euclideanspace RN (such as the unit circle in R2 and the unit sphere Sm in Rm+1),the Levi-Civita connection is simply the Euclidean derivatives followed by a

10 CHAN AND SHENprojection as pointed out above. In this case, a feature point f can be seenas an ordinary vector in RN, and the de�nition of the strength functione(f;p) =qkfx(p)k2 + kfy(p)k2now only involves the ordinary di�erentiations of vectors and is exactlykrfk =pkrf1k2 + krf2k2 + � � �+ krfNk2;if f = (f1; f2; � � � ; fN) 2 RN. Moreover, the diminishing ow �elds F of theenergy functionals can be computed explicitly. When M = Sm is embed-ded in Rm+1, this work has been carried out recently in Tang, Sapiro andCaselles [19], and was also worked out independently by Osher [11]. For ageneral embedded feature manifold, for example, the diminishing ow �eldF = @�x(@xf) + @�y (@yf)for the L2 energy E is simply given byF = �f �f;(3)where, � is the at Laplacian on , and �f is the orthogonal projectionfrom TfRN onto TfM . Spheres Sm in Rm+1 are the most interesting to imageprocessing. For the sphere, it was shown by Tang, Sapiro and Caselles [19]and Osher [11] that the diminishing ow for the L2 energy isF (f) = �f+ krfk2f;(4)and for the TV energy,FTV(f) = r �� rfkrfk�+ krfk f:(5)Osher [11] established these formulae by the method of Lagrange multiplieranalysis, and Tang, Sapiro and Caselles [19] proved them by working outexplicitly the variational problem. Here we give a simpler geometric proofbased on the special form of the Levi-Civita connection (2).Proof of Eq. (4) and (5) (see Figure 2). Take the TV diminishing owEq. (5) for example. For sphere Sm, the projection at f is�fv = v� (v � f)f:

RESTORATION OF NON-FLAT IMAGE FEATURES 11S

T S2

2

)grad( f )

|grad( f )|div(

fF

TV(f)

grad( f )grad( f )| f|

f

- Figure 2. The TV diminishing ow.Hence for any distribution f : ! Sm,FTV(f) = �fr � rfkrfk= r � rfkrfk � �r � rfkrfk � f� f= r � rfkrfk � �r � f � rfkrfk � krf k� f= r � rfkrf k + krfk f;since f � rf = rf2=2 = 0. �This continuous model provides a universal language for restoring non- atfeatures. However, the model assumes that the readers have some knowledgeof modern di�erential geometry, concepts such as the Levi-Civita connec-tion. It is unavoidable in this continuous formulation since the variationalformulation is global. Fortunately, there does exist a way that one can cir-cumvent this technical di�culty. We now develop a simpler model for \dig-ital" image domains, namely graphs, where di�erentiation of vector �elds(i.e. Levi-Civita connection) can be avoided. Besides, the discrete modelshave their own computational advantages as discussed later in the paper.2.2. Models for discrete image domains. Since numerically it is alwaysnecessary to lay down a �nite discrete grid to approximate the continuousimage domain , it is practically convenient and important to study directly

12 CHAN AND SHENnon- at features over a discrete grid, or more generally, a graph. This is themain task of this section. Perona took the �rst step in [12].Let G[n] be an undirectional graph over a �nite set of nodes n (\n"stands for \numerical"), and G is the dictionary of edges. We write � � �if nodes � and � are linked by an edge. For any node � 2 n, denote byN� all its neighbors f� 2 n : � � �g:We start with the de�nition of the concept of LRD, which shall play animportant role in our models. A locally Riemannian distance (LRD) is apiecewise smooth continuous function dldl : M �M ! R+ = [0; 1);such that(1) For any f; g 2M , dl(f; g) = dl(g; f).(2) dl(f; g) = 0, if and only if f = g.(3) dl(f; g) + dl(f; h) � dl(g; h).(4) dl(f; g) = d(f; g) +O(d2(f; g)) as d(f; g)! 0.Here d is the induced Riemannian distance de�ned in the previous section.Remark 2. Conditions (1){(3) are the basic ingredients of a distance, whichauthorize dl to \legally" measure the amplitude of noise. Condition (4)restricts our attention to only those distances that are locally Riemannian.The motivation is as follows. Suppose the feature Riemannian manifold islocally at (i.e. locally isometric to Euclidean spaceRm). Then condition (4)ensures that the \non- at Laplacian" (see the remarks at the end of thissection) is locally exactly the ordinary Laplacian. In this way, many classicalanalytical results may be transplanted easily.EXAMPLE 2. Perona's distance.Consider the unit circle S1 as the concerned feature manifold. For anytwo points f = ei�1 and g = ei�2 , de�nedperona(f; g) =p2(1� cos(�1 � �2)):Symmetry is immediate. Now if dperona(f; g) = 0. Then cos(�1 � �2) = 1,which implies that f = g. It is also clear that the leading term of dperona(f; g)is the Riemannian distance when f and g are close. Therefore, to show dperona

RESTORATION OF NON-FLAT IMAGE FEATURES 13is a locally Riemannian distance, it su�ces to verify the triangular relation.The proof follows. Formulasin(�1 + �2) = sin �1 cos �2 + sin �2 cos �1leads to j sin(�1 + �2)j � j sin(�1)j+ j sin(�2)j:Noticing that j sin �j =r1� cos 2�2 ;we obtain p1� cos(2�1 + 2�2) �p1� cos 2�1 +p1� cos 2�2:Now that �1 and �2 are arbitrary, this last inequality holds even without thefactor 2, which eventually leads to (by noticing that �3 � �2 = (�1 � �2) +(�3 � �1))p1� cos(�3 � �2) �p1� cos(�1 � �2) +p1� cos(�3 � �1):It is the triangular relation for dperona!It is the distance that Perona has proposed in his work [12] (though hedid not formulate it in this way).As an example of a non-LRD, considerdline(f; g) =r1� cos(2(�1 � �2))2 :It is not a local distance for S1 sincedline(1;�1) = 0:For the real projective line RP1, however, dline is an LRD.For a given LRD dl, we de�ne the associated strength of a given featuredistribution f to be e(f;�) = [ X�2N� d2l (f�; f�)] 12 ;which generalizes the continuous strength function in Eq. (1) or krfk whenf is a at function. Next, we can de�ne the l2 total energyE(f) = X�2n 12e2(f;�);

14 CHAN AND SHENand the total variation (TV)ETV(f) = X�2n e(f;�):For a given feature distribution f(0) and some weight constant �, de�ne the�tted l2 total energyE(f;�) = E(f) + � X�2n 12d2l (f(0)� ; f�);and the �tted TV total energyETV(f;�) = ETV(f) + � X�2n 12d2l (f(0)� ; f�):To establish the di�usion equations, we �rst compute the gradient �eldsfor the total energies. For the l2 total energy,@E(f)@f� = X�2N� @@f�d2l (f�; f�);(6)for the TV total energy,@ETV(f)@f� = X�2N� � @@f�d2l (f�; f�)��1=e(f;�) + 1=e(f; �)2 � :(7)Hence the corresponding �tted di�usion equations aredf�dt = � X�2N� @@f�d2l (f�; f�)� �2 @@f�d2l (f(0)� ; f�);(8)df�dt = � X�2N� � @@f�d2l (f�; f�)��1=e(f;�) + 1=e(f; �)2 �� �2 @@f�d2l (f(0)� ; f�):(9)Notice that here our starting point is the locally Riemannian distance dl.In his work on orientation di�usion, Perona [12] started with a choice ofenergy, which is originally inspired by physical considerations. For generalnon- at feature manifolds, dl (and the special case of dl = d, the Riemanniandistance) seems to be more fundamental.As an example, we now apply the discrete models to the restoration ofchromaticity. The continuous di�usion model without the �tting constraintwas �rst discussed in Tang, Sapiro and Caselles [18]. Our discrete modelhas the following advantages: 1) unlike the continuous model, one doesnot need to choose a numerical discretization scheme (often complicated)for the spatial derivatives; 2) the discrete model rigorously diminishes theenergy function; while in the continuous model, the numerical PDE's only

RESTORATION OF NON-FLAT IMAGE FEATURES 15asymptotically (i.e. as the step size of time or space tends to zero) diminishthe continuous energy functionals.EXAMPLE 3. Restoration of the chromaticity feature on S2.The 2-dimensional sphere S2 models the chromaticity feature of colorimages. Let I(p) denote the linear RGB vector value at pixel p. Then I(p)can be perceptually separated into the brightness component B = kI(p)k,and the chromaticity componentf(p) = I(p)B :f(p) records the saturation degree of colors. We now apply the discretemodels developed above to chromaticity restoration. Take the TV model (9)for example.Embed S2 in R3 and take dl to be the embedded Euclidean distance, i.e.dl(f; g) := kf� gkR3 =p(f� g)2; for any f; g 2 S2:Let rf denote the gradient for f 2 R3 and @=@f the gradient on S2. Thenfor any scalar function G(f) on R3,@@fG(f) = �frfG(f):Therefore, for any �xed feature point g 2 S2,@@fd2l (f; g) = �frf(f� g)2= 2�f(f� g) = �2�f(g)= �2(g� (g � f)f):The TV model (9) is explicitly given bydf�dt = X�2N��f�(f�)� 1e(f;�) + 1e(f; �)�+ ��f�(f(0)� ):(10)There are no spatial derivatives in the last equation! In computation, oneonly needs to take care of the time discretization, which is easy to handlewith the geodesic marching scheme we propose in Section 4. It is easilyseen that the �tted diminishing ow on the right-hand side is indeed on thetangent plane at f�, which is crucial to prevent f� from wandering away fromthe feature manifold in�nitesimally. This completes our example. We willreturn to it later in Section 4.More generally, suppose the feature manifoldM is isometrically embeddedin RN. Let �f denote the orthogonal projection from TfRN onto the tangent

16 CHAN AND SHENplane TfM . If we choose dl to be the straight line distance in RN (butrestricted on M), then the TV restoration equation is given bydf�dt = X�2N��f�(f� � f�)� 1e(f;�) + 1e(f; �)� + ��f�(f(0)� � f�):(11)Especially, ifM is the level set of a non-singular (i.e. with non-zero gradienteverywhere) function �(f) in Rm+1: � = 0, then�fv = v� (v � �f) �fk�fk2 ;where �f denotes the gradient of � and v 2 TfRm+1. In the continuous case,Osher [11] �rst proposed to consider the level-set feature manifolds (andindeed, both S1 and S2 are level-set manifolds).Some comments are in order now.Remark 3.1. Non- at Laplacian. Suppose the feature manifold M is in fact a atdomain in Rm. Take dl to be the natural Euclidean distanced(f�; f�) = kf� � f�k; in Rm:Then @d2l (f�; f�)@f� = 2(f� � f�);and �@E(f)@f� = 2 X�2N�(f� � f�);which is the at graph Laplacian acting on f (up to a constant). GraphLaplacians unlock many combinatorial secrets of a graph, a topic whichhas been recently well studied in the spectral graph theory (see Chung [4],for example).Therefore, formula (6) can be viewed as a \non- at" Laplacian.2. Nonlinear Adaptive Filter. The right-hand side of formula (7) gener-alizes the curvature term frequently appearing in image analysis (Rudinand Osher [14]; Rudin, Osher and Fatemi [15]) |r� rujruj� ;which leads to the total variational anisotropic di�usion of a gray levelimage u, and is a special but the most commonly applied case of Peronaand Malik's general concept of anisotropic di�usion [13]. For a generalgraph, formula (9) de�nes a non-linear adaptive di�usion. The feature

RESTORATION OF NON-FLAT IMAGE FEATURES 17di�uses faster near a node where the strength e is smaller, or equiva-lently, where the feature distribution is smoother. It adaptively slowsdown when the distribution gets rougher and the local strength grows.This adaptivity is particularly useful for preserving local characteristicarchitectures of a given feature distribution, such as the singularities of�ngerprint \waves," and feature \edges."3. The Graph Curvature Term. Suppose that M is isometrically embed-ded in RN, and dl is the restricted Euclidean distance. Denote by K�(f)the at ow �eld in RNX�2N�(f� � f�)� 1e(f;�) + 1e(f; �)� :Then the TV restoration equation (11) reads@f�@t = �f�K�(f) + ��f�(f(0)� � f�):We now show that K�(f) is exactly the graph version for the classicalcurvature term. Fix a node �. Let !e denote any of the directed edgesstarting from �. De�ne the directional derivative of a at feature f along!e by @f@!e = @f@!e ���� = @f@!e ���� := f� � f�;where � is the tail node of !e . Then the strength function is exactlye(f;�) = 24X!e��� @f@!e �235 12 ;which generalizes the magnitude of the ordinary gradient. Here !e � �includes all directed edges starting from �. Under this setup, it is easyto see that K�(f) = X!e�� @@!e 1e @f@!e ����;which is exactly in the form of graph divergence and gradient!3. Restoration of Orientation and Alignment: Equations andAlgorithmsIn this section, based on the general mathematical models constructedabove, we study the restoration of two classes of non- at features frequentlyencountered in gray level image analysis, namely, the orientation feature and

18 CHAN AND SHENthe alignment feature. For the time being, we shall only discuss the discretemodels.3.1. Restoration equations for orientation. Orientation lives on theunit circle S1, which can be naturally parameterized by the angle parameter�. De�ne the sawtooth function on R:D(x) = 8<:jxj jxj � �2� periodic extension for general x:Then the Riemannian distance d is given byd(ei�1 ; e�2) = D(�1 � �2);or simply, d(�1; �2) = D(�1� �2), which measures the length of the shortestarc on S1 that links the two feature points. Recall that in the previousexample, we discussed Perona's locally Riemannian distancedperona(�1; �2) =p2(1� cos(�1 � �2)) = 2 ����sin �1 � �22 ���� :The fact that it is an LRD is better seen geometrically through Figure 3,which shows that dperona measures the Euclidean distance of the two featurepoints ei�1 and ei�2 in R2 (instead of measuring along S1).x

e

e

i θ 1

i θ 2

θ − θ2 1

2|| sin

Figure 3. The Perona distance.Remark 4. The Perona distance belongs to a wider class of isometrically em-bedded distances (IED). An IED, denoted by die, is the result of an isometricembedding of S1 in R2 (see Figure 4). Precisely, this means to establish asmooth (C2, say) injective mappingF : S1 ! R2;

RESTORATION OF NON-FLAT IMAGE FEATURES 19which satis�es the isometry conditionkdFkR2 = kdskS1;for any in�nitesimal segment ds on the circle and its image dF . The IEDassociated with F is de�ned bydie(f; g) = kF (f)� F (g)kR2;(12)for any f; g in S1. die is an LRD because of the isometry condition. Sucha distance is biased, or not rotationally invariant. It can be useful in occa-sions where we do know a priori that certain directions are more robust orimportant than the others.F

R2

x

x

f

g

x

x

f)(F

(g)F

d (f, g) ieFigure 4. The isometrically embedded distance (IED).One advantage of dperona overD is the smoothness of d2perona, on which wetake derivatives in the restoration formulae Eq.(6{9). Remarkably, dperonais also rotationally invariant. In what follows, we shall only discuss thisdistance function.We now derive the parametric restoration equations for dperona. Parame-terize the circle by � ! ei� . Then the tangent vector @=@� (or equivalently,iei�) is an orthonormal basis for the tangent space at each point. All vectorsappearing in the di�usion equations are expressible using this basis. Assumef� = ei�� . Then @f�@t = @��@t @@�� ;@d2perona(f�; f�)@f� = 2@(1� cos(�� � ��))@�� @@�� = 2 sin(�� � ��) @@�� ;@d2perona(f�; f(0)� )@f� = 2@(1� cos(�� � �(0)� ))@�� @@�� = 2 sin(�� � �(0)� ) @@�� :

20 CHAN AND SHENThen the �tted l2 di�usion equation (8) becomesd��dt = X�2N� 2 sin(�� � ��) + � sin(�(0)� � ��);(13)which is exactly the equation that Perona �rst studied in [12]. Similarly, the�tted TV di�usion equation isd��dt = X�2N� sin(�� � ��)� 1e(�;�) + 1e(�; �)� + � sin(�(0)� � ��);(14)where, according to the previous section, the strength ise(�;�) = [ X�2N� 2(1� cos(�� � ��))] 12Remark 5. Notice that the parametric equations (13) and (14) are equivalentto equations (8) and (9). This is one of the main advantages of the discrete(graph) model | it is less sensitive to the discontinuities in the raw data �(0).In fact, a graph is \blind" to discontinuities since there is no in�nitesimaldistance. For the continuous model, the parametric equations (i.e. on �) arenot equivalent to the corresponding embedded equations (i.e. on f, treatedas a vector in R2), when the given raw data �(0) has discontinuities.Equations (13) and (14) generalize classical at equations. To see it,consider a smooth orientation distribution ��, that is,�� � �� = O(�);for all � 2 N� and some small characteristic parameter �. ThenX�2N� sin(�� � ��) = X�2N�(�� � ��) + O(�3):The leading term is exactly the at Laplacian on �. This is generally truefor any choice of locally Riemannian distances because of the last conditionin the de�nition list.Similarly, the strength function can be shown to bee(�;�) = [ X�2N� (�� � ��)2] 12 (1 +O(�2)):For example, for a typical internal cross node � = (i; j) in image processing,its neighbors are�u = (i; j � 1); �d = (i; j + 1); �l = (i� 1; j); �r = (i+ 1; j):

RESTORATION OF NON-FLAT IMAGE FEATURES 21Thus the leading term of the strength function is simply the �nite di�erenceapproximation to the magnitude of the gradientp2hkr�k = p2h"�@�@x�2 + �@�@y�2# 12 ;where h is the discretization step size.3.2. Restoration equations for alignment. For alignment, direction ofa line segment is unimportant (see Figure 5). This amounts to saying thatthe feature manifold is the real projective line RP1.Energy Increases. Largest Energy Energy DecreasesFigure 5. The alignment feature.A line segment in the direction of � is denoted by l�. Then l� and l�+�denote the same line segment. A one-to-one correspondence between RP1and the unit circle S1 can be established by representing each line segmentl� by a point ei2�. This embedding technique has already been practiced inimage processing (see Perona [12], and Granlund and Knuttson [6]).The Riemannian metric on RP1 is given byd(l�1 ; l�2) = j�1 � �2j;whenever j�1 � �2j < �=2.The line distance mentioned in Section 2 is given bydline(l�1 ; l�2) =r1� cos(2(�1 � �2))2 = j sin(�1 � �2)j:dline is in fact the pull-back of dperona under the circle representation ofRP1. If fact, any LRD dl on the circle (an IED, especially) can be pulledback to RP1 by de�ningdRP1l (l�1 ; l�2) = 12dl(ei2�1 ; ei2�2):In what follows, we shall only work with dline.

22 CHAN AND SHENSimilar to the orientation di�usion, we obtain the �tted l2 di�usion equa-tions for dline d��dt = X�2N� sin(2�� � 2��) + �2 sin(2�(0)� � 2��):(15)The �tted TV di�usion equation isd��dt = X�2N� 12 sin(2�� � 2��)� 1e(�;�) + 1e(�; �)�+ �2 sin(2�(0)� � 2��);(16)where, the local strength is given bye(�;�) = 24 X�2N� 1� cos(2�� � 2��)2 35 12 :We shall only discuss these two equations in the following. Our laternumerical experiment shows that the alignment restoration model is partic-ularly useful for �ngerprint processing (also see Perona [12]).3.3. Algorithms. The parametric restoration equations (13{16) are all non-linear. The good news from the discrete model is that the steady equationsare all algebraic. Various existing solvers for non-linear algebraic equationscan thus make their contributions, though careful modi�cations should bemade.In this section, we discuss three numerical approaches. We �rst study thel2 restoration equations in details, and then point out how to make suitableadjustments for the TV restoration equations. Let n denote the graphimage domain. For the purpose of image processing, one can be happy withthe standard rectangular grid, where each (internal) node � = (i; j) has fourneighboring nodes � = (i; j�1); (i�1; j) such that � � �. Considering thatour models and algorithms may be applicable even out of the scope of imageprocessing, in the following, we still state the algorithms using the abstractnotation for nodes: �; �; � � � .Recall from the previous subsections that we try to solve directly thesystem of steady restoration equations2X��� sin(�� � ��) + � sin(�� � ��) = 0; � 2 n:(17)

RESTORATION OF NON-FLAT IMAGE FEATURES 23Here � is the given raw (noisy) angle distribution. The associated totalenergy is given byE(�;�) = 2X�� (1� cos(�� � � )) + � X�2n(1� cos(�� � ��));where � � denotes all distinct edges.3.3.1. The non-linear Gauss-Seidel method. We �rst propose a nonlinearGauss-Seidel method for solving it numerically. It proceeds as follows.Choose an initial guess �0 (�0 = �, say) and assign a linear ordering forall nodes: � � � < � < � < < � � � :For convenience, for each node �, de�neN�� := f� 2 N� : � < �g; N+� := f� 2 N� : � > �g:Let k denote the global updating clock and � a certain node. Supposethat in the beginning of step kj�, we have the following data available�k� : � < �; �k�1� ; �k�1� : � > �:We try to update �k�1� to �k� by the end of step kj�.Set x = �k�. According to the restoration equation (17), we demand that2 X�2N+� sin(�k�1� � x) + 2 X�2N�� sin(�k� � x) + � sin(�� � �k�1� ) = 0:(18)De�ne C = 2 X�2N+� cos(�k�1� ) + 2 X�2N�� cos(�k�);S = 2 X�2N+� sin(�k�1� ) + 2 X�2N�� sin(�k�):Then S cosx� C sin x = �� sin(�� � �k�1� );(19)De�ne r =pC2 + S2; ! = \(C; S):Here \(a; b) is the angle in the direction of (a; b). Then Eq. (19) simpli�esto sin(! � x) = ��r sin(�� � �k�1� );

24 CHAN AND SHENwhich gives (mod 2�)x = ! + arcsin ��r sin(�� � �k�1� )� ;or x = ! � arcsin ��r sin(�� � �k�1� )�� �:Here we take the principle range of arcsin to be [��=2; �=2]. To determinewhich one we should choose, consider the updated orientation distribution�0 = (�k� : � < �; x; �k�1� : � > �):We require that the total l2 energyE(�0) = 2X�� (1� cos(�0� � �0 ))can be as small as possible. This implies that x should be chosen so that2 X�2N�� (1� cos(�k� � x)) + 2 X�2N+� (1� cos(�k�1� � x))is the smallest, or equivalently,2 X�2N�� cos(�k� � x) + 2 X�2N+� cos(�k�1� � x)can be the biggest. This amounts to maximizingC cosx+ S sin xunder the constraint (19). Hence x is given by�k� = x = ! + arcsin ��r sin(�� � �k�1� )� :(20)This completes the kj�{step of the nonlinear Gauss-Seidel updating.Remark 6. Careful readers may have noticed that in the updating equa-tion (18), we do not replace �k�1� by x in the �tting term. This is oneimportant lesson we learned from numerical experiments. If one permitsthat replacement, then the algorithm tends to giving a constant solution.Gauss-Seidel is simple, local and reliable, but with a relatively slow con-vergence compared with the other schemes we propose below.

RESTORATION OF NON-FLAT IMAGE FEATURES 253.3.2. Linearized iteration method (LIM). Let sinc(x) denote the functionsin x=x. Then Eq. (17) can be written asX���(�� � ��)2 sinc(�� � ��) + �(�� � ��) sinc(�� � ��) = 0:(21)The linearized iteration method generates the iteration as follows. Take aguess �0 (taken to be �, say) for the initial step 0. Suppose in the beginningof step k, �k�1 is available. We update �k�1 to �k by solving a linear algebraicsystem. De�ne a� = 2 sinc(�k�1� � �k�1 ); if � � ;(22) c� = � sinc(�� � �k�1� ); for all � 2 n:(23)Based on Eq (21), for each �, we require thatX���a��(�k� � �k�) + c�(�� � �k�) = 0:This leads to an ordinary linear systemA�k = bwith b = (c��� : � 2 n)T and A = (A�� : �; � 2 n):A�� = c� +X���a��;A�� = 8<:�a�� ; if � 2 N�;0; else:Under this linearization, the non-linear updating can be achieved usingany numerical linear algebra solver. LIM converges geometrically, fasterthan the nonlinear Gauss-Seidel scheme above.3.3.3. The time marching method. In this subsection, we approach the steadydistribution by time marching according to the discrete di�usion equation.Recall that the �tted di�usion model using l2 energy is given byd��dt = 2X��� sin(�� � ��) + � sin(�� � ��); � 2 n:(24)Take a marching step size h and set�k� = ��(kh):Then the system of di�erential equations (24) can be solved using variousnumerical methods.

26 CHAN AND SHENIntroduce the non- at Perona Laplacian symbol �0�0 � = X��� sin( � � �):The simplest single-step method that ignores the non-linearity of the di�u-sion equation is the Euler method�k+1� = �k� + h2�0�k� + h� sin(�� � �k�);for each step k. In practice, this is implemented by choosing two smallpositive free parameters a and b:�k+1� = �k� + a�0�k� + b sin(�� � �k�):Numerically, Euler method su�ers from low accuracy and slow conver-gence. The major advantage is its simplicity and easy implementation, es-pecially for such a non-linear problem. It can also be easily parallelized.Moreover, from the practical point of view, the exact steady distribution isnot so unique and critical for human vision. Numerical experiments showthat near-steady distributions are usually already good enough for human vi-sion and recognition of characteristic patterns residing in a given raw (noisy)feature distribution. This is the good news Euler method.From the numerical point of view, one can also try other high order single-step methods such as Runge-Kutta 2. But both methods require a smalltime step to ensure stability. In order to achieve faster marching by allowinglarger step size h, we discuss the implicit trapezoidal scheme (or second-orderAdams-Moulton method, see Golub and Ortega [5]) combined with the ideaof linearization in the preceding subsection. The major advantage of such ascheme is its stability and permission for a large marching step size.Recall that for a system of equations liked��dt = f�(�); � 2 ;the implicit trapezoidal schemes at step k is given by�k+1� = �k� + h2 [f�(�k) + f�(�k+1)]:Applying it to our problem, we have�k+1� = �k� + [h�0�k� + h�2 sin(�� � �k�)] + [h�0�k+1� + h�2 sin(�� � �k+1� )];or�k+1� � [h�0�k+1� + h�2 sin(�� � �k+1� )] = �k� + [h�0�k� + h�2 sin(�� � �k�)]:

RESTORATION OF NON-FLAT IMAGE FEATURES 27Next, similar to the previous method, we linearize the left hand side by thesubstitution Xk+1 sincXk ( sinXk+1;(25)for any expression Xk+1 involving the current unknown �k+1. De�nea� = h sinc(�k� � �k ); if � � ;c� = h�2 sinc(�� � �k�); � 2 n;b� = �k� + h�0�k� + h�2 sin(�� � �k�) + c���; � 2 n:Then the above implicit scheme leads to a linear system of equations A�k+1 =b with b = (b� : � 2 n)T and A = (A�� : �; � 2 n):A�� = c� +X���a��;A�� = 8<:�a�� ; if � 2 N�;0; else:This linear system is bandlimited, symmetric and positive de�nite for suf-�ciently small h and can thus be solved easily using the standard numericallinear algebra solvers.3.3.4. Algorithms for the TV restoration equations. With the algorithmsready for the �tted l2 energy, it is easy to modify each one separately forthe �tted TV energy. We summarize them brie y below.The non-linear Gauss-Seidel method. The modi�cation is done by comput-ing the local strengths e's explicitly without involving x. Therefore, Eq. (19)holds with a new pair of constants (C; S). The discussion there is also valid.For the at case, the idea appeared in Chan and Vese [3].The linearized iteration method (LIM). The sin{sinc linearization techniqueextends to the TV model by modifying the constant a�� toa�� = sinc(�k�1� � �k�1� ) � 1e(�k�1;�) + 1e(�k�1; �)� :This generalizes the algorithm for the classical at case0 = �r � � rujruj�+ �(u� u0);where Vogel and Oman [20] proposed the linear iteration scheme0 = �r � � rukjruk�1j� + �(uk � u0):

28 CHAN AND SHENThe time marching method. The modi�cation is done by modifying (25) toXk+1 sincXke(�k ; �) ( sinXk+1e(�k+1; �) :Remark 7.1. In practice, the local strength function e is usually modi�ed toea =pa2 + e2;for some small parameter a (a = 10�4, for example). This modi�cationavoids the zero denominator in all the above TV formulae; On the otherhand, it produces an intermediate e�ect between the TV energy and thel2 energy. If a is large enough, then this modi�ed TV model gets closerto the l2 energy. This regularization technique is commonly practiced inthe classical literature (see Marquina and Osher [8], for example).2. For the time marching method, as Marquina and Osher [8] recently pro-posed for the continuous case, instead of solving the direct in�nitesimalsteepest descent equation (16)@��@t = X�2N� 12 sin(2�� � 2��)� 1ea(�;�) + 1ea(�; �)� + �2 sin(2�(0)� � 2��);one can solve its preconditioned version@��@t = X�2N� 12 sin(2�� � 2��)�1 + ea(�;�)ea(�; �)�+ �2ea(�;�) sin(2�(0)� � 2��):Since the \diagonal" multiplier ea(�;�) is non-zero, the preconditionedequation still marches to the original steady solution. However, fromthe numerical point of view, this modi�cation dramatically improves themarching speed and numerical stability.4. Restoration of Chromaticity: Algorithms and DiscussionsIn this section, we discuss the restoration models for the chromaticityfeature living on the unit sphere S2 and the related computational issues. Weshall discuss the computational issues only for the discrete models developedin Section 2. For the work on the continuous pure di�usion model (i.e.without the �tting term), see Tang, Sapiro and Caselles [18, 19].

RESTORATION OF NON-FLAT IMAGE FEATURES 294.1. The continuous models. Let f : ! S2 be a given raw noisy chro-maticity distribution. In Section 2, we have discussed the continuous restora-tion model corresponding to the L2 and TV regularization energies:ft = �f+ krfk2f+ ��f(f(0)� f); (L2)ft = r � � rfkrfk� + krfkf+ ��f(f(0) � f): (TV)Here we have chosen the distance function for the �tting term to be therestricted Euclidean distance in R3 (not the geodesic distance on S2). Theorthogonal projection is given by�f : TfR3! TfS2; �fv := v � (v � f)f:Since �ff = 0, the two equations simplify toft = �f+ krfk2f+ ��ff(0);(26) ft = r � � rfkrfk� + krfkf+ ��ff(0):(27)One can start the evolutions with the initial conditionf��t=0 = f(0);for instance.There are two practical problems with these continuous models. Take theTV model for instance. First, if a numerical scheme is sought for Eq. (27),we must take a special care for the diminishing ow termFTV(f) = r �� rfkrfk�+ krfk f;to make sure that the numerical FTV(f) does lie on the tangent plane TfS2.If the distribution f is smooth, this tangency condition causes no seriousproblem since it is in any case satis�ed to the leading order. But our taskis to denoise and restore feature distributions with poor regularities! Wehave to permit discontinuities and singularities. In this realistic situation,typically, the numerical diminishing ow FTV overshoots the tangent planeTfS2. This can of course be �xed by arti�cially projecting it back onto thetangent plane. But by doing so, we introduce a new degree of uncertaintyregarding the diminishing e�ect of the algorithm.Besides this overshooting problem, another issue is that we lack a rigorousexplanation for why the numerical evolution truly minimizes the continu-ous energy functional ETV(f;�) on S2. The gap between continuous energy

30 CHAN AND SHENfunctionals and numerical PDE's cannot be smoothed away easily becauseof the discretization error.These are the intrinsic problems with the continuous model. They disp-pear in the discrete formulation, since the latter starts with discrete ener-gies and results in algebraic equations (on S2), which need no numericaldiscretization for spatial derivatives.4.2. The discrete models. Let f(0) : n ! S2 be the given raw discretechromaticity distribution. By taking dl to be the restricted Euclidean dis-tance, we have established in Section 2 the discrete restoration equationscorresponding to the l2 and TV energies:df�dt = 2 X�2N��f� f� + ��f�f(0)� ; � 2 n; (l2)df�dt = 2 X�2N��f� f� � 1e(f;�) + 1e(f; �)� + ��f�f(0)� ; � 2 n: (TV)De�ne the weight w��(f) = 1e(f;�) + 1e(f; �);for any � � �. Then w�� = w��. Rewrite the above equations using thelinearity of the projection operatordf�dt = �f� 0@X�2N� 2f� + �f(0)� 1A =: F�(f); � 2 n;df�dt = �f� 0@X�2N�w��f� + �f(0)� 1A =: FTV� (f); � 2 n:The steady restoration equations are thus given byF�(f) = �f� 0@X�2N� 2f� + �f(0)� 1A = 0; � 2 n;(28) FTV� (f) = �f� 0@X�2N�w��f� + �f(0)� 1A = 0; � 2 n:(29)Here 0 denotes the zero vector in TfS2. These are \algebraic" equations onthe chromaticity sphere S2.Nowwe discuss the numerical implementations of both the marching equa-tions and the steady algebraic equations.

RESTORATION OF NON-FLAT IMAGE FEATURES 314.2.1. Geodesic marching for the discrete evolution equation. Take the TVmodel df�dt = FTV� (f); � 2 n;for example. Set fn� = f�(n�t). The classical Euler method is a at marching~fn� = fn�1� + �tFTV� (fn�1);followed by a projection (see Figure 6)fn� = ~fn�k~fn�k :f f

f

n n-1

F TV

-

n

f n-1

)(

geodesic marching

α αα

α

Figure 6. A at marching followed by a projection, and thegeodesic marching.To compress these two steps into a single one, we propose the geodesicmarching:fn� = cos ��tkFTV� (fn�1)k� fn�1� + sin ��tkFTV� (fn�1)k� FTV� (fn�1)kFTV� (fn�1)k :(30)It means that fn�1� marches for distance �tkFTV� (fn�1)k along the big cir-cle (i.e. the geodesic) determined by fn�1� and the ow vector FTV� (fn�1).Therefore, the marching always stays on the sphere. It is easy to see that forsmall �t, the above two-step classical scheme is equivalent to the geodesicmarching up to the leading order. Precisely, the latter always marches a dis-tance O(�t2) farther than the former (see Figure 6). The geodesic marchingscheme is intrinsic for the geometry of the feature manifold, and thus hasthe most general meaning for numerical restorations of non- at features.

32 CHAN AND SHEN4.2.2. Fixed-point iteration, �ltering, and the Convex Cone Theorem. Againtake the system of TV equationsFTV� (f) = �f� 0@X�2N�w��f� + �f(0)� 1A = 0; � 2 n;for example. It can be interpreted as: for the optimal restoration f, at eachnode �, P�2N� w�� f� + �f(0)� is parallel to f�.This explanation inspires the iteration scheme:~fn� = X�2N�w�� (fn�1)fn�1� + �f(0)� ;fn� = ~fn�=k~fn�k:Or, to stabilize the algorithm (and even lead to new insights), de�neh��(f) = w��(f)P 2N� w� (f) + �; h��(f) = �P 2N� w� (f) + �:We consider h as an adaptive lowpass �lter. Then the above algorithmstabilizes to ~fn� = X�2N� hn�1�� fn�1� ++hn�1�� f(0)� ;(31) fn� = ~fn�=k~fn�k:(32)Here hn�1�� = h��(fn�1).This �ltering formulation immediately rewards us with one importantproperty. A region � on S2 is said to be a convex cone if its spanning cone~� in R3 ~� = R+� � = fr� v : r � 0;v 2 �gis convex.Theorem 1 (Convex Cone Theorem). Suppose the raw data f(0)� 's belong toa convex cone �. If we start the above iterations using f(0) as the initial data,then for any step n, all fn�'s belong to �.The simple proof is left for our readers. Since RGB values are nonnegative,the chromaticity feature lives on the positive cone (one eighth of the sphere),which is convex. Therefore, the above �xed point iteration scheme keeps newfeature points from wandering away from the positive cone on the sphere.This another advantage of the discrete model.

RESTORATION OF NON-FLAT IMAGE FEATURES 33The above �xed-point iteration is similar to the Gauss-Jacobi method innumerical linear algebra. In the same fashion, we can construct the sphericalGauss-Seidel-like iteration. Assign a linear ordering for all the pixel nodes:� � � < � < � < < � � � :Let n denote the Gauss-Seidel clock and � a pixel under consideration.Suppose in the beginning of step nj�, we have available the data(fn� : � < �; fn�1� ; fn�1� : � > �):(33)Within step nj�, we update fn�1� to fn� by~fn� = X�2N�� h0��fn� + X�2N+� h0��fn�1� + h0��f(0)� ;fn� = ~fn�=k~fn�k:Here the lowpass �lter coe�cients (h0��) are de�ned similar to the previousalgorithm, but for the distribution given by (33). The Convex Cone Theoremstill holds for this non-linear Gauss-Seidel method.5. Numerical Examples and Applications5.1. Denoising by the l2 model: 1-D example (Figure 7). Figure 7displays the restoration e�ect of the l2 model for a noisy orientation distribu-tion. The �rst subplot shows the angle distributions, and the rest three arethe corresponding orientation distributions. In the �rst subplot, the solidline represents the clean angle distribution. Then uniform random noise isadded to produce the dotted dash line. The restoration with the a prioriLagrange multiplier � = 0:01 is given by the dash line. Small � leads tosmooth interiors. However, the resulting weak e�ect of the �tting term maysmear orientation \edges," as clearly observable from the �gure.5.2. Restoration by the TVmodel: 1-D example (Figure 8). Similarto Figure 7, we apply the TV model (14) to restore the noisy orientationdistribution. We have chosen � = 1, and the small parameter a discussed inSection 3.3.4 to be 0:0001.Compared to the l2 model, the TV model clearly works better. Therestoration of orientation \edges" is more faithful. This is generally truewhen the underlying clean feature distribution undergoes abrupt changes.

34 CHAN AND SHEN0 50 100 150 200 250 300

1

1.5

2

2.5

Denoising using the fitted L2 model (λ=0.01) : the angle distributions.

Ang

le D

istr

ibut

ion

θ

Noisy Clean Denoised

0 50 100 150 200 250 3000

20

40

60The clean orientation distribution

0 50 100 150 200 250 3000

20

40

60The noisy orientation distribution

0 50 100 150 200 250 3000

20

40

60The denoised orientation distributionFigure 7. Denoising using the l2 model Eq. (13). The ori-entation edges are smoothed out. (See Section 5.1.)5.3. Restoring wave lines of a �ngerprint (Figure 9). For �ngerprints,the alignment pattern of wave lines is the most important feature. Therefore,instead of Eq. (13), we use Eq. (15), which yields better results.We �rst apply the enhancement and di�usion techniques to the originalraw �ngerprint image to obtain the alignment distribution. This was doneby computing the gradient ow of the processed image. We then have the

RESTORATION OF NON-FLAT IMAGE FEATURES 350 50 100 150 200 250 300

1

1.5

2

2.5

Denoising and restoring edges by the TV model (λ=1, a=.0001) : the angle distributions.

Ang

le D

istr

ibut

ion

θ

Noisy Clean Restored

0 50 100 150 200 250 3000

20

40

60The clean orientation distribution

0 50 100 150 200 250 3000

20

40

60The noisy orientation distribution

0 50 100 150 200 250 3000

20

40

60The denoised orientation distributionFigure 8. Denoising and restoration of edges through theTV model Eq. (14). The orientation edges are faithfully re-stored. (See Section 5.2.)\heads of the arrows" removed. The alignment pattern has been superim-posed on the �ngerprint image in the top subplot Figure 9. Notice that itis indeed a noisy distribution regardless our smoothing e�ort.In the bottom subplot, we apply the line model Eq. (15) with � = 0:1 todenoise the above distribution. The result is very successful. The alignment

36 CHAN AND SHENpattern coincides very well with the �ngerprint except on a couple of localregions where the original image is not clear or damaged.Alignment distribution obtained from the denoised image by the gradient method .

20 40 60 80 100 120 140 160 180 200 220

20

40

60

80

100

120

140

160

180

200

Denoised alignment distribution (λ=0.1).

20 40 60 80 100 120 140 160 180 200 220

20

40

60

80

100

120

140

160

180

200Figure 9. Restoring the alignment feature of �ngerprintsby the l2 model Eq. (15). (See Section 5.3.)5.4. Restoration of optical ows by the l2 model (Figure 10). InFigure 10, we show an application of the l2 model for denoising optical ows.

RESTORATION OF NON-FLAT IMAGE FEATURES 3750 100 150

0

20

40

60

80

100

120

140

160

180

Noisy optical flow: a football hits a soccer ball

0 50 100 150

0

20

40

60

80

100

120

140

160

180

Restoration by the L2 model with λ=0.01

Angle distribution of the raw optical flow

50 100 150 200

20

40

60

80

100

120

140

160

180

200

The restored angle distribution

50 100 150 200

20

40

60

80

100

120

140

160

180

200Figure 10. Restoration of optical ows by the l2 modelEq. (13). (See Section 5.4.)The upper left subplot shows the noisy optical ow of an imagined videorecording the spinless rigid motion of a football and a soccer ball. Thebackground is noisy.The upper right subplot shows the restored optical ow obtained by the l2model. We have chosen � = 0:01. The length of each arrow is not changed.The two subplots at bottom plot the corresponding angle distributions.Observe that the motion near the boundaries is smeared due to thesmoothing e�ect of the l2 energy.

38 CHAN AND SHEN0 50 100 150

0

20

40

60

80

100

120

140

160

180

Optical flow: a football hits a soccer ball

0 50 100 150

0

20

40

60

80

100

120

140

160

180

Restoration by the TV model with λ=1, a=0.0001

The angle distribution of the raw optical flow

50 100 150 200

20

40

60

80

100

120

140

160

180

200

The restored angle distribution

50 100 150 200

20

40

60

80

100

120

140

160

180

200Figure 11. Restoration of optical ows by the TV modelEq. (14). (See Section 5.5.)5.5. Restoration of optical ows by the TV model (Figure 11).Similar to Figure 10, in Figure 11, we have tested the TV model with � = 1and a = 0:0001. TV model restores the boundaries better than the l2 model.5.6. Restoration of chromaticity (Figure 12 and 13). In Figure 12,we apply the discrete chromaticity restoration model for a clown image withGaussian noise (the top image). We �rst separate the vector RGB image Iinto the brightness component B = kIk, and the chromaticity componentf(0) = kIk=B. Then we apply the �xed-point TV �ltering algorithm (31) tof(0) and get the optimal restoration f. Finally, we use the original brightness

RESTORATION OF NON-FLAT IMAGE FEATURES 39The raw image with Gaussian noise

20 40 60 80 100 120 140

10

20

30

40

50

60

70

80

90

100

The new image with restored chromaticity. λ=5.

20 40 60 80 100 120 140

10

20

30

40

50

60

70

80

90

100 Figure 12. Restoration of chromaticity fromGaussian noiseby the discrete TV model and spherical �ltering Eq. (31).The brightness component remains unchanged. (See Section5.6. The raw image is from the courtesey of G. Sapiro.)value B to assemble the new image Inew = f � B, which is shown in thebottom subplot. (One can also apply the classical at restoration model to

40 CHAN AND SHENthe scalar �eld B along with the chromaticity restoration. See Tang, Sapiroand Caselles [18].)Raw R−channel

50 100 150

20

40

60

80

100

Restored R−channel

50 100 150

20

40

60

80

100

Raw G−channel

50 100 150

20

40

60

80

100

Restored G−channel

50 100 150

20

40

60

80

100

Raw B−channel

50 100 150

20

40

60

80

100

Restored B−channel

50 100 150

20

40

60

80

100Figure 13. TV Restoration of chromaticity from Gaussiannoise: the e�ect on each channel. (See Section 5.6.)In Figure 13, we plot the raw and restored images (from Figure 12) channelby channel.

RESTORATION OF NON-FLAT IMAGE FEATURES 41The restoration is quite successful. The visible noisy red and green dotshave been almost completely removed. The eyes and dark lines resumetheir original black color. The nose and lips are now smoothly red. Therestoration e�ect can be improved further if one applies an extra step of at(or linear) restoration to the brightness component B (see Tang, Sapiro andCaselles [18]).5.7. Concluding remarks. We have developed the general mathemati-cal models for denoising and restoration of non- at image features. Therestoration models for three types of important non- at features are studiedin details. They are the orientation feature, the alignment feature, and thechromaticity feature. Algorithms resolving the nonlinearity of the modelsare constructed. Numerical evidences show that generally, the TV type ofmodels perform better than the L2 models.AcknowledgmentsIt is a great pleasure to thank Professor Perona for bringing his workto our attention and for helpful discussions on the subject. We are alsoenormously grateful to Professors Sapiro and Osher for their numerous dis-cussions, suggestions, and generous help.References[1] P. V. Blomgren and T. F. Chan. Modular solvers for constrained image restoration.Technical report, CAM 97-52, UCLA Department of Mathematics, 1997.[2] A. Chambolle and P. L. Lions. Image recovery via total variational minimization andrelated problems. Numer. Math., 76:167{188, 1997.[3] T. Chan and L. Vese. Variational image restoration and segmentation models andapproximations. submitted to IEEE Tran. Image Process., 1999.[4] F. R.K. Chung. Spectral Graph Theory, volume 92 of Regional Conference Series inMathematics. Amer. Math. Soc., 1997.[5] G. H. Golub and J. M. Ortega. Scienti�c Computing and Di�erential Equations.Academic Press, 1992.[6] G. H. Granlund and H. Knuttson. Signal Processing for Computer Vision. Kluwer,Boston, 1995.[7] S. Helgason. Di�erential Geometry and Symmetric Spaces. Academic Press, 1962.[8] A. Marquina and S. Osher. Explicit algorithms for a new time dependent model basedon level set motion for nonlinear deblurring and noise removal. Technical report, CAM99-5, UCLA Department of Mathematics, 1999.[9] J.-M. Morel and S. Solimini. Variational Methods in Image Segmentation, volume 14of Progress in Nonlinear Di�erential Equations and Their Applications. Birkh�auser,Boston, 1995.

42 CHAN AND SHEN[10] D. Mumford and J. Shah. Optimal approximations by piecewise smooth functions andassociated variational problems. Comm. Pure Applied. Math., XLII:577{685, 1989.[11] S. Osher. Private communication. 1999.[12] P. Perona. Orientation di�usion. IEEE Trans. Image Proc., 7(3):457{467, 1998.[13] P. Perona and J. Malik. Scale-space and edge detection using anisotropic di�usion.IEEE Trans. Pattern Anal. Machine Intell., 12:629{639, 1990.[14] L. Rudin and S. Osher. Total variation based image restoration with free local con-straints. Proc. 1st IEEE ICIP, 1:31{35, 1994.[15] L. Rudin, S. Osher, and E. Fatemi. Nonlinear total variation based noise removalalgorithms. Physica D, 60:259{268, 1992.[16] G. Sapiro. Color snakes. Technical report, HPL-95-113, Hewlett Packard ComputerPeripherals Laboratory, 1995.[17] G. Sapiro and D. Ringach. Anisotropic di�usion of multivalued images with applica-tions to color �ltering. IEEE Tran. Image Process., 5:1582{1586, 1996.[18] B. Tang, G. Sapiro, and V. Caselles. Color image enhancement via chromaticitydi�usion. Technical report, ECE-University of Minnesota, 1999.[19] B. Tang, G. Sapiro, and V. Caselles. Direction di�usion. Technical report, ECE-University of Minnesota. To appear in International Conference in Computer Vision,1999.[20] C. R. Vogel and M. E. Oman. Iterative methods for total variation denoising. SIAMJ. Sci. Statist. Compt., 17:227{238, 1996.[21] J. Weickert. Anisotropic Di�usion in Image Processing. Teubner-Verlag, Stuttgart,Germany, 1998.