Pattern Recognition Letters 32 (2011) 2047–2052

Contents lists available at SciVerse ScienceDirect

Pattern Recognition Letters

journal homepage: www.elsevier .com/locate /patrec

Feature detection and tracking in optical flow on non-flat manifolds

Sheraz Khan a,c,d,⇑, Julien Lefevre b, Habib Ammari c, Sylvain Baillet a,e

a Department of Neurology, Medical College of Wisconsin, Milwaukee, USAb Aix-Marseille Univ, Département d’Informatique de Luminy /CNRS, LSIS, UMR 6168, 13397 Marseille, Francec Center of Applied Mathematics, Ecole Polytechnique, Franced Department of Neurology, MGH/Harvard Medical School, Boston, USAe Montreal Neurological Institute, McGill University, Montreal, Canada

a r t i c l e i n f o

Article history:Received 22 September 2010Available online 19 September 2011Communicated by A. Shokoufandeh

Keywords:Optical flowHelmholtz–Hodge decompositionFeature detectionImage processingRiemannian formalismVector fields

0167-8655/$ - see front matter � 2011 Elsevier B.V. Adoi:10.1016/j.patrec.2011.09.017

⇑ Corresponding author at: Department of NeuroHospital, Harvard Medical School, Boston, MA 02129,fax: +1 617 948 5966.

E-mail address: sheraz@nmr.mgh.harvard.edu (S. K

a b s t r a c t

Optical flow is a classical approach to estimating the velocity vector fields associated to illuminatedobjects traveling onto manifolds. The extraction of rotational (vortices) or curl-free (sources or sinks) fea-tures of interest from these vector fields can be obtained from their Helmholtz–Hodge decomposition(HHD). However, the applications of existing HHD techniques are limited to flat, 2D domains. Here wedemonstrate the extension of the HHD to vector fields defined over arbitrary surface manifolds. We pro-pose a Riemannian variational formalism, and illustrate the proposed methodology with synthetic andempirical examples of optical-flow vector field decompositions obtained on a variety of surface objects.

� 2011 Elsevier B.V. All rights reserved.

1. Introduction � a non-diverging component, deriving from the curl of a scalar

The optical flow in a scene is the apparent motion of objects,induced by the observed temporal variations of local brightness.Under certain conditions, it provides an adequate model for objects’displacement fields, Horn and Schunck (1981). In most applications,however, the definition of the optical flow is restricted to two-dimensional (2D), i.e. flat, surface manifolds. This hinders somepotentially high-impact applications of the optical flow, e.g. thestudy of turbulent flow over general surfaces. We recently intro-duced a new variational framework to estimate optical-flow vectorfields on non-flat surfaces using a Riemannian formulation, Lefèvreand Baillet (2008). In most applications, features of interest such assources, sinks and traveling waves, need to be extracted from time-varying optical-flow vector fields, and be interpreted as meaningfulcomponents of the kinematics of the observed phenomena.

The Helmholtz–Hodge decomposition (HHD) of vector fields hasproven to be a useful technique in that respect, Chorin and Mars-den (1993). HHD proceeds by decomposing vector fields definedin 2D flat domains, Guo et al. (2005) or full 3D geometry, Tonget al. (2003), into the sum of:

� a curl-free component, deriving from the gradient of a scalarpotential U;

ll rights reserved.

logy, Massachusetts GeneralUSA. Tel.: +1 617 643 5634;

han).

(in 2D) or vector (in 3D) potential A;� an harmonic component H, with zero Laplacian.

Singularities or vanishing points in the optical-flow vector fieldcan be detected using these components, as shown in e.g., Guo et al.(2005), and as we shall detail in Section 3.3 of this article. As alreadymentioned, a large number of new fields of application would ben-efit from a generic and principled approach to the HHD of vectorfields on surface manifolds: e.g., experimental fluid dynamics andturbulence (Corpetti et al., 2003; Palit et al., 2005); physiologicalmodeling (Guo et al., 2006); structural and functional neuroimag-ing (Lefevre et al., 2009), and the compressed representation oflarge, distributed vector fields (Scheuermann and Tricoche, 2005).

It has been suggested that approximations to surface-basedHHD could be achieved using polyhedral surfaces, in the localEuclidean geometry of the manifold (Polthier and Preuss, 2003).This however proves to be problematic, when the vector fieldsare defined over highly-curved surface supports. Indeed, we haveshown previously that the surface curvature plays a crucial rolein the robust estimation of vector fields within the tangent bundleof the surface support (Lefèvre and Baillet, 2008). Our results alsoshowed that the non-flatness properties of the surface may causeconvergence issues in the estimation process of the vector field.We note however that new results in the Euclidian domain thatdo not require a meshed tessellation of the surface were publishedrecently and indicate a promising alternative to our approach(Petronetto et al., 2009).

2048 S. Khan et al. / Pattern Recognition Letters 32 (2011) 2047–2052

We show in this article how the HHD can be performed ontonon-flat domains and we extended the study to the extraction ofbasic kinetic features of interest from the HHD components. In linewith our previous developments (Lefèvre and Baillet, 2008), thepresent results were worked in the framework of differentialgeometry on 2-Riemannian manifolds and a numerical implemen-tation using the Finite Element Method (FEM). Important conceptsand notations from this framework are revisited in Section 2. Sec-tion 3 introduces the proposed framework for HHD. The method isthen featured with a variety of results in Section 4.

We emphasize that the software implementation of the meth-ods are available for download, as a plugin to the Brainstorm aca-demic software for electromagnetic brain imaging (http://neuroimage.usc.edu/brainstorm, requires Matlab and has beenoptimized for multithreaded computation).

2. Technical background

This section is a summary of the concepts and notations that areappropriate to the 2-Riemannian manifold framework and its FEMimplementation. More details can be found in (Do Carmo, 1993;Lefèvre and Baillet, 2008).

2.1. Differential geometry

Let us consider M, a surface (or 2-Riemannian manifold)equipped by a metric or scalar product g(�, �) to measure distancesand angles on the surface. M can be parameterized with localcharts (x1,x2). Thus, it is possible to obtain a normal vector at eachsurface point:

np ¼@

@x1� @

@x2:

Note that np does not depend on the (x1,x2) parametrization.Let U :M! R be a function defined on the surface, and its dif-

ferential dU : TM! R, acting on TM, the space of vector fields. Wethen can write:

dUðv1;v2Þ ¼@U@x1

v1 þ@U@x2

v2:

Let us then define the gradient and divergence operatorsthrough duality:

dUðVÞ ¼ gð$MU;VÞ;ZM

UdivMH ¼ �ZM

gðH;$MUÞ:

We emphasize that the gradient and divergence measures are alsoindependent of the surface parametrization.

Lastly, let us define two additional functional spaces that will beuseful for a theorem proposed in Section 4: H1ðMÞ is the space ofdifferentiable functions on the manifoldM, and C1ðMÞ is the spaceof vector fields, which satisfy good properties for regularity (as dis-cussed in (Druet et al., 2004; Lefèvre and Baillet, 2008)).

2.2. Optical flow

Under the seminal hypothesis of the conservation of a scalarfield I along streamlines, the optical flow V is a vector field whichsatisfies:

@t I þ gðV ;rMIÞ ¼ 0: ð1Þ

Note that the scalar product g(�, �) is modified by the local curva-ture of the surface domainM. The solution to Eq. (1) is not uniqueas long as the components of V(p, t) orthogonal to rMI are leftunconstrained. This so-called ‘aperture problem’ has beenaddressed by a large number of methods using e.g., regularization

approaches. These latter may be formalized as the minimizationproblem of an energy functional, which both includes the regular-ity of the solution and the agreement with the model:

EðVÞ ¼ZM

@I@tþ gðV ;rMIÞ

� �2

dlþ kZMCðVÞdl; ð2Þ

where dl is a volume form of the manifold M.In this study, we used a regularity operator that is quadratic in

the generalized gradient of the expected vector field:

CðVÞ ¼ TrðtrV � rVÞ; ð3Þ

we note that the gradient of a vector field must be defined as thecovariant derivative associated to the manifold M, so that it canbe considered as an intrinsic tensor. We refer to the general con-cepts of differential geometry for more information (Do Carmo,1993).

3. Helmholtz–Hodge decomposition on a 2-Riemannianmanifold

We now introduce an extended framework to perform HHD onsurfaces and show that it can be applied to any vector field definedon a 2-Riemannian manifold, M.

3.1. Definitions and theorem

Let us first define the scalar and vectorial curls by:

CurlM : H1ðMÞ ! C1ðMÞ curlM : C1ðMÞ ! H1ðMÞCurlMA ¼ $MA� n; curlMH ¼ divMðH � nÞ:

Note that these intrinsic expressions are not dependent on theparametrization of the surface.

As established in (Polthier and Preuss, 2003), we assume thatMis a closed manifold, i.e. it has no boundaries. Given V, a vector fieldin C1ðMÞ, there exists functions U and A (up to an additive con-stant) in H1ðMÞ and a vector field H in C1ðMÞ such that:

V ¼ $MU þ CurlMAþ H; ð4Þ

where

curlMð$MUÞ ¼ 0; divMðCurlMAÞ ¼ 0;divMH ¼ 0; curlMH ¼ 0:

To prove this theorem, we show how U and A may be con-structed. Following classical constructions, we consider U and Aas minimizers of the two following convex functionals, which arenot necessarily unique:ZMkV � $MUk2dl; ð5ÞZ

MkV � CurlMAk2dl; ð6Þ

where k�k is the norm associated to the Riemannian metric g(�, �).These two functionals carry a global minimum on H1ðMÞ, which

satisfies:

8/ 2 H1ðMÞ;ZM

gðV ;$M/Þdl ¼ZM

gð$MU;$M/Þdl; ð7Þ

8/ 2 H1ðMÞ;ZM

gðV ;CurlM/Þdl ¼ZM

gðCurlMA;CurlM/Þdl: ð8Þ

These two equations will play a key role in the subsequentnumerical computations, as detailed below. The rest of the proofis provided in the Appendix.

S. Khan et al. / Pattern Recognition Letters 32 (2011) 2047–2052 2049

3.2. Discretization

Eqs. (7) and (8) are crucial since they pave the way to numericalimplementation, when H1ðMÞ is approximated by a subspace of fi-nite elements (e.g., continuous linear piecewise functions).

We now detail the numerical implementation of Eqs. (9) and(10), which are defined over a surface tessellation cM that approx-imates the ideal manifold. Let N be the number of nodes in the tes-sellation (Fig. 1).

In the FEM implementation, we define N continuous piecewiseaffine functions /i, decreasing from a value of 1 at node i, to 0 at theother triangle nodes.

Using these basis functions (/1, . . . ,/n), we can writeU = (U1, . . . ,Un)T, A = (A1, . . . ,An)T, and Eqs. (7) and (8) become,using array notations:ZM

gð$M/i;$M/jÞ� �

i;j

U ¼ZM

gðV ;$M/iÞ� �

i

; ð9ÞZM

gðCurlM/i;CurlM/jÞ� �

i;jA ¼

ZM

gðV ;CurlM/iÞ� �

i; ð10Þ

where [�]i,j is a N � N square matrix and [�]i is a column vector oflength N.

The harmonic component H of the vector field V is obtained as:

H ¼ V � $MU � CurlMA: ð11Þ

The gradient of each basis function is constant on each triangleand can be simply computed with geometrical quantities (see Lefè-vre and Baillet (2008)). Hence Eq. (9) reads:

XT3i;j

hi

khik2 �hj

khjk2AðTÞ" #

i;j

U ¼XT3i

AðTÞV � hi

khik2

" #i

; ð12Þ

where hi is the height of triangle taken from vertex i, andAðTÞ is thesurface area of triangle T (Fig. 1).

Similarly, Eq. (10) is discretized as follows:

XT3i;j

hi

khik2�n

!� hj

khjk2�n

!AðTÞ

" #i;j

A¼XT3i

AðTÞV � hi

khik2�n

!" #i

; ð13Þ

where n is the normal to triangle T.

Fig. 1. Illustration of local computations and notation

3.3. HHD feature detection as critical points of potentials

The critical points of a vector field are often obtained from theeigenvalues of their Jacobian matrix. However, it has been shownthat feature detection from critical points in global potential fieldsis less sensitive to noise in the data (Tong et al., 2003) and there-fore less prone to false positive detections than with methodsbased on the extraction of local Jacobian eigenvalues (Mann andRockwood, 2002). We therefore define critical points in the flowas local extrema of the divergence-free potential A (representingrotation) and curl-free potential U (representing divergence).

A sink (respectively, a source) is thus defined as a local maxi-mum (respectively, minimum) of U. Counterclockwise (respec-tively, clockwise) vortices are featured by local maximum(respectively, minimum) of A. Further, since H is a curl-free vectorfield with zero divergence, traveling objects may be detected fromvector elements bearing the largest magnitudes in the H vectorfield.

4. Results

The proposed approach was first evaluated by detecting sourcesand sinks on a spherical surface bearing a rotating vector field, asshown in Fig. 2. A possible real-life situation with such vector fieldpatterns would be the air flow over a rotating spherical projectile.

The HHD decomposition was applied on this vector field andvortices were identified from the critical points in A, which wastextured over the surface, as shown in Fig. 2. In the case of a coun-ter-clockwise rotating vector field, the vortex was found as themaxima in the distribution of A; for a clockwise rotating vectorfield, the vortex was identified from the minima of A over thesurface.

Secondly, the HHD was tested on the relatively more curvedsurface of the classical bunny (Fig. 3). Vector fields with sourcesand sinks were synthesized to mimic the optical flow of patternswith increasing or decreasing surface extent. Rotating vectors fieldwere also generated to mimic the optical flow generated by a rotat-ing vortex.

Fig. 3(a) shows a snapshot of the rotating vector field with a tex-ture overlay of A. The two vortices of the original vector field couldbe successfully identified using this approach. Similarly, Fig. 3(b)

s necessary to FEM, on a triangular surface mesh.

Fig. 2. Simulation of a rotating vector fields over a spherical surface object (shown with green arrows). (a) Counter-clockwise rotating vector field and (b) clockwise rotatingvector field. The divergence-free potentials A obtained from the HHD of the vector fields were mapped onto the surface as a colored texture. In both cases, the vortices wereidentified as the extrema in the A map (maximum in (a), minimum in (b)). Values of A are shown in arbitrary, scaled units. (For interpretation of the references to colour inthis figure legend, the reader is referred to the web version of this article.)

2050 S. Khan et al. / Pattern Recognition Letters 32 (2011) 2047–2052

shows a snapshot of a diverging vector field with two sources,overlaid with a texture of the values of U, thereby revealing thecorrect locations of the sources in the vector field.

Fig. 3(c) and (d) extend further these results using vector fieldswith both rotating and diverging components. It is shown thatfeatures of different categories can be extracted at once from a vec-tor field. Our colormap shows sources and clockwise vortices inblue (local minima); sinks and counter-clockwise vortices in red(local maxima).

In another set of simulations, a traveling source and a travelingvortex were tracked on the surface of the bunny manifold, by iden-tifying the critical points of the U and A HHD scalar fields (Fig. 4(a)).Additionally, a surface patch of constant illumination was dis-placed onto the surface using the advection equation along a pre-defined vector field (Lefèvre and Baillet, 2008) that defined atrajectory. This object was tracked with the vectors of largest normin H vector field (Fig. 4(b)).

Fig. 4 shows snapshots of the tracking simulation. A black arrowhas been used to trace path of sources and vortex in Fig. 4(a) andmoving the object in Fig. 4(b).

We further tested the HHD using experimental functional brainimaging data obtained from magnetoencephalography (MEG)(Baillet et al., 2001). Brain responses were recorded over 151MEG sensors following repeated somatosensory stimulation ofthe median nerve of the right hand (200 trials), sampled at1024 Hz. Cortical currents were constrained on a surface tessella-tion of the subject’s brain containing about 50,000 triangle nodes.The brain surface tessellation was obtained from the tissue seg-mentation of T1 weighted magnetic resonance image data usingbrainVISA (http://brainvisa.info). The cortical currents were esti-mated with a regularized minimum-norm inverse model as de-tailed in (Baillet et al., 2001). The vector field of the optical flowof these currents were computed using the method described in(Lefèvre and Baillet, 2008), as implemented by us in the Brain-Storm software package (http://neuroimage.usc.edu/brainstorm).HHD was then applied to detect sources and sinks in the vectorfields of the cortical currents. The source pattern at about 30 msafter stimulus onset obtained from the event-related average ofall trials revealed the expected primary somatosensory brain re-sponse. Fig. 5 shows a strong divergent pattern of the vector fieldover the central sulcus contralateral to the side of the stimulation.

Computations to obtain all HHD components on the 50,000-node brain surface tessellation took about 10 s using a conven-tional workstation with 8 cores.

Additional results are featured as Supplementary material.

5. Conclusion

In this contribution, we developed a framework for the HHD ofoptical flow vector fields over 2-Riemannian manifolds. We sug-gest that features of interest regarding the kinetics supported byvector fields can be extracted and tracked through the potentialcomponents of the HHD. The required computations are simpleand are made available to the academic community as part asthe distribution of BrainStorm. Evaluation of this framework underreal and simulated environments may open promising applicationsin the emerging field of multi-dimensional imaging.

Appendix A

We show that the scalar potentials U and A can be obtained upto a constant. If we consider two minimizers U1 and U2 for thefunctional in (5) then from Eq. (7), we get:

8/ 2 H1ðMÞ;ZM

gð$MðU1 � U2Þ;$M/Þdl ¼ 0:

Green’s formula allows to transform this equation (given that Mhas no boundaries) as follows:

8/ 2 H1ðMÞ;ZM

/divM$MðU1 � U2Þdl ¼ 0;

which corresponds to the Laplace equation:

DMðU1 � U2Þ ¼ 0;

where DM ¼ divM$M is the Laplace–Beltrami operator.The same approach can be followed for two minimizers A1 and

A2 of the functional in Eq. (6). Hence, writing R = U1 � U2 andR0 = A1 � A2, we obtained the following two conditions:

DMR ¼ 0; DMR0 ¼ 0:

Multiplying by R the first equation and integrating on the man-ifold we get:

Fig. 3. Vector fields were synthesized, as shown using green arrows, on the classical bunny surface. HHD potentials U and A are overlaid as textures onto the surface: (a)rotating vector field with two vortices and its A component; (b) diverging vector field with two sources and its U component; (c) rotating and diverging vector field and its Acomponent; (d) rotating and diverging vector field and its U component. In all four scenarios, the sources, sinks and vortices correspond to the extrema in the surface texturesof A or U. Values of the HHD potentials are scaled and shown in arbitrary units. (For interpretation of the references to colour in this figure legend, the reader is referred to theweb version of this article.)

Fig. 4. Tracking of sinks, vortices (a) and of a traveling surface patch of constant illumination (b) on the bunny surface manifold. Symbols were used for each of these objectsalong their respective trajectory (shown as dark line curves).

S. Khan et al. / Pattern Recognition Letters 32 (2011) 2047–2052 2051

Fig. 5. HHD applied to vector fields of cortical currents revealed by time-resolved imaging of MEG sources following stimulation of the right median nerve (see text body fordetails regarding data collection). (a) Textured map of scalar potentials U obtained from the HHD of the vector field (show with green arrows) of cortical currents estimated onthe surface of the brain from MEG recordings. (b) Zoomed view over the square frame in (a). The color map uses arbitrary units for U, which were normalized to itsinstantaneous maximum. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

2052 S. Khan et al. / Pattern Recognition Letters 32 (2011) 2047–2052

ZM

RDMRdl ¼ 0:

Applying Green’s formula yields:ZM

gð$MR;$MRÞdl ¼ 0:

It follows that $MR must equal zero and R must be a constant. Thesame approach can be applied to R0, which must be constant.

We then define the conditions on the vector field H. We firstwrite H ¼ V � $MU þ CurlMA, and obtain:

divMH ¼ divMV � divM$MU and curlMH

¼ curlMV � curlMCurlMA;

since divMCurlMA ¼ 0 and curlM$MU ¼ 0. It follows that:

8/ 2 H1ðMÞ;ZM

/divMHdl ¼ZM

/ðdivMV � divM$MUÞdl:

Using Green’s formula and Eq. (7), we obtain:

8/ 2 H1ðMÞ;ZM

/divMHdl

¼ZM

gðV ;$M/Þ �ZM

gð$MU;$M/Þdl ¼ 0:

We have then proved that divMH ¼ 0. The same strategy applies toobtain curlMH ¼ 0.

Appendix B. Supplementary data

Supplementary data associated with this article can be found, inthe online version, at doi:10.1016/j.patrec.2011.09.017.

References

Baillet, S., Mosher, J., Leahy, R., 2001. Electromagnetic brain mapping. IEEE SignalProcess. Mag. 18 (6), 14–30.

Chorin, A., Marsden, J., 1993. A Mathematical Introduction to Fluid Mechanics.Springer.

Corpetti, T., MTmin, E., PTrez, P., 2003. Extraction of singular points fromdense motion fields: an analytic approach. J. Math. Imaging Vision 19,175–198.

Do Carmo, M., 1993. Riemannian Geometry. Birkhauser.Druet, O., Hebey, E., Robert, F., 2004. Blow-Up Theory for Elliptic PDEs in

Riemannian Geometry. Princeton University Press, Princeton, N.J (Chapter.Background Material, pp. 1–12).

Guo, Q., Mandal, M., Li, M., 2005. Efficient Hodge–Helmholtz decomposition ofmotion fields. Pattern Recognition Lett. 26 (4), 493–501.

Guo, Q., Mandal, M., Liu, G., Kavanagh, K., 2006. Cardiac video analysis using Hodge–Helmholtz field decomposition. Comput. Biol. Med. 36 (1), 1–20.

Horn, B., Schunck, B., 1981. Determining Optical Flow. Artif. Intell. 17 (1-3),185–203.

Lefèvre, J., Baillet, S., 2008. Optical flow and advection on 2-Riemannian manifolds:A common framework. IEEE Trans. Pattern Anal. Machine Intell. 30 (6),1081–1092 http://cogimage.dsi.cnrs.fr/publications/2008/LB08.

Lefevre, J., Leroy, F., Khan, S., Dubois, J., Huppi, P., Baillet, S., Mangin, J., 2009.Identification of Growth Seeds in the Neonate Brain through Surfacic HelmholtzDecomposition. In: 21st Information Processing in Medical Imaging,Williamsburg.

Mann, S., Rockwood, A., 2002. Computing singularities of 3D vector fields withgeometric algebra. Proc. IEEE Visual., 283–289.

Palit, B., Basu, A., Mandal, M., 2005. Applications of the discrete Hodge Helmholtzdecomposition to image and video processing. Lect. Notes Comput. Sci. 3776,497.

Petronetto, F., Paiva, A., Lage, M., Tavares, G., Lopes, H., Lewiner, T., 2009. MeshlessHelmholtz–Hodge decomposition. IEEE Trans. Visual. Comput. Graphics,338–349.

Polthier, K., Preuss, E., 2003. Identifying vector fields singularities using a discreteHodge decomposition. Visual. Math. 3, 113–134.

Scheuermann, G., Tricoche, X., 2005. Topological Methods for Flow Visualization.The Visualization Handbook, 341–356.

Tong, Y., Lombeyda, S., Hirani, A., Desbrun, M., 2003. Discrete multiscale vector fielddecomposition. ACM Trans. Graphics 22 (3), 445–452.