on a nonlinear generalization of sparse coding and dictionary

On A Nonlinear Generalization of Sparse Coding and DictionaryLearning

Yuchen Xie [email protected]

Qualcomm Technologies, Inc., San Diego, CA 92121 USA

Jeffrey Ho [email protected] Vemuri [email protected]

University of Florida, Gainesville, FL 32611 USA

Abstract

Existing dictionary learning algorithms arebased on the assumption that the data arevectors in an Euclidean vector space Rd,and the dictionary is learned from the train-ing data using the vector space structure ofRd and its Euclidean L2-metric. However,in many applications, features and data of-ten originated from a Riemannian manifoldthat does not support a global linear (vectorspace) structure. Furthermore, the extrin-sic viewpoint of existing dictionary learningalgorithms becomes inappropriate for model-ing and incorporating the intrinsic geometryof the manifold that is potentially importantand critical to the application. This paperproposes a novel framework for sparse cod-ing and dictionary learning for data on a Rie-mannian manifold, and it shows that the ex-isting sparse coding and dictionary learningmethods can be considered as special (Eu-clidean) cases of the more general frameworkproposed here. We show that both the dic-tionary and sparse coding can be effectivelycomputed for several important classes ofRiemannian manifolds, and we validate theproposed method using two well-known clas-sification problems in computer vision andmedical imaging analysis.

1. Introduction

Dictionary learning has been widely used in machinelearning applications such as classification, recogni-

Proceedings of the 30 th International Conference on Ma-chine Learning, Atlanta, Georgia, USA, 2013. JMLR:W&CP volume 28. Copyright 2013 by the author(s).

tion, image restoration and others (e.g., (Aharon et al.,2006)). Under this model, each data point is as-sumed to be generated linearly using only a smallnumber of atoms, and this assumption of linear spar-sity is responsible for much of its generalization powerand success. The underlying linear process requiresthat the data points as well as the atoms are vectorsin a vector space Rd, and the dictionary is learnedfrom the input data using the vector space structureof Rd and its metric (typically the L2-norm). How-ever, for many applications such as those in direc-tional statistics (e.g., (Mardia & Jupp, 1999)), ma-chine learning (e.g., (Yu & Zhang, 2010)), computervision (e.g., (Turaga et al., 2008)), and medical im-age analysis, data and features are often presented aspoints on known Riemannian manifolds, e.g., the spaceof symmetric positive-definite matrices (Fletcher &Joshi, 2007), hyperspheres for parameterizing square-root densities (Srivastava et al., 2007), Stiefel andGrassmann manifolds (Mardia & Jupp, 1999), etc..While these manifolds are equipped with metrics, theirlack of vector space structures is a significant hin-drance for dictionary learning, and primarily becauseof this, it is unlikely that the existing dictionary learn-ing methods can be extended to manifold-valued datawithout serious modifications and injections of newideas. This paper takes a small step in this directionby proposing a principled extension of the linear sparsedictionary learning in Euclidean space Rd to generalRiemannian manifolds.

By the Nash embedding theorem (Nash, 1956), anyabstractly-defined Riemannian (data) manifoldM canbe isometrically embedded in some Euclidean spaceRd, i.e., M can be considered as a Riemannian sub-manifold of Rd. It is tempting to circumvent the lackof linear structure by treating points inM as points inthe embedding space Rd and learning the dictionary in

On A Nonlinear Generalization of Sparse Coding and Dictionary Learning

Rd. Unfortunately, this immediately raises two thornyissues regarding the suitability of this approach. First,for most manifolds, such as Grassmann and Stiefelmanifolds, there simply does not exist known canoni-cal embedding into Rd (or such embedding is difficultto compute), and although Nash’s theorem guaranteesthe existence of such embedding, its non-uniqueness isa difficult issue to resolve as different embeddeings areexpected to produce different results. Second, even inthe case when the existing methods can be applied, dueto their extrinsic nature (both the vector space struc-ture and metric structure in Rd are extrinsic to M),important intrinsic properties of the data manifold arestill very difficult to capture using an extrinsic dictio-nary. For example, a linear combination of atoms in Rddoes not in general define a proper feature (a point inM), and the inadequacy can be further illustrated byanother simple example: it is possible that two pointsx, y ∈ M have a large geodesic distance separatingthem but under an embedding i :M→ Rd, i(x), i(y)are near each other in Rd. Therefore, sparse coding us-ing dictionary learned in Rd is likely to code i(x), i(y)(and hence x, y) using the same set of atoms with sim-ilar coefficients. This is undesirable for classificationand clustering applications that use sparse coding co-efficients as discriminative features, and for dictionarylearning to be useful for manifold-valued data, sparsecoding must reflect some degree of similarity betweenthe two samples x, y ∈M as measured by the intrinsicmetric of M.

While the motivation for seeking an extension of theexisting dictionary learning framework to the moregeneral nonlinear (manifold) setting has been outlinedabove, it is by no means obvious how the extensionshould be correctly formulated. Let x1, · · · , xn denotea collection of data points on a Riemannian manifoldM. An important goal of dictionary learning onM isto compute a collection of atoms {a1, · · · , am} ⊂ M,also points inM, such that each data point xi can begenerated using only a small number of atoms (spar-sity). In the Euclidean setting, this is usually formu-lated as

minD,w1,··· ,wn

n∑i=1

‖xi −Dwi‖2 + Sp(wi), (1)

where D is the matrix with columns composed of theatoms ai, wi the sparse coding coefficients for xi andSp(w) the sparsity-promoting regularizer. One im-mediate technical hurdle that any satisfactory gener-alization must overcome is the generalization of thecorresponding sparse coding problem (with a fixedD above), and in particular, the crucial point is theproper generalization of linear combination xi = Dwi

for data and atoms belonging to the manifoldM thatdoes not support a (global) vector space structure.

Once the nonlinear sparse coding has been properlygeneralized (Section 3), the dictionary learning algo-rithm can then be formulated by formally modifyingeach summand in Equation 1 so that the sparse codingof a data xi with respect to the atoms {a1, · · · , am} ⊂M is now obtained by minimizing (log denoting theRiemannian logarithm map (do Carmo, 1992))

minwi

‖m∑j=1

wij logxiaj‖2xi

+ Sp(wi), (2)

with the important affine constraint that∑mj=1 wij =

1, where wi = (wi1, . . . , wim)T . Mathematically, thelack of global vector space structure onM is partiallycompensated by the local tangent space TxM at eachpoint x, and global information can be extracted fromthese local tangent spaces using Riemannian geometryoperations such as covariant derivatives, exponentialand logarithm maps. We remark that this formula-tion is completely coordinate-independent since eachlogxi

aj is coordinate-independent, and Equation 2 isa direct generalization of its Euclidean counterpart thisis the individual summand in Equation 1. Computa-tionally, the resulting optimization problem given byEquation 2 can be effectively minimized. In partic-ular, the gradient of the objective function in somecases admits a closed-form formula, and in general, itcan be evaluated numerically to provide the input forgradient-based optimization algorithms on manifolds,e.g., (Edelman et al., 1998).

Before moving onto the next section, we remark thatour context is very different from the context of man-ifold learning that is perhaps better known in the ma-chine learning community. For the latter, the manifoldM on which the data reside is not known and the focusis on estimating this unknown M and characterizingits geometry. For us, however, M and its geometry(Riemannian metric) are known, and the goal is notto estimateM from the data but to compute a dictio-nary as a finite subset of points in M.

2. Preliminaries

This section presents a brief review of the necessarybackground material from Riemannian geometry thatare needed in the later sections and we refer to (Spi-vak, 1979) for more details. A manifold M of dimen-sion d is a topological space that is locally homeo-morphic to open subsets of the Euclidean space Rd.With a globally defined differential structure, mani-fold M becomes a differentiable manifold. The tan-


gent space at x ∈ M, denoted by TxM, is a vectorspace that contains all the tangent vectors to M atx. A Riemannian metric on M associates to eachpoint x ∈ M an inner product 〈·, ·〉x in the tangentspace TxM. Let xi, xj be two points on the mani-fold M. A geodesic γ : [0, 1] →M is a smooth curvewith vanishing covariant derivative of its tangent vec-tor field, and in particular, the Riemannian distancebetween two points xi, xj ∈ M, dist2M(xi, xj), is theinfimum of the lengths of all geodesics joining xi, xj .Let v ∈ TxM be a tangent vector at x. There existsa unique geodesic γv satisfying γv(0) = x with ini-tial tangent vector v, and the Riemannian exponentialmap (based at x) is defined as expx(v) = γv(1). Theinverse of the exponential map expx is the log map,denoted as logx : M → TxM. We remark that ingeneral the domain of expx is not the entire tangentspace TxM and similarly, logx is not defined on all ofM. However, for technical reasons, we will assume inthe following sections that M is a complete Rieman-nian manifold (Spivak, 1979) such that logx is definedeverywhere in M for all x ∈ M, and the subtle tech-nical point when exp, log are not defined everywherewill be addressed in a future work. Under this as-sumption, there are two important consequences: 1)the geodesic distance between xi and xj can be com-puted by the formula distM(xi, xj) = ‖ logxi

(xj)‖xi,

and 2) the squared distance function dist2M(x,−) is asmooth function for all x ∈M.

3. Nonlinear Sparse Coding

In linear sparse coding, a collection of m atomsa1, · · · , am, are given that form the columns of the(overcomplete) dictionary matrix D. The sparse cod-ing of a feature vector x ∈ Rd is determined by thefollowing l0-minimization problem:

minw∈Rm

‖w‖0, s.t. x = FD(w), (3)

where the function FD : Rm → Rd is defined asFD(w) = Dw. In the proposed nonlinear generaliza-tion of sparse coding to a Riemannian manifold M,the main technical difficulty is the proper interpreta-tion of the function FD(w) : Rm →M in the manifoldsetting, where the atoms a1, · · · , am are now points inM and D now denotes the set of atoms since it is nolonger possible to stack together the atoms to form amatrix as in the linear case.

Moving to the more general manifold setting, we haveforsaken the vector space structure in Rd, and at thesame time, we are required to work only with notionsthat are coordinate independent (do Carmo, 1992).This latter point is related to the fact that onM there

does not have a special point such as the origin (zerovector) in Rd, and the subtlety of this point seems tohave been under-appreciated. In particular, the notionof sparsity that we are accustomed to is very much de-pendent on the choice (location) of the origin.

Sparsity, Coordinate Invariance and AffineConstraint To illustrate the above point, let x ∈ Rdbe a sparse vector (according to D) such that x =a1 + a2 + a3, i.e., x can be reconstructed using onlythree atoms in D. Changing the coordinates by trans-lating the origin to a new vector t, each atom ai be-comes ai− t, and similarly for x, we have x− t. Underthis new coordinates with a different origin, x− t canno longer be reconstructed using the three (translated)atoms a1− t, a2− t, a3− t, and most likely, in this newcoordinates, the same point cannot be reconstructedby a small number of atoms, i.e., it is not a sparsevector with respect to the dictionary D. This is notsurprising because linear sparse coding considers eachpoint in Rd as a vector whose specification requiresa reference point (the origin). However, in nonlinearsetting, each point cannot be considered as a vectorand therefore, must be considered as a point, and thisparticular viewpoint is the main source of differencesbetween linear and nonlinear sparse coding.

Fortunately, we can modify the usual notion spar-sity using affine constraint to yield a coordinate-independent notion of sparsity: a vector is a (affine)sparse vector if it can be written as an affine linearcombination of a small number of vectors:

x = w1a1 + w2a1 + ...+ wsas, w1 + ...+ ws = 1.

It is immediately clear that that the notion of affinesparsity is a coordinate-independent notion as the lo-cation of the origin is immaterial (thanks to the affineconstraint), and it is this notion of affine sparsity thatwill be generalized. We note that the affine constraintwas also used in (Yu et al., 2009; Yu & Zhang, 2010) fora related but different reason. We also remark that theusual exact recovery results (e.g., (Elad, 2010)) that es-tablish the equivalence between the l0-problem aboveand its l1-relaxation remain valid for affine sparsitysince the extra constraint introduced here is convex.

Re-interpreting FD It is natural to require thatour proposed definition for FD on a manifoldM mustreduce to the usual linear combination of atoms ifM =Rd. Furthermore, it is also necessary to require thatFD(w) ∈ M and it is well-defined and computablewith certain intrinsic properties ofM playing a crucialrole. An immediate candidate would be

FD(w) = argminx∈MΨD,w(x), (4)


where ΨD,w(x) is the function of weighted sum ofsquared distances to the atoms:

ΨD,w(x) =

m∑i=1

wi dist2M(x, ai).

While this definition uses the Riemannian geodesic dis-tances, the intrinsic quantities that the learning algo-rithm is supposed to incorporate, it suffers from twomain shortcomings. First, since there is no guaranteesthat the global minimum of ΨD,w must be unique,FD(w) is no longer single-valued but a multi-valuedfunction in general. Second, much more importantly,FD(w) may not exist at all for some w (e.g., when itcontains both positive and negative weights wi). How-ever, if we are willing to accept a multi-valued gener-alization FD(w), then the second shortcoming can besignificantly remedied by defining

FD(w) = CP(ΨD,w(x)), (5)

where CP(ΨD,w(x)) denotes the set of critical pointsof ΨD,w(x), points y ∈ M with vanishing gradient:∇ΨD,w(y) = 0.

We remark that under the affine constraint w1 + · · ·+wm = 1, not all wi can be zero simultaneously; there-fore, ΨD,w(x) cannot be a constant (zero) function,i.e., ΨD,w(x) 6= M. Since a global minimum of asmooth function must also be its critical point, thisimplies that the existence of critical points is less ofa problem than the existence of a global minimum.This can be illustrated using the simplest nontrivialRiemannian manifold of positive reals M = R+ con-sidered as the space of 1× 1 positive-definite matricesequipped with its Fisher Information metric (see nextsection): for x, y ∈M,

dist2M(x, y) = log2(x

y).

For any two atoms a1, a2 ∈M, the function

ΨD,w(x) = w1 dist2M(x, a1) + w2 dist2M(x, a2)

does not have a global minimum if w1w2 < 0. How-ever, it has one unique critical point aw1

1 aw22 ∈ M.

In particular, for any number of atoms a1, ..., am andw ∈ Rm satisfying the affine constraint, ΨD,w(x) hasone unique critical point

FD(w) = aw11 · · · awm

m .

Furthermore, we have

Proposition 1 If M = Rd, then FD(w) is single-valued for all w such that w1 + · · ·+ wm = 1 and

FD(w) = w1a1 + · · ·+ wmam.

The proof is straightforward since the unique criticalpoint y is defined by the condition ∇ΨD,w(y) = 0 andΨD,w(y) = w1‖y− a1‖2 + · · ·+wm‖y− am‖2. Thanksto the affine constraint, we have

y = w1a1 + · · ·+ wmam,

and FD(w) = y.

While the acceptance of multi-valued FD(w) may bea source of discomfort or even annoyance at first, welist the following five important points that stronglysuggest the correctness of our generalization:

1. FD(w) = y incorporates the geometry of M inthe form of geodesic distances to the atoms, andin particular, for two nearby points x, y ∈ M,their sparse codings are expected in general to besimilar.

2. The generalization reduces to the correct form forRd.

3. FD(w) is effectively computable because any crit-ical point of a smooth function (e.g., ΨD,w) canbe reached via gradient decent or ascent with ap-propriate initial point.

4. The above point also shows the viability of us-ing FD(w) as an approximation to a data x. Es-sentially, the approximation looks for the criticalpoint of ΨD,w that is near x, and such criticalpoint, if exists, can be obtained by gradient de-scent or ascent from x.

5. While FD(w) is multi-valued, it is continuous inthe following sense: let t1, t2, ... ∈ Rm be a se-quence such that tn → tRm and y1, y2, ... ∈ Msuch that yi ∈ FD(ti) for each i and yn → y ∈Mas n → ∞. Then, y ∈ FD(t). This follows frominterpreting the gradient∇ΨD,ti as a smooth fam-ily of vector fields on M and yi are the zeros ofthese vector fields.

We believe that in generalizing sparse coding to Rie-mannina manifolds, it is not possible to preserve everydesirable property enjoyed by the sparse coding in theEuclidean space. In particular, in exchange for themulti-valued FD(w), we have retained enough usefulfeatures and properties that will permit us to pursuitdictionary learning on Riemannian manifolds.

4. Dictionary Learning on RiemannianManifolds

In this section, we present a dictionary learning al-gorithm based on the nonlinear sparse coding frame-work described above. Given a collection of features


x1, . . . , xn ∈ Rd, existing dictionary learning meth-ods such as (Olshausen & Field, 1997; Lewicki & Se-jnowski, 2000; Aharon et al., 2006) compute a dictio-nary D ∈ Rd×m with m atoms such that each featurexi can be represented as a sparse linear combinationof these atoms xi ≈ Dwi, where wi ∈ Rm. Using l1regularization on wi, the learning problem can be suc-cinctly formulated as an optimization problem (Mairalet al., 2010; Yang et al., 2009):

minD,wi

n∑i=1

(‖xi −Dwi‖22 + λ‖wi‖1

), (6)

where λ is a regularization parameter. There areseveral notable recent extensions of this well-knownformula, and they include online dictionary learning(Mairal et al., 2010) and dictionary learning using dif-ferent regularization schemes such as group-structuredsparsity (Szabo et al., 2011; Huang et al., 2011) andlocal-coordinate constraints (Wang et al., 2010). Forour nonlinear generalization, the main point is to makesense of the data fidelity term ‖xi−Dwi‖22 in the man-ifold setting. Formally, this is not difficult because,using previous notations, the data fidelity term can bedefined analogously using

distM(xi,FD(w))2.

We remark that FD(wi) is a multi-valued function andthe above equation should be interpreted as finding anpoint x̂i ∈ FD(wi) such that distM(xi, x̂i)

2 is small,i.e., x̂i is a good approximation of xi. However, the dis-tance distM(xi, x̂i) is generally difficult to compute,and to circumvent this difficulty, we propose a heuris-tic argument for approximating this distance using areadily computable function. The main idea is to notethat x̂i a critical point of the function ΨD,wi(x) andthe equation ∇ΨD,wi

(x̂i) = 0 implies that

m∑j=1

wij logx̂i(aj) = 0

since the LHS is precisely the gradient∇ΨD,wi

(x̂i) (Spivak, 1979). Therefore, if xi is a pointnear x̂i, the tangent vector (at xi)

∑mj=1 wij logxi

(aj)should be close to zero. In particular, this immediatelysuggests1 using

‖m∑j=1

wij logxi(aj)‖2xi

as a substitute for the distance distM(xi, x̂i)2.

1The validity of this approximation and the heuristicargument will be examined more closely in a future work.

This heuristic argument readily leads to the followingoptimization problem for dictionary learning on M:

minW,D

n∑i=1

‖m∑j=1

wij logxi(aj)‖2xi

+ λ‖W‖1

s.t.

m∑j=1

wij = 1, i = 1, . . . , n,

(7)

where W ∈ Rn×m and wij denotes its (i, j) compo-nent. Similar to Euclidean dictionary learning, theoptimization problem can be solved using an iterativealgorithm that iteratively performs

1. Sparse Coding: fix the dictionary D and opti-mize the sparse coefficients W.

2. Codebook Optimization: fix the sparse coeffi-cients W and optimize the dictionary D.

The first step is the regular sparse coding problem, andbecause the optimization domain is in Rn×m, manyfast algorithms are available. The codebook optimiza-tion step is considerably more challenging for two rea-sons. First, the optimization domain is no longer Eu-clidean but the manifoldM. Second, for a typical Rie-mannian manifold M, its Riemannian logarithm mapis very difficult to compute. Nevertheless, there arealso many Riemannian manifolds that have been ex-tensively studied by differential geometers with knownformulas for their exp/log maps, and these results sub-stantially simplify the computational details. The fol-lowing two subsection will present two such examples.For optimization on the manifold M, we use a linesearch-based algorithm to update the dictionary D,and the main idea is to determine a descent directionv (as a tangent vector at a point x ∈M) and performthe search on a geodesic in the direction v. The for-mal similarity between Euclidean and Riemannian linesearch is straightforward and tansparent, and the maincomplication in the Riemannian setting is the compu-tation of geodesics. We refer to (Absil et al., 2008) formore algorithm details and convergence analysis.

4.1. Symmetric Positive-Definite Matrices

Let P (d) denote the space of d×d symmetric positive-definite (SPD) matrices. The tangent space TXP (d)at every point X ∈ P (d) can be naturally identifiedwith Sym(d), the space of d × d symmetric matrices.The general linear group GL(d) acts transitively onP (d) : X −→ GXGT , where X ∈ P (d) and G ∈ GL(d)is a d × d invertible matrix. Let Y,Z ∈ TMP (d) betwo tangent vectors at M ∈ P (d), and define an inner-product in TMP (d) using the formula

〈Y,Z〉M = tr(YM−1ZM−1), (8)


where tr is the matrix trace. This above formula de-fines a Riemannian metric on P (d) that is invariant un-der the GL(d)-action (Pennec et al., 2006; Fletcher &Joshi, 2007), and the corresponding geometry on P (d)has been studied extensively by differential geometers(see (Helgason, 2001) and (Terras, 1985)), and ininformation geometry, this is the Fisher Informationmetric for P (d) considered as the domain for parame-terizing zero-mean normal distributions on Rd (Amari& Nagaoka, 2007). In particular, the formulas for com-puting geodesics, Riemannian exponential and log-arithm maps are well-known: The geodesic passingthrough M ∈ P (d) in the direction of Y ∈ TMP (d)is given by the formula

γ(t) = GExp(G−1Y G−T t

)GT , t ∈ R, (9)

where Exp denotes the matrix exponential and G ∈GL(d) is a square root of M such that M = GGT .Consequently, the Riemannian exponential map at Mwhich maps Y ∈ TMP (d) to a point in P (d) is givenby the formula

expM (Y ) = GExp(G−1Y G−T

)GT ,

and given two positive-definite matrices X,M ∈ P (d),the Riemannian logarithmic map logM : P (d) →TMP (d) is given by

LogM (X) = GLog(G−1XG−T )GT ,

where Log denotes the matrix logarithm. Finally, thegeodesic distance between M and X is given by theformula

dist(M,X) = ‖ logM (X)‖ =

√tr(Log2(G−1XG−T )).

The above formulas are useful for specializing Equa-tion 7 to P (d): let X1, . . . , Xn ∈ P (d) denote a collec-tion of d × d SPD matrices, and A1, . . . , Am ∈ P (d)the m atoms in the dictionary D. We have

m∑j=1

wij logXi(Aj) =

m∑j=1

wij Gi Log(G−1i AjG−Ti )GTi ,

where Gi ∈ GL(d) such that GiGTi = Xi. With l1

regularization, dictionary learning using Equation 7now takes the following precise form for P (d):

minW,D

n∑i=1

m∑j=1

m∑k=1

wijwiktr(LijLik) + λ‖W‖1

s.t.

m∑j=1

wij = 1, i = 1, . . . , n,

where Lij = log(G−1i AjG−Ti ). The resulting optimiza-

tion problem can be solved using the method outlinedpreviously.

4.2. Spheres and Square-Root DensityFunctions

We next study dictionary learning on spheres, the mostwell-known class of closed manifolds. In the context ofmachine learning and vision applications, spheres canbe used to parameterize the square roots of densityfunctions. More specifically, for a probability densityfunction p and its (continuous) square root ψ =

√p,

we have ∫S

ψ2ds = 1.

By expanding ψ using orthonormal basis functions(e.g., spherical harmonics if S is a sphere), the aboveequation allows us to identify ψ as a point on the unitsphere in a Hilbert space (see e.g., (Srivastava et al.,2007)). In other words, for a collection of density func-tions, we can consider them as a collection of pointsin some high-dimensional sphere, a finite-dimensionalsphere spanned by these points in the unit Hilbertiansphere. Under this identification, the classical Fisher-Rao metric (Rao, 1945) for the density functions p cor-responds exactly to the canonical metric on the sphere,and the differential geometry of the sphere is straight-forward: Given two points ψi, ψj on a d-dimensionalunit sphere Sd, the geodesic distance is just the anglebetween ψi and ψj considered as vectors in Rd+1

dist(ψi, ψj) = cos−1(〈ψi, ψj〉). (10)

The geodesic started at ψi in the direction v ∈ Tψi(Sd)

is given by the formula

γ(t) = cos(t)ψi + sin(t)v

|v|, (11)

and the exponential and logarithm maps are given bythe formulas:

expψi(v) = cos(|v|)ψi + sin(|v|) v

|v|,

logψi(ψj) = u cos−1(〈ψi, ψj〉)/

√〈u, u〉,

where u = ψj − 〈ψi, ψj〉ψi. We remark that in orderto ensure that the exponential map is well-defined, werequire that |v| ∈ [0, π), and similarly, the log maplogψi

(x) is defined on the sphere minus the antipodalpoint of ψi (x 6= −ψi).

Using these formulas, we can proceed as before to spe-cialize Equation 7 to the sphere Sd: Let x1, . . . , xn de-note a collection of square-root density functions con-sidered as points in a high-dimensional sphere Sd, andai, . . . , am ∈ Sd the atoms in the dictionary D. W isan n×m matrix. Using l1 regularization, Equation 7


takes the following form for Sd:

minW,D

n∑i=1

‖m∑j=1

wij cos−1(〈xi, aj〉)uij|uij |‖2xi

+ λ‖W‖1

s.t.

m∑j=1

wij = 1, i = 1, . . . , n,

where uij = aj − 〈xi, aj〉xi. The resulting optimiza-tion problem can be efficiently solved using the methodoutlined earlier.

5. Experiments

This section presents the details of the two classifica-tion experiments used in evaluating the proposed dic-tionary learning and sparse coding algorithms. Themain idea of the experiments is to transform each(manifold) data point xi into a feature vector wi ∈ Rmusing its sparse coefficients wi encoded with respect tothe trained dictionary. In practice, this approach offerstwo immediate advantages. First, the sparse featurewi is encoded with respect to a dictionary that is com-puted using the geometry of M, and therefore, it isexpected to be more discriminative and hence usefulthan the data themselves (Yang et al., 2009). Sec-ond, using wi as the discriminative feature allows usto train a classifier in the Euclidean space Rm, avoid-ing the more difficult problem of training the classi-fier directly on M. Specifically, let x1, . . . , xn ∈ Mdenote the n training data. A dictionary (or code-book) D = {a1, . . . , am} with m atoms is learned fromthe training data using the proposed method, and foreach xi, we compute its sparse feature wi ∈ Rm us-ing the learned dictionary D. For classification, wetrain a linear Support Vector Machine (SVM) usingw1, ..., wn as the labelled features. During testing, atest feature y ∈M is first sparse coded using D to ob-tain its sparse feature w, and the classification resultis computed by applying the trained SVM classifier tothe sparse feature w. The experiments are performedusing two public available datasets: Brodatz texturedataset and OASIS brain MRI dataset, and trainingand testing data are allocated by a random binary par-tition of the available data giving the same number oftraining and testing data.

For comparison, we use the following three alternativemethods: 1) Geodesic K-nearest neighbor (GKNN),2) SVM on vectorized data and 3) SVM on fea-tures sparse coded using a dictionary trained byKSVD (Aharon et al., 2006). GKNN is a K-nearestneighbor classifier that uses Riemannian distance onM for determining neighbors, and it is a straight-forward method for solving classification problems on

Figure 1. Samples of Brodatz textures used in the first ex-periment.

Table 1. The texture classification accuracy for differentmethods on the Brodatz dataset.

Class SVM KSVD+SVM GKNN Proposed

16 93.36 94.92 95.70 99.0232 88.67 90.82 91.11 95.70

manifolds. In the experiments, K is set to 5. For thesecond method (SVM), we directly vectorize the man-ifold data xi to form their Euclidean features (withoutsparse coding) w̃i and an SVM is trained using theseEuclidean features. The third method (SVM+KSVD)applies the popular KSVD method to train a dictio-nary using the Euclidean features w̃i, and it uses alinear SVM on the sparse features encoded used thisdictionary. All SVMs used in the experiments aretrained using the LIBSVM package (Chang & Lin,2011). We remark that the first method (GKNN) usesthe distance metric intrinsic to the manifold M with-out sparse feature transforms. The second and thirdmethods are extrinsic in nature and they completelyignore the geometry of M.

5.1. Brodatz Texture Dataset

In this experiment, we evaluate the dictionary learn-ing algorithm for SPD matrices using Brodatz texturedataset (Brodatz, 1966). Using a similar experimen-tal setup as in (Sivalingam et al., 2010), we construct16-texture and 32-texture sets using the images fromBrodatz dataset. Some sample textures are shown inFigure 1. Each 256× 256 texture image is partitionedinto 64 non-overlapping blocks of size 32 × 32. In-side each block, we compute a 5 × 5 covariance ma-trix FF> (Tuzel et al., 2006) summing over the block,

where F =(I,∣∣ ∂I∂x

∣∣ , ∣∣∣ ∂I∂y ∣∣∣ , ∣∣∣ ∂2I∂x2

∣∣∣ , ∣∣∣ ∂2I∂y2

∣∣∣)T . To ensure

that each covariance matrix is positive-definite, we useFF> + σE, where σ is a small positive constant andE is the identity matrix. For k-class problem, where


Figure 2. From left to right, three sample images from theOASIS dataset belonging to the Young, Middle-aged andOld groups, respectively.

Table 2. The classification accuracy for different methodson the OASIS dataset. There are three binary classifica-tions (YM: Young vs Middle-aged, MO: Middled-aged vsOld and YO: Young vs Old), and YMO is three-class clas-sification (Young, Middle-aged and Old).

SVM KSVD+SVM GKNN Proposed

YM 90.04 91.29 91.84 98.34MO 97.32 98.08 100 100YO 98.12 99.46 100 100YMO 91.97 94.04 93.18 98.62

k=16 or 32, the size of the dictionary D is set to 5k.The texture classification results are reported in Ta-ble 1. Our method outperforms the three compara-tive methods. Among the three comparative methods,sparse feature transform outperforms the one without(KSVD+SVM vs SVM) and the intrinsic method hasadvantage over extrinsic ones (GKNN vs. SVM andKSVD+SVM). Not surprisingly, our method utilizingboth intrinsic geometry and sparse feature transformoutperforms all three comparative methods.

5.2. OASIS Dataset

In this experiment, we evaluate the dictionary learn-ing algorithm for square-root densities using the OA-SIS database (Marcus et al., 2007). OASIS datasetcontains T1-weighted MR brain images from a cross-sectional population of 416 subjects. Each MRI scanhas a resolution of 176 × 208 × 176 voxels. The agesof the subjects range from 18 to 96. We divide theOASIS population into three (age) groups: young sub-jects (40 or younger), middle-aged subjects (between40 and 60) and old subjects (60 or older), and theclassification problem is to classify each MRI imageaccording to its age group. Sample images from thethree age groups are shown in Figure 2, and the subtledifferences in anatomical structure across different agegroups are apparent. The MR images in the OASISdataset are first aligned (with respect to a template)using the nonrigid group-wise registration method de-

scribed in (Joshi et al., 2004). For each image, weobtain a displacement field, and the histogram of thedisplacement vectors is computed for each image asthe feature for classification (Chen et al., 2010). Inour experiment, the number of bins in each directionis set to 4, and the resulting 64-dimensional histogramis used as the feature vector for the SVM+KSVD andSVM methods, while the square root of the histogramis used in GKNN and our method. The dictionar-ies (KSVD+SVM and our method) in this experi-ment have 100 atoms. We use five-fold cross valida-tion and report the classification results in Table 2.The pattern among the three comparative methods inthe previous experiment is also observed in this ex-periment, confirming the importance of intrinsic ge-ometry and sparse feature transform. We note thatall four methods produce good results for the two bi-nary classifications involving Old group, which can bepartially explained by the clinical observation that re-gional brain volume and cortical thickness of adults arerelatively stable prior to reaching age 60 (Mortametet al., 2005). For the more challenging problem of clas-sifying young and middle-aged subjects, our methodsignificantly outperforms the other three, demonstrat-ing again the effectiveness of combining intrinsic geom-etry and sparse feature transform for classifying man-ifold data.

6. Summary and Conclusions

We have proposed a novel dictionary learning frame-work for manifold-valued data. The proposed dic-tionary learning is based on a novel approach tosparse coding that uses the critical points of functionsconstructed from the Riemannian distance function.Compared with the existing (Euclidean) sparse cod-ing, the loss of the global linear structure is compen-sated by the local linear structures given by the tan-gent spaces of the manifold. In particular, we haveshown that using this generalization, the nonlinear dic-tionary learning for manifold-valued data shares manyformal similarities with its Euclidean counterpart, andwe have also shown that the latter can be consideredas a special case of the former. We have presentedtwo experimental results that validate the proposedmethod, and the two classification experiments pro-vide a strong support for the viewpoint advocated inthis paper that for manifold-valued data, their sparsefeature transforms should be formulated in the contextof an intrinsic approach that incorporates the geome-try of the manifold.

Acknowledgement This research was supported bythe NIH grant NS066340 to BCV.


References

Absil, P.A., Mahony, R., and Sepulchre, R. Optimiza-tion algorithms on matrix manifolds. Princeton UnivPr, 2008.

Aharon, M., Elad, M., and Bruckstein, A. K-SVD: An algo-rithm for designing overcomplete dictionaries for sparserepresentation. IEEE Trans. on Signal Processing, 54(11):4311–4322, 2006.

Amari, S. and Nagaoka, H. Methods of Information Geom-etry. American Mathematical Society, 2007.

Brodatz, P. Textures: a photographic album for artists anddesigners, volume 66. Dover New York, 1966.

Chang, C.-C. and Lin, C.-J. LIBSVM: A library for sup-port vector machines. ACM Transactions on IntelligentSystems and Technology, 2:27:1–27:27, 2011.

Chen, T., Rangarajan, A., and Vemuri, B.C. Caviar: Clas-sification via aggregated regression and its application inclassifying oasis brain database. In ISBI, 2010.

do Carmo, M. Riemannian Geometry. Birkhauser, 1992.

Edelman, A., Arias, T.A., and Smith, S.T. The geome-try of algorithms with orthogonality constraints. Arxivpreprint physics/9806030, 1998.

Elad, M. Sparse and Redundant Representations: FromTheory to Applications in Signal and Image Processing.Springer, 2010.

Fletcher, P.T. and Joshi, S. Riemannian geometry for thestatistical analysis of diffusion tensor data. Signal Pro-cessing, 87(2):250–262, 2007.

Helgason, S. Differantial Geometry, Lie Groups, and Sym-metric Spaces. American Mathematical Society, 2001.

Huang, J., Zhang, T., and Metaxas, D. Learning withstructured sparsity. The Journal of Machine LearningResearch, 999888:3371–3412, 2011.

Joshi, S., Davis, B., Jomier, M., and Gerig, G. Unbi-ased diffeomorphic atlas construction for computationalanatomy. NeuroImage, 23:S151–S160, 2004.

Lewicki, M.S. and Sejnowski, T.J. Learning overcom-plete representations. Neural computation, 12(2):337–365, 2000.

Mairal, J., Bach, F., Ponce, J., and Sapiro, G. Onlinelearning for matrix factorization and sparse coding. TheJournal of Machine Learning Research, 11:19–60, 2010.

Marcus, D.S., Wang, T.H., Parker, J., Csernansky, J.G.,Morris, J.C., and Buckner, R.L. Open access seriesof imaging studies (oasis): cross-sectional mri data inyoung, middle aged, nondemented, and demented olderadults. Journal of Cognitive Neuroscience, 19(9):1498–1507, 2007.

Mardia, K. and Jupp, P. Directional Statistics. Wiley,1999.

Mortamet, B., Zeng, D., Gerig, G., Prastawa, M., and Bul-litt, E. Effects of healthy aging measured by intracra-nial compartment volumes using a designed MR braindatabase. MICCAI, pp. 383–391, 2005.

Nash, J. The imbedding problem for riemannian manifolds.Annals of Mathematics, 63(1):20–63, 1956.

Olshausen, B.A. and Field, D.J. Sparse coding with anovercomplete basis set: A strategy employed by V1? Vi-sion Research, 37(23):3311–3325, 1997.

Pennec, X., Fillard, P., and Ayache, N. A riemannianframework for tensor computing. International Journalof Computer Vision, 66(1):41–66, 2006.

Rao, C.R. Information and accuracy attainable in the es-timation of statistical parameters. Bull. Calcutta Math.Soc, 37(3):81–91, 1945.

Sivalingam, R., Boley, D., Morellas, V., and Pa-panikolopoulos, N. Tensor sparse coding for region co-variances. In ECCV, pp. 722–735, 2010.

Spivak, M. A comprehensive introduction to differentialgeometry. Publish or perish Berkeley, 1979.

Srivastava, A., Jermyn, I., and Joshi, S. Riemannian anal-ysis of probability density functions with applications invision. In CVPR, 2007.

Szabo, Z., Poczos, B., and Lorincz, A. Online group-structured dictionary learning. In CVPR, pp. 2865–2872,2011.

Terras, A. Harmonic Analysis on Symmetric Spaces andApplications. Springer-Verlag, 1985.

Turaga, P., Veeraraghavan, A., and Chellappa, R. Statis-tical analysis on stiefel and grassmann manifolds withapplications in computer vision. In CVPR, 2008.

Tuzel, O., Porikli, F., and Meer, P. Region covariance: Afast descriptor for detection and classification. In ECCV,pp. 589–600, 2006.

Wang, J., Yang, J., Yu, K., Lv, F., Huang, T., and Gong,Y. Locality-constrained linear coding for image classifi-cation. In CVPR, pp. 3360–3367, 2010.

Yang, J., Yu, K., Gong, Y., and Huang, T. Linear spatialpyramid matching using sparse coding for image classi-fication. In CVPR, pp. 1794–1801, 2009.

Yu, K. and Zhang, T. Improved local coordinate codingusing local tangents. In ICML, pp. 1215–1222, 2010.

Yu, K., Zhang, T., and Gong, Y. Nonlinear learning usinglocal coordinate coding. NIPS, 22:2223–2231, 2009.

on a nonlinear generalization of sparse coding and dictionary

Documents