information-theoretic dictionary learning for image classiﬁcation · 2018-09-04 · datasets...

1

Information-theoretic Dictionary Learning for ImageClassification

Qiang Qiu, Vishal M. Patel, Member, IEEE, and Rama Chellappa, Fellow, IEEE.

Abstract—We present a two-stage approach for learning dic-tionaries for object classification tasks based on the principleof information maximization. The proposed method seeks adictionary that is compact, discriminative, and generative. In thefirst stage, dictionary atoms are selected from an initial dictionaryby maximizing the mutual information measure on dictionarycompactness, discrimination and reconstruction. In the secondstage, the selected dictionary atoms are updated for improvedreconstructive and discriminative power using a simple gradientascent algorithm on mutual information. Experiments using realdatasets demonstrate the effectiveness of our approach for imageclassification tasks.

Index Terms—Dictionary learning, information theory, mutualinformation, entropy, image classification.

I. INTRODUCTION

Sparse signal representations have recently drawn muchtraction in vision, signal and image processing [1], [2], [3],[4]. This is mainly due to the fact that signals and images ofinterest can be sparse in some dictionary. Given a redundantdictionary D and a signal y, finding a sparse representationof y in D entails solving the following optimization problem

x = arg minx‖x‖0 subject to y = Dx, (1)

where the `0 sparsity measure ‖x‖0 counts the number ofnonzero elements in the vector x. Problem (1) is NP-hard andcannot be solved in a polynomial time. Hence, approximatesolutions are usually sought [3], [5], [6], [7].

The dictionary D can be either based on a mathematicalmodel of the data [3] or it can be trained directly from thedata [8]. It has been observed that learning a dictionary directlyfrom training rather than using a predetermined dictionary(such as wavelet or Gabor) usually leads to better representa-tion and hence can provide improved results in many practicalapplications such as restoration and classification [1], [2], [4],[9].

Various algorithms have been developed for the task oftraining a dictionary from examples. One of the most com-monly used algorithms is the K-SVD algorithm [10]. Givena set of examples {yi}ni=1, K-SVD finds a dictionary D thatprovides the best representation for each example in this setby solving the following optimization problem

(D, X) = arg minD,X‖Y −DX‖2F subject to ∀i ‖xi‖0 ≤ T0,

(2)

Q. Qiu, V. M. Patel, and R. Chellappa are with the Center for AutomationResearch, UMIACS, University of Maryland, College Park, MD 20742 USA(e-mail: [email protected], {pvishalm, rama}@umiacs.umd.edu)

where xi represents the ith column of X, Y is the matrixwhose columns are yi and T0 is the sparsity parameter. Here,the Frobenius norm is defined as ‖A‖F =

√∑ij A

2ij . The

K-SVD algorithm alternates between sparse-coding and dic-tionary update steps. In the sparse-coding step, D is fixed andthe representation vectors xis are found for each example yi.Then, the dictionary is updated atom-by-atom in an efficientway.

Dictionaries can be trained for both reconstruction andclassification applications. In the late nineties, Etemand andChellappa proposed a linear discriminant analysis (LDA)based basis selection and feature extraction algorithm forclassification using wavelet packets [11]. Recently, similaralgorithms for simultaneous sparse signal representation anddiscrimination have also been proposed in [12], [13], [14],[15]. Some of the other methods for learning discriminativedictionaries include [16], [17], [18], [19], [20], [21], [12].Additional techniques may be found within these references.

In this paper, we propose a general method for learningdictionaries for image classification tasks via informationmaximization. Unlike other previously proposed dictionarylearning methods that only consider learning only recon-structive and/or discriminative dictionaries, our algorithm canlearn reconstructive, compact and discriminative dictionariessimultaneously. Sparse representation over a dictionary withcoherent atoms has the multiple representation problem. Acompact dictionary consists of incoherent atoms, and encour-ages similar signals, which are more likely from the sameclass, to be consistently described by a similar set of atomswith similar coefficients [21]. A discriminative dictionaryencourages signals from different classes to be described byeither a different set of atoms, or the same set of atoms but withdifferent coefficients [13], [15], [17]. Both aspects are criticalfor classification using sparse representation. The additionalreconstructive requirement to a compact and discriminativedictionary enhances the robustness of the discriminant sparserepresentation [13]. All these three criteria are critical forclassification using sparse representation.

Our method of training dictionaries consists of two mainstages involving greedy atom selection and simple gradientascent atom updates, resulting in a highly efficient algorithm.In the first stage, dictionary atoms are selected in a greedy wayfrom an initial dictionary by maximizing the mutual informa-tion measure on dictionary compactness, discrimination andreconstruction. In the second stage, the dictionary is updatedfor improved discrimination and reconstruction via a simplegradient ascent method that maximizes the mutual information

2

(a) Four subjects

0

5

−2 −1 0 1

−5.5

−5

−4.5

−4

−3.5

−3Our approach (acc: 91.89%)

0

0.5

1

1.5 −0.5 0 0.5 1

−0.4

−0.2

0

0.2

0.4

0.6SOMP (acc: 36.03%)

−1

0

1

2 −20

2

−0.5

0

0.5

1

1.5

2

2.5MMI-1 (acc: 32.43%)

−1

0

1

2 −20

2

−1

−0.5

0

0.5

1

1.5

2MMI-2 (acc: 38.73%)

(b) Sparse representation for images of four subjects (Sparsity = 3)

(c) Four digits

−2

−1

0

1 −1 0 1 2

−1

−0.5

0

0.5

1

1.5Our approach (acc: 93.19%)

−0.5

0

0.5

1 −0.5 0 0.5 1

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

SOMP (acc: 73.46%)

−0.5

0

0.5

1 −0.5 0 0.5 1

−0.2

0

0.2

0.4

0.6

0.8

1

1.2

MMI-1 (acc: 70.06%)

−0.5

0

0.5

1 00.5

1

−0.2

0

0.2

0.4

0.6

0.8

1

1.2

MMI-2 (acc: 70.06%)

(d) Sparse representation for four handwritten digits (Sparsity = 3)

Fig. 1: Sparse representation using dictionaries learned by different approaches (SOMP [22], MMI-1 and MMI-2 [21]). Forvisualization, sparsity 3 is chosen, i.e., no more than three dictionary atoms are allowed in each sparse decomposition. Whensignals are represented at once as a linear combination of a common set of atoms, sparse coefficients of all the samples becomepoints in the same coordinate space. Different classes are represented by different colors. The recognition accuracy is obtainedthrough linear SVMs on the sparse coefficients. Our approach provides more discriminative sparse representation which leadsto significantly better classification accuracy (best viewed in color).

(MI) between the signals and the dictionary, as well as thesparse coefficients and the class labels.

Fig. 1 presents a comparison in terms of the discrimina-tive power of the information-theoretic dictionary learningapproach presented in this paper with three state-of-the-artmethods. Scatter plots of sparse coefficients obtained usingthe different methods show that our method provides morediscriminative sparse representation, leading to significantlybetter classification accuracy.

The organization of the paper is as follows. Section IIdefines and formulates the information theoretic dictionarylearning problem. In Section III, the proposed dictionary learn-ing algorithm is detailed. Experimental results are presentedin Section IV and Section V concludes the paper with a briefsummary and discussion.

II. BACKGROUND AND PROBLEM FORMULATION

Suppose we are given a set of N signals (images) in ann-dim feature space Y = [y1, ...,yN], yi ∈ Rn. Given thatsignals are from p distinct classes and Nc signals are fromthe c-th class, c ∈ {1, · · · , p}, we denote Y = {Yc}pc=1,where Yc = [yc

1, · · · ,ycNc

] are signals in the c-th class.When the class information is relevant, similarly, we defineX = {Xc}pc=1, where Xc = [xc

1, · · · ,xcNc

] is the sparserepresentation of Yc.

Given a sample y at random, the entropy (uncertainty) ofthe class label in terms of class prior probabilities is defined

asH(C) =

∑c

p(c) log

(1

p(c)

).

The mutual information which indicates the decrease in un-certainty about the pattern y due to the knowledge of theunderlying class label c is defined as

I(Y;C) = H(Y)−H(Y|C),

where H(Y|C) is the conditional entropy defined as

H(Y|C) =∑y,c

p(y, c) log

(1

p(y|c)

).

Given Y and an initial dictionary Do with `2 normal-ized columns, we aim to learn a compact, reconstructiveand discriminative dictionary D∗ via maximizing the mutualinformation between D∗ and the unselected atoms Do\D∗ inDo, between the sparse codes XD∗ associated with D∗ andthe signal class labels C, and finally between the signals Yand D∗, i.e.,

arg maxDλ1I(D; Do\D) + λ2I(XD ;C) + λ3I(Y; D) (3)

where {λ1, λ2, λ3} are the parameters to balance the contri-butions from compactness, discriminability and reconstructionterms, respectively.

It is widely known that inclusion of additional criteria, suchas a discriminative term, in a dictionary learning framework

3

often involves challenging optimization algorithms [17], [18],[12]. As discussed above, compactness, discriminability andreconstruction terms are all critical for classification usingsparse representation. Maximizing mutual information enablesa simple way to unify all three criteria for dictionary learning.As suggested in [23] and [21], maximizing mutual informationcan also lead to a sub-modular objective function, i.e., a greedyyet near-optimal approach, for dictionary learning.

A two-stage approach is adopted to satisfy (3). In the firststage, each term in (3) is maximized in a unified greedymanner and involves a closed-form evaluation, thus atoms canbe greedily selected from the initial dictionary while satisfying(3). In the second stage, the selected dictionary atoms areupdated using a simple gradient ascent method to furthermaximize

λ2I(XD ;C) + λ3I(Y; D).

III. INFORMATION-THEORETIC DICTIONARY LEARNING

In this section, we present the details of our Information-theoretic Dictionary Learning (ITDL) approach for classifica-tion tasks. The dictionary learning procedure is divided intotwo main steps: Information-theoretic Dictionary Selection(ITDS) and Information-theoretic Dictionary Update (ITDU).In what follows, we describe these steps in detail.

A. Dictionary Selection

Given input signals Y and an initial dictionary Do, we se-lect a subset of dictionary atoms D∗ from Do via informationmaximization, i.e., maximizing (3), to encourage the signalsfrom the same class to have very similar sparse representationyet have the discriminative power. In this section, we illustratewhy each term in (3) describes the dictionary compactness,discrimination and representation, respectively. We also showthat how each term in (3) can be maximized in a unified greedymanner that involves closed-form computations. Therefore, ifwe start with D∗ = ∅, and greedily select the next best atomd∗ from Do\D∗ which provides an information increase to(3), we obtain a set of dictionary atoms that is compact,reconstructive and discriminative at the same time. To thisend, we consider in detail each term in (3) separately.

1) Dictionary compactness I(D∗; Do\D∗): The dictionarycompactness I(D∗; Do\D∗) has been studied in our earlywork [21]. We summarize [21] to complete our information-driven dictionary selection discussion. [21] suggests dictionarycompactness is required to avoid the multiple sparse represen-tation problem for better classification performance. In [21],we first model sparse representation through a Gaussian Pro-cess model to define the mutual information I(D∗; Do\D∗).A compact dictionary can be then obtained as follows: we startwith D∗ = ∅ and iteratively choose the next best dictionaryitem d∗ from Do\D∗ which provides a maximum increase inmutual information, i.e.,

arg maxd∗∈Do\D∗I(D∗∪d∗; Do\(D∗∪d∗))−I(D∗; Do\D∗).(4)

It has been proved in [23] that the above greedy algorithmserves a polynomial-time approximation that is within (1 −1/e) of the optimum.

2) Dictionary Discrimination I(XD∗ ;C): Using any pur-suit algorithm such as OMP [6], we initialize the sparsecoefficients XDo for input signals Y and an initial dictionaryDo. Given XD∗ are sparse coefficients associated with thedesired set of atoms D∗ and C are the class labels for inputsignals Y, based on [24], an upper bound on the Bayes errorover sparse representation E(XD∗) is obtained as

1

2(H(C)− I(XD∗ ;C)).

This bound is minimized when I(XD∗ ;C) is maximized.Thus, a discriminative dictionary D∗ is obtained via

arg maxD∗

I(XD∗ ;C). (5)

We maximize (5) using a greedy algorithm initialized by D∗ =∅ and iteratively choosing the next best dictionary atom d∗

from Do\D∗ which provides a maximum mutual informationincrease, i.e.,

arg maxd∗∈Do\D∗

I(XD∗∪d∗ ;C)− I(XD∗ ;C), (6)

where I(XD∗ ;C) is evaluated as follows

I(XD∗ ;C) = H(XD∗)−H(XD∗ |C) (7)

= H(XD∗)−p∑

c=1

p(c)H(XD∗ |c).

Entropy measures in (7) involve computation of probabilitydensity functions p(XD∗) and p(XD∗ |c). We adopt the kerneldensity estimation method [25] to non-parametrically estimatethe probability densities. Using isotropic Gaussian kernels (i.e.Σ = σ2I, where I is the identity matrix), the class dependentdensity for the c-th class can be estimated as

p(x|c) =1

Nc

Nc∑j=1

KG(x− xcj , σ

2I), (8)

where KG is a d-dim Gaussian kernel defined as

KG(x,Σ) =1

(2π)d2 |Σ| 12

exp

(−1

2xTΣ−1x

). (9)

With p(c) = Nc

N , we can estimate p(x) as

p(x) =∑c

p(x|c)p(c).

3) Dictionary Representation I(Y; D∗): A representativedictionary D∗ maximizes the mutual information betweendictionary atoms and the signals, i.e.,

arg maxD∗

I(Y; D∗). (10)

We obtain a representative dictionary via a similar greedymanner as discussed above. That is, we iteratively choose thenext best dictionary atom d∗ from Do\D∗ which provides themaximum increase in mutual information,

arg maxd∗∈Do\D∗

I(Y; D∗ ∪ d∗)− I(Y; D∗). (11)

4

By assuming the signals are drawn independently and usingthe chain-rule of entropies, we can evaluate I(Y; D∗) as

I(Y; D∗) = H(Y)−H(Y|D∗) (12)

= H(Y)−N∑i=1

H(yi|D∗).

H(Y) is independent of dictionary selection and can beignored. To evaluate H(yi|D∗) in (12), we define p(yi|D∗)through the following relation holding for each input signalyi,

yi = D∗xi + ri,

where ri is a Gaussian residual vector with variance σ2r . Such

a relation can be written in a probabilistic form as,

p(yi|D∗) ∝ exp(− 1

2σ2r

||yi −D∗xi||2).

4) Selection of λ1, λ2 and λ3: The parameters λ1, λ2 andλ3 in (3) are data dependent and estimated as respective ratiosbetween the maximal information gained from an dictionaryatom to the compactness, discrimination and reconstructionmeasures, i.e.,

λ1 = 1, (13)

λ2 =maxi I(Xdi

;C)

maxi I(di; Do\di),

λ3 =maxi I(Y; di)

maxi I(di; Do\di).

For each term in (3), only the first greedily selected atom basedon (4), (6) and (11), respectively are involved in parameterestimation. In more detail, the first greedily selected atomusing (4) alone provides the value of maxi I(di; D

o\di), thefirst atom using (6) provides maxi I(Xdi

;C), and the firstatom using (11) provides maxi I(Y; di). This leads to anefficient process in finding parameters.

B. Dictionary Update

A representative and discriminative dictionary D producesthe maximal MI between the sparse coefficients and the classlabels, as well as the signals and the dictionary, i.e.,

maxD

λ2I(XD ;C) + λ3I(Y; D).

In the dictionary update stage, we update the set of selecteddictionary atoms D to further enhance the discriminability andrepresentation.

To achieve sparsity, we assume the cardinality of the setof selected atoms D is much smaller than the dimension ofthe signal feature space. Under such an assumption, the sparserepresentation of signals Y can be obtained as XD = D†Ywhich minimizes the representation error ‖Y − DXD‖2F ,where

D† = (DTD)−1DT.

Thus, updating dictionary atoms for improving discriminabil-ity while maintaining representation is transformed into find-ing D† that maximizes

I(D†Y;C).

1) A Differentiable Objective Function: To enable a simplegradient ascent method for dictionary update, we first ap-proximate I(D†Y;C) using a differentiable objective func-tion. I(X;C) can be viewed as the Kullback-Leibler (KL)divergence D(p‖q) between p(X, C) and p(X)p(C), whereX = D†Y. Motivated by [26], we approximate the KL di-vergence D(p‖q) with the quadratic divergence (QD), definedas

Q(p‖q) =

∫t

(p(t)− q(t))2 dt,

making I(X;C) differentiable. Due to the property that

D(p‖q) ≥ 1

2Q(p‖q),

by maximizing the QD, one can also maximize a lower boundto the KL divergence. With QD, I(X;C) can now be evaluatedas,

IQ(X;C) =∑c

∫x

p(x, c)2 dx (14)

− 2∑c

∫x

p(x, c)p(x)p(c) dx

+∑c

∫x

p(x)2p(c)2 dx.

In order to evaluate the individual terms in (14), we need toderive expressions for the kernel density estimates of variousdensity terms appearing in (14). Observe that for the twoGaussian kernels in (9), the following holds∫x

KG(x−si,Σ1)KG(x−sj ,Σ2) dx = KG(si−sj ,Σ1+Σ2).

(15)Using (8), p(c) = Nc

N and p(x, c) = p(x|c)p(c), we have

p(x, c) =1

N

Nc∑j=1

KG(x− xcj , σ

2I).

Similarly, since p(x) =∑

c p(x, c), we have

p(x) =1

N

N∑i=1

KG(x− xi, σ2I).

Inserting expressions for p(x, c) and p(x) into (14) and using(15), we get the following closed form

IQ(X;C) =1

N2

p∑c=1

Nc∑k=1

Nc∑l=1

KG(xck − xc

l , 2σ2I)

− 2

N2

p∑c=1

Nc

N

Nc∑j=1

N∑k=1

KG(xcj − xk, 2σ

2I)

+1

N2

(p∑

c=1

(Nc

N

)2)

N∑k=1

N∑l=1

KG(xk − xl, 2σ2I).

(16)

5

2) Gradient Ascent Update: For simplicity, we define a newmatrix Φ as

Φ , (D†)T.

Once we have estimated IQ(X;C) as a function of the data setin a differential form, where X = ΦTY, we can use gradientascent on IQ(X;C) to search for the optimal Φ maximizingthe quadratic mutual information with

Φk+1 = Φk + ν∂IQ∂Φ|Φ=Φk

where ν ≥ 0 defining the step size, and

∂IQ∂Φ

=

p∑c=1

Nc∑i=1

∂IQ∂xc

i

∂xci

∂Φ.

Since xci = ΦTyc

i , we get

∂xci

∂Φ= (yc

i )T .

Note that∂

∂xiKG(xi − xj , 2σ

2I) = KG(xi − xj , 2σ2I)

(xi − xj)

2σ2.

We have

∂

∂xci

IQ =1

N2σ2

Nc∑k=1

KG(xck − xc

i , 2σ2I)(xc

k − xci )

− 2

N2σ2

(p∑

c=1

(Nc

N

)2)

N∑k=1

KG(xk − xci , 2σ

2I)(xk − xci )

+1

N2σ2

p∑k=1

Nk +Nc

2N

Nk∑j=1

KG(xkj − xc

i , 2σ2I)(xk

j − xci ).

(17)

Once Φ is updated, the dictionary D can be updated usingthe relation Φ = (D†)T. Each update step requires O(N2)operations. Such dictionary updates guarantee convergence toa local maximum due to the fact that the quadratic divergenceis bounded [27].

C. Dictionary Learning Framework

Given a dictionary Do, a set of signals Y, the class labelsC and a sparsity level T , the supervised sparse coding methodgiven in Algorithm 1 represents these signals at once asa linear combination of a common subset of T atoms inD, where T is much smaller than the dimension of thesignal feature space to achieve sparsity. We obtain a sparserepresentation as each signal has no more than T coefficientsin its decomposition. The advantage of simultaneous sparsedecomposition for classification has been discussed in [13].Such simultaneous decompositions extract the internal struc-ture of given signals and neglects minor intra-class variations.The ITDS stage in Algorithm 1 ensures such common set ofatoms are compact, discriminative and reconstructive.

When the internal structures of signals from different classescan not be well represented in a common linear subspace, Al-gorithm 2 illustrates supervised sparse coding with a dedicatedset of atoms per class. It is noted in Algorithm 2 that both the

Input: Dictionary Do, signals Y, class labels C, sparsity levelT

Output: sparse coefficients X, reconstruction Ybegin

Initialization stage:1. Initialize X with any pursuit algorithm,i = 1, · · · , N minxi ‖yi −Doxi‖22 s.t. ‖xi‖0 ≤ T .

ITDS stage (shared atoms):2. Estimate λ1, λ2 and λ3 from Y, X and C;3. Find T most compact, discriminative and reconstructiveatoms:

D∗ ← ∅; Γ← ∅ ;for t=1 to T do

d∗ ← arg maxd∈Do\D∗

λ1[I(D∗ ∪ d;Do\(D∗ ∪ d))−I(D∗;Do\D∗)] + λ2[I(XD∗∪d;C)− I(XD∗ ;C)] +λ3[I(Y;D∗ ∪ d)− I(Y;D∗)];

D∗ ← D∗⋃

d∗;Γ← Γ

⋃γ∗, γ∗ is the index of d∗ in Do ;

end4. Compute sparse codes and reconstructions:

X← pinv(D∗)Y;Y ← D∗X;

5. return X, Y, D∗, Γ ;end

Algorithm 1: Sparse coding with global atoms.

Input: Dictionary Do, signals Y = {Yc}pc=1, sparsity level TOutput: sparse coefficients {Xc}pc=1, reconstruction {Yc}pc=1

beginInitialization stage:1. Initialize X with any pursuit algorithm,i = 1, · · · , N minxi ‖yi −Doxi‖22 s.t. ‖xi‖0 ≤ T .

ITDS stage (dedicated atoms):for c=1 to p do

2. Cc ← {ci|ci = 1 if yi ∈ Yc, 0 otherwise } ;3. Estimate λ1, λ2 and λ3 from Yc, X and Cc;4. Find T most compact, discriminative andreconstructive atoms for class c:

D∗ ← ∅; Γ← ∅ ;for t=1 to T do

d∗ ← arg maxd∈Do\D∗

λ1[I(D∗ ∪ d;Do\(D∗ ∪d))− I(D∗;Do\D∗)] + λ2[I(XD∗∪d;Cc)−I(XD∗ ;Cc)] + λ3[I(Yc;D

∗ ∪ d)− I(Yc;D∗)];

D∗ ← D∗⋃

d∗;Γ← Γ

⋃γ∗, γ∗ is the index of d∗ in Do ;

endD∗c ← D∗; Γc ← Γ;5. Compute sparse codes and reconstructions:

Xc ← pinv(D∗c)Yc;Yc ← D∗cXc;

end6. return {Xc}pc=1, {Yc}pc=1, {D∗c}pc=1, {Γc}pc=1 ;

endAlgorithm 2: Sparse coding with atoms per class.

discriminative and reconstructive terms in ITDS are handledon a class by class basis.

A sparse dictionary learning framework, such as K-SVD[10] which learns a dictionary that minimizes the reconstruc-tion error, usually consists of sparse coding and update stages.

6

Input: Dictionary Do, signals Y = {Yc}pc=1, class labels C,sparsity level T , update step ν

Output: Learned dictionary D, sparse coefficients X,reconstruction Y

beginSparse coding stage:Use supervised sparse coding to obtain {D∗c}pc=1.

ITDU stage:foreach class c do

[In the shared atom case, use the global label Cinstead of Cc, and one iteration is required as the sameD∗c is used for all classes.]Cc ← {ci|ci = 1 if yi ∈ Yc, 0 otherwise } ;Φ1 ← pinv(D∗c)T ;X← pinv(D∗c)Y;repeat

Φk+1 = Φk + ν∂IQ(X,Cc)

∂Φ|Φ=Φk ;

D∗ ← pinv(ΦTk+1);

X← pinv(D∗)Y;until convergence;D∗c ← D∗ ;

endforeach class c do

Xc ← pinv(D∗c)Yc;Yc ← D∗cXc;

endreturn {Xc}pc=1, {Yc}pc=1, {D∗c}pc=1 ;

endAlgorithm 3: Sparse coding with atom updates.

In K-SVD, at the coding stage, a pursuit algorithm is employedto select a set of atoms for each signal; and at the update stage,the selected atoms are updated through SVD for improvedreconstruction. Similarly, in Algorithm 3, at the coding stage,ITDS is employed to select a set of atoms for each class ofsignals; and at the update stage, the selected atoms are updatedthrough ITDU for improved reconstruction and discrimination.Algorithm 3 is also applicable to the case when sparse codingis achieved using global atoms.

IV. EXPERIMENTAL EVALUATION

This section presents an experimental evaluation on threepublic datasets: the Extended YaleB face dataset [28], theUSPS handwritten digits dataset [29], and the 15-Scenesdataset [30]. The Extended YaleB dataset contains 2414 frontalface images for 38 individuals. This dataset is challengingdue to varying illumination conditions and expressions. TheUSPS dataset consists of 8-bit 16×16 images of “0” through“9” and 1100 examples for each class. The 15-Scenes datasetcontains 4485 images falling into 15 scene categories. The 15categories include images of living rooms, kitchens, streets,industrials, etc.. In all of our experiments, linear SVMs onthe sparse coefficients are used for classifiers. First, we thor-oughly evaluate the basic behaviors of the proposed dictionarylearning method. Then we evaluate the discriminative powerof the ITDL dictionary over the full Extended YaleB dataset,the full USPS dataset, and the 15-Scenes dataset.

A. Evaluation with Illustrative ExamplesTo enable visualized illustrations, we conduct the first set of

experiments on the first four subjects in the Extended YaleBface dataset and the first four digits in the USPS digit dataset.Half of the data are used for training and the rest is used fortesting.

1) Comparing Atom Selection Methods: We initialize a128 sized dictionary using the K-SVD algorithm [10] onthe training face images of the first four subjects in theExtended YaleB dataset. A K-SVD dictionary only minimizesthe reconstruction error and is not yet optimal for classificationtasks. As shown later, one can also initialize the dictionarydirectly with training samples or even with random noise.Due to the fact that an ITDL dictionary converges to a localmaximum, a better initial dictionary generally helps ITDL interms of classification performance.

In Fig. 2, we present the recognition accuracy and thereconstruction error with different sparsity on the first foursubjects in the Extended YaleB dataset. The Root Mean SquareError (RMSE) is employed to measure the reconstruction error.To illustrate the impact of the compactness, discriminationand reconstruction terms in (3), we keep one term at atime for the three selection approaches, i.e., the compact, thediscriminative and the reconstructive method. The compactmethod is equivalent to MMI-1 discussed in [21].

Parameters λ1, λ2 and λ3 in (3) are estimated as discussed inSection III-A4. As the dictionary learning criteria becomes lesscritical when sparsity increases, i.e., more energies in signalsare actually preserved and good atoms are selected anyway, wefocus on curves in Fig. 2 when sparsity<20. Although sparsecoding methods generally perform well for face recognition,it is still easy to notice that the proposed ITDS method usingall three terms (red) significantly outperforms those whichoptimize just one of the three terms, compactness (black),discrimination (blue), and representation (green), in terms ofrecognition accuracy. For example, the discrimination termalone (blue) leads to a better initial but poor overall recog-nition performance. The proposed ITDS method also providesmoderate reconstruction error. In Fig. 2, it is interesting tonote that the reconstructive term in (3) delivers nearly identicalrecognition accuracy and RMSE to SOMP [22] with both theshared and dedicated atoms, given the different formulationsof two methods. The proposed dictionary selection using allthree terms provides a good local optimum to converge at thedictionary update stage.

2) Enhanced Discriminability with Atom Update: We illus-trate how the discriminability of dictionary atoms selected bythe ITDS method can be further enhanced using the proposedITDU method. We initialize a 128 sized K-SVD dictionaryfor the face images and a 64 sized K-SVD dictionary for thethe digit images. Sparsity 2 is adopted for visualization, as thenon-zero sparse coefficients of each image can now be plottedas a 2-D point. In Fig. 3, with a common set of atoms sharedover all classes, sparse coefficients of all samples becomepoints in the same 2-D coordinate space. Different classes arerepresented by different colors. The original images are alsoshown and placed at the coordinates defined by their non-zerosparse coefficients. The atoms to be updated in Fig. 3a and

7

0 5 10 15 20 25 3020

30

40

50

60

70

80

90

100

Sparsity

Recognitio

n A

ccura

cy

ITDSCompactDiscriminativeReconstructiveSOMP

(a) Recognition Rate

0 5 10 15 20 25 306

8

10

12

14

16

18

20

22

SparsityR

MS

E (

10

−3)


(b) RMSE

0 5 10 15 20 25 3030

40

50

60

70

80

90

100

Sparsity

Recognitio

n A

ccura

cy


(c) Recognition Rate

0 5 10 15 20 25 300

2

4

6

8

10

12

14

16

18

Sparsity

RM

SE

(1

0−

3)


(d) RMSE

Fig. 2: Recognition accuracy and RMSE on the YaleB dataset using different dictionary selection methods. We vary the sparsitylevel, i.e., the maximal number of dictionary atoms that are allowed in each sparse decomposition. In (a) and (b), a global setof common atoms are selected for all classes. In (c) and (d), a dedicated set of atoms are selected per class. In both cases, theproposed ITDS (red lines) provides the best recognition performance and moderate reconstruction error.

3d are selected using ITDS. We can see from Fig. 3 that theproposed ITDU method makes sparse coefficients of differentclasses more discriminative, leading to significantly improvedclassification accuracy. Fig. 4 shows that the ITDU methodalso enhances the discriminability of atoms dedicated to eachclass. It is noted that, though the dictionary update sometimesonly converges after a considerable number of iterations, basedon our experience, the first 50 to 100 iterations in general bringsignificant improvement in classification accuracy.

3) Enhanced Reconstruction with Atom Update: FromFig. 5d, we notice obvious errors in the reconstructed digits,shown in Fig. 5c with atoms selected from the initial K-SVD dictionary using ITDS. After 30 ITDU iterations, Fig. 5eshows that all digits are reconstructed correctly with a unifiedintra-class structure and limited intra-class variation. Thisleads to a more accurate classification as shown in Fig. 4.It is noted that Fig. 5 and Fig. 4 are results from the sameset of experiments. As can be seen from Fig. 5f, after ITDUconverges, all digits are reconstructed correctly with the trueunderlying intra-class structures, i.e., the left-slanted and right-slanted styles for both digits “1” and “0”. Fig. 5b shows theimages in Fig. 5c with 60% missing pixels.

4) Dictionary Initialization: In the experiments above, weinitialize a dictionary using the K-SVD algorithm. In fact, onecan also initialize a dictionary directly with randomly selectedtraining samples, or even with random noises. As shown inFig. 6 and Fig. 7, with random sample or random noiseinitialized dictionaries, we obtain comparable recognition andreconstruction performance to the K-SVD dictionary in Fig. 5.Due to the fact that an ITDL dictionary converges to a localmaximum shown in Fig. 8, a better initial dictionary generallyhelps ITDL in terms of classification performance.

B. Discriminability of ITDL Dictionaries

We evaluate the discriminative power of ITDL dictionariesover the complete USPS dataset, where we use 7291 imagesfor training and 2007 images for testing, and the ExtendedYaleB face dataset, where we randomly select half of the

TABLE I: Classification rate (%) on the USPS dataset.

Method Accuracy (%)k-NN 94.80SVM-Gauss 95.80SRC [31] 87.23SRC* [31] 96.10SDL-D [18] 96.44SRSC [15] 93.95FDDL [12] 96.31Proposed 98.28

TABLE II: Classification rate (%) on the 15 scenes dataset.

Method Accuracy (%)SRC [31] 70.58SRC* [31] 76.04ScSPM [32] 80.28KSPM [30] 76.73KC [33] 76.67LSPM [32] 65.32Proposed 81.13

images as training and the other half for testing, and finallythe 15-Scenes dataset, where we randomly use 100 images perclass for training and used the remaining data for testing.

For each dataset, we initialize a 512 sized dictionary fromK-SVD and set the sparsity to be 30. Then we perform 30 it-erations of dictionary update and report the peak classificationperformance. Here we adopt a dedicated set of atoms for eachclass and input the concatenated sparse representation into alinear SVM classifier. For the Extended YaleB face dataset,we adopt the same experimental setup in [20], and the resultsfor other compared methods are also taken from [20]. It isnoted that SRC [31] uses training samples as the dictionary.For a fair comparison, we need to constrain the total numberof training samples used in SRC based on the dictionary sizein other compared methods. For completeness, we also includeSRC with all training samples and denote it as SRC*, whichthough is not a fair comparison.

As shown in Table I, Table II, and Table III, our methodoutperforms some of the competitive discriminative dictionary

8

(a) Before update (Acc.= 30.63 %) (b) After 100 updates (Acc.= 42.34 %) (c) Converge after 489 updates (Acc.= 51.35 %)

(d) Before update (Acc.= 73.54 %) (e) After 50 updates (Acc.= 84.45%) (f) Converge after 171 updates (Acc.= 87.75%)

Fig. 3: Information-theoretic dictionary update with global atoms shared over classes. For a better visual representation, sparsity2 is chosen and a randomly selected subset of all samples are shown. The recognition rate associated with (a), (b), and (c) are:30.63%, 42.34% and 51.35%. The recognition rate associated with (d), (e), and (f) are: 73.54%, 84.45% and 87.75%. Notethat the proposed ITDU effectively enhances the discriminability of the set of common atoms (best viewed in color).

TABLE III: Classification rate (%) on the Extended YaleB facedataset.

Method Accuracy (%)SRC [31] 80.50SRC* [31] 97.20D-KSVD [19] 94.10LC-KSVD [20] 95.00K-SVD [10] 93.10LLC [34] 90.70Proposed 95.39

learning algorithms such as SDL-D [18], SRSC [15], D-KSVD[19], LC-KSVD [20], and FDDL [12]. Note that, our methodis flexible enough that it can be applied over any dictionarylearning schemes to enhance the discriminability.

V. CONCLUSION

We have presented an information theoretic approach todictionary learning that seeks a dictionary that is compact,reconstructive and discriminative for the task of image classi-fication. The algorithm consists of dictionary selection and

update stages. In the selection stage, an objective functionis maximized using a greedy procedure to select a set ofcompact, reconstructive and discriminative atoms from aninitial dictionary. In the update stage, a gradient ascent al-gorithm based on the quadratic mutual information is adoptedto enhance the selected dictionary for improved reconstructionand discrimination. Both the proposed dictionary selection andupdate methods can be easily applied for other dictionarylearning schemes.

ACKNOWLEDGMENT

The work was partially supported by a MURI from theOffice of Naval Research under the Grant N00014-10-1-0934.

REFERENCES

[1] R. Rubinstein, A. Bruckstein, and M. Elad, “Dictionaries for sparserepresentation modeling,” Proceedings of the IEEE, vol. 98, no. 6, pp.1045 –1057, Jun. 2010.

[2] J. Wright, Y. Ma, J. Mairal, G. Sapiro, T. Huang, and S. Yan, “Sparserepresentation for computer vision and pattern recognition,” Proceedingsof the IEEE, vol. 98, no. 6, pp. 1031 –1044, June 2010.

9

−0.5 0 0.5 1−0.2

0

0.2

0.4

0.6

0.8

1

1.2

Digit 1Digit 2Digit 3Digit 0

−0.5 0 0.5 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9


0 0.2 0.4 0.6 0.8 1−0.2

−0.1

0

0.1

0.2

0.3

0.4

0.5

0.6


−0.5 0 0.5 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9


(a) Before dictionary update (Acc.= 85.71%)

0 1 2 3 4−4

−3.5

−3

−2.5

−2

−1.5

−1

−0.5


2 3 4 5 6−5.5

−5

−4.5

−4

−3.5

−3

−2.5

−2

−1.5


0.5 1 1.5 2 2.5 3

0.8

1

1.2

1.4

1.6

1.8

2

2.2

2.4

2.6


1 2 3 4 5 6−6

−5.5

−5

−4.5

−4

−3.5

−3

−2.5

−2

−1.5

−1


(b) After 30 update iterations (Acc.= 89.11%)

1 2 3 4 5−5

−4.5

−4

−3.5

−3

−2.5

−2

−1.5


2 3 4 5−5

−4.5

−4

−3.5

−3

−2.5

−2


2 4 6 8 102

3

4

5

6

7

8

9

10


2 4 6 8 10−9

−8

−7

−6

−5

−4

−3

−2


(c) Converge after 57 update iterations (Acc.= 90.47%)

Fig. 4: Information-theoretic dictionary update with dedicated atoms per class. The first four digits in the USPS digit datasetare used. Sparsity 2 is chosen for visualization. In each figure, signals are first represented at once as a linear combination ofthe dedicated atoms for the class colored by red, then sparse coefficients of all signals are plotted in the same 2-D coordinatespace. The proposed ITDU effectively enhances the discriminability of the set of dedicated atoms.

[3] M. Elad, M. Figueiredo, and Y. Ma, “On the role of sparse and redundantrepresentations in image processing,” Proceedings of the IEEE, vol. 98,no. 6, pp. 972 –982, June 2010.

[4] V. M. Patel and R. Chellappa, “Sparse representations, compressivesensing and dictionaries for pattern recognition,” in Asian Conferenceon Pattern Recognition (ACPR), Beijing, China, 2011.

[5] S. Chen, D. Donoho, and M. Saunders, “Atomic decomposition by basispursuit,” SIAM J. Sci. Comp., vol. 20, no. 1, pp. 33–61, 1998.

[6] Y. C. Pati, R. Rezaiifar, and P. S. Krishnaprasad, “Orthogonal matchingpursuit: recursive function approximation with applications to waveletdecomposition,” Proc. 27th Asilomar Conference on Signals, Systemsand Computers, pp. 40–44, 1993.

[7] J. A. Tropp, “Greed is good: Algorithmic results for sparse approxima-tion,” IEEE Trans. Info. Theory, vol. 50, no. 10, pp. 2231–2242, Oct.2004.

[8] B. A. Olshausen and D. J. Field, “Emergence of simple-cell receptivefield properties by learning a sparse code for natural images,” Nature,vol. 381, no. 6583, pp. 607–609, 1996.

[9] V. M. Patel, T. Wu, S. Biswas, P. J. Phillips, and R. Chellappa,“Dictionary-based face recognition under variable lighting and pose,”IEEE Transactions on Information Forensics and Security, vol. 7, no. 3,pp. 954–965, June 2012.

[10] M. Aharon, M. Elad, and A. Bruckstein, “k-SVD: An algorithm fordesigning overcomplete dictionaries for sparse representation,” IEEETrans. on Signal Processing, vol. 54, no. 11, pp. 4311–4322, Nov. 2006.

[11] K. Etemand and R. Chellappa, “Separability-based multiscale basisselection and feature extraction for signal and image classification,”IEEE Trans. on Image Processing, vol. 7, no. 10, pp. 1453–1465, Oct.1998.

[12] M. Yang, X. F. L. Zhang, and D. Zhang, “Fisher discrimination dictio-nary learning for sparse representation,” in Proc. Intl. Conf. on ComputerVision, Barcelona, Spain, 2011.

[13] F. Rodriguez and G. Sapiro, “Sparse representations for image clas-sification: Learning discriminative and reconstructive non-parametricdictionaries,” Tech. Report, University of Minnesota, Dec. 2007.

[14] E. Kokiopoulou and P. Frossard, “Semantic coding by superviseddimensionality reduction,” IEEE Trans. Multimedia, vol. 10, no. 5, pp.806–818, Aug. 2008.

[15] K. Huang and S. Aviyente, “Sparse representation for signal classifica-tion,” in Neural Information Processing Systems, Vancouver, Canada,Dec. 2007.

[16] J. Mairal, F. Bach, and J. Ponce, “Task-driven dictionary learning,” IEEETPAMI, vol. 34, no. 4, pp. 791 –804, April 2012.

[17] J. Mairal, F. Bach, J. Pnce, G. Sapiro, and A. Zisserman, “Discriminative

10

(a) Original images (b) Noisy images (60% missing pixels)

K-svd dictionary

(c) K-SVD dictionary (d) Before update (Acc.= 85.71%) (e) After 30 iterations (Acc.= 89.11%) (f) After convergence (Acc.= 90.47%)

(g) Before update (Acc.=76.87%) (h) After 30 iterations (Acc.= 85.03%) (i) After convergence (Acc.= 85.71%)

Fig. 5: Reconstruction using K-SVD initialized dictionary with information-theoretic dictionary update (dedicated atoms perclass and sparsity 2 is used.). Each column of (c) shows the updated dictionary atoms used in (d), (e) and (f) respectively, andfrom the top to the bottom the two atoms in each row are the dedicated atoms for class ‘1’,‘2’,‘3’ and ‘0’. (d), (e) and (f)show the reconstruction to (a). (g), (h) and (i) show the reconstruction to (b). (b) are images in (a) with 60% missing pixels.Note that ITDU extracts the common internal structure of each class and eliminates the variation within the class, which leadsto more accurate classification.

Before updates 30 iterations After convergence

(a) Random sample dictionary (b) Before update (Acc.= 83.67%) (c) After 30 iterations (Acc.= 89.11%) (d) After convergence (Acc.= 89.11%)

Fig. 6: Reconstruction using random samples initialized dictionary with information-theoretic dictionary update (dedicatedatoms per class and sparsity 2 is used.). Each column of (a) shows the updated dictionary atoms used in (b), (c) and (d)respectively, and from the top to the bottom the two atoms in each row are the dedicated atoms for class ‘1’,‘2’,‘3’ and ‘0’.(b), (c) and (d) show the reconstruction to Fig.5a.

Before updates 30 iterations After convergence

(a) Random noise dictionary (b) Before update (Acc.= 68.02%) (c) After 30 iterations (Acc.= 84.35%) (d) After convergence (Acc.= 89.11%)

Fig. 7: Reconstruction using random noises initialized dictionary with information-theoretic dictionary update (dedicated atomsper class and sparsity 2 is used.). Each column of (a) shows the updated dictionary atoms used in (b), (c) and (d) respectively,and from the top to the bottom the two atoms in each row are the dedicated atoms for class ‘1’,‘2’,‘3’ and ‘0’. (b), (c) and(d) show the reconstruction to Fig.5a.

learned dictionaries for local image analysis,” in IEEE Computer SocietyConf. on Computer Vision and Patt. Recn., Anchorage, 2008.

[18] J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, “Superviseddictionary learning,” in Neural Information Processing Systems, Vancou-ver, Canada, Dec. 2008.

[19] Q. Zhang and B. Li, “Discriminative k-SVD for dictionary learning inface recognition,” in Proc. IEEE Computer Society Conf. on ComputerVision and Patt. Recn., San Francisco, CA, June 2010.

[20] Z. Jiang, Z. Lin, and L. S. Davis, “Learning a discriminative dictionaryfor sparse coding via label consistent k-SVD,” in IEEE Computer SocietyConf. on Computer Vision and Patt. Recn., Colorado Springs, June 2011.

[21] Q. Qiu, Z. Jiang, and R. Chellappa, “Sparse dictionary-based representa-tion and recognition of action attributes,” in Proc. Intl. Conf. ComputerVision, Barcelona, Spain, Nov. 2011.

[22] J. A. Tropp, A. C. Gilbert, and M. J. Strauss, “Algorithms for simulta-

neous sparse approximation. part i: Greedy pursuit,” Signal Processing,vol. 86, pp. 572–588, 2006.

[23] A. Krause, A. Singh, and C. Guestrin, “Near-optimal sensor placementsin Gaussian processes: Theory, efficient algorithms and empirical stud-ies,” JMLR, no. 9, pp. 235–284, 2008.

[24] M. E. Hellman and J. Raviv, “Probability of error, equivocation, and theChernoff bound,” IEEE Trans. on Info. Theory, vol. 16, pp. 368–372,1979.

[25] C. M. Bishop, Pattern Recognition and Machine Learning. Springer,2006.

[26] K. Torkkola, “Feature extraction by non parametric mutual informationmaximization,” JMLR, vol. 3, pp. 1415–1438, Mar. 2003.

[27] J. Kapur, “Measures of information and their applications,” 1994, wiley.[28] A. S. Georghiades, P. N. Belhumeur, and D. J. Kriegman, “From

few to many: Ilumination cone models for face recognition under

11

0 100 200 3000

0.02

0.04

0.06

0.08

0.1

Number of Iterations

∆D


(a) Dictionary atom changes

0 100 200 3000.048

0.05

0.052

0.054

0.056

0.058

0.06

0.062

Number of Iterations

RM

SE


(b) Reconstruction errors

Fig. 8: Convergence of ITDL in Algorithm 3. Initialized using random noises shown in Fig.7, one ITDL dictionary is learnedfor each of ‘1’,‘2’,‘3’ and ‘0’ classes. Let Dk be an ITDL dictionary after the k-th update, and ∆D = ||Dk −Dk−1||2F .

variable lighting and pose,” IEEE Trans. Pattern Analysis and MachineIntelligence, vol. 23, no. 6, pp. 643–660, June 2001.

[29] “USPS handwritten digit database.” in http://www-i6.informatik.rwth-aachen.de/ keysers/usps.html.

[30] S. Lazebnik, C. Schmid, and J. Ponce, “Beyond bags of features: Spatialpyramid matching for recognizing natural scene categories,” in IEEEComputer Society Conf. on Computer Vision and Patt. Recn., New York,NY, vol. 2, 2006, pp. 2169 – 2178.

[31] J. Wright, A. Yang, A. Ganesh, S. Sastry, and Y. Ma, “Robust facerecognition via sparse representation,” IEEE TPAMI, vol. 31, no. 2, pp.210–227, 2009.

[32] J. Yang, K. Yu, Y. Gong, and T. Huang, “Linear spatial pyramidmatching using sparse coding for image classification,” in Proc. IEEEComputer Society Conf. on Computer Vision and Patt. Rec., Miami, FL,June 2009.

[33] J. C. Gemert, J.-M. Geusebroek, C. J. Veenman, and A. W. Smeulders,“Kernel codebooks for scene categorization,” in Proc. European Conf.on Computer Vision, Marseiiles, France, Oct. 2008.

[34] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong, “Locality-constrained linear coding for image classification,” in Proc. IEEE Com-puter Society Conf. on Computer Vision and Patt. Recn., San Francisco,June 2010.

Qiang Qiu received his Bachelor’s degree with firstclass honors in Computer Science in 2001, and hisMaster’s degree in Computer Science in 2002, fromNational University of Singapore. He received hisPh.D. degree in Computer Science in 2012 fromUniversity of Maryland, College Park. During 2002-2007, he was a Senior Research Engineer at Institutefor Infocomm Research, Singapore. He is currentlya Postdoctoral Associate at the Department of Elec-trical and Computer Engineering, Duke University.His research interests include computer vision and

machine learning, specifically on face recognition, human activity recognition,image classification, and sparse representation.

Vishal M. Patel is a member of the researchfaculty at the University of Maryland Institute forAdvanced Computer Studies (UMIACS). He re-ceived the B.S. degrees in Electrical Engineeringand Applied Mathematics (with honors) and theM.S. degree in Applied Mathematics from NorthCarolina State University, Raleigh, NC, in 2004and 2005, respectively. He received his Ph.D. fromthe University of Maryland, College Park, MD,in Electrical Engineering in 2010. He received anORAU postdoctoral fellowship in 2010. His research

interests are in signal processing, computer vision and pattern recognition withapplications to biometrics and imaging. He is a member of Eta Kappa Nu, PiMu Epsilon and Phi Beta Kappa.

Rama Chellappa received the B.E. (Hons.) degreein Electronics and Communication Engineering fromthe University of Madras, India and the M.E. (withDistinction) degree from the Indian Institute of Sci-ence, Bangalore, India. He received the M.S.E.E. andPh.D. Degrees in Electrical Engineering from PurdueUniversity, West Lafayette, IN. During 1981-1991,he was a faculty member in the department of EE-Systems at University of Southern California (USC).Since 1991, he has been a Professor of Electricaland Computer Engineering (ECE) and an affiliate

Professor of Computer Science at University of Maryland (UMD), CollegePark. He is also affiliated with the Center for Automation Research andthe Institute for Advanced Computer Studies (Permanent Member) and isserving as the Chair of the ECE department. In 2005, he was named aMinta Martin Professor of Engineering. His current research interests spanmany areas in image processing, computer vision and pattern recognition.Prof. Chellappa has received several awards including an NSF PresidentialYoung Investigator Award, four IBM Faculty Development Awards, two paperawards and the K.S. Fu Prize from the International Association of PatternRecognition (IAPR). He is a recipient of the Society, Technical Achievementand Meritorious Service Awards from the IEEE Signal Processing Society.He also received the Technical Achievement and Meritorious Service Awardsfrom the IEEE Computer Society. He is a recipient of Excellence in teachingaward from the School of Engineering at USC. At UMD, he receivedcollege and university level recognitions for research, teaching, innovationand mentoring undergraduate students. In 2010, he was recognized as anOutstanding ECE by Purdue University. Prof. Chellappa served as the Editor-in-Chief of IEEE Transactions on Pattern Analysis and Machine Intelligenceand as the General and Technical Program Chair/Co-Chair for several IEEEinternational and national conferences and workshops. He is a Golden CoreMember of the IEEE Computer Society, served as a Distinguished Lecturer ofthe IEEE Signal Processing Society and as the President of IEEE BiometricsCouncil. He is a Fellow of IEEE, IAPR, OSA and AAAS and holds fourpatents.

information-theoretic dictionary learning for image classiﬁcation · 2018-09-04 · datasets...

Documents