cross-modal metric learning for auc optimization · 2017. 11. 7. · submitted to ieee transactions...

The University of Manchester Research

Cross-Modal Metric Learning for AUC Optimization

DOI:10.1109/TNNLS.2017.2769128

Document VersionAccepted author manuscript

Link to publication record in Manchester Research Explorer

Citation for published version (APA):Huo, J., Gao, Y., Shi, Y., & Yin, H. (2018). Cross-Modal Metric Learning for AUC Optimization. IEEE Transactionson NEural Networks and Learning Systems, 29(10), 4844-4856. [8246530].https://doi.org/10.1109/TNNLS.2017.2769128

Published in:IEEE Transactions on NEural Networks and Learning Systems

Citing this paperPlease note that where the full-text provided on Manchester Research Explorer is the Author Accepted Manuscriptor Proof version this may differ from the final Published version. If citing, it is advised that you check and use thepublisher's definitive version.

General rightsCopyright and moral rights for the publications made accessible in the Research Explorer are retained by theauthors and/or other copyright owners and it is a condition of accessing publications that users recognise andabide by the legal requirements associated with these rights.

Takedown policyIf you believe that this document breaches copyright please refer to the University of Manchester’s TakedownProcedures [http://man.ac.uk/04Y6Bo] or contact [email protected] providingrelevant details, so we can investigate your claim.

Download date:18. Dec. 2020

https://doi.org/10.1109/TNNLS.2017.2769128

https://www.research.manchester.ac.uk/portal/en/publications/crossmodal-metric-learning-for-auc-optimization(da83e9a1-77b2-41a4-8fbb-61170a536867).html

/portal/hujun.yin.html

https://www.research.manchester.ac.uk/portal/en/publications/crossmodal-metric-learning-for-auc-optimization(da83e9a1-77b2-41a4-8fbb-61170a536867).html

https://doi.org/10.1109/TNNLS.2017.2769128

SUBMITTED TO IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 1

Cross-Modal Metric Learning for AUCOptimization

Jing Huo, Yang Gao, Member, IEEE, Yinghuan Shi, and Hujun Yin, Senior Member, IEEE

Abstract—Cross-modal metric learning deals with learningdistance functions for cross-modal data matching. The existingmethods mostly focus on minimizing a loss defined on samplepairs. However, the numbers of intra-class and inter-class samplepairs can be highly imbalanced in many applications, andthis can lead to deteriorating or unsatisfactory performances.The Area Under the ROC curve (AUC) is a more meaningfulperformance measure for the imbalanced distribution problem.To tackle the problem as well as to make samples from differentmodalities directly comparable, a cross-modal metric learningmethod is presented by directly maximizing AUC. The methodcan be further extended to focus on optimizing partial AUC(pAUC), which is the AUC between two specific false positiverates. This is particularly useful in certain applications whereonly the performances assessed within predefined false positiveranges are critical. The proposed method is formulated as aLog Determinant (LogDet) regularized semi-definite optimizationproblem. For efficient optimization, a mini-batch proximal pointalgorithm is developed. The algorithm is experimentally verifiedstable with the size of sampled pairs which form a mini-batchat each iteration. Several datasets have been used in evaluation,including three cross-modal datasets on face recognition undervarious scenarios and a single modal dataset, the LabeledFaces in the Wild (LFW). Results demonstrate the effectivenessof the proposed methods and marked improvements overthe existing methods. Specifically, pAUC-optimized cross-modalmetric learning proves to be more competitive for performancemeasures such as Rank-1 and VR@FPR=0.1%.

Index Terms—Cross-modal metric learning, AUC optimization,positive semi-definite optimization, face recognition, multi-modalclassification

I. INTRODUCTION

W ITH the rapid development of data acquisition inincreasing number of modalities, multi-modal data is

becoming a commonality in real-world applications. As aresult, there is an increasing demand for cross-modal datamatching techniques that can determine whether two samplesof different modalities are of the same class. Applicationsinclude cross-modal face recognition (e.g. sketches to photosand near infrared to visible light images), cross-media retrieval(images to text), voices to images and so on. To solve such

This work was supported by the National Science Foundation of China(Grant Nos. 61432008, 61673203), the Young Elite Scientists SponsorshipProgram by CAST (YESS 20160035) and the Collaborative InnovationCenter of Novel Software Technology and Industrialization. J. Huo was alsosupported by a scholarship from the China Scholarship Council as a one-yearvisiting research student at the University of Manchester during 2015-2016.(Corresponding author: Yang Gao.)

J. Huo, Y. Gao and Y. Shi are with the State Key Laboratory for NovelSoftware Technology, Nanjing University, Nanjing 210046, China. (e-mail:[email protected], [email protected], [email protected]).

H. Yin is with the School of Electrical and Electronic Engineering, TheUniversity of Manchester, UK. (e-mail: [email protected]).

matching problems, cross-modal metric learning has beenintensively studied.

Generally, the goals of cross-modal metric learning are oftwo folds. The first is to remove data heterogeneity so thatdata of two modalities become comparable. For this objective,a common subspace based cross-modal metric function isadopted as in this paper. The cross-modal metric function,parameterized by two projection matrices, can be interpreted asfinding a common subspace to remove modality variations sothat distances of different modality are measurable. Previouswork that applied such metric function includes [1] and [2].However, most of these methods have used an alternatingoptimization technique to optimize the two metric parameters,which leads to non-convex optimization. Differently, themetric function in our approaches is written into a morecompact form parameterized by a Positive Semi-Definite(PSD) matrix, so the final optimization problem is convex.

For the second objective, pre-defined must-link (i.e. samelabeled or similar) and cannot-link (i.e. differently labeledor dissimilar) cross-modal distance constraints should besatisfied in the common subspace. However, as the numbersof must-link and cannot-link sample pairs constructed ona training (or given) dataset can be highly imbalancedin many applications. For example, in large scale facerecognition problem, each subject may only have a smallnumber of samples to construct the must-link constraints.However, all the samples of all the other subjects can beused to construct the cannot-link constraints and are ofvery large quantity, leading to highly imbalanced distanceconstraints. For other applications, the cross-modal datasetscan also be essentially unbalanced, such as rumor detection(text/image/video) [3], spam detection (network/text) [4],sentiment analysis (image/text) [5]. In these cases, directlyminimizing the loss of predicting sample pairs can lead tothe loss dominated by the loss of cannot-link constraints.The Area Under the ROC curve (AUC) is a arguablymore meaningful performance measure for such imbalanceddistribution problems [6]. We therefore propose to learncross-modal metrics that directly optimize AUC on pre-definedmust-link and cannot-link sample pairs. In general, formetric learning with must-link and cannot-link sample pairsgiven, AUC measures the probability for a randomly selectedmust-link sample pair to have smaller distance compared withthe distance of a randomly selected cannot-link sample pair.In practice, AUC optimization is written as the sum of all thelosses of separating the distances of a must-link sample pairand a cannot-link sample pair. Besides, for certain applications,partial AUC (pAUC), which is the AUC for a specific range


of false positive rate, is of more importance than the fullAUC. For example, in face recognition, it is often requiredthat a system should have a high true positive rate (TPR)with the false positive rate (FPR) of a very small scale suchas 0.1%. In this case, it is more meaningful to optimizethe pAUC with a FPR range of [0, 0.1]. To adapt to thesesituations, the proposed cross-modal metric learning methodis also extended for optimizing pAUC. The formulation ofAUC and pAUC optimization can be written into a consistentform and optimized in the same fashion.

For the final optimization, as the metric function isparameterized by a PSD matrix, the Log Determinant(LogDet) regularization is adopted to guarantee the metricparameters to remain PSD during the optimization. Besides,as the complexity of the AUC-optimized or pAUC-optimizedmetric learning is proportional to the number of must-linkdistance constraints times the number of cannot-link distanceconstraints, which is quartic of the number of training samples.For efficient optimization, we adopted a combination of theproximal point algorithm [7], [8] and mini-batch stochasticoptimization algorithm. Proximal point algorithm is used forLogDet optimization to guarantee the metric parameter PSD.Mini-batch stochastic optimization is to reduce the complexityof the AUC-optimization or pAUC-optimization. Specifically,during the optimization, both must-link and cannot-link samplepairs are sampled and the optimization is done on thesampled subsets, which form the mini-batches. The algorithmis experimentally proven to be stable with the size of sampledmust-link and cannot-link pairs. Only under the conditionwhen either the number of must-link or cannot-link pairs usedfor optimization at each iteration is extremely small (less than10), the performance may degrade.

The rest of this paper is organized as follows. InSection II, related work is briefly reviewed. Details of theproposed methods together with optimization techniques andimplementation details are given in Section III. In SectionIV, experiments and results are presented together withdiscussions. Finally, the study is concluded in Section V.

II. RELATED WORK

A. Single-modal Metric Learning

Metric learning is to learn metric functions that satisfypairwise must-link/cannot-link distance/similarity constraintsor triplet-based relative distance/similarity constraints [9],[10], [11], [12], [13]. It has been widely studied during thepast decades. According to the metric functions used, metriclearning can be categorized into two categories.

The first is the Mahalanobis metric learning. One drawbackis that it is mainly for data of single modality, making itunsuitable for cross-modal data matching and inapplicable ifsamples of two modalities are of different dimensionality. Xinget al. [14] were the first to study the problem of learningMahalanobis metric functions given pairs of must-link andcannot-link samples. Other representative work includes theInformation Theoretic Metric Learning (ITML) [15], whichlearns metrics subject to both must-link and cannot-linkdistance constraints together with minimizing the divergence

of the learned metric and Euclidean distance metric. Similarto ITML, the proposed method also adopts the LogDetregularization. However, we have used the proximal pointmethod in optimization, while ITML used Bregman projection.In [16], Weinberger et al. proposed a Large Margin NearestNeighbor (LMNN) method. The objective contains two parts,one is to minimize the distances of must-link pairs and theother is to subject to a set of triplet-based relative distanceconstraints by forcing margins between the distances ofmust-link and cannot-link pairs encoded in the triplets. Theproposed method also uses relative distance constraints andmust-link distance constraints for optimization. However, therelative distance constraints are tetrad based. A more detailedcomparison of LMNN and the proposed method is given inSection III. In [17], [18], metrics were optimized for variousperformance measures and AUC was one of them. Differently,the AUC in their methods are defined on ranking results whileours on the must-link and cannot-link constraints. In fact,these existing AUC optimization methods are based on triplets.Besides, our method can be easily extended to optimize partialAUC, and this would not be possible in the previous methods.

The second category is the bilinear similarity metriclearning. Because a bilinear similarity metric does not requirePSD constraints on metric parameters, it is usually moreefficient compared to Mahalanobis metric learning. Earlywork of this kind was also designed for single-modaldata. The Online Algorithm for Scalable Image Similaritylearning (OASIS) was proposed [19]. Bilinear similarityfunction was adopted and an online passive-aggressive basedlearning algorithm was developed for efficient optimization.In the work of [20], Liu et al. proposed to learn low-rankbilinear functions and the algorithm scaled linearly withinput dimensionality in both space and time. The work of[21] addresses the problem of efficiently learning similaritymeasure from high-dimensional sparse data. The similaritymeasure was modeled as a convex combination of rank-onematrices with specific sparsity structures.

B. Cross-modal Metric LearningRecently, there is an increase in the number of studies in

learning metric functions for cross-modal problems. Again,according to the metric function used, the studies can be putinto two categories. The first is the common subspace basedand the second is the similarity metric based.

For the first category, a cross-modal metric learningmethod [1] is proposed to find common subspace in whichboth the pairwise must-link and cannot-link cross-modaldistance constraints are satisfied. Zhou et al. [22] proposedto incorporate both homogeneous local information andheterogeneous constraints into a whole framework to learn across-modal metric, and the local information was adopted byusing an extension of Locally Linear Embedding (LLE). In[23], Quadrianto and Lampert proposed to learn projectionsto preserve neighbourhood information for multi-view data.Wu et al. [2] also proposed to learn a common subspacebased heterogeneous distance metric by optimizing distancesof related cross-modal data and by using two regularizationterms to incorporate prior information.


For the second category, a bilinear cross-modal similaritymetric learning method is proposed [24] and the learnedmetric is optimized to be low rank. Zhen et al. [25] proposeda probabilistic framework for learning pairwise similarities.It was capable of learning from both pairwise and relativeconstraints and suitable for active learning setting.

Our proposed method is also a common subspace based.However, different from the previous methods, which adoptalternating direction methods to optimize the projectionparameters, our method is convex, ensuring that theoptimization does not suffer from local minimum. Besides,the existing methods optimize metric functions to satisfyeither pairwise distance constraints or triplet based relativeconstraints, while the proposed method optimizes the AUCdefined on provided must-link and cannot-link constraints,which are tetrad based.

C. AUC Optimization

AUC optimization has been extensively studied over the pastdecades [6], [26], [27], [28]. Most of the algorithms weredesigned for learning classifiers for classification problems.Joachims [26] proposed to use the structural support vectormachine (svm) to optimize various performance measures(AUC was one of them). In [29], Narasimhan and Agarwalproposed to use the structural svm framework for efficientoptimization of partial AUC. The AUC optimization problemin an online manner was also studied [6], [27], [28]. Kar et al.[30] proposed an online and stochastic learning framework foroptimizing non-decomposable loss function (the loss functionof pAUC was one of them).

In AUC and pAUC optimization, the main problem is thecomputational cost. For AUC-optimized classifier learning, thecomplexity is quadratic of the number of training samples.To ease the computational cost, online learning has beendeveloped [6], [27], [28], [30] and another approach is to usecutting plane optimization [26], [29]. The computational costof AUC-optimized metric learning is higher than classifierlearning which is quartic of the training sample number.Perhaps this is the reason why little metric learning workrelated to AUC optimization has been done. However, asAUC is an important performance measure, it is beneficialto bring AUC optimization to metric learning. The proposedmetric learning method is designed to optimize AUC or pAUC.Further more, we have developed a mini-batch proximal pointalgorithm for efficient optimization, which differs with [17],[18] that adopt cutting plane method.

III. CROSS-MODAL METRIC LEARNING FOR AUC ANDPAUC OPTIMIZATION

A. Notations and Problem Statement

Suppose there are two sets of training samples in twomodalities X = {(xi, l

xi )|i = 1, 2, ..., nx} and Y =

{(yi, lyi )|i = 1, 2, ..., ny}. nx and ny are the numbers of

samples of the two modalities, respectively. xi ∈ Rdx is theith sample of the first modality of dimension dx with labellxi . yi ∈ Rdy is the ith sample of the second modality ofdimension dy with class label lyi . Suppose there are totally c

classes, i.e. lxi ∈ {1, 2, ..., c} and lyi ∈ {1, 2, ..., c}. Besides,define X ∈ Rdx×nx and Y ∈ Rdy×ny the matrices ofsamples of the two modalities. To learn a cross-modal distancefunction, two sets of must-link and cannot-link sample pairsare constructed and in some situations, the two sets arepre-given. Define S = {(i, j)|lxi = lyj } the set of indexof the constructed cross-modal must-link sample pairs andD = {(i, j)|lxi = lyj } the constructed cross-modal cannot-linkindex set. In this work, the aim is to learn a distance functionto measure the distance between two samples of differentmodalities:

DWx,Wy (xi,yj) = ∥WTx xi −WT

y yj∥22, (1)

where ∥·∥2 is the l2 norm. DWx,Wy (xi,yj) is the distancefunction and Wx ∈ Rdx×dc and Wy ∈ Rdy×dc are itsparameters.

The metric function can be interpreted as projecting samplesof two modalities into a common sub-space of dimensionalitydc and then distances are measured in the common subspace.However, the optimization of learning such a metric functionrelies on alternately optimizing Wx and Wy and the problemis non-convex. Therefore, the following transformed metricfunction is adopted instead,

DM(xi,yj) = ∥WTx xi −WT

y yj∥22

= [xTi ,−yT

j ]

[WxW

Tx WxW

Ty

WyWTx WyW

Ty

] [xi

−yj

]= zTijMzij = ⟨zijzTij ,M⟩F ,

(2)

where ⟨A,B⟩F = tr(ATB) is the Frobenius inner product,tr(·) is the trace operator. zij ∈ Rdx+dy is the concatenationof two feature vectors of the form [xT

i ,−yTj ]

T . M ∈ S(dx+dy)+

is the parameter of the newly defined metric, Sd+ denotes thecone of symmetric positive semi-definite (PSD) matrices ofsize d× d.

The PSD constraint is to ensure that the distance definedin Eq. (2) is non-negative. It is obvious to see that learningparameters of Wx and Wy is equivalent to learning parameterM.

B. Objective Functions

Most of the previous metric learning methods try to directlyadjust DM(·, ·) by minimizing loss functions defined ontraining pairs, so that the output of DM(·, ·) for must-linkpair is small and the output for cannot-link pair is large.However, the numbers of pairs in the must-link and cannot-linksets are usually imbalanced, making the losses defined onmust-link and cannot-link sets also imbalanced. AUC is amore efficient performance measure for this situation. AUCon the defined pairwise constraints is therefore optimized. Fora specific metric function, its AUC equals to the proportionof the number of correctly ranked distances of pairs. Simplydenote D as DM. Formally, the AUC value of D on thedefined pair sets S and D is as follows:


00 1

1

False Posi�ve Rate (FPR)

Tru

e P

osi

�ve

Ra

te (

TP

R)

Fig. 1. Illustration of partial AUC between FPRs α and β.

AUCD =1

|S||D|∑

(i,j)∈S

∑(k,l)∈D

(I[D(xi,yj) < D(xk,yl)]

+1

2I[D(xi,yj) = D(xk,yl)]

),

(3)

where I[·] is the indicator function which returns 1 if thestatement is true and 0 otherwise. |S| and |D| are the numbersof index pairs in set S and D respectively.

Besides AUC, partial AUC is also an important performancemeasure for applications that require high true positive ratesand low false positive rates. Partial AUC is the AUC in aspecific false positive rate (FPR) range. An illustration ofpartial AUC is given in Fig. 1. The partial AUC for a metricfunction D on the defined constraints in a false positive rate(FPR) range of [α, β] is defined as:

pAUCD(α, β) =

1

|S||Djα,jβ |∑

(i,j)∈S

∑(k,l)∈Djα,jβ

(I[D(xi,yj) < D(xk,yl)]

+1

2I[D(xi,yj) = D(xk,yl)]

),

(4)

where jα = ⌈|D|α⌉ + 1 and jβ = ⌊|D|β⌋ are two integers1.Note that we assume |D| ≥ 1

β−α , so that jα ≤ jβ . Djα,jβ

is a subset of D. By firstly sorting all the distances of thecannot-link pairs in set D in an ascending order using thedistance function D, then Djα,jβ are the indices of thosepairs that are ranked within the range of jαth position to jβ thposition.

Given the definition of pAUC, the full AUC can be seenas a special case by setting α = 0 and β = 1. Therefore,in the following, only the optimization of the partial AUC isdiscussed. Directly optimize the formulation in Eq. (4) can

1Note that the formulation of pAUC used in this paper does not take theAUC between FPRs [α, jα

|D| ] and [jβ|D| , β] into consideration, as can be seen

in Fig. 1. It is a very small potion for large scale problem. By using thisformulation, the definition of pAUC is consistent with that of full AUC andcan be optimized in a compact form.

lead to an NP-hard problem. We therefore try to optimize anapproximated formulation by replacing the indicator functionwith a convex loss function (hinge loss in this case). Themaximization of the correctly ranked pairs is equivalent to theminimization of incorrectly ranked pairs. Then by replacingthe indicator function with the loss function resulted in thefinal objective function [6], the objective of maximizing AUCor partial AUC is thus changed to minimizing the followingfunction:

L(M) =∑

(i,j)∈S

∑(k,l)∈Djα,jβ

[1 +D(xi,yj)−D(xk,yl)]+|S||Djα,jβ |

, (5)

where [a]+ = max(0, a).Directly minimizing Eq. (5) can lead to both the distances of

must-link pairs and the distances of cannot-link pairs turningunbounded. Therefore regularization terms are needed to avoidsuch situations. In this paper, the following two regularizationterms are used. The first is to force the distances of themust-link cross-modal pair remain small. The second is theLog Determinant (LogDet) divergence function. The LogDetdivergence of two metric parameters M and M0 is defined asLDg(M||M0) = g(M)−g(M0)−⟨∇g(M0),M−M0⟩ withg(M) = −logdet(M). By removing the constant term, it isequivalent to LD(M||M0) = tr(M−1

0 M) − logdet(M). Inthis paper, M0 is set as the identity matrix. The use of suchregularization has several nice properties. The first is that itimplicitly guarantees the metric parameter remains PSD duringoptimization. Besides, by setting M0 as the identity matrix,it regularizes tr(M) = ∥Wx∥2F + ∥Wy∥2F which implicitlycontrols the magnitude of the two projection matrices to avoidoverfitting. The last is that as M is PSD, the nuclear normof M is equal to the trace of the matrix. Thus tr(M) canpromote low-rankness of matrix M during optimization. Thefinal objective of the proposed method is to minimize thefollowing function:

F (M) =∑

(i,j)∈S

∑(k,l)∈Djα,jβ


+ γ∑

(i,j)∈S

D(xi,yj)

|S| + µ[tr(M−1

0 M)− logdet(M)],

(6)

where M ∈ S(dx+dy)+ and M0 is set as the identity matrix.

γ and µ are two regularization parameters which need totune. The first term in Eq. (6) relates to AUC or pAUCmaximization. The second term controls the distances of themust-link pairs staying small. The last term is the LogDetregularization term.

With the formulation of the proposed method at hand, wegive a brief discussion of the differences of the proposedmethod and LMNN. Both of them use relative distanceconstraints (distance of a cannot-link pair be greater than thatof a must-link pair) and pairwise must-link constraints foroptimization. However, the key differences are of two folds. 1)The relative distance constraints used in this paper are tetradbased. While in LMNN, the relative distance constraints are


triplet based. The proposed method is thus more general. Onedrawback of LMNN using triplets is that, as the constructionof triplets relies on data labels, it is not applicable forapplications where only must-link and cannot-link pairs aregiven. 2) LMNN can partly be seen as a special case ofour pAUC formulation under the following setting. SupposeLMNN selects one same class nearest neighbor y for a samplex to construct a must-link pair (x,y). Also suppose that forx, there are n different classes samples. LMNN then selectsk different classes nearest neighbors from all the n differentclasses samples to construct k cannot-link pairs. The objectiveof LMNN is to force the k cannot-link pairs have largerdistances than the must-link pair. This process is equivalentto optimizing the pAUC of the must-link pair and the ncannot-link pairs with the FPR range of [0, kn ].

C. OptimizationBeing introduced in the previous section, the objective of

the proposed AUC and pAUC optimization is to solve thefollowing optimization problem.

minM

F (M)

s.t. M ∈ S(dx+dy)+ .

(7)

Since there is a logdet(·) term in function F (M), the proximalpoint algorithm (PPA) together with mini-batch stochasticoptimization is adopted for efficient optimization. Beforeintroducing the optimization algorithm, we first transform theobjective function to a more compact form without summationterms. The mini-batch PPA optimization problem is thenintroduced, followed with implementation details.

1) Transform the Objective Function: We first transformthe first term in Eq. (6). Define an index matrix J ∈{0, 1}|S|×|Djα,jβ

|. The elements of J are defined as:

Jp(i,j),q(k,l) =

{1 1 +Di,j > Dk,l

0 1 +Di,j ≤ Dk,l, (8)

where (i, j) ∈ S and (k, l) ∈ Djα,jβ . Di,j and Dk,l areshort for DM(xi,yj) and DM(xk,yl) respectively. p(i, j) isdefined as the index of pair (xi,yj) in set S and q(k, l) theindex of pair (xk,yl) in set Djα,jβ . Thus Jp(i,j),q(k,l) is theelement at the p(i, j)th row and q(k, l)th column of matrixJ. It indicates whether the distance of the p(i, j)th must-linkpair plus one is correctly ranked before the distance of theq(k, l)th cannot-link pair using the distance function DM, ifit is incorrectly ranked, the value is 1. Define nj the numberof non-zero elements in J. Then the following transformationholds:∑

(i,j)∈S

∑(k,l)∈Djα,jβ


= ⟨∑

(i,j)∈S

∑(k,l)∈Djα,jβ

Jp(i,j),q(k,l)(zijzTij − zklz

Tkl)

|S||Djα,jβ |,M⟩F

+nj

|S||Djα,jβ |

= ⟨PA,M⟩F +nj

|S||Djα,jβ |,

(9)

where PA is defined as:

PA =∑

(i,j)∈S

∑(k,l)∈Djα,jβ

Jp(i,j),q(k,l)(zijzTij − zklz

Tkl)

|S||Djα,jβ |. (10)

From the above definition, it is apparent that duringoptimization, for different Mt at different time step, the indexmatrix J changes and thus PA also changes. Calculate PA

on the full set of S and Djα,jβ is computationally intensive ifboth the two sets are large. Stochastic optimization [31] hasbeen proved to be an efficient optimization method for manyapplications. For our optimization problem, as both must-linkpairs and cannot-link pairs are needed for optimization, weperform sampling in sets S and D to get two subsets St andDt at the tth time step. Suppose uniform sampling is adopted,the AUC or pAUC defined on sets S and D are assumed to beconsistent with that on St and Dt. At the tth time step, Pt

A

which is defined on St and Dt is calculated and thus becomesmore efficient.

The second term in Eq. (6) is equivalent to the following:

∑(i,j)∈S

D(xi,yj)

|S|=

1

|S|∑

(i,j)∈S

⟨zijzTij ,M⟩F

=1

|S|⟨∑

(i,j)∈S

zijzTij ,M⟩F = ⟨PS ,M⟩F ,

(11)

where PS = 1|S|

∑(i,j)∈S zijz

Tij . Differing with PA, PS does

not change during optimization and can be pre-calculated.Therefore the final optimization problem at the tth time step

is equivalent to:

minM

⟨PtA,M⟩F + γ⟨PS ,M⟩F + µ

[⟨I,M⟩F − logdet(M)

]s.t. M ∈ S(dx+dy)

+ ,(12)

where I is the identity matrix. Irrelevant terms are removedfrom the final objective function.

2) Proximal Point Algorithm: For minimizing the objectivefunction with a current solution Mt at the tth step, theProximal Point Algorithm (PPA) generates the next solutionMt+1 by (approximately) solving a perturbed problem of thefollowing form:

Mt+1 = arg minM∈S(dx+dy)

+

Ft(M) +1

2ηt∥M−Mt∥2F , (13)

where ηt is a parameter and Ft(M) = ⟨PtA,M⟩F +

γ⟨PS ,M⟩F + µ[⟨I,M⟩F − logdet(M)

]. The solution to the

optimization problem in Eq. (13) is:

Mt+1 = ϕ+λt

(Mt − ηt(P

tA + γPS + µI)

), (14)

where λt = ηtµ. The proof of this solution is given in Lemma3. And the whole algorithm is summarized in Algorithm 1.

To proof Lemma 3, a few preliminary information is firstpresented.


Lemma 1. Given a symmetry matrix X ∈ Sn of sizen × n with its eigenvalue decomposition X = UVUT ,where V = diag([v1, v2, ..., vn]) and assume that v1 ≥v2 ≥ ... ≥ vn. Let λ > 0 be given. For the two followingscalar functions ϕ+λ (x) :=

(√x2 + 4λ+ x

)/2 and ϕ−λ (x) :=(√

x2 + 4λ− x)/2 for all x ∈ R. We define their matrix

counterparts as:

X1 = ϕ+λ (X) := Udiag([ϕ+λ (v1), ..., ϕ+λ (vn)])U

T

X2 = ϕ−λ (X) := Udiag([ϕ−λ (v1), ..., ϕ−λ (vn)])U

T ,X ∈ Sn.(15)

Then the following holds X = X1−X2. Both X1 ∈ Sn++ andX2 ∈ Sn++ are positive definite and X1X2 = λI .

Proof. The proof can be easily obtained from the definitionof ϕ+λ and ϕ−λ .

Lemma 2. Given X ∈ Sn and η > 0, then the followingholds:

minY∈Sn++

{1

2η∥Y −X∥2F − µlogdet(Y)

}=

1

2η∥ϕ−λ (X)∥2F − µlogdet(ϕ+λ (X)),

(16)

where λ = ηµ.

Proof. The objective function in Eq. (16) is strictly convexand continuously differentiable. Thus its optimal solution Y∗

is unique and if exists it must satisfy the equation:

X = Y∗ − λ(Y∗)−1 (17)

By Lemma 1, Y∗ = ϕ+λ (X) is a solution to Eq. (17). Byplugging Y∗ = ϕ+λ (X) back into the objective function, onegets the lemma.

Lemma 3. The solution to the optimization problem in Eq.(13) is:

Mt+1 = ϕ+λt

(Mt − ηt(P

tA + γPS + µI)

), (18)

where λt = ηtµ.

Proof. As the optimization problem in Eq. (13) is equivalentto the following:

minM∈S(dx+dy)

+

1

2ηt∥M−Mt+ηt(P

tA+γPS+µI)∥2F−µlogdet(M),

(19)By Lemma 2, the solution to the above optimization problemis M∗ = ϕ+λt

(Mt−ηt(Pt

A+γPS+µI)), with λt = ηtµ.

3) Implementation Details: To further improve theefficiency of updating Pt

A, the following techniques are used.If mini-batch sampling is not adopted, then the update of Pt

A

can be done in the following form:

PtA =

∑(i,j)∈S

∑(k,l)∈Dt

jα,jβ

Jtp(i,j),q(k,l)(zijz

Tij − zklz

Tkl)

|S||Djα,jβ |

=∑

(i,j)∈S

ψt(zij).

(20)

We use ψt(zij) to denote the feature map of a must-link pair(xi,yj) at the time step of t. As ψt(zij) can be decomposedinto the following form:

ψt(zij) =

[XAijX

T XBijYT

YBTijX

T YCijYT

], (21)

where Aij ∈ Rnx×nx , Bij ∈ Rnx×ny and Cij ∈ Rny×ny arethree matrices of the following form:

Aij =∑

(k,l)∈Dtjα,jβ

Jtp(i,j),q(k,l)(eie

Ti − eke

Tk )

|S||Dtjα,jβ

|

Bij =∑

(k,l)∈Dtjα,jβ

Jtp(i,j),q(k,l)(−eie

Tj + eke

Tl )

|S||Dtjα,jβ

|

Cij =∑

(k,l)∈Dtjα,jβ

Jtp(i,j),q(k,l)(ej e

Tj − ele

Tl )

|S||Dtjα,jβ

|

(22)

where ei ∈ Rnx is the standard basis vector with the ithelement equal to 1. ei ∈ Rny is also the standard basisvector with the ith element equal to 1. By using the abovedecomposition and firstly computing all the Aij , Bij and Cij ,the matrix multiplication number in Eq. (20) can be reducedfrom |S||Djα,jβ | to 6 plus a matrix transpose operation.

The calculation of the three matrices can be done efficientlyin the following way. Take the calculation of Aij as anexample. Initialize Aij as an all zero matrix. By firstlycalculate all the distances of pairs in set Djα,jβ and thencompare those distances with 1+Dij . The ith diagonal elementof Aij is set to the number of cannot-link pairs that havesmaller distances than 1 + Dij divide |S||Djα,jβ |. For thecannot-link pair (xk,yl) that triggers the hinge loss, the kthdiagonal element of Aij subtract a number of one divided by|S||Djα,jβ |.

Therefore, the complexity of updating PtA mainly consists

of the following four parts. 1) Compute the distances ofpairs in Dt

jα,jβand S which is of O(|S| + |Dt

jα,jβ|). 2)

Compare the distances of must-link pairs plus 1 with thedistances of cannot-link pairs where the compare operationis of O(|S| × |Dt

jα,jβ|). 3) Set the values of all the Aij , Bij

and Cij which is of complexity O(|S|×|Dtjα,jβ

|). 4) 6 matrixmultiplications. Therefore by combining the above Pt

A updateprocedure with mini-batch sampling described in the previoussection, the complexity of all the four steps can be reduced.Specifically, as the sampling is done on pairs, not all samplesin sets X and Y are used. Therefore the complexity of the 6matrix multiplications is also reduced.

An illustration of the full optimization algorithm is givenin Algorithm 1, where the initialization of M0 is done byfirst performing principal component analysis (PCA) on dataof two modality to get to projection matrices Wx0 and Wy0.

Then M0 is set to[

Wx0WTx0 Wx0W

Ty0

Wy0WTx0 Wy0W

Ty0

]. ηt is updated

in a form of ηt+1 = min(ηt × ρ, τ), where τ is the maximumvalue of ηt to set and ρ is set to a value slightly greater than1.


Algorithm 1 Cross-modal Metric Learning for AUC/pAUCOptimization by Mini-batch Proximal Point AlgorithmInput:

Two sets of constructed (or pre-given) indices S and Dand training sample sets X and YRegularization parameters γ > 0 and µ > 0Parameter η0 > 0Two false positive rates α and βSampling ratio s1 on set S and ratio s2 on set DPrecalculate PS

Initialize M0

Output:Metric parameter Mt ∈ S(dx+dy)

+

1: Initialize t = 02: while not converged do3: Perform uniform sampling according to sampling ratios

s1 and s2 on sets S and D to get St and Dt, computeDt

jα,jβ

4: Calculate PtA in Eq. (20) on sets St and Dt

jα,jβ

5: Update Mt+1 = ϕ+λt

(Mt−ηt(Pt

A+γPS+µI)), where

λt = ηtµ6: Update ηt+1

7: t = t+ 18: end while

IV. EXPERIMENTAL RESULTS

The proposed method has been evaluated on several datasetsfor cross-modal matching. The first three are cross-modalface datasets including matching sketches to photos, matchingnear-infrared (NIR) to visible light (VIS) face images andmatching low resolution to high resolution face images. Asthe proposed method is also applicable for single modality,the Labeled Faces in the Wild (LFW) [32], [33] face imagedataset was also adopted in testing.

A. Dataset Information and Evaluation Protocols

CUHK Face Sketch FERET Dataset (CUFSF) [34]: TheCUFSF dataset was used for photo to sketch face recognition.It includes 1,194 persons from the FERET dataset [35]. Foreach person, there are one photo and one sketch drawn byan artist after viewing the photo. To evaluate on this dataset,we randomly split the dataset into two parts for ten times andeach time the first part was used for training and the other partfor testing. The first part contained images of 700 persons andthe second part contained images of 494 persons. For facefeature extraction, after face alignment and image filtering, aface region of 160×160 was cropped out. The 160×160 faceimages were also down sampled to size 96× 96 and 32× 32.Then SIFT features [36] were extracted on the images of threescales in a fixed cell size of 32× 32 with a separation of 16pixels. All the extracted features were concatenated to serve asthe final features of a face image. As this feature vector was ofvery high dimension, 13,696 (128×81+128×25+128), theextracted face features were reduced by PCA before applyingmetric learning. Results with various reduced dimensions werepresented.

1 5 10 20 30 40 50 60 70 80 90 100

All(700

)

Number of Sampled Must-link Constraints

0.4

0.5

0.6

0.7

0.8

0.9

1

Ran

k-1\

1-E

ER

\VR

\AU

C

Rank-11-EERVR@FPR=0.1%AUC

(a) CMLAUC

1 5 10 20 30 40 50 60 70 80 90 100

All(489

300)

Number of Sampled Cannot-link Constraints

0.4

0.6

0.8

1

Ran

k-1\

1-E

ER

\VR

\AU

C


(b) CMLAUC

1 5 10 20 30 40 50 60 70 80 90 100

All(700

)


0.4

0.5

0.6

0.7

0.8

0.9

1

Ran

k-1\

1-E

ER

\VR

\AU

C


(c) CMLpAUC

1050100

200

300

400

500

600

700

800

900

1000

All(489

300)


0.4

0.6

0.8

1

Ran

k-1\

1-E

ER

\VR

\AU

C Rank-11-EERVR@FPR=0.1%AUC

(d) CMLpAUC

Fig. 2. Results of CMLAUC and CMLpAUC with different number of sampledmust-link and cannot-link constraints on CUFSF dataset.

CASIA NIR-VIS 2.0 Dataset [37]: The CASIA NIR-VIS2.0 dataset was used for evaluating the VIS-NIR facerecognition. It contains 725 subjects. We followed the sameevaluation protocol on this dataset by [37]. The dataset wasdivided into two views. View 1 was used for parameter tuningand view 2 for testing. On this dataset, the feature extractionstrategy was the same as that on CUFSF dataset.

NJU-ID Dataset2: NJU-ID dataset contains images of 256persons. For each person, there are one card image and fiftyimages collected from a high resolution camera. The ID cardimage is of resolution 102 × 126 and the camera image isof resolution 640 × 480. To evaluate on this dataset, werandomly selected one of the fifty images for each person andthen divided the dataset into 10 folds according to identityinformation. The 10 folds were fully independent withoutoverlap. The same feature extraction method on CUFSFdataset was also adopted.

Labeled Faces in the Wild (LFW) Dataset: LFW datasetcontains 13,233 face images of 5,749 people. We followed therestricted setting of this dataset for testing. For comparisonwith related methods on this dataset, we adopted the featuresprovided in [38] for experiments. They were SIFT featuresextracted at 9 fixed points of three scales. Before testing ourmethods, the features were also reduced to a lower dimensionusing PCA.

Besides the SIFT features, on all four datasets, VGG-Face[39] was also used for extracting deep features for comparison.Face alignment method was adjusted to be consistent withthe one used in [39]. The resulting face feature was of 4,096dimensions. PCA was also applied before testing.

On the three cross-modal datasets, we used evaluationprotocols of AUC, 1-EER, VR@FPR=0.1% and Rank-1. EERdenotes Equal Error Rate, which is the rate where false positiverate equals to false negative rate. 1-EER can be obtained

2Available from: http://cs.nju.edu.cn/rl/Data.html


by plotting a straight line to connect (0, 1) and (1, 0), thetrue positive rate of the intersection point of this line withROC curve equals to 1-EER. It measures the degree of theROC curve approaching the point (0,1) with false positive rate0 and true positive rate 1. The third performance measure,VR@FPR=0.1%, is the verification rate at FPR=0.1%. Itequals to the true positive rate at FPR=0.1%. The firstthree performance measures are directly related to AUCoptimization. Rank-1 recognition rate is also provided as it is acommonly used performance measure in face recognition. Onthe LFW dataset, to compare with the state-of-the-art methods,accuracy was adopted as required in [32].

B. Parameter Settings

The proposed cross-modal metric learning for AUCoptimization (denoted as CMLAUC) and pAUC optimizationwith FPRs of range [0, 0.1] (CMLpAUC) were tested on all thedatasets. The gradient step related parameter η0 was set to0.05, ρ to 1.01 and τ to 0.5 on the three cross-modal datasets.The setting of η0 was 0.05, ρ 1.01 and τ 0.3 on the LFW.γ was firstly tuned by fixing µ to a small value 10−7 and

selected from [0.1, 0.5, 1, 1.5, 2, 3, 4, 5]. Then γ was fixed andµ was selected from [10−3, 10−4, 10−5, 10−6, 10−7]. Whiletuning these parameters, the sampling ratio s1 on set S andsampling ratio s2 on set D were both set to 1 on CUFSF,NJU-ID and LFW. On CASIA, s1 was set to 0.1 and s2 to0.01.

To verify the influence of the numbers of sampled must-linkand cannot-link constraints, we varied these numbers ateach iteration on the CUFSF and CASIA NIR-VIS 2.0datasets. Results on the CUFSF dataset are given in Fig. 2.The data dimension was set to 400; the influence of datadimension will be illustrated later. For AUC, the numbersof sampled must-link and cannot-link constraints wererespectively varied among [1, 2, ..., 5, 10, 20, ..., 100, 700] and[1, 2, ..., 5, 10, 20, ..., 100, 489300]. When testing the numberof must-link constraints, the number of cannot-link constraintswas set to 20. The number of must-link constraints was setto 10 when testing the number of cannot-link constraints. Ascan be seen, if the sampled number was larger than 10, theresults were quite stable for different numbers of both sampledmust-link and cannot-link constraints. For pAUC optimization,the number of sampled must-link was the same with AUCoptimization. The number of sampled cannot-link constraintswas varied among [10, 20, ..., 50, 100, 200, ..., 1000, 489300].As for pAUC with FPRs of [0, 0.1], at each iteration, onlythe top 0-0.1 cannot-link constraints were used for training.Therefore while sampling 10 cannot-link constraints, only thetop 1 was used. Similar results have been observed for varyingthe number of sampled must-link constraints. But for numberof sampled cannot-link constraints, the results were stable ifthe number of sampled cannot-link constraints was greaterthan 100 where the number of actually used constraints wasstill 10.

Since the CUFSF dataset is relatively small, we have alsotested on CASIA NIR-VIS 2.0 dataset. As can be seen inFig. 3, the results were similar to that on CUFSF dataset.

1 5 10 20 30 40 50 60 70 80 90 100

10%

(442

1)

All(442

11)


0.4

0.6

0.8

1

Ran

k-1\

1-E

ER

\VR

\AU

C


(a) CMLAUC

1 510 20 30 40 50 60 70 80 90 100

1%(1

5471

8)

10%

(154

7185

)

All(154

7185

0)


0.2

0.4

0.6

0.8

1

Ran

k-1\

1-E

ER

\VR

\AU

C


(b) CMLAUC

1 5 10 20 30 40 50 60 70 80 90 100

10%

(442

1)

All(442

11)


0.4

0.6

0.8

1

Ran

k-1\

1-E

ER

\VR

\AU

C


(c) CMLpAUC

1050100

200

300

400

500

600

700

800

90010

00

1%(1

5471

8)

10%

(154

7185

)

All(154

7185

0)


0.2

0.4

0.6

0.8

1

Ran

k-1\

1-E

ER

\VR

\AU

C


(d) CMLpAUC

Fig. 3. Results of CMLAUC and CMLpAUC with different number of sampledmust-link and cannot-link constraints on CASIA NIR-VIS 2.0 dataset.

100 200 300 400 500 600 700 800

Number of Dimension

0.4

0.5

0.6

0.7

0.8

0.9

1R

ank-

1\1-

EE

R\V

R\A

UC Rank-1

1-EERVR@FPR=0.1%AUC

(a) CMLAUC

100 200 300 400 500 600 700 800

Number of Dimension

0.4

0.5

0.6

0.7

0.8

0.9

1

Ran

k-1\

1-E

ER

\VR

\AU


(b) CMLpAUC

Fig. 4. Results of CMLAUC and CMLpAUC with different number ofdimensions on CUFSF dataset.

For both AUC and pAUC optimization, when the samplednumber of must-link constraints was greater than 10, theresults were stable for all the performance measures. If thesampled numbers of cannot-link constraints were greater than10 for AUC and 100 for pAUC, the results of AUC and pAUCwere stable.

C. Results on CUFSF Dataset

1) Influence of Dimension: We first present results ofthe influence of the dimension on CUFSF dataset. Thedimensions tested on this dataset were [100, 200, ..., 800].For this experiment, the sampling ratios for both must-linkand cannot-link constraints were set to 1. Using a smallersampling ratio would not influence much the results as it hasbeen shown in the previous part. All the four performancemeasures were evaluated. Results are provided in Fig. 4.As can be seen, for both AUC and pAUC optimization, allfour performance measures improved with the increase indimension. Specifically, the improvements tended to be smallwhen the dimension reached 800. Thus for both CMLAUC andCMLpAUC, we set the dimension to 800 for reporting results.

2) Comparison with Single Modal Methods: Table Ipresents the results of the proposed methods on CUFSF


TABLE ICOMPARISON WITH SINGLE MODAL METHODS ON CUFSF DATASET, VR

DENOTES THE VERIFICATION RATE AT FPR=0.1%

Methods 1-EER(%) VR(%) AUC Rank-1(%)PCA 85.5± 0.7 25.9± 4.8 0.936 35.0± 2.3KDA 94.5± 0.4 50.2± 6.1 0.987 61.0± 2.6LMNN 93.3± 0.5 34.7± 6.8 0.981 55.0± 1.4NCA 95.1± 0.4 42.6± 2.4 0.989 59.1± 2.0ITML 91.4± 0.8 34.8± 3.4 0.974 41.4± 2.0LDML 94.3± 0.4 42.9± 5.7 0.986 59.8± 1.7MahaAUC 95.8± 0.4 55.6± 2.8 0.991 67.4± 1.7MahapAUC 96.0± 0.5 58.3± 4.5 0.992 71.7± 1.5PCA-C 93.6± 0.3 37.2± 8.1 0.983 48.7± 1.8KDA-C 95.5± 0.2 37.4± 10.2 0.990 57.9± 1.3LMNN-C 94.2± 1.9 33.1± 9.3 0.985 54.2± 7.5NCA-C 94.6± 0.5 46.5± 1.5 0.988 52.4± 1.4ITML-C 95.3± 0.3 45.1± 5.2 0.990 55.1± 1.7LDML-C 93.1± 0.7 41.9± 3.8 0.981 47.6± 4.1CMLAUC 96.3± 0.4 58.0± 5.4 0.994 73.6± 1.3CMLpAUC 96.5± 0.3 59.7± 6.8 0.995 75.5± 1.5

dataset and comparisons with single modal methods. Thesix compared methods are, PCA [40], Kernel DiscriminantAnalysis (KDA) [41], LMNN [16] and NeighbourhoodComponents Analysis (NCA) [42], ITML [15], and LogisticDiscriminant-based Metric Learning (LDML) [38].

Since the proposed methods have two terms that mayinfluence the final results: the first is the cross-modal metric toremove modality variations, and the second is AUC or pAUCoptimization to learn discriminant metrics. The six methodswere compared in two settings. In the first setting, originalfeatures were used. Results of the compared methods are inthe first six rows of Table I. In the second setting, featuresprocessed with Canonical Correlation Analysis (CCA) [43]were used (denoted as the method with a suffix of ’-C’ in thetable). Features under this setting should have less modalityvariability compared with the first setting. The proposedmethods CMLAUC and CMLpAUC were using original features.Besides, we have also replaced the cross-modal metric in ourmethods with Mahalanobis metric for comparison, giving intwo results of AUC and pAUC (in FPRs of range [0, 0.1])optimization of Mahalanobis metric, denoted as MahaAUC andMahapAUC.

As can be seen in the first eight rows of the table, MahaAUCand MahapAUC outperformed almost all of the single modalbased methods. Though KDA was comparable with them withrespect to 1-EER and AUC. This illustrates the superiorityof AUC and pAUC optimization. The second observation isthat results of CMLAUC and CMLpAUC were better than allthe compared methods under both settings, with CMLpAUCthe best. Compared with MahaAUC and MahapAUC, CMLAUCand CMLpAUC should have better ability to remove modalityvariations, as demonstrated by the results of CMLAUC andCMLpAUC. Besides, compared with the methods with CCApreprocessing original features, the proposed methods are endto end with both the abilities to remove modality variationsand to learn discriminant metric. In this case, the superiorityis clear. Lastly, the results of pAUC optimization were betterthan that of AUC optimization for both Mahalanobis and

TABLE IICOMPARISON WITH MULTI-MODAL METHODS ON CUFSF DATASET, VR


Methods 1-EER(%) VR(%) AUC Rank-1(%)CSR 95.1± 0.3 51.9± 7.5 0.989 63.7± 2.0KCSR 96.0± 0.3 39.5± 13.2 0.991 66.8± 1.8CCA 93.6± 0.3 37.2± 8.1 0.983 48.7± 1.8KCCA 95.2± 0.3 41.3± 9.6 0.990 57.4± 1.5CDFE 91.8± 0.7 15.9± 4.8 0.974 48.6± 1.4MvDA 90.7± 0.7 27.7± 3.1 0.967 36.0± 1.7CMML 94.1± 0.5 43.7± 6.7 0.986 61.2± 1.9HMLCR 95.7± 0.3 51.0± 10.9 0.992 68.7± 1.8CMLAUC 96.3± 0.4 58.0± 5.4 0.994 73.6± 1.3CMLpAUC 96.5± 0.3 59.7± 6.8 0.995 75.5± 1.5

TABLE IIICOMPARISON WITH MULTI-MODAL METHODS ON CUFSF DATASET WITHDEEP FEATURES, VR DENOTES THE VERIFICATION RATE AT FPR=0.1%

Methods 1-EER(%) VR(%) AUC Rank-1(%)PCA 93.8± 0.2 46.0± 2.1 0.985 54.3± 2.3CCA 97.4± 0.3 72.9± 1.6 0.997 76.8± 2.1CSR 97.1± 0.3 63.0± 1.7 0.996 75.5± 2.0KCSR 97.4± 0.3 62.5± 2.3 0.997 73.7± 1.8HMLCR 97.2± 0.2 71.9± 2.1 0.997 77.8± 2.4CMLAUC 98.1± 0.2 73.2± 1.9 0.998 79.5± 2.0CMLpAUC 98.1± 0.1 74.7± 2.4 0.998 80.6± 1.6

cross-modal metrics. This is mainly because that the partialAUC of false positive range [0,0.1] is optimized. Comparedwith AUC optimization, pAUC optimizes with the top 10%most inseparable cannot-link constraints. There may be someredundancy in all the cannot-link constraints used in AUCoptimization. Therefore, pAUC optimization can outperformAUC optimizaiton.

3) Comparison with Multi-Modal Methods: Comparisonwith multi-modal methods are also given in Table II.Compared methods include Coupled Spectral Regression(CSR) [44], Kernel Coupled Spectral Regression (KCSR)[44], Canonical Correlation Analysis (CCA) [43], KernelCanonical Correlation Analysis (KCCA) [45], CommonDiscriminant Feature Extraction (CDFE) [46] and Multi-viewDiscriminant Analysis(MvDA) [47]. As can be seen, whencompared with the best of these compared methods (CSR),CMLAUC improved over CSR significantly, specifically withVR@FPR=0.1% and Rank-1. Rank-1 is not directly relatedto AUC optimization, however, as can be seen, both CMLAUCand CMLpAUC achieved good Rank-1 rates. This illustrates thatthe proposed methods can not only perform well for AUCrelated performance measures, but also for other performancemeasures.

4) Results of Deep Features: Besides SIFT feature, wehave also tested the proposed methods on deep features(VGG-Face). Results are summarized in Table III. With betterfeatures, performance on this dataset are improved to a largeextent compared with SIFT features. The proposed methodsstill improved over the compared methods.


200 400 600 800 1000 1200 1400 1600

Number of Dimension

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Ran

k-1\

1-E

ER

\VR

\AU


(a) CMLAUC

200 400 600 800 1000 1200 1400 1600

Number of Dimension

0.4

0.5

0.6

0.7

0.8

0.9

1

Ran

k-1\

1-E

ER

\VR

\AU

C


(b) CMLpAUC

Fig. 5. Results of CMLAUC and CMLpAUC with different number ofdimensions on CASIA NIR-VIS 2.0 dataset.

TABLE IVCOMPARISON WITH SINGLE MODAL METHODS ON CASIA NIR-VIS 2.0

DATASET, VR DENOTES THE VERIFICATION RATE AT FPR=0.1%

Methods 1-EER(%) VR(%) AUC Rank-1(%)PCA 81.0± 0.8 15.9± 2.1 0.889 28.4± 1.5KDA 93.5± 0.4 44.5± 5.3 0.981 63.7± 1.4LMNN 89.0± 1.1 29.1± 3.2 0.957 43.8± 1.6ITML 83.9± 0.9 19.4± 2.2 0.921 30.3± 2.5DML-eig 87.9± 0.6 28.8± 1.7 0.950 40.9± 1.4MahaAUC 93.1± 0.2 42.7± 7.1 0.980 72.0± 1.1MahapAUC 93.3± 0.3 57.1± 3.3 0.981 72.4± 1.0PCA-C 88.6± 0.4 29.0± 2.6 0.954 41.8± 1.4KDA-C 92.8± 0.4 34.9± 5.4 0.978 55.8± 1.3LMNN-C 89.3± 0.3 29.4± 3.3 0.959 44.1± 1.5ITML-C 90.5± 0.5 33.8± 2.4 0.967 45.1± 1.2DML-C 87.1± 1.2 24.2± 3.8 0.944 33.4± 4.7CMLAUC 93.9± 0.4 55.0± 4.7 0.984 74.2± 0.8CMLpAUC 94.1± 0.3 58.3± 3.8 0.985 74.9± 0.9

D. Results on CASIA NIR-VIS 2.0 Dataset

1) Influence of Dimension: The influence of the dimensionon CASIA NIR-VIS 2.0 dataset is shown in Fig. 5. Since thereare more training samples on CASIA NIR-VIS 2.0 dataset thanCUFSF, the reduced dimension on this dataset can be higherthan that on CUFSF. The tested dimensions on this datasetwere [200, 400, ..., 1600]. As can be seen, all the performancemeasures increased with the dimension. The results almostconverged when the dimension number reached 1,600. On thisdataset, for final results reporting, the dimension used was1,600.

2) Comparison with Single Modal Methods: Results ofsingle modal methods on CASIA NIR-VIS 2.0 dataset aregiven in Table IV. The results of CMLpAUC were betterthan those of CMLAUC. Besides, MahapAUC was betterthan MahaAUC. In fact, for 1-EER, AUC and Rank-1, thedifferences of AUC and pAUC optimization were small.However, for VR@FPR=0.1%, the differences were relativelylarge. This is mainly because CMLpAUC and MahapAUCoptimizes the left most of the ROC curve, leading to anadvantage over CMLAUC and MahaAUC with respect to theVR@FPR=0.1% performance measure. Besides, comparedwith the state-of-the-art single modal methods under twosettings, the four methods (MahaAUC, MahapAUC, CMLAUC,CMLpAUC) under the proposed framework achieved verycompetitive results.

TABLE VCOMPARISON WITH MULTI-MODAL METHODS ON CASIA NIR-VIS 2.0

DATASET, VR DENOTES THE VERIFICATION RATE AT FPR=0.1%


TABLE VIRESULTS OF DEEP FEATURES ON CASIA NIR-VIS 2.0 DATASET, VR


Methods 1-EER(%) VR(%) AUC Rank-1(%)PCA 96.4± 0.2 76.0± 1.7 0.995 82.1± 1.6CSR 98.3± 0.2 88.1± 1.1 0.998 91.0± 1.2KCSR 98.3± 0.2 89.5± 0.9 0.998 92.4± 0.8CCA 98.1± 0.2 88.6± 1.1 0.998 91.5± 1.2HMLCR 97.2± 0.3 78.6± 3.2 0.996 88.2± 0.6CMLAUC 98.0± 0.1 86.8± 1.0 0.998 91.0± 1.1CMLpAUC 97.9± 0.1 86.7± 0.9 0.998 91.0± 1.1

3) Comparison with Multi-Modal Methods: In Table V,comparisons with multi-modal methods are given. From thetable, CMLAUC and CMLpAUC achieved AUC values of 0.984and 0.985 respectively. The 1-EER for the two methods were93.9 ± 0.4% and 94.1 ± 0.3%. For these two performancemeasures, only KCSR was comparable with our methods. Forthe Rank-1 performance measure, which is not related to AUCoptimization, it is clear our methods also have an advantageover the compared methods. CMLpAUC at VR@FPR=0.1%was the best among all the methods, clearly demonstratingthe effectiveness of pAUC optimization.

4) Results of Deep Features: Table VI provides results ofdeep features on CASIA NIR-VIS 2.0. With deep features,CSR, KCSR, CCA, CMLAUC and CMLpAUC all achieved thebest AUC. KCSR was the best for the three other performancemeasures. The results of the proposed methods are quitecomparable with the results of KCSR. Comparing Table VIand Table V, deep features improved on the SIFT featuressignificantly.

E. Results on NJU-ID Dataset

1) Influence of Dimension: As the number of trainingsamples on NJU-ID dataset is relatively small, the maximumreduced dimension is 461. On this dataset, the dimensionnumber was varied between [50,100,200,300,400,461].Corresponding results are prsented in Fig. 6. On this dataset,the two sample ratios were both set to 1. With varingdimensions, the results of all the four adopted performancemeasures tended to increase with the dimension. Anotherobservation is that CMLAUC appeared more stable thanCMLpAUC. This is mainly due to that the number ofcannot-link constraints used by CMLAUC was larger than


100 200 300 400

Number of Dimension

0

0.2

0.4

0.6

0.8

Ran

k-1\

1-E

ER

\VR

\AU

C


(a) CMLAUC

100 200 300 400

Number of Dimension

0

0.2

0.4

0.6

0.8

Ran

k-1\

1-E

ER

\VR

\AU

C


(b) CMLpAUC

Fig. 6. Results of CMLAUC and CMLpAUC with different number ofdimensions on NJU-ID dataset.

TABLE VIICOMPARISON WITH SINGLE MODAL METHODS ON NJU-ID DATASET, VR


Methods 1-EER(%) VR(%) AUC Rank-1(%)PCA 64.9± 5.3 3.1± 3.1 0.701 23.0± 7.6KDA 71.9± 6.9 3.5± 4.8 0.781 27.8± 9.0LMNN 70.3± 6.3 3.9± 4.2 0.770 23.8± 5.0ITML 70.7± 3.9 3.1± 4.1 0.766 26.6± 4.8NCA 65.6± 3.8 0.4± 1.2 0.708 21.1± 5.7LDML 73.1± 4.9 6.6± 5.8 0.799 29.7± 8.5MahaAUC 73.5± 5.7 7.8± 6.6 0.815 32.9± 9.6MahapAUC 72.7± 6.0 8.2± 5.6 0.812 33.6± 10.5KDA-C 68.0± 6.0 3.9± 3.2 0.730 20.0± 8.2LMNN-C 69.6± 4.9 2.8± 4.2 0.749 21.5± 6.9ITML-C 68.0± 2.6 1.9± 2.7 0.724 14.5± 4.7NCA-C 68.8± 2.3 3.9± 3.7 0.749 21.9± 7.8LDML-C 69.2± 5.1 4.7± 3.6 0.754 24.7± 9.6CMLAUC 73.1± 4.2 8.6± 6.3 0.805 30.9± 11.8CMLpAUC 72.3± 4.2 8.6± 6.3 0.805 30.1± 9.4

that of CMLpAUC and this dataset is relatively small. ThusCMLpAUC could not use sufficient information for training.On this dataset, for final results reporting, we used thedimension of 461.

2) Comparison with Single Modal Methods: Results ofsingle modal methods on NJU-ID dataset are given inTable VII. As this dataset is relatively small, the resultson this dataset are generally worse than those on CUFSFand CASIA NIR-VIS 2.0. On this dataset, MahaAUC andMahapAUC were slightly better than CMLAUC and CMLpAUC.This may be because this dataset is smaller, cross-modal metricwith more parameters compared with Mahalanobis metric isprone to overfitting. However, MahaAUC, MahapAUC, CMLAUC,CMLpAUC still outperformed the compared methods as shownin Table VII.

3) Comparison with Multi-Modal Methods: Results ofcomparison with multi-modal methods on NJU-ID are given inTable VIII. Among all the compared methods, only CSR wascomparable with CMLpAUC. As the RID card face verificationis a real-world application, which requires low FPR rate andhigh TPR rate, the VR@FPR=0.1% performance measure onthis dataset is low, indicating that there are still rooms forimprovement. We further give results of deep features on thisdataset.

4) Results of Deep Features: Results of Deep Featureson NJU-ID are summarized in Table IX. The best resultswere achieved by CMLpAUC, with CSR and HMLCR achieved

TABLE VIIICOMPARISON WITH MULTI-MODAL METHODS ON NJU-ID DATASET, VR



TABLE IXRESULTS OF DEEP FEATURES ON NJU-ID DATASET, VR DENOTES THE

VERIFICATION RATE AT FPR=0.1%

Methods 1-EER(%) VR(%) AUC Rank-1(%)PCA 90.3± 4.5 27.3± 15.7 0.949 65.6± 12.4CCA 93.0± 4.0 27.8± 7.5 0.971 71.4± 10.1CSR 94.9± 3.2 56.2± 9.9 0.985 87.5± 7.2KCSR 91.0± 2.7 29.8± 18.9 0.967 73.0± 9.2HMLCR 94.9± 1.9 53.1± 11.8 0.985 87.9± 6.0CMLAUC 94.5± 2.7 50.5± 14.6 0.984 87.1± 4.2CMLpAUC 95.3± 2.5 58.7± 13.0 0.986 90.2± 5.0

comparable results. Compared with the results of SIFTfeatures, results of deep features are much better. However,the best result of VR@FPR=0.1% was 58.7 ± 13.0, whichis still far from satisfactory. Therefore, there is still need forfurther improvement even with deep features.

F. Results on LFW Dataset

1) Influence of Dimension: The results of our methods onLFW dataset with different dimensions are shown in Fig. 7.The dimension was varied from 25 to 305. For both CMLAUCand CMLpAUC, the results increased with the dimension.The improvement in accuracy became convergent when thedimension exceeded 250. Different from LDML whose bestdimension was 35, it can be seen the best dimension of theproposed methods was around 300. For the following results,the dimension was set to 300.

2) Comparison with Other Methods: On LFW, theproposed methods were compared with two single modalmetric learning methods, LDML [38] and DML-eig [48]. Therestricted setting of LFW was used. Note that LMNN is notsuitable for this scenario as no label information is available.Results of our methods and the compared methods are givenin Table X. Results of the compared methods were cited fromtheir papers for the original and squared root SIFT features.The results for VGG-Face features were obtained using thesource codes provided by the authors. The results of CMLAUCand CMLpAUC were similar on this dataset. Compared withLDML and DML-eig, our methods achieved better results.This illustrates the strength of our methods compared withthose pair-based metric learning methods.


25 65 105 145 185 225 265 305

Number of Dimension

0.77

0.78

0.79

0.8

0.81

0.82

0.83

Acc

urac

y

(a) CMLAUC

25 65 105 145 185 225 265 305

Number of Dimension

0.75

0.76

0.77

0.78

0.79

0.8

0.81

Acc

urac

y

(b) CMLpAUC

Fig. 7. Results of CMLAUC and CMLpAUC with different number ofdimensions on LFW dataset.

TABLE XCOMPARISON OF ACCURACY WITH OTHER METHODS ON LFW

Method Original Square Root VGG-FaceLDML 76.6± 0.7 77.5± 0.5 97.4± 1.0DML-eig 80.6± 1.7 81.3± 2.3 96.7± 0.9CMLAUC 82.5± 1.3 83.0± 1.3 97.8± 0.8CMLpAUC 82.5± 1.2 83.4± 0.8 97.7± 0.9

G. Comparison of Training Time

A comparison of training time of our methods and LMNNon CUFSF dataset is given in Table XI. For LMNN, we usedthe code provided by its authors and the nearest neighbor wasused for constructing triplets. The sub-sample parameter of thecode was set to 0.3. For both CMLAUC and CMLpAUC, threesettings were tested. Under the first setting, the numbers ofmust-link and cannot-link constraints were set to 100. Underthe other two settings, the numbers of must-link constraintswere both set to 700 and the numbers of cannot-linkconstraints were set 1,000 and 10,000 respectively. In TableXI, the two numbers in brackets denote the sampled numbersof must-link and cannot-link constraints. All the experimentswere run on the same computer with an Intel Core i5-4278Uprocessor and 8GB RAM running Windows 10. Both LMNNand our methods were tested using MATLAB R2014a. For theprojection step of our methods with eigenvalue decomposition,the implementation of [7] was adopted.

From Table XI, it can be seen that our methods are fasterwhen the dimension is less than 600. With increases in thedimension, both LMNN and our methods increase in trainingtime. However, as the implementation of LMNN is not basedon optimizing PSD matrix, there is no additional operationat each iteration to project matric parameter onto PSD cone.Differently, the operation ϕ+λ in our algorithms in fact relieson a full eigenvalue decomposition at each iteration, whichis time consuming. That is why the increase in training timeis faster than that of LMNN. This indicates there is room forimprovement to make our methods more efficient, for example,using coordinate descent [49] for LogDet optimization. Thesecond observation is the training time of setting samplednumbers as (100,100), (700,1000) and (700,10000) is similar.There is little increase in training time with the samplednumber increases. This is mainly because sampled numbersof constraints only influence the update of matrix PA. Withthe techniques in Section III to update matrix PA, the time

TABLE XICOMPARISON OF TRAINING TIME ON CUFSF DATASET (IN SECONDS)

Methods\Dimension 200 400 600 800LMNN 357 689 1114 1661CMLAUC(100, 100) 101 381 979 2068CMLpAUC(100, 100) 101 378 979 2088CMLAUC(700, 1000) 110 414 1048 2199CMLpAUC(700, 1000) 113 409 1041 2193CMLAUC(700, 10000) 138 433 1068 2223CMLpAUC(700, 10000) 120 414 1049 2213

for updating of matrix PA grows slowly with increase in thenumber of training pairs used.

V. CONCLUSION

For cross-modal metric learning, most of the previousmethods minimize loss defined on must-link and cannot-linkpairs. Since the numbers of must-link and cannot-link pairscan be highly imbalanced and AUC is a more meaningfulperformance measure. We propose to learn cross-modalmetrics that directly maximize AUC defined on must-linkand cannot-link pairs. An extension of our proposed methodcan also optimize partial AUC. The proposed methods areformulated as a LogDet regularized semi-definite optimizationproblem. For efficient optimization, a mini-batch proximalpoint algorithm is proposed. Extensive experiments haveverified that both proposed CMLAUC and CMLpAUC methodsachieved marked improvements over the existing methods.Specifically, CMLpAUC works well for performance measureslike VR@FPR=0.1% and Rank-1. Future work includesimproving the efficiency of the optimization methods, forexample, using coordinate descent based methods for LogDetoptimization.

REFERENCES

[1] A. Mignon and F. Jurie, “CMML: a new metric learning approach forcross modal matching,” in Proc. Asian Conf. Comput. Vis., 2012, pp.1–14.

[2] L. Wu, L. Du, B. Liu, G. Xu, Y. Ge, Y. Fu, J. Li, Y. Zhou, and H. Xiong,“Heterogeneous metric learning with content-based regularization forsoftware artifact retrieval,” in Proc. Int. Conf. Data Mining, 2014, pp.610–619.

[3] L. Wu, J. Li, X. Hu, and H. Liu, “Gleaning wisdom from the past: Earlydetection of emerging rumors in social media,” in Proc. SIAM Int. Conf.Data Mining. SIAM, 2017, pp. 99–107.

[4] L. Wu, X. Hu, F. Morstatter, and H. Liu, “Adaptive spammer detectionwith sparse group modeling.” in Proc. Int. Conf. Web Social Media,2017, pp. 319–326.

[5] Q. You, J. Luo, H. Jin, and J. Yang, “Robust image sentiment analysisusing progressively trained and domain transferred deep networks.” inProc. AAAI Conf. Artif. Intell., 2015, pp. 381–388.

[6] P. Zhao, R. Jin, T. Yang, and S. C. Hoi, “Online AUC maximization,”in Proc. Int. Conf. Mach Learn., 2011, pp. 233–240.

[7] C. Wang, D. Sun, and K.-C. Toh, “Solving log-determinant optimizationproblems by a newton-cg primal proximal point algorithm,” SIAM J.Optim., vol. 20, no. 6, pp. 2994–3013, 2010.

[8] J. Yang, D. Sun, and K.-C. Toh, “A proximal point algorithm forlog-determinant optimization with group lasso regularization,” SIAM J.Optim., vol. 23, no. 2, pp. 857–893, 2013.

[9] X. Xu, W. Li, and D. Xu, “Distance metric learning using privilegedinformation for face verification and person re-identification,” IEEETrans. Neural Netw. Learn. Syst., vol. 26, no. 12, pp. 3150–3162, 2015.


[10] J. Li, X. Lin, X. Rui, Y. Rui, and D. Tao, “A distributed approachtoward discriminative distance metric learning,” IEEE Trans. NeuralNetw. Learn. Syst., vol. 26, no. 9, pp. 2111–2122, 2015.

[11] H. Jia, Y. Cheung, and J. Liu, “A new distance metric for unsupervisedlearning of categorical data,” IEEE Trans. Neural Netw. Learn. Syst.,vol. 27, no. 5, pp. 1065–1079, 2016.

[12] C. Shen, J. Kim, F. Liu, L. Wang, and A. Van Den Hengel, “Efficientdual approach to distance metric learning,” IEEE Trans. Neural Netw.Learn. Syst., vol. 25, no. 2, pp. 394–406, 2014.

[13] S. Ying, Z. Wen, J. Shi, Y. Peng, J. Peng, and H. Qiao, “Manifoldpreserving: An intrinsic approach for semisupervised distance metriclearning,” IEEE Trans. Neural Netw. Learn. Syst., vol. PP, no. 99, pp.1–12, 2017.

[14] E. P. Xing, A. Y. Ng, M. I. Jordan, and S. Russell, “Distance metriclearning with application to clustering with side-information,” Proc. Adv.Neural Inf. Process. Syst., vol. 15, pp. 505–512, 2003.

[15] J. V. Davis, B. Kulis, P. Jain, S. Sra, and I. S. Dhillon,“Information-theoretic metric learning,” in Proc. Int. Conf. Mach Learn.,2007, pp. 209–216.

[16] K. Q. Weinberger and L. K. Saul, “Distance metric learning for largemargin nearest neighbor classification,” J. Mach. Learn. Res., vol. 10,pp. 207–244, 2009.

[17] B. McFee and G. R. Lanckriet, “Metric learning to rank,” in Proc. Int.Conf. Mach Learn., 2010, pp. 775–782.

[18] D. Lim, G. Lanckriet, and B. McFee, “Robust structural metric learning,”in Proc. Int. Conf. Mach Learn., 2013, pp. 615–623.

[19] G. Chechik, V. Sharma, U. Shalit, and S. Bengio, “Large scale onlinelearning of image similarity through ranking,” J. Mach. Learn. Res.,vol. 11, pp. 1109–1135, 2010.

[20] W. Liu, C. Mu, R. Ji, S. Ma, J. R. Smith, and S.-F. Chang, “Low-ranksimilarity metric learning in high dimensions,” in Proc. AAAI Conf. Artif.Intell., 2015, pp. 2792–2799.

[21] K. Liu, A. Bellet, and F. Sha, “Similarity learning for high-dimensionalsparse data.” in Proc. Int. Conf. Artif. Intell. Statist., 2015, pp. 653–662.

[22] P. Zhou, L. Du, M. Fan, and Y.-D. Shen, “An LLE based heterogeneousmetric learning for cross-media retrieval,” in Proc. SIAM Int. Conf. DataMining, pp. 64–72.

[23] N. Quadrianto and C. H. Lampert, “Learning multi-view neighborhoodpreserving projections,” in Proc. Int. Conf. Data Mining, 2011, pp.425–432.

[24] C. Kang, S. Liao, Y. He, J. Wang, W. Niu, S. Xiang, and C. Pan,“Cross-modal similarity learning: A low rank bilinear formulation,” inProc. Int. Conf. Inform. Knowl. Manage., 2015, pp. 1251–1260.

[25] Y. Zhen, P. Rai, H. Zha, and L. Carin, “Cross-modal similarity learningvia pairs, preferences, and active supervision,” in Proc. AAAI Conf. Artif.Intell., 2015, pp. 3203–3209.

[26] T. Joachims, “A support vector method for multivariate performancemeasures,” in Proc. Int. Conf. Mach Learn., 2005, pp. 377–384.

[27] W. Gao, R. Jin, S. Zhu, and Z.-H. Zhou, “One-pass AUC optimization,”in Proc. Int. Conf. Mach Learn., 2013, pp. 906–914.

[28] Y. Ding, P. Zhao, S. C. Hoi, and Y.-S. Ong, “An adaptive gradient methodfor online auc maximization,” in Proc. AAAI Conf. Artif. Intell., 2015,pp. 2568–2574.

[29] H. Narasimhan and S. Agarwal, “A structural svm based approach foroptimizing partial auc,” in Proc. Int. Conf. Mach Learn., 2013, pp.516–524.

[30] P. Kar, H. Narasimhan, and P. Jain, “Online and stochastic gradientmethods for non-decomposable loss functions,” in Proc. Adv. NeuralInf. Process. Syst., 2014, pp. 694–702.

[31] L. Bottou, “Online algorithms and stochastic approximations,” in OnlineLearn. Neural Netw., D. Saad, Ed. Cambridge University Press, 1998.

[32] G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeledfaces in the wild: A database for studying face recognition inunconstrained environments,” Univ. of Massachusetts, Amherst, Tech.Rep. 07-49, October 2007.

[33] G. B. H. E. Learned-Miller, “Labeled faces in the wild: Updates andnew reporting procedures,” Univ. of Massachusetts, Amherst, Tech. Rep.UM-CS-2014-003, May 2014.

[34] W. Zhang, X. Wang, and X. Tang, “Coupled information-theoreticencoding for face photo-sketch recognition,” in Proc. IEEE Conf.Comput. Vis. Pattern Recognit., 2011, pp. 513–520.

[35] P. J. Phillips, H. Moon, S. A. Rizvi, and P. J. Rauss, “The feretevaluation methodology for face-recognition algorithms,” IEEE Trans.Pattern Anal. Mach. Intell., vol. 22, no. 10, pp. 1090–1104, 2000.

[36] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,”Int. J. Comput. Vis., vol. 60, no. 2, pp. 91–110, 2004.

[37] S. Z. Li, D. Yi, Z. Lei, and S. Liao, “The casia nir-vis 2.0 face database,”in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, 2013,pp. 348–353.

[38] M. Guillaumin, J. Verbeek, and C. Schmid, “Is that you? metric learningapproaches for face identification,” in Proc. Int. Conf. Comput. Vis.,2009, pp. 498–505.

[39] O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition,”in Proc. Brit. Mach. Vis. Conf., 2015.

[40] M. A. Turk and A. P. Pentland, “Face recognition using eigenfaces,” inProc. IEEE Conf. Comput. Vis. Pattern Recognit., 1991, pp. 586–591.

[41] D. Cai, X. He, and J. Han, “Speed up kernel discriminant analysis,”VLDB J., vol. 20, no. 1, pp. 21–33, 2011.

[42] J. Goldberger, S. Roweis, G. Hinton, and R. Salakhutdinov,“Neighbourhood components analysis,” in Proc. Adv. Neural Inf.Process. Syst., 2004, pp. 513–520.

[43] H. Hotelling, “Relations between two sets of variates,” Biometrika, pp.321–377, 1936.

[44] Z. Lei and S. Z. Li, “Coupled spectral regression for matchingheterogeneous faces,” in Proc. IEEE Conf. Comput. Vis. PatternRecognit., 2009, pp. 1123–1128.

[45] P. L. Lai and C. Fyfe, “Kernel and nonlinear canonical correlationanalysis,” Int’l J. Neural Syst., vol. 10, no. 05, pp. 365–377, 2000.

[46] D. Lin and X. Tang, “Inter-modality face recognition,” in Proc. Eur.Conf. Comput. Vis., 2006, pp. 13–26.

[47] M. Kan, S. Shan, H. Zhang, S. Lao, and X. Chen, “Multi-viewdiscriminant analysis,” in Proc. Eur. Conf. Comput. Vis., 2012, pp.808–821.

[48] Y. Ying and P. Li, “Distance metric learning with eigenvalueoptimization,” J. Mach. Learn. Res., vol. 13, pp. 1–26, 2012.

[49] J. Friedman, T. Hastie, and R. Tibshirani, “Sparse inverse covarianceestimation with the graphical lasso,” Biostatistics, vol. 9, no. 3, pp.432–441, 2008.

Jing Huo received the Ph.D. degree from theDepartment of Computer Science and Technologyof Nanjing University in 2017. She is currently anassistant researcher in the Department of ComputerScience and Technology of Nanjing University,China. Her research interests lie in machine learningand computer vision. Her work currently focuseson metric learning, subspace learning and theirapplications to heterogeneous face recognition.

Yang Gao received the Ph.D. degree from theDepartment of Computer Science and Technology ofNanjing University in 2000. He is a Professor at theDepartment of Computer Science and Technology,Nanjing University. His research interests includeartificial intelligence and machine learning. He haspublished more than 100 papers in top conferencesand journals in and outside of China. He is a memberof the IEEE.


Yinghuan Shi is currently an associate professorin the Department of Computer Science andTechnology of Nanjing University, China. Hereceived his Ph.D. and B.Sc. degrees fromDepartment of Computer Science of NanjingUniversity in 2013 and 2007, respectively. He wasa visiting scholar in University of North Carolina atChapel Hill, and University of Technology Sydney,respectively. His research interests include computervision and medical image analysis. He has publishedmore than 40 research papers in related journals

and conferences such as TPAMI, TBME, TNNLS, TCYB, CVPR, AAAI,ACMMM, MICCAI and IPMI. He serves as a program committee memberfor several conferences, and also as a referee for several journals.

Hujun Yin is a Senior Lecturer (AssociateProfessor) at the University of Manchester, School ofElectrical and Electronic Engineering. He receivedthe PhD degree in neural networks from Universityof York, and MSc degree in signal processingand BEng degree in electronic engineering fromSoutheast University. His main areas of researchand expertise are neural networks, self-organisinglearning, image processing, face recognition, timeseries and bio-/neuro-informatics. He has publishedover 150 peer-reviewed articles in a range of topics

from density modelling, image processing, face recognition, text mining andknowledge management, gene expression analysis and peptide sequencing,novelty detection, to financial time series modelling and decoding neuronalresponses. He is a senior member of the IEEE.

cross-modal metric learning for auc optimization · 2017. 11. 7. · submitted to ieee transactions...

Documents