Distance Measure Machines

Alain Rakotomamonjy Abraham Traore Maxime Berar Remi Flamary Nicolas CourtyLITIS EA4108

Universite de Rouenand

Criteo AI LabsCriteo Paris

LITIS EA4108Universite de Rouen

LITIS EA4108Universite de Rouen

Lagrange UMR7293Universite de

Nice Sophia-Antipolis

IRISA UMR6074Universite de

Bretagne-Sud

Abstract

This paper presents a distance-based discrimi-native framework for learning with probabilitydistributions. Instead of using kernel meanembeddings or generalized radial basis kernels,we introduce embeddings based on dissimilar-ity of distributions to some reference distribu-tions denoted as templates. Our frameworkextends the theory of similarity of Balcanet al. (2008) to the population distributioncase and we show that, for some learningproblems, some dissimilarity on distributionachieves low-error linear decision functionswith high probability. Our key result is toprove that the theory also holds for empiricaldistributions. Algorithmically, the proposedapproach consists in computing a mappingbased on pairwise dissimilarity where learninga linear decision function is amenable. Our ex-perimental results show that the Wassersteindistance embedding performs better than ker-nel mean embeddings and computing Wasser-stein distance is far more tractable than esti-mating pairwise Kullback-Leibler divergenceof empirical distributions.

1 Introduction

Most discriminative machine learning algorithms havefocused on learning problems where inputs can be rep-resented as feature vectors of fixed dimensions. Thisis the case of popular algorithms like support vectormachines (Scholkopf & Smola, 2002) or random forest(Breiman, 2001). However, there exists several practical

situations where it makes more sense to consider inputdata as set of distributions or empirical distributionsinstead of a larger collection of single vector. As anexample, multiple instance learning (Dietterich et al. ,1997) can be seen as learning of a bag of feature vec-tors and each bag can be interpreted as samples froman underlying unknown distribution. Applications re-lated to political sciences (Flaxman et al. , 2015) orastrophysics Ntampaka et al. (2015) have also consid-ered this learning from distribution point of view forsolving some specific machine learning problems. Thispaper also addresses the problem of learning decisionfunctions that discriminate distributions.

Traditional approaches for learning from distributionsis to consider reproducing kernel Hilbert spaces (RKHS)and associated kernels on distributions. In this largercontext, several kernels on distributions have been pro-posed in the literature such as the probability productkernel (Jebara et al. , 2004), the Battarachya kernel(Bhattacharyya, 1943) or the Hilbertian kernel on prob-ability measures of Hein & Bousquet (2005). Anotherelegant approach for kernel-based distribution learninghas been proposedby Muandet et al. (2012). It consistsin defining an explicit embedding of a distribution as amean embedding in a RKHS. Interestingly, if the kernelof the RKHS satisfies some mild conditions then all theinformation about the distribution is preserved by thismean embedding. Then owing to this RKHS embed-ding, all the machinery associated to kernel machinescan be deployed for learning from these (embedded)distributions.

By leveraging on the flurry of distances between distri-butions (Sriperumbudur et al. , 2010), it is also possibleto build definite positive kernel by considering general-ized radial basis function kernels of the form

K(µ, µ′) = e−σd2(µ,µ′), (1)

where µ and µ′ are two distributions, σ > 0 a param-eter of the kernel and d(·, ·) a distance between twodistributions satisfying some appropriate properties so

arX

iv:1

803.

0025

0v3

[cs

.LG

] 1

5 N

ov 2

018

Alain Rakotomamonjy, Abraham Traore, Maxime Berar, Remi Flamary, Nicolas Courty

x1

x 2

Class 1Class 2Class 3

Figure 1: Illustrating the principle of the dissimilarity-based distribution embedding. We want to discriminateempirical normal distributions in R2; their discriminative feature being the correlation between the two variables.An example of these normal distributions are given in the left panel. The proposed approach consists in computinga embedding based on the dissimilarity of all these empirical distributions (the blobs) to few of, them that serveas templates. Our theoretical results show that if we take enough templates and there is enough samples in eachtemplate then with high-probability, we can learn a linear separator that yields few errors. This is illustratedin the 3 other panels in which we represent each of the original distribution as a point after projection in andiscriminant 2D space of the embeddings. From left to right, the dissimilarity embedding respectively considers10, 45 and 90 templates and we can indeed visualize that using more templates improve separability.

as to make K definite positive (Haasdonk & Bahlmann,2004).

As we can see, most works in the literature address thequestion of discriminating distributions by consideringeither implicit or explicit kernel embeddings. However,is this really necessary? Our observation is that thereare many advantages of directly using distances or evendissimilarities between distributions for learning. Itwould avoid the need for two-stage approaches, com-puting the distance and then the kernel, as proposedby Poczos et al. (2013) for distribution regression orestimating the distribution and computing the kernelas introduced by Sutherland et al. (2012). Using ker-nels limits the choice of distribution distances as theresulting kernel has to be definite positive. For in-stance, Poczos et al. (2012) used Reyni divergencesfor building generalized RBF kernel that turns out tobe non-positive. For the same reason, the celebratedand widely used Kullback-Leibler divergence does notqualify for being used in a kernel. This work aims atshowing that learning from distributions with distancesor even dissimilarity is indeed possible. Among all avail-able distances on distributions, we focus our analysis onWasserstein distances which come with several relevantproperties, that we will highlight later, compared toother ones (e.g Kullback-Leibler divergence).

Our contributions, depicted graphically in Figure 1,are the following : (a) We show that by following theunderlooked works of Balcan et al. (2008), learning todiscriminate population distributions with dissimilar-ity functions comes at no expense. While this mightbe considered a straightforward extension, we are not

aware of any work making this connection. (b) Ourkey theoretical contribution is to show that Balcan’sframework also holds for empirical distributions if theused dissimilarity function is endowed with nice con-vergence properties of the distance of the empiricaldistribution to the true ones. (c) While these conver-gence bounds have already been exhibited for distancessuch as the Wasserstein distance or MMD, we provethat this is also the case for the Bures-Wassersteinmetric.(d) Empirically, we illustrate the benefits ofusing this Wasserstein-based dissimilarity functionscompared to kernel or MMD distances in some simu-lated and real-world vision problem, including 3D pointcloud classification task.

2 Framework

In this section, we introduce the global setting andpresent the theory of learning with dissimilarity func-tions of Balcan et al. (2008).

2.1 Setting

Define X as an non-empty subset of Rd and let Pdenotes the set of all probability measures on a mea-surable space (X ,A), where A is σ-algebra of subsetsof X . Given a training set {µi, yi}ni=1, where µi ∈ Pand yi ∈ {−1, 1}, drawn i.i.d from a probability dis-tribution P on P × {−1, 1}, our objective is to learna decision function h : P 7→ {−1, 1} that predicts themost accurately as possible the label associated to anovel measure µ. In summary, our goal is to learnto classify probability distributions from a supervised

Alain Rakotomamonjy, Abraham Traore, Maxime Berar, Remi Flamary, Nicolas Courty

setting. While we focus on a binary classification, theframework we consider and analyze can be extendedto multi-class classification.

2.2 Dissimilarity function

Most learning algorithms for distributions are basedon reproducing kernel Hilbert spaces and leverage onkernel value k(µ, µ′) between two distributions wherek(·, ·) is the kernel of a given RKHS.

We depart from this approach and instead, we considerlearning algorithms that are built from pairwise dis-similarity measures between distributions. Subsequentdefinitions and theorems are recalled from Balcan et al.(2008) and adapted so as to suit our definition of

bounded dissimilarity.

Definition 1. A dissimilarity function over P is anypairwise function D : P× P 7→ [0,M ].

While this definition emcompasses many functions,given two probability distributions µ and µ′, we ex-pect D(µ, µ′) to be large when the two distributionsare “dissimilar” and to be equal to 0 when they aresimilar. As such any bounded distance over P fits intoour notion of dissimilarity, eventually after rescaling.Note that unbounded distance which is clipped aboveM also fits this definition of dissimilarity.

Now, we introduce the definition that characterizesdissimilarity function that allows one to learn a decisionfunction producing low error for a given learning task.

Definition 2. Balcan et al. (2008) A dissimilarityfunction D is a (ε, γ)-good dissimilarity function for alearning problem L if there exists a bounded weightingfunction w over P, with w(µ) ∈ [0, 1] for all µ ∈ P,such that a least 1 − ε probability mass of distribu-tion examples µ satisfy : Eµ′∼P [w(µ′)D(µ, µ′)|`(µ) =`(µ′)] + γ ≤ Eµ′∼P [w(µ′)D(µ, µ′)|`(µ) 6= `(µ′)] . Thefunction `(µ) denotes the true labelling function thatmaps µ to its labels y.

In other words, this definition translates into: a dissim-ilarity function is “good” if with high-probability, theweighted average of the dissimilarity of one distributionto those of the same label is smaller with a margin γ tothe dissimilarity of distributions from the other class.

As stated in a theorem of Balcan et al. (2008), sucha good dissimilarity function can be used to define anexplicit mapping of a distribution into a space. Inter-estingly, it can be shown that there exists in that spacea linear separator that produces low errors.

Theorem 1. Balcan et al. (2008) if D is an (ε, γ)-good dissimilarity function, then if one draws a setS from P containing n = ( 4M

γ )2 log( 2δ ) positive ex-

amples S+ = {ν1, · · · , νn} and n negative exam-

ples S− = {ζ1, · · · , ζn}, then with probability 1 − δ,the mapping ρS : P 7→ R2n defined as ρS(µ) =(D(µ, ν1), · · · ,D(µ, νn),D(µ, ζ1), · · · ,D(µ, ζn)) has theproperty that the induced distribution ρS(P) in R2n

has a separator of error at most ε+ δ at margin at leastγ/4.

The above described framework shows that under somemild conditions on a dissimilarity function and if weconsider population distributions, then we can benefitfrom the mapping ρS . However, in practice, we do haveaccess only to empirical version of these distributions.Our key theoretical contribution in Section 3 provesthat if the number of distributions n is large enoughand enough samples are obtained from each of thesedistributions, then this framework is applicable withtheoretical guarantees to empirical distributions.

3 Learning with empiricaldistributions

In what follows, we formally show under which con-ditions an (ε, γ)-good dissimilarity function for somelearning problems, applied to empirical distributionsalso produces a mapping inducing low-error linear sep-arator.

Suppose that we have at our disposal a dataset com-posed of {µi, yi = 1}ni=1 where each µi is a distribution.However, each µi is not observed directly but insteadwe observe it empirical version µi = 1

Ni

∑Nij=1 δxi,j with

xi,1,xi,2, · · ·xi,Nii.i.d∼ µi. For a sake of simplicity, we

assume in the sequel that the number of samples forall distributions are equal to N . Suppose that weconsider a dissimilarity D and that there exist a func-tion g1 such that D satisfies a property of the form

P(D(µ, µ) > ε

)≤ g1(K,N, ε, d) then following theo-

rem holds:

Theorem 2. For a given learning problem, if thedissimilarity D is an (ε, γ)-good dissimilarity func-tion on population distributions, with w(µ) = 1, ∀µand K a parameter depending on this dissimilaritythen, for a parameter λ ∈ (0, 1), if one draws a set

S from P containing n = 32M2

γ2 log( 2δ2(1−λ) ) positive

examples S+ = {ν1, · · · , νn} and n negative examplesS− = {ζ1, · · · , ζn}, and from each distribution νi or ζi,one draws N samples so that δ2λ ≥ Ng1(K,N, ε4 , d)samples so as to build empirical distributions {νi}or {ζi}, then with probability 1 − δ, the mappingρS : P 7→ R2n defined as

ρS(µ) =1

M(D(µ, ν1), · · · ,D(µ, νn),D(µ, ζ1), · · · ,D(µ, ζn))

has the property that the induced distribution ρS(P)

Alain Rakotomamonjy, Abraham Traore, Maxime Berar, Remi Flamary, Nicolas Courty

in R2n has a separator of error at most ε+δ and marginat least γ/4.

Let us point out some relevant insights from this theo-rem. At first, due to the use of empirical distributionsinstead of population one, the sample complexity ofthe learning problem increases for achieving similarerror as in Theorem 1. Secondly, note that λ has atrade-off role on the number n of samples νi and ζi andthe number of observations per distribution. Hence,for a fixed error ε+ δ at margin γ/4, having less sam-ples per distribution has to be paid by sampling moreobservations.

The proof of Theorem 1 has been postponed to theappendix. It takes advantage of the following keytechnical result on empirical distributions.

Lemma 1. Let D be a dissimilarity on P×P such thatD is bounded by a constant M . Given a distributionµ ∈ P of class y and a set of independent distributions{νj}nj=1, randomly drawn from P, which have the same

label y and denote as µ and {νi} their empirical versioncomposed of N observations. Let us assume that thereexists a function g1 and a constant K > 0 so that

for any µ ∈ P, P(D(µ, µ) > ε

)≤ g1(K,N, ε, d), with

typically g1 tends towards 0 as N or ε goes to ∞. Thefollowing concentration inequality holds for any ε > 0 :

P

(∣∣∣∣∣ 1nn∑i=1

D(µ, νi)−Eν∼P[D(µ, ν)|`(µ) = `(ν)]

∣∣∣∣∣ > ε

)≤ Ng1(K,N,

ε

4, d) + 2e−n

ε2

2M2 .

This lemma tells us that, with high probability, themean average of the dissimilarity between an empiri-cal distribution and some other empirical distributionof the same class does not differ much from the ex-pectation of this dissimilarity measured on populationdistributions. Interestingly, the bound on the probabil-ity is composed of two terms : the first one is related tothe dissimilarity function between a distribution andits empirical version while the second one is due tothe empirical version of the expectation (resulting thusfrom Hoeffding inequality). The detailed proof of thisresult is given in the supplementary material. Notethat in order for the bound to be informative, we expectg1 to have a negative exponential form in N . Anotherversion of this lemma is proven in supplementary wherethe concentration inequality for the dissimilarity is on|D(µ, ν)−D(µ, ν)|.

From a theoretical point of view, there is only one rea-son for choosing one (ε, γ)-good dissimilarity functionon population distributions from another. The ratio-nale would be to consider the dissimilarity functionwith the fastest rate of convergence of the concentra-

tion inequality Pr(D(µ, µ) > ε), as this rate will impactthe upper bound in Theorem 1.

From a more practical point of view, several factorsmay motivate the choice of a dissimilarity function :computational complexity of computing D(µi, µj), itsempirical performance on a learning problem and adap-tivity to different learning problems ( e.g without theneed for carefully adapting its parameters to a newproblem.)

4 (ε− γ) Good dissimilarity fordistributions

We are interested now in characterizing convergenceproperties of some dissimilarities (or distance or di-vergence) on probability distributions so as to makethem fit into the framework. Mostly, we will focus ourattention on divergences that can be computed in anon-parametric way.

4.1 Optimal transport distances

Based on the theory of optimal transport, these dis-tances offer means to compare data probability distri-butions. More formally, assume that X is endowedwith a metric dX . Let p ∈ (0,∞), and let µ ∈ P andν ∈ P be two distributions with finite moments of orderp (i.e.

∫X dX (x, x0)pdµ(x) <∞ for all x0 in X ). then,

the p-Wasserstein distance is defined as:

Wp(µ, ν) =

(inf

π∈Π(µ,ν)

∫∫X×X

dX (x, y)pdπ(x, y)

) 1p

.

(2)Here, Π(µ, ν) is the set of probabilistic couplings π on(µ, ν). As such, for every Borel subsets A ⊆ X , wehave that µ(A) = π(X × A) and ν(A) = π(A × X ).We refer to (Villani, 2009, Chaper 6) for a completeand mathematically rigorous introduction on the topic.Note when p = 1, the resulting distance belongs to thefamily of integral probability metrics Sriperumbuduret al. (2010). OT has found numerous applications inmachine learning domain such as multi-label classifica-tion (Frogner et al. , 2015), domain adaptation (Courtyet al. , 2017) or generative models (Arjovsky et al. ,2017). Its efficiency comes from two major factors: i) ithandles empirical data distributions without resortingfirst to parametric representations of the distributionsii) the geometry of the underlying space is leveragedthrough the embedding of the metric dX . In some veryspecific cases the solution of the infimum problem isanalytic. For instance, in the case of two Gaussiansµ ∼ N (m1,Σ1) and ν ∼ N (m2,Σ2) the Wassersteindistance with dX (x, y) = ‖x− y‖2 reduces to:

W 22 (µ, ν) = ||m1 −m2||22 + B(Σ1,Σ2)2 (3)

Alain Rakotomamonjy, Abraham Traore, Maxime Berar, Remi Flamary, Nicolas Courty

where B(, ) is the so-called Bures metric Bures (1969):

B(Σ1,Σ2)2 = trace(Σ1 +Σ2−2(Σ1/21 Σ2Σ

1/21 )1/2). (4)

If we make no assumption on the form of the distribu-tions, and distributions are observed through samples,the Wasserstein distance is estimated by solving a dis-crete version of Equation 2 which is a linear program-ming problem.

One of the necessary condition for this distance tobe relevant in our setting is based on non-asymptoticdeviation bound of the empirical distribution to thereference one. For our interest, Fournier & Guillin(2015) have shown that for distributions with finitemoments, the following concentration inequality holds

P(Wp(µ, µ) > ε) ≤ C exp (−KNεd/p)

where C and K are constants that can be computedfrom moments of µ. This bound shows that the Wasser-stein distance suffers the dimensionality and as such aWasserstein distance embedding for distribution learn-ing is not expected to be efficient especially in high-dimension problems. However, a recent work of Weed& Bach (2017) has also proved that under some hy-pothesis related to singularity of µ better convergencerate can be obtained (some being independent of d).Interestingly, we demonstrate in what follows that theestimated Wasserstein distance for Gaussians using Bu-res metric and plugin estimate of m and Σ has a betterbound related to the dimension.

Lemma 2. Let µ be a d-dimensional Gaussian distri-bution and m and Σ the sample mean and covarianceestimator of µ obtained from N samples. Assume thatthe true covariance matrix of µ satisfies ‖Σ‖2 ≤ CΣtherandom vectors v used for computing these estimatesare so that ‖v‖2 ≤ Cv and The squared-Wassersteindistance between the empirical and true distributionsatisfies the following deviation inequality:

P(W2(µ, µ)) > ε) ≤ 2d exp

(− −Nε2/(8d4)

CvCΣ + 2Cvε/(3d2)

)

+ exp

(−( N1/2

24√Cv

ε2 − 1)1/2

)

This novel deviation bound for the Gaussian 2-Wasserstein metric tells us that if the empirical data(approximately) follows a high-dimensional Gaussiandistribution then it makes more sense to estimate themean and covariance of the distribution and then toapply Bures-Wasserstein distance rather to apply di-rectly a Wasserstein distance estimation based on thesamples.

4.2 Kullback-Leibler divergence and MMD

Two of the most studied and analyzed diver-gences/distances on probability distribution are theKullback-Leibler divergence and Maximum Mean Dis-crepancy. Several works have proposed non-parametricapproaches for estimating these distances and haveprovided theoretical convergence analyses of these esti-mators.

For instance, Nguyen et al. (2010) estimate the KLdivergence between two distributions by solving aquadratic programming problem which aims a findinga specific function in a RKHS. They also proved thatthe convergence rate of such estimator in in O(N

12 ).

MMD has originally been introduced by Gretton et al.(2007) as a mean for comparing two distributions

based on a kernel embedding technique. It has beenproved to be easily computed in a RKHS. In addition,its empirical version benefits from nice uniform bound.Indeed, given two distribution µ and ν and their empir-ical version based on N samples µ and ν, the followinginequality holds (Gretton et al. , 2012):

P(

MMD2(µ, ν)−MMD2(µ, ν) > ε)≤ exp

(− ε2N2

8K2

)where N2 = bN/2c, K is a bound on k(x,x′),∀x,x′,and k(·, ·) is the reproducing kernel of the RKHS inwhich distributions have been embedded. We can notethat this bound is independent of the underlying di-mension of the data.

Owing to this property we can expect MMD to pro-vide better estimation of distribution distance for high-dimension problems than WD for instance. Note how-ever that for MMD-based two sample test, Ramdaset al. (2015) has provided contrary empirical evidenceand have shown that for Gaussian distributions, as di-mension increases MMD2(µ, ν) goes to 0 exponentiallyfast in d. Hence, we will postpone our conclusion onthe advantage of one measure distance on another toour experimental analysis.

4.3 Discriminating normal distributions withthe mean

(ε, γ)-goodness of a dissimilarity function is a propertythat depends on the learning problem. As such, it isdifficult to characterize whether a dissimilarity will begood for all problems. In the sequel we characterizethis property for these three dissimilarities, on a mean-separated Gaussian distribution problem. (Muandetet al. , 2012) used the same problem as their numericaltoy problem. We show that even in this simple case,MMD suffers high dimensionality more than the twoother dissimilarities.

Alain Rakotomamonjy, Abraham Traore, Maxime Berar, Remi Flamary, Nicolas Courty

Consider a binary distribution classification problemwhere samples from both classes are defined by Gaus-sian distributions in Rd. Means of these Gaussian dis-tribution follow another Gaussian distribution whichmean depends on the class while covariance are fixed.Hence, we have µi ∼ N (mi,Σ) with mi ∼ N (m?

−1,Σ0)if yi = −1 and mi ∼ N (m?

+1,Σ0) if yi = +1 where Σand Σ0 are some definite-positive covariance matrix.We suppose that both classes have same priors. Wealso denote D? = ‖m?

−1 −m?+1‖22 which is a key com-

ponent in the learnability of the problem. Intuitively,assuming that the volume of each µi as defined by thedeterminant of Σ is smaller than the volume of Σ0, thelarger D? is the easier the problem should be. Thisidea appears formally in what follows.

Based on Wasserstein distance between two normaldistributions with same covariance matrix, we haveW (µi, µj)

2 = ‖mi − mj‖22. In addition, given a µiwith mean mi, regardless of its class, we have, withk ∈ {−1,+1}:

Eµj :mj∼N (m?k,Σ0)[‖mi−mj‖22] = ‖mi−m?

k‖22 +Tr(Σ0)

Given α ∈]0, 1], we define the subset of Rd,

E−1 = {m : (m−m−1)>(m?+1 −m?

−1) ≤ 1− α2

D?

Informally, E−1 is an half-space containing of m−1 forwhich all points are nearer to m−1 than m+1 with amargin defined by 1−α

2 D?. In the same way, we defineE+1 as :

E+1 = {m : (m−m?+1)>(m?

−1 −m?+1) ≤ 1− α

2D?}

Based on these definition, we can state that W (·, ·)is a (ε, γ) good dissimilarity function with γ = αD?,ε = 1

2

∫Rd\E−1

dN (m−1,Σ0) + 12

∫Rd\E+1

dN (m+1,Σ0)

and w(µ) = 1,∀µ. Indeed, it can be shown that for agiven µi with yi = −1, if mi ∈ E−1 then

‖mi −m?−1‖22 + α‖m?

−1 −m?+1‖22︸ ︷︷ ︸

γ

≤ ‖mi −m?+1‖22

With a similar reasoning, we get an equivalent inequal-ity for µi of positive label. Hence, we have all theconditions given in Definition 2 for the Wassersteindistance to be an (ε, γ) good dissimilarity function forthis problem. Note that the γ and ε naturally dependon the distance between expected means. The largerthis distance is, the larger the margin and the smallerε are.

Following the same steps, we can also prove that for thisspecific problem of discriminating normal distribution,the Kullback-Leibler divergence is also a (ε, γ) gooddissimilarity function. Indeed, for µ1 and µ2 following

two Normal distributions with same covariance matrixΣ0, we have KL(µ1, µ2) = ‖m2 −m1‖2Σ−1

0

. And fol-

lowing exactly the same steps as above, but replacinginner product m>m′ with m>Σ−1

0 m′ leads to similarmargin γ = α‖m?

−1 −m?+1‖2Σ−1

0

and similar definition

of ε.

While the above margins γ for KL and WD are validfor any Σ0, if we assume Σ0 = σ2I, then accordingto Ramdas et al. (2015), the following approximationholds for this problem

MMD2(µ1, µ2) ≈ ‖m1 −m2‖22σ2k(1 + 2σ2/σ2

k)d/2+1

where σk is the bandwidth of the kernel embedding,leading to a (ε− γ) distribution with margin

α‖m?

1 −m?2‖22

σ2k(1 + 2σ2/σ2

k)d/2+1

From these margin equations for all the dissimilarities,we can drive similar conclusions to those of Ramdaset al. (2015) on test power. Regardless on the choice ofthe kernel embedding bandwidth, the margin of MMDis supposed to decrease with respect to the dimen-sionality either polynomially or exponentially fast. Assuch, even in this simple setting, MMD is theoreticallyexpected to work worse than KL divergence or WD.

In practice, we need to compute these KL, WD orMMD distance from samples obtained i.i.d from theunknown distribution µ and ν. The problem of esti-mating in a non-parametric way some φ-divergence,especially the Kullback-Leibler divergence have beenthoroughly studied by Nguyen et al. (2007, 2010). ForKL divergence, this estimation is obtained by solv-ing a quadratic programming problem. In a nutshell,compared to Kullback-Leibler divergence, Wassersteindistance benefits from a linear programming problemcompared to a quadratic programming problem. Inaddition, unlike KL-divergence, Wasserstein distancetakes into account the properties of X and as such itdoes not diverge for distributions that do not sharesupport.

5 Numerical experiments

In this section, we have analyzed and compared theperformances of Wasserstein distances based embed-ding for learning to classify distributions. Several toyproblems, similar to those described in Section 4.3 havebeen considered as well as a computer-vision real-worldproblem.

Alain Rakotomamonjy, Abraham Traore, Maxime Berar, Remi Flamary, Nicolas Courty

100 200 300 400 500 600Number of learning samples

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Accu

racy

Dimension=50Support Measure MachinesMMD+ Linear SVMMMD + kernel SVMWD+ Linear SVMWD + kernel SVMBures + Linear SVMBures + kernel SVM

0 20 40 60 80 100Dimension

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Accu

racy

#Gaussian=250, #Samples per Gaussian=30

Support Measure MachinesMMD+ Linear SVMMMD + kernel SVMWD+ Linear SVMWD + kernel SVMBures + Linear SVMBures + kernel SVM

Figure 2: Comparing performances of Support Measure Machines and Wasserstein distance + classifier.

5.1 Competitors

Before describing the experiments we discuss the al-gorithms we have compared. We have considered twovariants of our approach. The first one embeds thedistributions based on ρS by using, unless specified, alldistributions available in the training set. The Wasser-stein distance is approximated using its entropic regu-larized version with λ = 0.01 for all problems. Then,we learn either a linear SVM or a Gaussian kernel clas-sifier resulting in two methods dubbed in the sequelas WD+linear and WD+kernel. As discussed insection 3, we can use the closed-form Bures Wassresteindistance when we suppose that the distributions areGaussian. Assuming that the samples come from aNormal distribution, plugging-in the empirical meanand covariance estimation into the Bures-Wassersteindistance 3 gives us a distance that we can use as anembedding. In the experiments, these approaches arenamed Bures+linear and Bures+kernel. In thefamily of integral probability metrics, we used the sup-port measure machines of Muandet et al. (2012), de-noted as SMM. We have considered its non-linearversion which used an Gaussian kernel on top of theMMD kernel. In SMM, we have thus two kernel hy-perparameters. In order to evaluate the choice of thedistance, we have also used the MMD distance in ad-dition to the Wasserstein distance in our framework.These approaches are denoted as MMD+linear andMMD+kernel. Note that we have not reportedbased on samples-based approaches such as SVM sinceMuandet et al. (2012) have already reported that theyhardly handle distributions.

Kullback-Leibler divergence can replace the Wasser-stein distance in our framework. For instance, we havehighlighted that for the problem in Section 4.3, KL-divergence is an (ε, γ) good dissimilarity function. Wehave thus implemented the non-parametric estimationof the KL-divergence based on quadratic programming

(Nguyen et al. , 2010, 2007). After few experimentson the toy problems, we finally decided to not reportperformance of the KL-divergence based approach dueto its poor computational scalability as illustred in thesupplementary material.

5.2 Simulated problem

These problems aim at studying the performances ofour models in controlled setting. The toy problemcorresponds to the one described in Section 4.3 but with3 classes. For all classes, mean of a given distributionfollows a normal distribution with mean m? = [1, 1]and covariance matrix σI with σ = 5. For class i,the covariance matrix of the distribution is defined asσiI+ui(I1 +I−1) where I1 and I−1 are respectively thesuper and sub diagonal matrices. The {σi} are constantwhereas ui follows a uniform distribution dependingon the class. We have kept the number of empiricalsamples per distribution fixed at N = 30.

For these experiments, we have analyzed the effect ofthe number of training examples n (which is also thenumber of templates) and the dimensionality d of thedistribution. Approaches are then evaluated on of 2000test distributions. 20 trials have been considered foreach n and dimension d. We define a trial as follows:we randomly sample the n number of distributionsand compute all the embeddings and kernels. Forlearning, We have performed cross-validation on allparameters of all competitors. This involves all kerneland classifier parameters. Details of all parameters andhyperparameters are given in the appendix.

Left plot in Figure 2 represents the averaged classifi-cation accuracy with N = 30 samples per classes andd = 50 for increasing number n of empirical trainingdistribution examples. Right plot represents the samebut for fixed n = 250 and increasing dimensionality d.

From the left panel, we note that MMD-based distance

Alain Rakotomamonjy, Abraham Traore, Maxime Berar, Remi Flamary, Nicolas Courty

Method Scenes 3DPC 3DPC-CV

SMM 51.58 ± 2.46 92.79 92.99 ± 0.99MMD linear 24.83 ± 1.22 91.89 91.84 ± 1.13MMD kernel 27.02 ± 4.09 90.54 92.66 ± 1.02WD linear 61.58 ± 1.34 97.30 95.52 ± 0.89WD kernel 60.70 ± 2.49 96.86 94.89 ± 0.80BW linear 62.30 ± 1.32 64.86 63.52 ± 5.72BW kernel 62.06 ± 1.34 70.72 72.20 ± 1.51

Table 1: Performances of competitors on real-worldproblems. 3DPC and 3DPC-CV columns report per-formances on original train/test split and for randomsplits. Bold denotes best test accuracy and under-line show statistically equivalent performance underWilocoxon signrank test with p = 0.01.

fails in achieving good performances regardless of howthey are employed (kernel or distance based classifier).WDMM performs better than MMD especially as thenumber of training distrubtions increases. For n = 600,the difference in performance is almost 30% of accuracywhen considering distance-based embeddings. We alsoremark that the Bures-Wasserstein metric naturallyfits to this Normal distribution learning problem andachieves perfect performances for n ≥ 200.

Right panel shows the Impact of the dimensionality ofthe problem on the classification performance. We notethat again MMD-based approaches do not perform asgood as Wasserstein-based ones. Whereas MMD topsbelow 70%, our non-linear WDMM method achievesabout 90% of classification rate across a large range ofdimensionality.

5.3 Natural scene categorization

We have also compared the performance the differentapproaches on a computer vision problem. For thispurpose, we have reproduced the experiments carriedout by Muandet et al. (2012). Their idea is to consideran image of a scene as an histogram of codewords,where the codewords have been obtained by k-meansclustering of 128-dim SIFT vector and thus to usethis histogram as a discrete probability distribution forclassifying the images. Details of the feature extractionpipeline can be found in the paper Muandet et al.(2012). The only difference our experimental set-up isthat we have used an enriched version of the dataset1

they used. Similarly, we have used 100 images perclass for training and the rest for testing. Again, allhyperparameters of all competing methods have beenselected by cross-validation.

The averaged results over 10 trials are presented inTable 1. Again, the plot illustrates the benefit of

1The dataset is available at http://www-cvr.ai.uiuc.edu/ponce_grp/data/

Wasserstein-distance based approaches (through fullynon-parametric distance estimation or through the es-timated Bures-Wasserstein metric) compared to MMDbased methods. We believe that the gain in perfor-mance for non-parametric methods is due to the abilityof the Wasserstein distance to match samples of onedistribution to only few samples of the other distribu-tion. By doing so, we believe that it is able to capturein an elegant way complex interaction between samplesof distributions.

5.4 3D point cloud classification

3D point cloud can be considered as samples from adistribution. As such, a natural tool for classifyingthem is to used metrics or kernels on distributions. Inthis experiment, we have benchmarked all competitorson a subset of the ModelNet10 dataset . Among the10 classes in that dataset, we have extracted the nightstand, desk and bathtub classes which respectively have400, 400 and 212 training examples and 172,172 and100 test examples. Experiments and model selectionhave been run as in previous experiments. In Table1, we report results based on original train and testsets and results using 50 − 50 random splits and re-samplings. Again, we highlight the benefit of using theWasserstein distance as an embedding and contrarilyto other experiments, the Bures-Wasserstein metricyields to poor performance as the object point cloudshardly fit a Normal distribution leading thus to modelmisspecification.

6 Conclusion

This paper introduces a method for learning to discrim-inate probability distributions based on dissimilarityfunctions. The algorithm consists in embedding the dis-tributions into a space of dissimilarity to some templatedistributions and to learn a linear decision function inthat space. From a theoretical point of view, whenconsidering population distributions, our framework isan extension of the one of Balcan et al. (2008). But weprovide a theoretical analysis showing that for embed-dings based on empirical distributions, given enoughsamples, we can still learn a linear decision functionswith low error with high-probability with empiricalWasserstein distance. The experimental results illus-trate the benefits of using empirical dissimilarity ondistributions on toy problems and real-world data.

Futur works will be oriented toward analyzing a moregeneral class of regularized optimal transport diver-gence, such as the Sinkhorn divergence Genevay et al.(2017) in the context of Wasserstein distance measuremachines. Also, we will consider extensions of thisframework to regression problems, for which a direct

Alain Rakotomamonjy, Abraham Traore, Maxime Berar, Remi Flamary, Nicolas Courty

application is not immediate.

References

Arjovsky, Martin, Chintala, Sumit, & Bottou, Leon.2017. Wasserstein Generative Adversarial Networks.Pages 214–223 of: ICML.

Balcan, Maria-Florina, Blum, Avrim, & Srebro,Nathan. 2008. A theory of learning with similarityfunctions. Machine Learning, 72(1-2), 89–112.

Bhattacharyya, Anil. 1943. On a measure of diver-gence between two statistical populations defined bytheir probability distributions. Bull. Calcutta Math.Soc., 35, 99–109.

Breiman, Leo. 2001. Random forests. Machine learn-ing, 45(1), 5–32.

Bures, Donald. 1969. An extension of Kakutani’stheorem on infinite product measures to the tensorproduct of semifinite σ-algebras. Transactions of theAmerican Mathematical Society, 135, 199–212.

Courty, Nicolas, Flamary, Remi, Tuia, Devis, & Rako-tomamonjy, Alain. 2017. Optimal transport for do-main adaptation. IEEE Transactions on Pattern Anal-ysis and Machine Intelligence.

Dietterich, Thomas G, Lathrop, Richard H, & Lozano-Perez, Tomas. 1997. Solving the multiple instanceproblem with axis-parallel rectangles. Artificial intel-ligence, 89(1-2), 31–71.

Flaxman, Seth R, Wang, Yu-Xiang, & Smola, Alexan-der J. 2015. Who supported Obama in 2012?: Ecolog-ical inference through distribution regression. Pages289–298 of: Proceedings of the 21th ACM SIGKDDInternational Conference on Knowledge Discovery andData Mining. ACM.

Fournier, Nicolas, & Guillin, Arnaud. 2015. On therate of convergence in Wasserstein distance of theempirical measure. Probability Theory and RelatedFields, 162(3-4), 707–738.

Frogner, C., Zhang, C., Mobahi, H., Araya, M., &Poggio, T. 2015. Learning with a Wasserstein Loss.In: NIPS.

Genevay, A., Peyre, G., & Cuturi, M. 2017. LearningGenerative Models with Sinkhorn Divergences. ArXive-prints, June.

Gretton, Arthur, Borgwardt, Karsten M, Rasch,Malte, Scholkopf, Bernhard, & Smola, Alex J. 2007.A kernel method for the two-sample-problem. Pages513–520 of: Advances in neural information processingsystems.

Gretton, Arthur, Borgwardt, Karsten M, Rasch,Malte J, Scholkopf, Bernhard, & Smola, Alexander.2012. A kernel two-sample test. Journal of MachineLearning Research, 13(Mar), 723–773.

Haasdonk, Bernard, & Bahlmann, Claus. 2004. Learn-ing with distance substitution kernels. Pages 220–227of: Joint Pattern Recognition Symposium. Springer.

Hein, Matthias, & Bousquet, Olivier. 2005. Hilbertianmetrics and positive definite kernels on probabilitymeasures. Pages 136–143 of: AISTATS.

Jebara, Tony, Kondor, Risi, & Howard, Andrew. 2004.Probability product kernels. Journal of MachineLearning Research, 5(Jul), 819–844.

Muandet, Krikamol, Fukumizu, Kenji, Dinuzzo,Francesco, & Scholkopf, Bernhard. 2012. Learningfrom distributions via support measure machines.Pages 10–18 of: Advances in neural information pro-cessing systems.

Nguyen, XuanLong, Wainwright, Martin J, & Jordan,Michael I. 2007. Nonparametric estimation of thelikelihood ratio and divergence functionals. Pages2016–2020 of: Information Theory, 2007. ISIT 2007.IEEE International Symposium on. IEEE.

Nguyen, XuanLong, Wainwright, Martin J, & Jordan,Michael I. 2010. Estimating divergence functionalsand the likelihood ratio by convex risk minimization.IEEE Transactions on Information Theory, 56(11),5847–5861.

Ntampaka, Michelle, Trac, Hy, Sutherland, Dougal J,Battaglia, Nicholas, Poczos, Barnabas, & Schneider,Jeff. 2015. A machine learning approach for dynamicalmass measurements of galaxy clusters. The Astrophys-ical Journal, 803(2), 50.

Poczos, Barnabas, Xiong, Liang, Sutherland, Dou-gal J, & Schneider, Jeff. 2012. Nonparametric ker-nel estimators for image classification. Pages 2989–2996 of: Computer Vision and Pattern Recognition(CVPR), 2012 IEEE Conference on. IEEE.

Poczos, Barnabas, Singh, Aarti, Rinaldo, Alessandro,& Wasserman, Larry A. 2013. Distribution-Free Dis-tribution Regression. Pages 507–515 of: AISTATS.

Ramdas, Aaditya, Reddi, Sashank Jakkam, Poczos,Barnabas, Singh, Aarti, & Wasserman, Larry A. 2015.On the decreasing power of kernel and distance basednonparametric hypothesis tests in high dimensions.Pages 3571–3577 of: AAAI.

Scholkopf, Bernhard, & Smola, Alexander J. 2002.Learning with kernels: support vector machines, regu-larization, optimization, and beyond. MIT press.

Alain Rakotomamonjy, Abraham Traore, Maxime Berar, Remi Flamary, Nicolas Courty

Sriperumbudur, Bharath K, Fukumizu, Kenji, Gret-ton, Arthur, Scholkopf, Bernhard, & Lanckriet,Gert RG. 2010. Non-parametric estimation of integralprobability metrics. Pages 1428–1432 of: InformationTheory Proceedings (ISIT), 2010 IEEE InternationalSymposium on. IEEE.

Sutherland, Dougal J, Xiong, Liang, Poczos,Barnabas, & Schneider, Jeff. 2012. Kernels on samplesets via nonparametric divergence estimates. arXivpreprint arXiv:1202.0302.

Villani, C. 2009. Optimal transport: old and new.Grund. der mathematischen Wissenschaften. Springer.

Weed, Jonathan, & Bach, Francis. 2017. Sharp asymp-totic and finite-sample rates of convergence of empiri-cal measures in Wasserstein distance. arXiv preprintarXiv:1707.00087.