Normalized Wasserstein for Mixture Distributions with Applications inAdversarial Learning and Domain Adaptation

Yogesh BalajiDepartment of Computer Science

University of Marylandyogesh@cs.umd.edu

Rama ChellappaUMIACS

University of Marylandrama@umiacs.umd.edu

Soheil FeiziDepartment of Computer Science

University of Marylandsfeizi@cs.umd.edu

Abstract

Understanding proper distance measures between dis-tributions is at the core of several learning tasks such asgenerative models, domain adaptation, clustering, etc. Inthis work, we focus on mixture distributions that arise nat-urally in several application domains where the data con-tains different sub-populations. For mixture distributions,established distance measures such as the Wasserstein dis-tance do not take into account imbalanced mixture propor-tions. Thus, even if two mixture distributions have identicalmixture components but different mixture proportions, theWasserstein distance between them will be large. This oftenleads to undesired results in distance-based learning meth-ods for mixture distributions. In this paper, we resolve thisissue by introducing the Normalized Wasserstein measure.The key idea is to introduce mixture proportions as opti-mization variables, effectively normalizing mixture propor-tions in the Wasserstein formulation. Using the proposednormalized Wasserstein measure leads to significant per-formance gains for mixture distributions with imbalancedmixture proportions compared to the vanilla Wassersteindistance. We demonstrate the effectiveness of the proposedmeasure in GANs, domain adaptation and adversarial clus-tering in several benchmark datasets.

1. Introduction

Quantifying distances between probability distributionsis a fundamental problem in machine learning and statis-tics with several applications in generative models, domainadaptation, clustering, etc. Popular probability distancemeasures include optimal transport measures such as theWasserstein distance [22] and divergence measures such asthe Kullback-Leibler (KL) divergence [4].

Classical distance measures, however, can lead to someissues for mixture distributions. A mixture distribution isthe probability distribution of a random variable X where

X = Xi with probability πi for 1 ≤ i ≤ k. k is thenumber of mixture components and π = [π1, ..., πk]T is thevector of mixture (or mode) proportions. The probabilitydistribution of each Xi is referred to as a mixture compo-nent (or, a mode). Mixture distributions arise naturally indifferent applications where the data contains two or moresub-populations. For example, image datasets with differ-ent labels can be viewed as a mixture (or, multi-modal) dis-tribution where samples with the same label characterize aspecific mixture component.

If two mixture distributions have exactly same mixturecomponents (i.e. same Xi’s) with different mixture pro-portions (i.e. different π’s), classical distance measures be-tween the two will be large. This can lead to undesired re-sults in several distance-based machine learning methods.To illustrate this issue, consider the Wasserstein distancebetween two distributions PX and PY , defined as [22]

W (PX ,PY ) := minPX,Y

E [‖X − Y ‖] , (1)

marginalX(PX,Y ) = PX , marginalY (PX,Y ) = PY

where PX,Y is the joint distribution (or coupling) whosemarginal distributions are equal to PX and PY . When noconfusion arises and to simplify notation, in some equa-tions, we use W (X,Y ) notation instead of W (PX ,PY ).

The Wasserstein distance optimization is over all jointdistributions (couplings) PX,Y whose marginal distribu-tions match exactly with input distributions PX and PY .This requirement can cause issues when PX and PY aremixture distributions with different mixture proportions. Inthis case, due to the marginal constraints, samples belong-ing to very different mixture components will have to becoupled together in PX,Y (e.g. Figure 1(a)). Thus, usingthis distance measure can then lead to undesirable outcomesin problems such as domain adaptation. This motivates theneed for developing a new distance measure to take into ac-count mode imbalances in mixture distributions.

In this paper, we propose a new distance measure thatresolves the issue of imbalanced mixture proportions for

1

arX

iv:1

902.

0041

5v2

[cs

.LG

] 2

9 O

ct 2

019

P1 P2

Figure 1. An illustration of the effectiveness of the proposed Normalized Wasserstein measure in domain adaptation. The source domain(shown in red) and the target domain (shown in blue) have two modes with different mode proportions. (a) The couplings computed byestimating Wasserstein distance between source and target distributions (shown in yellow lines) match several samples from incorrect anddistant mode components. (b,c) Our proposed normalized Wasserstein measure (3) constructs intermediate mixture distributions P1 and P2

(shown in green) with similar mixture components to source and target distributions, respectively, but with optimized mixture proportions.This significantly reduces the number of couplings between samples from incorrect modes and leads to 42% decrease in target loss indomain adaptation compared to the baseline.

multi-modal distributions. Our developments focus on aclass of optimal transport measures, namely the Wasser-stein distance Eq (1). However, our ideas can be extendednaturally to other distance measures (eg. adversarial dis-tances [6]) as well.

Let G be an array of generator functions with k compo-nents defined as G := [G1, ...,Gk]. Let PG,π be a mix-ture probability distribution for a random variable X whereX = Gi(Z) with probability πi for 1 ≤ i ≤ k. Throughoutthe paper, we assume that Z has a normal distribution.

By relaxing the marginal constraints of the classicalWasserstein distance (1), we introduce the NormalizedWasserstein measure (NW measure) as follows:

WN (PX ,PY )

:= minG,π(1),π(2)

W (PX ,PG,π(1)) +W (PY ,PG,π(2)).

There are two key ideas in this definition that help re-solve mode imbalance issues for mixture distributions.First, instead of directly measuring the Wasserstein dis-tance between PX and PY , we construct two intermediate(and potentially mixture) distributions, namely PG,π(1) andPG,π(2) . These two distributions have the same mixturecomponents (i.e. same G) but can have different mixtureproportions (i.e. π(1) and π(2) can be different). Second,mixture proportions, π(1) and π(2), are considered as op-timization variables. This effectively normalizes mixtureproportions before Wasserstein distance computations. Seean example in Figure 1 (b, c) for a visualization of PG,π(1)

and PG,π(2) , and the re-normalization step.In this paper, we show the effectiveness of the proposed

Normalized Wasserstein measure in three application do-mains. In each case, the performance of our proposed

method significantly improves against baselines when inputdatasets are mixture distributions with imbalanced mixtureproportions. Below, we briefly highlight these results:

Domain Adaptation: In Section 4, we formulate theproblem of domain adaptation as minimizing the normal-ized Wasserstein measure between source and target fea-ture distributions. On classification tasks with imbalanceddatasets, our method significantly outperforms baselines(e.g. ∼ 20% gain in synthetic to real adaptation on VISDA-3 dataset).

GANs: In Section 5, we use the normalized Wassersteinmeasure in GAN’s formulation to train mixture models withvarying mode proportions. We show that such a generativemodel can help capture rare modes, decrease the complexityof the generator, and re-normalize an imbalanced dataset.

Adversarial Clustering: In Section 6, we formulatethe clustering problem as an adversarial learning task usingNormalized Wasserstein measure.

2. Normalized Wasserstein Measure

In this section, we introduce the normalized Wassersteinmeasure and discuss its properties. Recall that G is an arrayof generator functions defined as G := [G1, ...,Gk] whereGi : Rr → Rd. Let G be the set of all possible G functionarrays. Let π be a discrete probability mass function withk elements, i.e. π = [π1, π2, · · · , πk] where πi ≥ 0 and∑i πi = 1. Let Π be the set of all possible π’s.Let PG,π be a mixture distribution, i.e. it is the proba-

bility distribution of a random variable X such that X =Gi(Z) with probability πi for 1 ≤ i ≤ k. We assume thatZ has a normal density, i.e. Z ∼ N (0, I). We refer to Gand π as mixture components and proportions, respectively.

The set of all such mixture distributions is defined as:

PG,k := {PG,π : G ∈ G, π ∈ Π} (2)

where k is the number of mixture components. Given twodistributions PX and PY belonging to the family of mixturedistributions PG,k, we are interested in defining a distancemeasure agnostic to differences in mode proportions, butsensitive to shifts in mode components, i.e., the distancefunction should have high values only when mode compo-nents of PX and PY differ. If PX and PY have the samemode components but differ only in mode proportions, thedistance should be low.

The main idea is to introduce mixture proportions as op-timization variables in the Wasserstein distance formulation(1). This leads to the following distance measure which werefer to as the Normalized Wasserstein measure (NW mea-sure), WN (PX ,PY ), defined as:

minG,π(1),π(2)

W (PX ,PG,π(1)) +W (PY ,PG,π(2)) (3)

k∑j=1

π(i)j = 1 i = 1, 2,

π(i)j ≥ 0 1 ≤ j ≤ k, i = 1, 2.

Since the normalized Wasserstein’s optimization (3) in-cludes mixture proportions π(1) and π(2) as optimizationvariables, if two mixture distributions have similar mix-ture components with different mixture proportions (i.e.PX = PG,π(1) and PY = PG,π(2) ), although the Wasser-stein distance between the two can be large, the introducednormalized Wasserstein measure between the two will bezero. Note that WN is defined with respect to a set of gen-erator functions G = [G1, ...,Gk]. However, to simplifythe notation, we make this dependency implicit. We wouldlike to point our that our proposed NW measure is a semi-distance measure (and not a distance) since it does not sat-isfy all properties of a distance measure. Please refer toAppendix for more details.

To compute the NW measure, we use an alternating gra-dient descent approach similar to the dual computation ofthe Wasserstein distance [1]. Moreover, we impose the πconstraints using a soft-max function. Please refer to Ap-pendix. C for more details.

To illustrate how NW measure is agnostic to mode im-balances between distributions , consider an unsuperviseddomain adaptation problem with MNIST-2 (i.e. a datasetwith two classes: digits 1 and 2 from MNIST) as the sourcedataset, and noisy MNIST-2 (i.e. a noisy version of it) asthe target dataset (details of this example is presented inSection 4.2). The source dataset has 4/5 digits one and 1/5digits two, while the target dataset has 1/5 noisy digits oneand 4/5 noisy digits two. The couplings produced by esti-

mating the Wasserstein distance between the two distribu-tions is shown in yellow lines in Figure 1-a. We observethat there are many couplings between samples from in-correct mixture components. The normalized Wassersteinmeasure, on the other hand, constructs intermediate mode-normalized distributions P1 and P2, which get coupled tothe correct modes of source and target distributions, respec-tively (see panels (b) and (c) in Figure 1)).

3. Theoretical ResultsFor NW measure to work effectively, the number of

modes k in NW formulation (Eq. (3)) must be chosen appro-priately. For instance, given two mixture distributions withk components each, Normalized Wasserstein measure with2k modes would always give 0 value. In this section, weprovide some theoretical conditions under which the num-ber of modes can be estimated accurately. We begin bymaking the following assumptions for two mixture distri-butions X and Y whose NW distance we wish to compute:

• (A1) If mode i in distribution X and mode j in distri-bution Y belong to the same mixture component, thentheir Wasserstein distance is ≤ ε i.e., if Xi and Yj cor-respond to the same component, W (PXi ,PYj ) < ε.

• (A2) The minimum Wasserstein distance between anytwo modes of one mixture distribution is at least δ i.e.,W (PXi ,PXj ) > δ and W (PYi ,PYj ) > δ ∀i 6= j.Also, non-overlapping modes between X and Y areseparated by δ i.e., for non-overlapping modes Xi andYj , W (PXi

,PYj) > δ. This ensures that modes are

well-separated.

• (A3) We assume that each mode Xi and Yi have den-sity at least η i.e., PXi ≥ η ∀i, PYi ≥ η ∀i. Thisensures that every mode proportion is at least η.

• (A4) Each generator Gi is powerful enough to captureexactly one mode of distribution PX or PY .

Theorem 1 Let PX and PY be two mixture distributionssatisfying (A1)-(A4) with n1 and n2 mixture components,respectively, where r of them are overlapping. Let k∗ =n1 + n2 − r. Then, k∗ is smallest k for which NW (k) issmall (O(ε)) and NW (k)−NW (k− 1) is relatively large(in the O(δη) )

The proof is presented in Appendix. A. All assumptionsmade are reasonable and hold in most practical situations:(A1)-(A3) enforces that non-overlapping modes in mixturedistribitions are separated, and overlapping modes are closein Wasserstein distance. To enforce (A4), we need to pre-vent multi-mode generation in one mode of G. This canbe satisfied by using the regularizer in Eq. (11). Note thatin the above theorem, k∗ is the optimal k that should be

used in the Normalized Wasserstein formulation. The theo-rem presents a way to estimate k∗. Please refer to Section 7for experimental results. In many applications like domainadaption, however, the number of components k is knownbeforehand, and this step can be skipped.

4. Normalized Wasserstein for Domain Adap-tation under covariate and label shift

In this section, we demonstrate the effectiveness of theNW measure in Unsupervised Domain Adaptation (UDA)both for supervised (e.g. classification) and unsupervised(e.g. denoising) tasks. Note that the term unsupervised inUDA means that the label information in the target domainis unknown while unsupervised tasks mean that the labelinformation in the source domain is unknown.

First, we consider domain adaptation for a classifica-tion task. Let (Xs, Ys) represent the source domain while(Xt, Yt) denote the target domain. Since we deal with theclassification setup, we have Ys, Yt ∈ {1, 2, ..., k}. A com-mon formulation for the domain adaptation problem is totransform Xs and Xt to a feature space where the distancebetween the source and target feature distributions is suffi-ciently small, while a good classifier can be computed forthe source domain in that space [6]. In this case, one solvesthe following optimization:

minf∈F

Lcl (f(Xs), Ys) + λ dist (f(Xs), f(Xt)) (4)

where λ is an adaptation parameter and Lcl is the empiri-cal classification loss function (e.g. the cross-entropy loss).The distance function between distributions can be adver-sarial distances [6, 21], the Wasserstein distance [20], orMMD-based distances [14, 15].

When Xs and Xt are mixture distributions (which isoften the case as each label corresponds to one mixturecomponent) with different mixture proportions, the use ofthese classical distance measures can lead to the computa-tion of inappropriate transformation and classification func-tions. In this case, we propose to use the NW measureas the distance function. Computing the NW measure re-quires training mixture components G and mode propor-tions π(1), π(2). To simplify the computation, we make useof the fact that labels for the source domain (i.e. Ys) areknown, thus source mixture components can be identifiedusing these labels. Using this information, we can avoidthe need for computing G directly and use the conditionalsource feature distributions as a proxy for the mixture com-ponents as follows:

Gi(Z)dist= f(X(i)

s ), (5)

X(i)s = {Xs|Ys = i}, ∀1 ≤ i ≤ k,

where dist= means matching distributions. Using (5), the for-

mulation for domain adaptation can be written as

minf∈F

minπ

Lcl (Xs, Ys) + λW

(∑i

π(i)f(X(i)s ), f(Xt)

).

(6)

The above formulation can be seen as a version of instanceweighting as source samples in X(i)

s are weighted by πi.Instance weighting mechanisms have been well studied fordomain adaptation [23, 24]. However, different from theseapproaches, we train the mode proportion vector π in anend-to-end fashion using neural networks and integrate theinstance weighting in a Wasserstein optimization. Of morerelevance to our work is the method proposed in [3], wherethe instance weighting is trained end-to-end in a neural net-work. However, in [3], instance weights are maximizedwith respect to the Wasserstein loss, while we show that themixture proportions need to minimized to normalize modemismatches. Moreover, our NW measure formulation canhandle the case when mode assignments for source embed-dings are unknown (as we discuss in Section 4.2). This casecannot be handled by the approach presented in [3].

For unsupervised tasks when mode assignments forsource samples are unknown, we cannot use the simplifiedformulation of (5). In that case, we use a domain adaptationmethod solving the following optimization:

minf∈F

Lunsup (Xs) + λWN (f(Xs), f(Xt)) , (7)

where Lunsup(Xs) is the loss corresponding to the desiredunsupervised learning task on the source domain data.

4.1. UDA for supervised tasks

4.1.1 MNIST→MNIST-M

In the first set of experiments1, we consider adaptation be-tween MNIST→ MNIST-M datasets. We consider threesettings with imbalanced class proportions in source andtarget datasets: 3 modes, 5 modes, and 10 modes. Moredetails can be found in Table. 9 of Appendix.

We use the same architecture as [6] for feature networkand discriminator. We compare our method with the follow-ing approaches: (1) Source-only which is a baseline modeltrained only on source domain with no domain adaptationperformed, (2) DANN [6], a method where adversarial dis-tance between source and target distibutions is minimized,and (3) Wasserstein [20] where Wasserstein distance be-tween source and target distributions is minimized. Table 1summarizes our results of this experiment. We observe thatperforming domain adaptation using adversarial distance

1Code available at https://github.com/yogeshbalaji/Normalized-Wasserstein

and Wasserstein distance leads to decrease in performancecompared to the baseline model. This is an outcome of notaccounting for mode imbalances, thus resulting in negativetransfer, i.e., samples belonging to incorrect classes are cou-pled and getting pushed to be close in the embedding space.Our proposed NW measure, however, accounts for modeimbalances and leads to a significant boost in performancein all three settings.

Table 1. Mean classification accuracies (in %) averaged over 5runs on imbalanced MNIST→MNIST-M adaptation

Method 3 modes 5 modes 10 modesSource only 66.63 67.44 63.17

DANN 62.34 57.56 59.31Wasserstein 61.75 60.56 58.22

NW 75.06 76.16 68.57

4.1.2 VISDA

In the experiment of Section 4.1.1 on digits dataset, mod-els have been trained from scratch. However, a commonpractice used in domain adaptation is to transfer knowledgefrom a pretrained network (eg. models trained on Ima-geNet) and fine-tune on the desired task. To evaluate theperformance of our approach in such settings, we consideradaptation on the VISDA dataset [18]; a recently proposedbenchmark for adapting from synthetic to real images.

We consider a subset of the entire VISDA dataset con-taining the following three classes: aeroplane, horse andtruck. The source domain contains (0.55, 0.33, 0.12) frac-tion of samples per class, while that of the target domainis (0.12, 0.33, 0.55). We use a Resnet-18 model pre-trainedon ImageNet as our feature network. As shown in Table 2,our approach significantly improves the domain adaptationperformance over the baseline and other compared methods.

Table 2. Mean classification accuracies (in %) averaged over 5runs on synthetic to real adaptation on VISDA dataset (3 classes)

Method Accuracy (in %)Source only 53.19

DANN 68.06Wasserstein 64.84

NW 73.23

4.1.3 Mode balanced datasets

The previous two experiments demonstrated the effective-ness of our method when datasets are imbalanced. In thissection, we study the case where source and target domainshave mode-balanced datasets – the standard setting con-sidered in the most domain adaptation methods. We per-form experiment on MNIST→MNIST-M adaptation using

the entire dataset. Table 3 reports the results obtained. Weobserve that our approach performs on-par with the stan-dard wasserstein distance minimization.

Table 3. Domain adaptation on mode-balanced datasets:MNIST→MNIST-M. Average classification accuracies averagedover 5 runs are reported

Method Accuracy (in %)Source only 60.22

DANN 85.24Wasserstein 83.47

NW 84.16

4.2. UDA for unsupervised tasks

For unsupervised tasks on mixture datasets, we use theformulation of Eq (7) to perform domain adaptation. Toempirically validate this formulation, we consider the im-age denoising problem. The source domain consists of dig-its {1, 2} from MNIST dataset as shown in Fig 2(a). Notethat the color of digit 2 is inverted. The target domain is anoisy version of the source, i.e. source images are perturbedwith random i.i.d Gaussian noiseN (0.4, 0.7) to obtain tar-get images. Our dataset contains 5, 000 samples of digit1 and 1, 000 samples of digit 2 in the source domain, and1, 000 samples of noisy digit 1 and 5, 000 samples of noisydigit 2 in the target. The task is to perform image denois-ing by dimensionaly reduction, i.e., given a target domainimage, we need to reconstruct the corresponding clean im-age that looks like the source. We assume that no (source,target) correspondence is available in the dataset.

To perform denoising when the (source, target) corre-spondence is unavailable, a natural choice would be to min-imize the reconstruction loss in source while minimizingthe distance between source and target embedding distribu-tions. We use the NW measure as our choice of distancemeasure. This results in the following optimization:

minf,g

Ex∼Xs‖g(f(x))− x‖22 + λWN (f(Xs), f(Xt))

where f(.) is the encoder and g(.) is the decoder.As our baseline, we consider a model trained only on

source using a quadratic reconstruction loss. Fig 2(b) showssource and target embeddings produced by this baseline. Inthis case, the source and the target embeddings are distantfrom each other. However, as shown in Fig 2(c), using theNW formulation, the distributions of source and target em-beddings match closely (with estimated mode proportions) .We measure the L2 reconstruction loss of the target domain,errrecons,tgt = Ex∼Xt‖g(f(x)) − x‖22, as a quantitativeevaluation measure. This value for different approaches isshown in Table 4. We observe that our method outperformsthe compared approaches.

Figure 2. Domain adaptation for image denoising. (a) Samples from source and target domains. (b) Source and target embeddings learntby the baseline model. (c) Source and target embeddings learnt by minimizing the proposed NW measure. In (b) and (c), red and greenpoints indicate source and target samples, respectively.

Table 4. errrecons,tgt for an image denoising taskMethod errrecons,tgt

Source only 0.31Wasserstein 0.52

NW 0.18Training on target (Oracle) 0.08

5. Normalized Wasserstein GAN

Learning a probability model from data is a fundamen-tal problem in statistics and machine learning. Building onthe success of deep learning, a recent approach to this prob-lem is using Generative Adversarial Networks (GANs) [8].GANs view this problem as a game between a generatorwhose goal is to generate fake samples that are close to thereal data training samples, and a discriminator whose goalis to distinguish between the real and fake samples.

Most GAN frameworks can be viewed as methods thatminimize a distance between the observed probability dis-tribution, PX , and the generative probability distribution,PY , where Y = G(Z). G is referred to as the gener-ator function. In several GAN formulations, the distancebetween PX and PY is formulated as another optimizationwhich characterizes the discriminator. Several GAN archi-tectures have been proposed in the last couple of years. Asummarized list includes GANs based on optimal trans-port measures (e.g. Wasserstein GAN+Weight Clipping[1], WGAN+Gradient Penalty [9]), GANs based on diver-gence measures (e.g. the original GAN’s formulation [8],DCGAN [19], f -GAN [17]), GANs based on moment-matching (e.g. MMD-GAN [5, 11]), and other formula-tions (e.g. Least-Squares GAN [16], BigGAN [2], etc.)

If the observed distribution PX is a mixture one, the pro-posed normalized Wasserstein measure (3) can be used tocompute a generative model. Instead of estimating a singlegenerator G as done in standard GANs, we estimate a mix-ture distribution PG,π using the proposed NW measure. Werefer to this GAN as the Normalized Wasserstein GAN (orNWGAN) formulated as the following optimization:

minG,π

WN (PX ,PG,π). (8)

In this case, the NW distance simplifies as

minG,π

WN (PX ,PG,π)

= minG,π

minG′,π(1),π(2)

W (PX ,PG′,π(1)) +W (PG,π,PG′,π(2))

= minG,π

W (PX ,PG,π). (9)

There are couple of differences between the proposedNWGAN and the existing GAN architecures. The gener-ator in the proposed NWGAN is a mixture of k models,each producing πi fraction of generated samples. We se-lect k a priori based on the application domain while π iscomputed within the NW distance optimization. Modelingthe generator as a mixture of k neural networks has alsobeen investigated in some recent works [10, 7]. However,these methods assume that the mixture proportions π areknown beforehand, and are held fixed during the training. Incontrast, our approach is more general as the mixture pro-portions are also optimized. Estimating mode proportionshave several important advantages: (1) we can estimate raremodes, (2) an imbalanced dataset can be re-normalized, (3)by allowing each Gi to focus only on one part of the distri-bution, the quality of the generative model can be improved

NWGAN WGAN MGAN

Figure 3. Mixture of Gaussian experiments. In all figures, red points indicate samples from the real data distribution while blue pointsindicate samples from the generated distribution. NWGAN is able to capture rare modes in the data and produces a significantly bettergenerative model than other methods.

while the complexity of the generator can be reduced. Inthe following, we highlight these properties of NWGAN ondifferent datasets.

5.1. Mixture of Gaussians

First, we present the results of training the NWGAN ona two dimensional mixture of Gaussians. The input datais a mixure of 9 Gaussians, each centered at a vertex of a3 × 3 grid as shown in Figure 3. The mean and the covari-ance matrix for each mode are randomly chosen. The modeproportion for mode i is chosen as πi = i

45 for 1 ≤ i ≤ 9.Generations produced by NWGAN using k = 9 affine

generator models on this dataset is shown in Figure 3. Wealso compare our method with WGAN [1] and MGAN [10].Since MGAN does not optimize over π, we assume uniformmode proportions (πi = 1/9 for all i). To train WGAN, anon-linear generator function is used since a single affinefunction cannot model a mixture of Gaussian distribution.

To evaluate the generative models, we report the follow-ing quantitative scores: (1) the average mean error whichis the mean-squared error (MSE) between the mean vectorsof real and generated samples per mode averaged over allmodes, (2) the average covariance error which is the MSEbetween the covariance matrices of real and generated sam-ples per mode averaged over all modes, and (3) the π esti-mation error which is the normalized MSE between the πvector of real and generated samples. Note that comput-ing these metrics require mode assignments for generatedsamples. This is done based on the closeness of generativesamples to the ground-truth means.

We report these error terms for different GANs in Ta-ble 5. We observe that the proposed NWGAN achieves bestscores compared to the other two approaches. Also, fromFigure 3, we observe that the generative model trained byMGAN misses some of the rare modes in the data. This isbecause of the error induced by assuming fixed mixture pro-

portions when the ground-truth π is non-uniform. Since theproposed NWGAN estimates π in the optimization, evenrare modes in the data are not missed. This shows the im-portance of estimating mixture proportions specially whenthe input dataset has imbalanced modes.

Table 5. Quantitative Evaluation on Mixture of GaussiansMethod Avg. µ error Avg. Σ error π errorWGAN 0.007 0.0003 0.0036MGAN 0.007 0.0002 0.7157

NWGAN 0.002 0.0001 0.0001

5.2. A Mixture of CIFAR-10 and CelebA

One application of learning mixture generative modelsis to disentangle the data distribution into multiple compo-nents where each component represents one mode of theinput distribution. Such disentanglement is useful in manytasks such as clustering (Section 6). To test the effective-ness of NWGAN in performing such disentanglement, weconsider a mixture of 50, 000 images from CIFAR-10 and100, 000 images from CelebA [12] datasets as our input dis-tribution. All images are reshaped to be 32× 32.

To highlight the importance of optimizing mixture pro-portion to produce disentangled generative models, wecompare the performance of NWGAN with a variation ofNWGAN where the mode proportion π is held fixed asπi = 1

k (the uniform distribution). Sample generations pro-duced by both models are shown in Figure 4. When π isheld fixed, the model does not produce disentangled repre-sentations (in the second mode, we observe a mix of CI-FAR and CelebA generative images.) However, when weoptimize π, each generator produces distinct modes.

Fixing π Learning π

Figure 4. Sample generations of NWGAN with k = 2 on a mixture of CIFAR-10 and CelebA datasets for fixed and optimized π’s. Whenπ is fixed, one of the generators produces a mix of CIFAR and CelebA generative images (boxes in red highlight some of the CelebAgenerations in the model producing CIFAR+CelebA). However, when π is optimized, the model produces disentangled representations.

6. Adversarial ClusteringIn this section, we use the proposed NW measure to for-

mulate an adversarial clustering approach. More specif-ically, let the input data distribution have k underlyingmodes (each representing a cluster), which we intend torecover. The use of deep generative models for perform-ing clustering has been explored in [25] (using GANs)and [13](using VAEs). Different from these, our approachmakes use of the proposed NWGAN for clustering, and thusexplicitly handles data with imbalanced modes.

Let PX be observed empirical distribution. Let G∗ andπ∗ be optimal solutions of NWGAN optimization (9). Fora given point xi ∼ PX , the clustering assignment is com-puted using the closest distance to a mode i.e.,

C(xi) = arg min1≤j≤k

minZ

[‖xi −Gj(Z)‖2

]. (10)

To perform an effective clustering, we require each modeGj to capture one mode of the data distribution. Withoutenforcing any regularization and using rich generator func-tions, one model can capture multiple modes of the datadistribution. To prevent this, we introduce a regularizationterm that maximizes the weighted average Wasserstein dis-tances between different generated modes. That is,

R =∑

(i,j)|i>j

πiπjW (Gi(Z),Gj(Z)) . (11)

This term encourages diversity among generative modes.With this regularization term, the optimization objective ofa regularized NWGAN becomes

minG,π

W (PX ,PG,π)− λregR

where λreg is the regularization parameter.We test the proposed adversarial clustering method on an

imbalanced MNIST dataset with 3 digits containing 3, 000

samples of digit 2, 1, 500 samples of digit 4 and 6, 000 sam-ples of digit 6. We compare our approach with k-meansclustering and Gaussian Mixture Model (GMM) in Table 6.Cluster purity, NMI and ARI scores are used as quantitativemetrics (refer to Appendix E.3 for more details). We ob-serve that our clustering technique is able to achieve goodperformance over the compared approaches.

Table 6. Clustering results on Imbalanced MNIST datasetMethod Cluster Purity NMI ARIk-means 0.82 0.49 0.43GMM 0.75 0.28 0.33NW 0.98 0.94 0.97

7. Choosing the number of modesAs discused in Section 3, choosing the number of modes

(k) is crucial for computing NW measure. While this infor-mation is available for tasks such as domain adaptation, itis unknown for others like generative modeling. In this sec-tion, we experimentally validate our theoretically justifiedalgorithm for estimating k. Consider the mixture of Gaus-sian dataset with k = 9 modes presented in Section 5.1. Onthis dataset, the NWGAN model (with same architecture asthat used in Section 5.1) was trained with varying numberof modes k. For each setting, the NW measure between thegenerated and real data distribution is computed and plot-ted in Fig 5. We observe that k = 9 satisfies the conditiondiscussed in Theorem 1: optimal k∗ is the smallest k forwhich NW (k) is small, NW (k − 1) − NW (k) is large,and NW (k) saturates after k∗.

8. ConclusionIn this paper, we showed that Wasserstein distance, due

to its marginal constraints, can lead to undesired resultswhen when applied on imbalanced mixture distributions.

Figure 5. Choosing k: Plot of NW measure vs number of modes

To resolve this issue, we proposed a new distance measurecalled the Normalized Wasserstein. The key idea is to op-timize mixture proportions in the distance computation, ef-fectively normalizing mixture imbalance. We demonstratedthe usefulness of NW measure in three machine learningtasks: GANs, domain adaptation and adversarial clustering.Strong empirical results on all three problems highlight theeffectiveness of the proposed distance measure.

9. AcknowledgementsBalaji and Chellappa were supported by MURI pro-

gram from the Army Research Office (ARO) under thegrant W911NF17-1-0304. Feizi was supported by theUS National Science Foundation (NSF) under the grantCDS&E:1854532, and Capital One Services LLC.

References[1] Martin Arjovsky, Soumith Chintala, and Leon Bottou.

Wasserstein GAN. arXiv preprint arXiv:1701.07875,2017. 3, 6, 7, 11

[2] Andrew Brock, Jeff Donahue, and Karen Simonyan.Large scale GAN training for high fidelity natural im-age synthesis. CoRR, abs/1809.11096, 2018. 6

[3] Qingchao Chen, Yang Liu, Zhaowen Wang, Ian Was-sell, and Kevin Chetty. Re-weighted adversarial adap-tation network for unsupervised domain adaptation. InThe IEEE Conference on Computer Vision and PatternRecognition (CVPR), June 2018. 4

[4] Thomas M Cover and Joy A Thomas. Elements ofinformation theory. John Wiley & Sons, 2012. 1

[5] Gintare Karolina Dziugaite, Daniel M Roy, andZoubin Ghahramani. Training generative neural net-works via maximum mean discrepancy optimization.arXiv preprint arXiv:1505.03906, 2015. 6

[6] Yaroslav Ganin and Victor Lempitsky. Unsuperviseddomain adaptation by backpropagation. In Francis

Bach and David Blei, editors, Proceedings of the32nd International Conference on Machine Learning,volume 37 of Proceedings of Machine Learning Re-search, pages 1180–1189, Lille, France, 07–09 Jul2015. PMLR. 2, 4, 14

[7] Arnab Ghosh, Viveka Kulharia, Vinay P Nambood-iri, Philip HS Torr, and Puneet K Dokania. Multi-agent diverse generative adversarial networks. CoRR,abs/1704.02906, 6:7, 2017. 6

[8] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza,Bing Xu, David Warde-Farley, Sherjil Ozair, AaronCourville, and Yoshua Bengio. Generative adversarialnets. In Advances in neural information processingsystems, pages 2672–2680, 2014. 6

[9] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky,Vincent Dumoulin, and Aaron Courville. Improvedtraining of Wasserstein GANs. arXiv preprintarXiv:1704.00028, 2017. 6, 12, 14

[10] Quan Hoang, Tu Dinh Nguyen, Trung Le, and DinhPhung. MGAN: training generative adversarial netswith multiple generators. 2018. 6, 7, 14

[11] Yujia Li, Kevin Swersky, and Rich Zemel. Genera-tive moment matching networks. In Proceedings of the32nd International Conference on Machine Learning(ICML-15), pages 1718–1727, 2015. 6

[12] Ziwei Liu, Ping Luo, Xiaogang Wang, and XiaoouTang. Deep learning face attributes in the wild. InProceedings of International Conference on ComputerVision (ICCV), 2015. 7

[13] Francesco Locatello, Damien Vincent, Ilya O. Tol-stikhin, Gunnar Ratsch, Sylvain Gelly, and BernhardScholkopf. Clustering meets implicit generative mod-els. CoRR, abs/1804.11130, 2018. 8

[14] Mingsheng Long, Yue Cao, Jianmin Wang, andMichael I. Jordan. Learning transferable features withdeep adaptation networks. In Proceedings of the 32ndInternational Conference on Machine Learning, pages97–105, 2015. 4

[15] Mingsheng Long, Jianmin Wang, and Michael I. Jor-dan. Unsupervised domain adaptation with residualtransfer networks. CoRR, abs/1602.04433, 2016. 4

[16] Xudong Mao, Qing Li, Haoran Xie, Raymond YKLau, and Zhen Wang. Multi-class generative adversar-ial networks with the l2 loss function. arXiv preprintarXiv:1611.04076, 2016. 6

[17] Sebastian Nowozin, Botond Cseke, and RyotaTomioka. f-GAN: training generative neural sam-plers using variational divergence minimization. InAdvances in Neural Information Processing Systems,pages 271–279, 2016. 6

[18] Xingchao Peng, Ben Usman, Neela Kaushik, JudyHoffman, Dequan Wang, and Kate Saenko. Visda:The visual domain adaptation challenge. CoRR,abs/1710.06924, 2017. 5

[19] Alec Radford, Luke Metz, and Soumith Chintala. Un-supervised representation learning with deep convolu-tional generative adversarial networks. arXiv preprintarXiv:1511.06434, 2015. 6

[20] Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu.Wasserstein distance guided representation learningfor domain adaptation. In AAAI, pages 4058–4065.AAAI Press, 2018. 4

[21] Eric Tzeng, Judy Hoffman, Kate Saenko, and TrevorDarrell. Adversarial discriminative domain adapta-tion. In Computer Vision and Pattern Recognition(CVPR), volume 1, page 4, 2017. 4

[22] Cedric Villani. Optimal transport: old and new, vol-ume 338. Springer Science & Business Media, 2008.1

[23] Hongliang Yan, Yukang Ding, Peihua Li, QilongWang, Yong Xu, and Wangmeng Zuo. Mind the classweight bias: Weighted maximum mean discrepancyfor unsupervised domain adaptation. In 2017 IEEEConference on Computer Vision and Pattern Recog-nition, CVPR 2017, Honolulu, HI, USA, July 21-26,2017, pages 945–954, 2017. 4

[24] Yaoliang Yu and Csaba Szepesvari. Analysis of ker-nel mean matching under covariate shift. In Proceed-ings of the 29th International Conference on MachineLearning, ICML 2012, Edinburgh, Scotland, UK, June26 - July 1, 2012, 2012. 4

[25] Yang Yu and Wen-Ji Zhou. Mixture of gans for clus-tering. In Proceedings of the Twenty-Seventh Inter-national Joint Conference on Artificial Intelligence,IJCAI-18, pages 3047–3053. International Joint Con-ferences on Artificial Intelligence Organization, 2018.8

Appendices

A. Proof of Theorem 1For NW measure to normalize mode proportions appro-

priately, we need a good estimate of the number of modeproportions. Theorem 1 provides conditions under whichthe mode proportions can provable be estimated.

Let PX and PY be two mixture distributions whose NWmeasure we wish to compute. Let PX and PY have n1 andn2 modes respectively, with r modes overlapping. Let k∗ =n1 + n2 − r. We make the following assumptions

• (A1) If mode i in distribution X and mode j in distri-bution Y belong to the same mixture component, thentheir Wasserstein distance is ≤ ε i.e., if Xi and Yj cor-respond to the same component, W (PXi ,PYj ) < ε.

• (A2) The minimum Wasserstein distance between anytwo modes of one mixture distribution is at least δ i.e.,W (PXi

,PXj) > δ and W (PYi

,PYj) > δ ∀i 6= j.

Also, non-overlapping modes between X and Y areseparated by δ i.e., for non-overlapping modes Xi andYj , W (PXi

,PYj) > δ. This ensures that modes are

well-separated.

• (A3) We assume that each mode Xi and Yi have den-sity at least η i.e., PXi ≥ η ∀i, PYi ≥ η ∀i. Thisensures that every mode proportion is at least η.

• (A4) Each generator Gi is powerful enough to captureexactly one mode of distribution PX or PY .

Lemma 1 NW (k) is a monotonically decreasing functionwith respect to k.

This is because inNW (k+1), we add one additional modecompared to NW (k). If we have π(1), π(2) for this newmode to be 0 and give the same assignements as NW (k)to the rest of the modes, NW (k + 1) = NW (k). Sincecomputing NW (k) contains a minimization over mode as-signments, the NW (k + 1) ≤ NW (k)∀k. Hence, it ismonotonically decreasing.

Lemma 2 NW (k∗) ≤ ε

This is because at k = k∗, we can make the followingmode assignments.

• Assign n1+n2−r modes of NW to each of n1+n2−rnon-overlapping modes in PX and PY with the samemixture .

• Assign the remaining r modes of NW to the overlap-ping modes of either PX or PY . WLOG, let us assumewe assign them to r overlapping modes of PX .

• Choose π(1) to be same as π for PX , with 0 to non-overlapping components of PY

• Choose π(2) to be same as π for PY , with 0 to non-overlapping components of PX

Let us denote NOv(X) to be non-overlapping modesof X , Ov(X) to be overlapping modes of X , NOv(Y ) tobe non-overlapping modes of Y , and Ov(Y ) to be overlap-ping modes of Y . Then, under the mode assignments givenabove, NW (k∗) can be evaluated as,

WN (PX ,PY )

:= minG,π(1),π(2)

W (PX ,PG,π(1)) +W (PY ,PG,π(2)).

=∑

i∈NOv(X)

πXi W (PXi,PXi

) +∑

i∈Ov(X)

πXi W (PXi,PXi

)+

∑i∈NOv(Y )

πYi W (PYi,PYi

) +∑

i∈Ov(Y )

πYi W (PYi,PXi

)

= 0 + 0 + 0 +∑

i∈Ov(Y )

πYi W (PYi,PXi

)

≤ ε

The last step follows from (A1) i.e., overlapping modes areseparated by a Wasserstein distance of ε.

Lemma 3 NW (k∗ − 1) ≥ δ2η

By assumption (A2), we know that any two modes haveseparation of at least δ. In the distribution PX + PY , thereare n1 +n2− r unique cluster centers, each pair of clustersat a Wasserstein distance δ distance apart. In NW (k∗ − 1),generators have n1 +n2−r−1 modes, which is 1 less thanthe number of modes in PX + PY . Now, let us assume thatNW (k∗ − 1) < δ

2η. Then,

W (PX ,PG,π(1)) +W (PY ,PG,π(2)) <δ

2η

Since each mode of PX and PY has density at least η (by(A3)), the above condition can be satisfied only if

∀i ∈ [n1],∃j ∈ [k∗ − 1] s.t. W (PXi ,PGj ) <δ

2(12)

∀i ∈ [n2],∃j ∈ [k∗ − 1] s.t. W (PYi,PGj

) <δ

2(13)

Accounting for rmode overlap betweenX and Y , there willbe n1 + n2 − r unique constraints in Eq. (12) and Eq. (13).Since, G has only k∗ − 1 modes, by Pigeonhole principle,there should be at least one pair (i, j) that is matched to thesame Gj . WLOG, let us consider both i and j to belong toPX , although each can either belong to PX or PY . Then,

W (PXi,Gk) <

δ

2

W (PXj,Gk) <

δ

2

Then, by triangle inequality, W (PXi ,PXj ) < δ. This con-tradicts assumption (A2). Hence NW (k∗ − 1) ≥ δ

2η

Theorem 1 Let PX and PY be two mixture distributionssatisfying (A1)-(A4) with n1 and n2 mixture components,respectively, where r of them are overlapping. Let k∗ =n1 + n2 − r. Then, k∗ is smallest k for which NW (k) issmall (O(ε)) and NW (k)−NW (k− 1) is relatively large(in the O(δη) )

Proof: From Lemma 2 and Lemma 1, we know thatNW (k) ≤ ε ∀k ≥ k∗. Similarly, from Lemma 3 andLemma 1, we NW (k) ≥ δ

2η ∀k < k∗. Hence, k∗

is the smallest k for which NW (k) is small (O(ε)) andNW (k) − NW (k − 1) is relatively large (in the O(δη) ).Hence, proved.

B. Properties of Normalized Wasserstein mea-sure

The defined NW measure is not a distance because itdoes not satisfy the properties of a distance measure.

• In general, WN (PX ,PY ) 6= 0. However, ifPX ∈ PG,π , WN (PX ,PX) = 0. Moreover, if∃G, π s.t. WN (PG,π,PX) < ε (i.e., PG,π approxi-mates PX within ε factor), then WN (PX ,PX) ≤ 2ε.This follows from the definition of NW measure. So,when the class of generators are powerful enough, thisproperty is satisfied within 2ε approximation

• Normalized Wasserstein measure is symmetric.WN (PX ,PY ) = WN (PY ,PX)

• Normalized Wasserstein measure does not satisfy tri-angle inequality.

C. Optimizing Normalized Wasserstein usingduality

NW measure between two distributions PX and PY isdefined as

minG,π(1),π(2)

W (PX ,PG,π(1)) +W (PY ,PG,π(2))

Similar to [1], using the dual of Wasserstein distance, wecan write the above optimization as

minG,π(1),π(2)

[max

D1∈1−LipE[D1(X)]− E[

∑i

π(1)i D1(Gi(Z))]+

maxD2∈1−Lip

E[D2(Y )]− E[∑i

π(2)i D2(Gi(Z))]

](14)

Here, D1 andD2 are 1-Lipschitz functions, and π(1) andπ(2) are k−dimesional vectors lying in a simplex i.e.,∑

i

π(1)i = 1,

∑i

π(2)i = 1

To enforce these constraints, we use the softmax function asfollows.

π(1) = softmax(π(1)), π(2) = softmax(π(2))

The new variables π(1) and π(2) become optimization vari-ables. The softmax function ensures that the mixture prob-abilities π(1) and π(2) lie in a simplex.

The above equations are optimized using alternating gra-dient descent given by the following algorithm.

Algorithm 1 Optimizatizing Normalized Wasserstein1: Training iterations = Niter2: Critic iterations = Ncritic3: for t = 1 : Niter do4: Sample minibatch x ∼ PX , y ∼ PY5: Sample minibatch z ∼ N (0, 1)6: Compute Normalized Wasserstein as

NW = E[D1(x)]− E[∑i

π(1)i D1(Gi(z))]+

E[D2(y)]− E[∑i

π(2)i D2(Gi(z))]

7: for k = 1 : Ncritic do8: Maximize NW w.r.t D1 and D2

9: Minimize NW w.r.t π(1) and π(2)

10: end for11: Minimize NW w.r.t G12: end for

D. Comparative analysis of mixture distribu-tions

In this section, we propose a test using a combinationof Wasserstein distance and NW measure to identify if twomixture distributions differ in mode components or modeproportions. Such a test can provide better understandingwhile comparing mixture distributions. Suppose PX andPY are two mixture distributions with the same mixturecomponents but different mode proportions. I.e., PX andPY both belong to PG,k. In this case, depending on thedifference between π(1) and π(2), the Wasserstein distancebetween the two distributions can be arbitrarily large. Thus,using the Wasserstein distance, we can only conclude thatthe two distributions are different. In some applications,it can be informative to have a test that determines if twodistributions differ only in mode proportions. We propose

a test based on a combination of Wasserstein and the NWmeasure for this task. This procedure is shown in Table. 7.We note that computation of p-values for the proposed testis beyond the scope of this paper.

We demonstrate this test on 2D Mixture of Gaussians.We perform experiments on two settings, each involvingtwo datasetsD1 andD2, which are mixtures of 8 Gaussians:

Setting 1: Both D1 and D2 have same mode compo-nents, with the ith mode located at (r cos( 2πi

8 ), r sin( 2πi8 )).

Setting 2: D1 and D2 have shifted modecomponents. The ith mode of D1 is located at(r cos( 2πi

8 ), r sin( 2πi8 )), while the ith mode of D2 is

located at (r cos( 2πi+π8 ), r sin( 2πi+π

8 )).In both the settings, the mode fraction ofD1 is πi = i+2

52 ,and that ofD2 is πi = 11−i

52 . We use 2, 000 data points fromD1 and D2 to compute Wasserstein distance and the NWmeasure in primal form by solving a linear program. Thecomputed distance values are reported in Table 8. In setting1, we observe that the Wasserstein distance is large whilethe NW measure is small. Thus, one can conclude that thetwo distributions differ only in mode proportions. In setting2, both Wasserstein and NW measures are large. Thus, inthis case, distributions differ in mixture components as well.

E. Additional resultsE.1. CIFAR-10

We present the results of training NWGAN on CIFAR-10 dataset. We use WGAN-GP [9] with Resnet-based gen-erator and discriminator models as our baseline method.The proposed NWGAN was trained with k = 4 modes us-ing the same network architectures as the baseline. Sam-ple generations produced by each mode of the NWGAN isshown in Figure 6. We observe that each generator modelcaptures distinct variations of the entire dataset, thereby ap-proximately disentangling different modes in input images.For quantitative evaluation, we compute inception scoresfor the baseline and the proposed NWGAN. The inceptionscore for the baseline model is 7.56, whereas our modelachieved an improved score of 7.89.

E.2. Domain adaptation under uniform mode pro-portions

In this section, we present results on domain adaptationon mode-balanced VISDA dataset – source and target do-mains contain 3 classes - aeroplane, horse and truck withuniform mode proportion. The results of performing adap-tation using NW measure in comparison with classical dis-tance measures are reported in Table 10. We observe thatNW measure performs on-par with the compared methodson this dataset. This experiment demonstrates the effec-tiveness of NW measure on a range of settings – when thesource and target datasets are balanced in mode proportions,

Table 7. Comparative analysis of two mixture distributionsWasserstein

distanceNW

measure ConclusionHigh High Distributions differ in mode components

High LowDistributions have the same components,

but differ in mode proportionsLow Low Distributions are the same

Figure 6. Sample generations produced by the proposed NWGAN trained on CIFAR-10 with k = 4 generator modes.

Table 8. Hypothesis test between two MOG - D1 and D2

Setting Wasserstein Distance NW measureSetting 1 1.51 0.06Setting 2 1.56 0.44

NW becomes equivalent to Wasserstein distance and mini-mizing it is no worse than minimizing the classical distancemeasures. On the other hand, when mode proportions ofsource and target domains differ, NW measure renormal-izes the mode proportions and effectively performs domainadaptation. This illustrates the usefulness of NW measurein domain adaptation problems.

E.3. Adversarial clustering: Quantitative metrics

• Cluster purity: Cluster purity measures the extent towhich clusters are consistent i.e., if each cluster con-stains similar points or not. To compute the cluster pu-rity, the cardinality of the majority class is computedfor each cluster, and summed over and divided by thetotal number of samples.

• ARI - Adjusted Rand Index: The rand index computesthe similarity measure between two clusters by con-sidering all pairs of samples, and counting the pairsof samples having the same cluster in the ground-truthand predicted cluster assignments. Adjusted rand in-dex makes sure that ARI score is in the range (0, 1)

• NMI - Normalized Mutual Information: NMI is thenormalized version of the mutual information betweenthe predicted and the ground truth cluster assignments.

E.4. Adversarial clustering of CIFAR+CelebA

In this section, we show the results of performing ad-versarial clustering on a mixture of CIFAR-10 and CelebAdatasets. The same dataset presented in Section 3.2 of themain paper is used in this experiment (i.e) the dataset con-tains CIFAR-10 and CelebA samples in 1 : 2 mode pro-portion. NWGAN was trained with 2 modes - each em-ploying Resnet based generator-discriminator architectures(same architectures and hyper-parameters used in Section3.2 of main paper). Quantitative evaluation of our approachin comparison with k − means is given in Table 11. Weobserve that our approach outperforms k −means cluster-ing. However, the clustering quality is poorer that the oneobtained on imbalanced MNIST dataset. This is becausethe samples generated on MNIST dataset had much betterquality than the one produced on CIFAR-10. So, as long asthe underlying GAN model produces good generations, ouradversarial clustering algorithm performs well.

F. Architecture and hyper-parameters

Implementation details including model architecturesand hyperparameters are presented in this section:

F.1. Mixture models for Generative AdversarialNetworks (GANs)

F.1.1 Mixture of Gaussians

As discussed in Section 3.1 of the main paper, the inputdataset is a mixture of 8 Gaussians with varying mode pro-portion. Normalized Wasserstein GAN was trained withlinear generator and non-linear discriminator models using

Table 9. MNIST → MNIST-M settingsConfig 3 modes 5 modes 10 modesClasses {1, 4, 8} {0, 2, 4, 6, 8} {0, 1, . . . 9}

Proportion ofsource samples

{0.63, 0.31,

0.06}{0.33, 0.26, 0.2,

0.13, 0.06}{0.15, 0.15, 0.15, 0.12, 0.12

0.11, 0.05, 0.05, 0.05, 0.05}Proportion oftarget samples

{0.06, 0.31,

0.63}{0.06, 0.13, 0.2,

0.26, 0.33}{0.05, 0.05, 0.05, 0.05, 0.11

0.12, 0.12, 0.15, 0.15, 0.15}

Table 10. Domain adaptation on mode-balanced datasets: VISDA.Average classification accuracies averaged over 5 runs are reported

Method Classification accuracy (in %)Source only 63.24

DANN 84.71Wasserstein 90.08

Normalized Wasserstein 90.72

Table 11. Performance of clustering algorithms on CI-FAR+CelebA dataset

Method Cluster Purity NMI ARIk-means 0.667 0.038 0.049

Normalized Wasserstein 0.870 0.505 0.547

the architectures and hyper-parameters as presented in Ta-ble 12. The architecture used for training Vanilla WGANis provided in Table 13. The same architecture is used forMGAN, however we do not use theReLU non-linearities inthe Generator function (to make the generator affine so thatthe model is comparable to ours). For WGAN and MGAN,we use the hyper-parameter details as provided in the re-spective papers – [9] and [10].

F.1.2 CIFAR-10 + CelebA

To train models on CIFAR-10 + CelebA dataset (Section3.2 of the main paper), we used the Resnet architectures ofWGAN-GP [9] with the same hyper-parameter configura-tion for the generator and the discriminator networks. InNormalized WGAN, the learning rate of mode proportionπ was 5 times the learning rate of the discriminator.

F.2. Domain adaptation for mixture distributions

F.2.1 Digit classification

For MNIST→MNIST-M experiments (Section 4.1.1 of themain paper), following [6], a modified Lenet architecturewas used for feature network, and a MLP network wasused for domain classifier. The architectures and hyper-parameters used in our method are given in Table 14. Thesame architectures are used for the compared approaches -Source only, DANN and Wasserstein.

F.2.2 VISDA

For the experiments on VISDA dataset with three classes(Section 4.1.2 of the main paper), the architectures andhyper-parameters used in our method are given in Ta-ble 15. The same architectures are used for the comparedapproaches: source only, Wasserstein and DANN.

F.2.3 Domain adaptation for Image denoising

The architectures and hyper-parameters used in our methodfor image denoising experiment (Section 4.2 of the main pa-per) are presented in Table 16. To perform adaptation usingNormalized Wasserstein measure, we need to train the in-termediate distributions PG,π(1) and PG,π(2) (as discussedin Section 2, 4.2 of the main paper). We denote the genera-tor and discriminator models corresponding to PG,π(1) andPG,π(2) as Generator (RW) and Discriminator (RW) respec-tively. In practice, we noticed that the Generator (RW) andDiscriminator (RW) models need to be trained for a certainnumber of iterations first (which we call initial iterations)before performing adaptation. So, for these initial itera-tions, we set the adaptation parameter λ as 0. Note that theencoder, decoder, generator (RW) and discriminator (RW)models are trained during this phase, but the adaptation isnot performed. After these initial iterations, we turn theadaptation term on. The hyperparameters and model archi-tectures are given in Table 16. The same architectures areused for Source only and Wasserstein.

F.3. Adversarial clustering

For adversarial clustering in imbalanced MNIST dataset(Section 5 of the main paper), the architectures and hyper-parameters used are given in Table 17.

F.4. Hypothesis testing

For hypothesis testing experiment (Section 6 of the mainpaper), the same model architectures and hyper-parametersas the MOG experiment (Table 12) was used.

Table 12. Architectures and hyper-parameters: Mixture of Gaussians with Normalized Wasserstein GANGenerator Discriminator

Linear(2→ 64) Linear(2→ 128)Linear(64→ 64) LeakyReLU(0.2)Linear(64→ 64) Linear(128→ 128)Linear(64→ 2) LeakyReLU(0.2)

Linear(128→ 2)Hyperparameters

Discriminator learning rate 0.00005Generator learning rate 0.00005

π learning rate 0.01Batch size 1024Optimizer RMSProp

Number of critic iters 10Weight clip [−0.003, 0.003]

Table 13. Architectures: Mixture of Gaussians with vanilla WGAN modelGenerator Discriminator

Linear(2→ 512) + ReLU Linear(2→ 512) + ReLULinear(512→ 512) + ReLU Linear(512→ 512) + ReLULinear(512→ 512) + ReLU Linear(512→ 512) + ReLU

Linear(512→ 2) Linear(512→ 2)

Table 14. Architectures and hyper-parameters: Domain adaptation for MNIST→MNIST-M experimentsFeature network

Conv(3→ 32, 5× 5 kernel) + ReLU + MaxPool(2)Conv(32→ 48, 5× 5 kernel) + ReLU + MaxPool(2)Domain discriminator Classifier

Linear(768→ 100) + ReLU Linear(768→ 100) + ReLULinear(100→ 1) Linear(100→ 100) + ReLU

Linear(100→ 10)Hyperparameters

Feature network learning rate 0.0002Discriminator learning rate 0.0002

Classifier learning rate 0.0002π learning rate 0.0005

Batch size 128Optimizer Adam

Number of critic iters 10Weight clipping value [−0.01, 0.01]

λ 1

Table 15. Architectures and hyper-parameters: Domain adaptation on VISDA datasetFeature network

Resnet-18 model pretrained on ImageNettill the penultimate layer

Domain discriminator ClassifierLinear(512→ 512) + LeakyReLU(0.2) Linear(512→ 3)Linear(512→ 512) + LeakyReLU(0.2)Linear(512→ 512) + LeakyReLU(0.2)

Linear(512→ 1)Hyperparameters

Feature network learning rate 0.000001Discriminator learning rate 0.00001

Classifier learning rate 0.00001π learning rate 0.0001

Batch size 128Optimizer Adam

Number of critic iters 10Weight clipping value [−0.01, 0.01]

λ 1

Table 16. Architectures and hyper-parameters: Domain adaptation for image denoising experimentEncoder Decoder

Conv(3→ 64, 3× 3 kernel) Linear(2→ 128)+ReLU + MaxPool(2) Conv(128→ 64, 3× 3 kernel)

Conv(64→ 128, 3× 3 kernel) + ReLU + Upsample(2)+ReLU + MaxPool(2) Conv(64→ 64, 4× 4 kernel)

Conv(128→ 128, 3× 3 kernel) + ReLU + Upsample(4)+ReLU + MaxPool(2) Conv(64→ 3, 3× 3 kernel)

Conv(128→ 128, 3× 3 kernel)Linear(128→ 2)

Domain discriminatorLinear(2→ 64) + ReLULinear(64→ 64) + ReLU

Linear(64→ 1)Generator (RW) Discriminator (RW)Linear(2→ 128) Linear(2→ 128) + ReLU

Linear(128→ 128) Linear(128→ 128) + ReLULinear(128→ 2) Linear(128→ 1)

HyperparametersEncoder learning rate 0.0002Decoder learning rate 0.0002

Domain Discriminator learning rate 0.0002Generator (RW) learning rate 0.0002

Discriminator (RW) learning rate 0.0002π learning rate 0.0005

Batch size 128Optimizer Adam

Number of critic iters 5Initial iters 5000

Weight clipping value [−0.01, 0.01]λ 1

Table 17. Architectures and hyper-parameters: Mixture models on imbalanced-MNIST3 datasetGenerator Discriminator

ConvTranspose(100→ 256, 4× 4 kernel, stride 1) Spectralnorm(Conv(1→ 64, 4× 4 kernel, stride 2))Batchnorm + ReLU LeakyReLU(0.2)

ConvTranspose(256→ 128, 4× 4 kernel, stride 2) Spectralnorm(Conv(64→ 128, 4× 4 kernel, stride 2))Batchnorm + ReLU LeakyReLU(0.2)

ConvTranspose(128→ 64, 4× 4 kernel, stride 2) Spectralnorm(Conv(128→ 256, 4× 4 kernel, stride 2))Batchnorm + ReLU LeakyReLU(0.2)

ConvTranspose(64→ 1, 4× 4 kernel, stride 2) Spectralnorm(Conv(256→ 1, 4× 4 kernel, stride 1))Tanh()

HyperparametersDiscriminator learning rate 0.00005

Generator learning rate 0.0001π learning rate 0.001

Batch size 64Optimizer RMSProp

Number of critic iters 5Weight clip [−0.01, 0.01]

λreg 0.01