Towards Distributed Coevolutionary GANs

Abdullah Al-Dujaili, Tom Schmiedlechner, Erik Hemberg and Una-May O’Reillyaldujail@mit.edu, tschmied@mit.edu, hembergerik@csail.mit.edu, unamay@csail.mit.edu

CSAIL, MIT, USA

AbstractGenerative Adversarial Networks (GANs) have become one ofthe dominant methods for deep generative modeling. Despitetheir demonstrated success on multiple vision tasks, GANs aredifficult to train and much research has been dedicated towardsunderstanding and improving their gradient-based learningdynamics. Here, we investigate the use of coevolution, a classof black-box (gradient-free) co-optimization techniques anda powerful tool in evolutionary computing, as a supplementto gradient-based GAN training techniques. Experiments ona simple model that exhibits several of the GAN gradient-based dynamics (e.g., mode collapse, oscillatory behavior,and vanishing gradients) show that coevolution is a promisingframework for escaping degenerate GAN training behaviors.

IntroductionGenerative modeling aims to learn functions that expressdistributional outputs. In a standard setup, generative mod-els take a training set drawn from a specific distribution andlearn to represent an estimate of that distribution. By estimate,we mean either an explicit density estimation, the ability togenerate samples, or the ability to do both [14]. GANs [15]are a framework for training generative deep models via anadversarial process. They have been applied with celebratedsuccess to a growing body of applications. Typically, a GANpairs two networks, viz. a generator and a discriminator. Thegoal of the generator is to produce a sample (e.g., an image)from a latent code such that the distribution of the producedsamples are indistinguishable from the true data (training set)distribution. In tandem, the discriminator plays the role ofa critic to do the assessment and tell whether the samplesare true data or generated by the generator. Concurrently,the discriminator is trained to discriminate optimally (max-imize its accuracy), while the generator is trained to foolthe discriminator (minimize its accuracy). Despite their wit-nessed success, it is well known that GANs are difficult tooptimize. From a game theory perspective, GAN trainingcan be seen as a two-player minimax game. Since the twonetworks are differentiable, optimizing the minimax GANobjective is typically carried out by (variants of) simultane-ous gradient-based updates to their parameters. While it hasbeen shown that simultaneous gradient updates converge if

Copyright c© 2018, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

they are made in function space, the same proof does notapply to these updates in parameter space [15]. On the onehand, a zero gradient is a necessary condition for standard op-timization to converge. On the other hand, equilibrium is thecorresponding necessary condition in a two-player game [4].In practice, gradient-based GAN training often oscillateswithout ultimately reaching an equilibrium. Moreover, a vari-ety of degenerate behaviors have been observed—e.g., modecollapse [5], discriminator collapse [27], and vanishing gra-dients [2]. These unstable learning dynamics have been thefocus of several investigations by the deep learning commu-nity, seeking a principled theoretical understanding as well aspractical algorithmic improvements and heuristics [2, 3, 16].Two-player minimax black-box optimization and games havebeen a topic of recurrent interest in the evolutionary comput-ing community [24, 38]. In seminal work, Hillis [19] showedthat more efficient sorting programs can be produced by com-petitively co-evolving them versus their testing programs.Likewise, Herrmann [18] proposed a two-space genetic algo-rithm as a general technique to solve minimax optimizationproblems and used it to solve a parallel machine schedulingproblem with uncertain processing times. In competitive co-evolution, two different populations, namely solutions andtests, coevolve against each other [13]. The quality of a solu-tion is determined by its performance when interacting withthe tests. Reciprocally, a test’s quality is determined by itsperformance when interacting with the solutions, leading towhat is commonly referred to as evolutionary arms race [10].

In this paper, we propose to pair a coevolutionary algo-rithm with conventional GAN training, asking whether thecombination is powerful enough to more frequently avoiddegenerate training behaviors. The motivation behind ourproposition is of two-fold. First, most of the pathologicalbehaviors encountered with gradient-based GAN traininghave been identified and studied by the evolutionary comput-ing community decades ago—e.g., focusing, relativism, andloss of gradients [35, 47]. Second, there has been a growingbody of work, which shows that the performance of gradient-based methods can be rivaled by evolutionary-based coun-terparts when combined with sufficient computing resourcesand data [42, 32, 26, 43]. The aim of this paper is to bridgethe gap between works of the deep learning and evolution-ary computing communities towards a better understandingof gradient-based and gradient-free GAN dynamics. Indeed,

arX

iv:1

807.

0819

4v2

[cs

.NE

] 2

2 A

ug 2

018

one can see that the Nash Equilibrium solution concept incoevolutionary literature [37] is not that different from thenotion of GAN mixtures in GAN literature [4].

We report the following contributions: i) For a simple para-metric generative modeling problem [27] that exhibits severaldegenerate behaviors with gradient-based training, we vali-date the effectiveness of combining coevolution with gradient-based updates (mutations). ii) We present Lipizzaner, acoevolutionary framework to train GANs with gradient-basedmutations (for neural net parameters) and gradient-free mu-tations (for hyperparameters) and learn a mixture of GANs.iii) Finally, we provide the Lipizzaner framework andexperiment code for public use.1

Related WorkTraining GANs Several gradient-based GAN training vari-ants have been proposed to improve and stabilize its dy-namics. One variant category is focused on improving train-ing techniques for single-generator single-discriminator net-works. Examples include modifying the generator’s objec-tive [46], the discriminator’s objective [30], or both [3, 41].Some of these propositions are theoretically well-founded,but convergence still remains elusive in practice. The sec-ond category employs a framework of multiple generatorsand/or multiple discriminators. Examples include trainingmultiple discriminators [11]; training an array of specializeddiscriminators, each of which looks at a different random low-dimensional projection of the data [33]; sequentially trainingand adding new generators with boosting techniques [44];training a cascade of GANs [45]; training multiple generatorsand discriminators in parallel (GAP) [22]; training a classifier,a discriminator, and a set of generators [20]; and optimizinga weighted average reward over pairs of generators and dis-criminators (MIX+GAN) [4]. For a theoretical view on GANtraining, the reader may refer to [2, 27, 4].

Coevolutionary Algorithms for Minimax Problems.Variants of competitive coevolutionary algorithms have beenused to solve minimax formulations in the domains of con-strained optimization [7], mechanical structure optimiza-tion [6], and machine scheduling [18]. These early coevo-lutionary propositions were tailored to symmetric minimaxproblems. In practice, the symmetry property may not al-ways hold. In fact, mode collapse in GANs may arise fromasymmetry [14]. To address this issue, asymmetric fitnessevaluation was presented in [21] and analyzed in [8]. Further,Qiu et al. [38] attempt to overcome the limitations of existingcoevolutionary approaches in solving minimax optimizationproblems using differential evolution.

MethodsNotation We adopt a mix of notation used in [4, 27]. LetG = {Gu, u ∈ U} denote the class of generators, where Gu

is a function indexed by u that denotes the parameters of thegenerators. Likewise, let D = {Dv, v ∈ V} denote the class

1https://github.com/ALFA-group/lipizzaner-gan

of discriminators, where Dv is a function parameterized byv. Here U ,V ⊆ Rp represent the parameters space of thegenerators and discriminators, respectively. Further, let G∗be the target unknown distribution that we would like to fitour generative model to. Formally, the goal of GAN trainingis to find parameters u and v so as to optimize the objectivefunction

minu∈U

maxv∈VL(u, v) , where

L(u, v) = Ex∼G∗ [φ(Dv(x))] + Ex∼Gu [φ(1−Dv(x))] , (1)

and φ : [0, 1] → R, is a concave function, commonly re-ferred to as the measuring function. In the recently proposedWasserstein GAN [3], φ(x) = x, and we use the same for therest of the paper. In practice, we have access to a finite num-ber of training samples x1, . . . , xS ∼ G∗. Therefore, onecan use the empirical version 1

S

∑Si=1 φ(Dv(xi)) to estimate

Ex∼G∗ [φ(Dv(x))] . The same holds for Gu. Further, let Sube a distribution supported on U and Sv be a distributionsupported on V .

Basic Coevolutionary Dynamics. With coevolutionary al-gorithms, the two search spaces U and V can be searchedwith two different populations: the generator populationPu = {u1, . . . , uT } and the discriminator population Pv ={v1, . . . , vT }, where T is the population size. In a predator-prey interaction, the two populations coevolve: the generatorpopulation Pu aims to find generators which evaluate to lowL values with the discriminator population Pv whose goalis to find discriminators which evaluate to high L valueswith the generator population. This is realized by harness-ing the neo-Darwanian notions of heredity and survival ofthe fittest, as outlined in Algorithm 1. Over multiple genera-tions (iterations), the fitness of each generator ui ∈ Pu anddiscriminator vj ∈ Pv are evaluated based on their interac-tions with one or more discriminators from Pv and generatorsfrom Pu, respectively (Lines 2 to 7). Based on their fitnessrank (Lines 8 to 11), the current population individuals areemployed in producing next population of generators anddiscriminators with the help of mutation: a genetic-like varia-tion operator (Lines 12 to 13), where the mutated individualsreplace the current population if they exhibit a better fitness.In gradient-free scenarios, Gaussian mutations are usuallyapplied [38, 1]. With GANs (which are differentiable nets),we propose to use gradient-based mutations for the genera-tors and discriminators net parameters, i.e., Pu and Pv aremutated with a gradient step computed by back-propagatingthrough one (or more) of their fitness updates (right-handside of Lines 5 and 6). Note that the coevolutionary dynamicsare not restricted to tuning net parameters. Non-differentiable(hyper)parameters can also be incorporated. In our frame-work, we tune the learning rates for the generator and dis-criminator populations with Gaussian mutations.

Spatial Coevolution Dynamics. The basic coevolutionarysetup (as adapted for GAN training in Algorithm 1) has beenthe subject of several studies (e.g., [47, 31]) analyzing de-generate behaviors such as focusing, relativism, and loss of

Algorithm 1 BasicCoevGANs(Pu, Pv,L, {αi}, {βi}, I)Input:Pu : generator population Pu : discriminator population{αi} : selection probability {βi} : mutation probabilityI : number of generations L : GAN objective functionReturn:Pu : evolved generator populationPv : evolved discriminator population

1: for i in range(I) do// Evaluate Pu and Pv

2: fu1...uT← 0

3: fv1...vT← 0

4: for each ui in Pu , each vj in Pv do5: fui

−= L(ui, vj)

6: fvj += L(ui, vj)

7: end for// Sort Pu and Pv

8: u1...T ← us(1)...s(T ) with s(i) = argsort(fu1...uT, i)

9: v1...T ← vs(1)...s(T ) with s(j) = argsort(fv1...vT, j)

// Selection10: u1...T ← us(1)...s(T ) with s(i) = argselect(u1...T , i, {αi})11: v1...T ← vs(1)...s(T ) with s(j) = argselect(v1...T , j, {αj})

// Mutation & Replacement12: u1...T ← replace({ui}, {u′i}) with u′i = mutate(ui, βi)

13: v1...T ← replace({vj}, {v′j}) with v′j = mutate(vj , βj)14: end for15: return Pu, Pv

gradients; which correspond to mode collapse, discrimina-tor collapse, and vanishing gradients in the GAN literature,respectively. Consequently, this has led to the emergence ofmore stable setups such as spatial coevolution, where indi-viduals from both populations are distributed spatially (e.g.,on a grid), with local interactions governing fitness evalua-tion, selection, and mutation. This is different from the basiccoevolutionary setup in which individuals from the two popu-lations test each other either exhaustively or employ randomsampling to realize interactions [31]. Spatial coevolution hasshown to be substantially successful over several non-triviallearning tasks due to its ability to maintain diversity in thepopulation for long periods and to foster continuing armsraces. We refer the reader to [49, 31] for detailed numericalexperiments on the efficiency of spatial coevolution. In thecontext of GAN training, we distribute the generator and dis-criminator populations over a two-dimensional toroidal gridwhere each cell holds one (or more) individual(s) from thegenerator population and one (or more) individual(s) from thediscriminator population. During the coevolutionary process,each cell (and the individuals therein) interacts with its neigh-boring cells. A cell’s neighborhood is defined by its adjacentcells and specified by its size sn. A five-cell neighborhood(one center and four adjacent cells) is a commonly used setup.Note that for an m×m-grid, there exist m2 neighborhoods.For the kth neighborhood in the grid, we refer to the set ofgenerator individuals in its center cell by P k,1

u ⊂ Pu and theset of generator individuals in the rest of the neighborhoodcells by P k,2

u , . . . , P k,sn , respectively. Furthermore, we de-note the union of these sets by P k

u = ∪sni=1Pk,iu ⊆ Pu, which

represents the kth generator neighborhood. Note that, withsn = 5 and for the k′th neighborhood whose center cell’s gen-erator individuals P k′,1

u = P k,ju for some j ∈ {2, . . . , sn},

we haveP ku∩P k′

u = P k,1u ∪P k′,1

u . Furthermore, |P ku | = |P k′

u |for all k, k′ ∈ {1, . . . ,m2} and we denote this number byN . The same notation and terminology is adopted for thediscriminator population, with P k

v ⊆ Pv representing thekth discriminator neighborhood. As shown in Algorithm 2,

each neighborhood k runs an instance of Algorithm 1 withthe generator and discriminator populations being P k

u andP kv , respectively. The difference is that the evolved popula-

tions (Line 15 of Algorithm 1) are used to update only theindividuals of the center cells P k,1

u , P k,1v rather than P k

u , P kv

(Lines 4 and 5 of Algorithm 2). Since there are m2 neighbor-hoods, all of the populations individuals will get updated asPu = ∪m2

k=1Pku , Pv = ∪m2

k=1Pkv . The m2 instances of Algo-

rithm 1 can run in parallel in a synchronous or asynchronousfashion (in terms of reading/writing to the populations). Inour implementation, we opted for the asynchronous modefor three reasons. First, asynchronous variant scales moreefficiently with lower communication overhead among cells.Second, with asynchronous mode, different cells are oftenin different stages of the training process (i.e., compute dif-ferent generations). Individuals from previous or upcominggenerations may therefore be used during the training pro-cess, which further increases the diversity as well [35, 37].Third, several works have concluded that asynchronous co-evolutionary computing produces slightly better results withless function evaluations [34].

Generator Neighborhood As A Generator Mixture. To-wards the end of training, |Pu| generators will be availablefor use as generative models. And instead of using one,we propose to choose one of the generator neighborhoods{P k

u }1≤k≤m2 as a mixture of generators according to a givenperformance metric g : UN × RN → R (e.g., inceptionscore [41]). That is, the best generator mixture P ∗u ∈ UN andthe corresponding mixture weights w∗ ∈ [0, 1]N—Recallthat in a neighborhood, there are N generators (and N dis-criminators). Hence, the N -dimensional mixture weight vec-tor w.—is defined as follows

P ∗u ,w∗ = argmax

Pku ,w

k:1≤k≤m2

g( ∑ui∈Pk

uwi∈wk

wiGui

), (2)

where wi represents the mixture weight of (or the probabil-ity that a data point comes from) the ith generator in theneighborhood, with

∑wi∈wk wi = 1. One may think of

{wk}1≤k≤m2 as hyperparameters of the proposed frame-work that can be set a priori (e.g., uniform mixture weightswi = 1

N ). Nevertheless, the system is flexible enough toincorporate learning these weights in tandem with the coevo-lutionary dynamics as discussed next.

Evolving Mixture Weights. With anm×m-grid, we havem2 mixture weight vectors {wk}1≤k≤m2 , which we wouldlike to learn and optimize such that our performance met-ric g is maximized across all the m2 generator neighbor-hoods. To this end, we view {wk}1≤k≤m2 as a populationof m2 individuals whose fitness measures are evaluated by ggiven the corresponding generator neighborhoods. In otherwords, the fitness of the kth individual (weight vector wk)is g(∑

ui∈Pku ,wi∈wk wiGui

). After each step of spatial co-

evolution of the generator and discriminator populations, themixture weight vectors {wk}1≤k≤m2 are updated with anevolution strategy (e.g., (1+1)-ES [29, Algorithm 2.1]), where

selection and mutation based on the neighborhoods’ g values(Line 7 of Algorithm 2). This concludes the description of ourcoevolutionary proposition for training GANs with gradient-based mutations as summarized in Algorithm 2. Fig. 1 pro-vides a pictorial illustration of the grid. We refer to our pythonimplementation of Algorithm 2 by Lipizzaner.

Algorithm 2 CoevGANs(Pu, Pv,L, {αi}, {βi})Input:Pu : generator population Pu : discriminator population{αi} : selection probability {βi} : mutation probabilityI : number of population generations per training step m : side length of the spatial square gridL : GAN objective functionReturn:P∗u : evolved generator mixture w∗ : evolved mixture weight vector

1: repeat// Spatial Coevolution of Generator & Discriminator Populations

2: parfor k in range(m2) do3: P̂k

u , P̂kv ← BasicCoevGANs(Pk

u , Pkv ,L, {αi}, {βi}, I)

4: Pk,1u ← TopN(P̂k

u , n = |Pk,1u |)

5: Pk,1v ← TopN(P̂k

v , n = |Pk,1v |)

6: end parfor// Generator Mixture Weights Evolution

7: w1, . . . ,wm2←(1+1)-ES(w1, . . . ,wm2

, g, {Pku )}) . See [29, Algorithm

2.1])8: until training converged9: P∗u ,w

∗ ← argmaxPku ,wk:1≤k≤m2 g

(∑ui∈P

ku

wi∈wk

wiGui

)10: return P∗u ,w

∗

ExperimentsTwo different types of experiments were conducted: 1) Toelaborate the capability of coevolutionary algorithms to solvetypical problems of GANs, we used the theoretical modelproposed in [27] that exhibits degenerate training behaviorin a typical framework and compare them when trained witha simple coevolutionary counterpart. 2) We then show theability of Lipizzaner to match state-of-the-art GANs on com-monly used image-generation datasets [4, 3].

Theoretical GAN ModelSetup. To investigate coevolutionary dynamics for GANtraining, we make use of the simple problem introducedin [27]. Formally, the generator set is defined as

G =

{1

2N (µ1, 1) +

1

2N (µ2, 1) | µ ∈ R2

}. (3)

On the other hand, the discriminator set is expressed as fol-lows.

D = {I[`1,r1] + I[`2,r2] | `, r ∈ R2 s.t. `1 ≤ r1 ≤ `2 ≤ r2} .(4)

Given a true distribution G∗ with parameters µ∗, the GANobjective of this simple problem can be written as

minµ

max`,rL(µ, `, r) , where

L(µ, `, r) = Ex∼G∗ [D`,r(x)] + Ex∼Gµ [1−D`,r(x)] . (5)

While being simple to understand and demonstrate, thisGAN variant exhibits the relevant dynamics we are elabo-rating. We conducted several experiments to understand theperformance of the coevolutionary framework in its simplestform in comparison to the standard gradient-based dynamic.

Fig. 1: Topology of a 3× 3-grid (m = 3) with a neighborhood sizeof sn = 5. A neighborhood of the 5th cell is highlighted in light red.Each cell has a population size of one (one generator Gu and onediscriminator Dv). The corresponding neural net parameters u andv are updated with gradient-based mutations, while the respectivehyperparameters (e.g., learning rate αu and αv) are updated withGaussian-based mutations based on the interactions of each cell withits neighbors. Each cell has the mixture weight vectorwk for theirrespective neighborhood, which is optimized with an evolutionaryalgorithm according to a given performance metric g.

Unless stated otherwise, we used Algorithm 1 with 120 runsper experiment, each run is set with 100 generations and apopulation size of 10. We also use Gaussian mutation with astep size of 1 as the only genetic operator.

Results. Fig. 4 shows the convergence of the parame-ters `1, `2, r1, r2, µ1, µ2 using different variants of gradient-based and coevolutionary dynamics. One can observe that theµ1 and µ2 under coevolutionary dynamics consistently con-verge to the true values µ∗1 and µ∗2, respectively. Furthermore,we investigated coevolutionary behavior for the followingscenarios that have been shown to be critical for traditionalpure gradient-based GAN training methods [5, 27]:

Mode collapse. Being one of the most-observed failuresof GANs in real-world problems, mode collapse often oc-curs when attempting to learn models from highly complexdistributions, e.g. images of high visual quality [5]. In thisscenario, the generator is unable to learn the full underlyingdistribution of the data, and attempts to fool the discriminatorby producing samples from a small part of this distribution.Vice versa, the discriminator learns to distinguish real andfake values by focusing on another part of the distribution– which leads to the generator specializing on this area, andfurthermore to oscillating behavior. In our experiments, weused the same setting as Li et al. [27], initializing µ1 and µ1

to values in the interval of [−10, 10], with a step size of 0.1.Fig. 2 shows the average success rate with the given initial-ization values. In accordance with [27], we define success asthe ability to reach a distance less than 0.1, between the bestgenerator of the last generation and the optimal generator G∗.From the figure, we see that coevolutionary GAN training isable to step out of mode collapse scenarios, where µ1 = µ2—Note the high success rate along the diagonal of Fig. 2 (b) incomparison to best of gradient-based dynamics in (a).

Discriminator collapse. This term describes a phenomenonwhere the discriminator is stuck in a local minimum [27]. Dueto their local nature of updates, gradient-based dynamics aregenerally not able to escape these local minima without fur-ther enhancements—a problem that global optimizers likeevolutionary algorithms handle better. Our results in Fig. 3 (a)

1.0 0.5 0.0 0.5 1.0

1.0

0.5

0.0

0.5

1.0 0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Optimal training

-1.0 0.0 1.0

-1.0

0.0

1.0

Coevolutionary training

0.0

0.2

0.4

0.6

0.8

1.0

(a) (b)

Fig. 2: Heatmap of success probability for random generator (µ1

and µ2) initializations for (a) a variant of gradient-based dynamics(adapted from [27]) and (b) coevolutionary GAN training dynamics.For each square, the individuals of the generator population areinitialized within the corresponding range.

negative positive

positive

negative

0.0

0.2

0.4

0.6

0.8

1.0

10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0x

0.3

0.2

0.1

0.0

0.1

0.2

0.3

p(x)

q(x)

Discriminator collapse - Generation 1[left0, right0][left1, right1]

10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0x

0.3

0.2

0.1

0.0

0.1

0.2

0.3

p(x)

q(x)

Discriminator collapse - Generation 100[left0, right0][left1, right1]

(a) (b) (c)

Fig. 3: (a) Heatmap of success probability for random discrimi-nator (`1, r1, `2, r2) initializations for coevolutionary GAN train-ing dynamics. The axes refer to initial fitness values for both theleft ([`1, r1]) and the right ([`2, r2]) bounds, leading to four differ-ent quadrants. (b) First and (c) last generation of a coevolutionarytrained GAN, initialized in a discriminator collapse setup, whichcorresponds to the bottom left quadrant of (a). In other words, foreach square, the individuals of the discriminator population are ini-tialized randomly such that the signs of their fitness values matchthose of the corresponding square.

support this proposition, using the same setup as in the pre-viously described experiment. In particular, note the highsuccess probability for the bottom left quadrant, where bothbounds of the discriminator lie where the fitness value (Eq. 5)is less than 0. Fig. 3 (b) shows an example of such bounds. Inthis setup, gradient-based dynamics force the bounds to col-lapse (i.e., `1 = r1, `2 = r2, see [27, Fig. 2 (c)]). On the otherhand, coevolution is able to step out of the local minimumand converges to near-optimality as shown in Fig. 3 (c)—with more generations, the left bound asymptotically movestowards −∞. For this scenario, the parameters of G werefixed to µ1 = −1, µ2 = 2.5 during the whole evolutionaryprocess.

GAN for ImagesSetup. If not stated otherwise, the experiments were con-ducted with Algorithm 2 on a grid size of 2x2, and a popula-tion size of one per cell (i.e. one generator and one discrimi-nator); despite this small size, the shown results are alreadypromising in solving the pathologies described above. Weleave experiments with larger grid size for future work andupcoming versions of this paper. At the end of each genera-tion, the current cells individual is replaced with the highestranked offspring individual created from the neighborhood.For gradient-based mutations of the neural net parameters, weuse the Adam optimizer [23] with an initial learning rate of0.0002, which is altered with a mutation space ofN (0, 1e−7)per generation. The mixture weights are updated by an (1+ 1) ES, with the mutation space of N (0, 0.01). Regarding

0 1000 2000 3000 4000 5000 6000 7000 8000

4

2

0

2

4left0left1right0right1muhat0muhat1

First order dynamics, mode collapse

0 20 40 60 80 1008

6

4

2

0

2

4

6

Coev Alternating, optimal discriminator (Symmetric)

left0left1right0right1µ1µ2µ1'µ2'

0 20 40 60 80 1003

2

1

0

1

2

3

Optimal discriminator dynamics

0 20 40 60 80 1008

6

4

2

0

2

4

6

Coev Alternating, optimal discriminator (Asymmetric)

left0left1right0right1µ1µ2µ1'µ2'

0 1000 2000 3000 4000 50003

2

1

0

1

2

3

First order dynamics, converging behavior

0 20 40 60 80 1004

2

0

2

4

6

8

Coev Alternating (Symmetric)left0left1right0right1µ1µ2µ1'µ2'

0 1000 2000 3000 4000 50001.2

1.0

0.8

0.6

0.4

0.2

0.0

0.2

First order dynamics, vanishing gradient

0 20 40 60 80 1004

2

0

2

4

6

8Coev Alternating (Asymmetric)

left0left1right0right1µ1µ2µ1'µ2'

(a) (b)

Fig. 4: Parameters convergence for the theoretical GAN model with(a) gradient-based [27] and (b) coevolutionary-based (Algorithm 1)dynamics; the curves trace the best individuals (i.e., u1 and v1) ofeach generation.

the neural network topology, we used a four-layer perceptronwith 700 neurons for MNIST [25], and the more complexdeconvolutional GAN architecture [39] for the CelebA [28]dataset. We use the classic GAN setup [15] instead of recentpropositions (e.g., WGAN [3]). This simplifies the observa-tion of interesting pathologies, which can be more compli-cated to precipitate with stable GAN implementations.

Results. As stated, we conducted our experiments on twodifferent datasets, which were selected because of their abil-ity to show the behaviors we are primarily interested in. TheMNIST dataset [25] has been widely used in research, andespecially appropriate for showing mode collapse due to itslimited target space (namely the characters 0-9). Fig. 5 il-lustrates this behavior, and how Lipizzaner is able toprevent collapsing on few specific modes (numbers). Both re-sults were generated after 400 generations of training on theMNIST dataset with the above-mentioned four-layer percep-tron. We furthermore show comparable promising results forthe CelebA dataset [28], which contains more than 200, 000images of over 10, 000 celebrities’ faces. Fig. 6 shows that a

(a) Source data (b) Mode collapse (c) Lipizzaner

Fig. 5: Results on the MNIST dataset. (a) contains samples fromthe original dataset, while (b) shows a typical example for mode col-lapse; the generator is primarily focused on the characters 1, 9 and 7.The data sampled from a generator trained with Lipizzaner in(c) shows that coevolution is able to create higher diversity amongthe covered modes.

(a) Source data (b) Before collapse (c) First collapsed generation (d) 10 generations after collapse

Fig. 6: Sequence of images generated without Lipizzaner basedon the CelebA dataset (b) before, (c) during, and (d) 10 generationsafter the systems mode or discriminator collapses. The originalimages in (a) are shown for comparison. This figure illustrates that,without further optimizations, DCGAN is mostly not able to stepout of this scenario.

non-coevolutionary DCGAN [39] collapses at a certain pointand is unable to recover even after 10 more generations (witheach generation processing the whole dataset). Fig. 7 showsthat the same GAN wrapped in the Lipizzaner frameworkis able to step out of the collapse in only the next generation.We furthermore note that, while the DCGAN collapse is eas-ily repeatable, Lipizzaner was able to completely avoidthis scenario in most of our experiments.

ConclusionIn this paper, we have investigated coevolutionary (in par-ticular competitive) algorithms as an option to enhance theperformance of gradient-based GAN training methods. Wepresented Lipizzaner, a framework that combines theadvantages of gradient-based optimization for GANs withthose of coevolutionary systems, and allows scaling over adistributed spatial grid topology. As demonstrated, our frame-work shows promising results on the conducted experiments,even without scaling to larger dimensions than other compara-ble approaches [4] do. Even better results may be achieved byincluding improved GAN types like the recently introducedWGAN [3].

References[1] Al-Dujaili et al.. On the application of Danskin’s theorem to derivative-free minimax opti-

mization. Int. Workshop on Global Optimization, 2018.[2] Arjovsky and Bottou. Towards principled methods for training generative adversarial networks.

arXiv:1701.04862, 2017.

(a) Before collapse (b) Collapsed generation (c) One generation after collapse (d) After 30 generations

Fig. 7: Sequence of CelebA images generated by Lipizzaner(a) before, (b) during, and (c) one generation after the systems modeor discriminator collapses. Additionally, (d) shows the results gener-ated after 30 generations. Especially when compared to Figure 6,this illustrates how Lipizzaner is able to overcome collapsedGANs.

[3] Arjovsky et al.. Wasserstein gan. arXiv:1701.07875, 2017.[4] Arora et al.. Generalization and equilibrium in generative adversarial nets (gans).

arXiv:1703.00573, 2017.[5] Arora and Zhang. Do gans actually learn the distribution? an empirical study.

arXiv:1706.08224, 2017.[6] Barbosa. A coevolutionary genetic algorithm for a game approach to structural optimization.

In ICGA, 1997.[7] Barbosa. A coevolutionary genetic algorithm for constrained optimization. In CEC, 1999.[8] Branke et al. New approaches to coevolutionary worst-case optimization. In PPSN, 2008.[9] Cliff and Miller. Tracking the red queen: Measurements of adaptive progress in co-

evolutionary simulations. In ISAL, 1995.[10] Dawkins and Krebs. Arms races between and within species. Proc. R. Soc. Lond. B, 1979.[11] Durugkar et al. Generative multi-adversarial networks. arXiv:1611.01673, 2016.[12] Ficici and Pollack. A game-theoretic memory mechanism for coevolution. In GECCO, 2003.[13] Floreano and Mattiussi. Bio-inspired artificial intelligence: theories, methods, and technolo-

gies. MIT press, 2008.[14] Goodfellow. Nips 2016 tutorial: Generative adversarial networks. arXiv:1701.00160, 2016.[15] Goodfellow et al. Generative adversarial nets. In NIPS, 2014.[16] Gulrajani et al. Improved training of wasserstein gans. In NIPS, 2017.[17] Harper. Evolving robocode tanks for evo robocode. Genetic Programming and Evolvable

Machines, 2014.[18] Herrmann. A genetic algorithm for minimax optimization problems. In CEC, 1999.[19] Hillis. Co-evolving parasites improve simulated evolution as an optimization procedure. Phys-

ica D: Nonlinear Phenomena, 1990.[20] Hoang et al. Multi-generator gernerative adversarial nets. arXiv:1708.02556, 2017.[21] Jensen. A new look at solving minimax problems with coevolutionary genetic algorithms. In

Metaheuristics: computer decision-making, 2003.[22] Jiwoong Im et al. Generative adversarial parallelization. arXiv:1612.04021, 2016.[23] Kingma and Ba. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014.[24] Laskari et al. Particle swarm optimization for minimax problems. In CEC, 2002.[25] Yann LeCun. The mnist database of handwritten digits, 1998.[26] Lehman et al. Es is more than just a traditional finite-difference approximator.

arXiv:1712.06568, 2017.[27] Li et al. Towards understanding the dynamics of generative adversarial networks.

arXiv:1706.09884, 2017.[28] Liu et al. Deep learning face attributes in the wild. In ICCV, 2015.[29] Loshchilov. Surrogate-assisted evolutionary algorithms. PhD thesis, 2013.[30] Metz et al. Unrolled generative adversarial networks. arXiv:1611.02163, 2016.[31] Mitchell. Coevolutionary learning with spatially distributed populations. Computational intel-

ligence: principles and practice, 2006.[32] Morse et al. Simple evolutionary optimization can rival stochastic gradient descent in neural

networks. In GECCO, pages 477–484. ACM, 2016.[33] Neyshabur et al. Stabilizing gan training with multiple random projections. arXiv:1705.07831,

2017.[34] Nielsen et al. Novel efficient asynchronous cooperative co-evolutionary multi-objective algo-

rithms. In CEC, 2012.[35] Nolfi and Floreano. Coevolving predator and prey robots: Do “arms races” arise in artificial

evolution? Artificial life, 1998.[36] Oliehoek et al. The parallel nash memory for asymmetric games. In GECCO, 2006.[37] Popovici et al. Coevolutionary principles. In Handbook of natural computing, 2012.[38] Qiu et al. A new differential evolution algorithm for minimax optimization in robust design.

IEEE transactions on cybernetics, 2017.[39] Radford et al. Unsupervised representation learning with deep convolutional generative adver-

sarial networks. arXiv:1511.06434, 2015.[40] Rosin and Belew. New methods for competitive coevolution. Evolutionary computation,

5(1):1–29, 1997.[41] Salimans et al. Improved techniques for training gans. In NIPS, 2016.[42] Salimans et al. Evolution strategies as a scalable alternative to reinforcement learning.

arXiv:1703.03864, 2017.[43] Stanley and Clune. Welcoming the era of deep neuroevolution, 2017.[44] Tolstikhin et al. Adagan: Boosting generative models. In NIPS, 2017.[45] Wang et al. Ensembles of generative adversarial networks. arXiv:1612.00991, 2016.[46] Warde-Farley et al. Improving generative adversarial networks with denoising feature match-

ing. 2016.[47] Watson and Pollack. Coevolutionary dynamics in a minimal substrate. In GECCO, 2001.[48] Wierstra et al. Natural evolution strategies. In Congress on Computational Intelligence, 2008.[49] Williams and Mitchell. Investigating the success of spatial coevolution. In GECCO, 2005.