Text to Image Synthesis with Mutual Information Optimization

Lihang LiuUT Austin

lihangliu@utexas.edu

Mei WangUT Austin

meiwang@utexas.edu

AbstractText to image synthesis is an interesting anduseful task to explore recently. The goal isto generate images from captions, i.e. takingunstructured textual descriptions as input andusing them to generate relevant images. Thistask combines two challenging components oflanguage modeling and image generation andcan be considered to be more difficult thancaption generation. In this paper, we proposeda method that combines both the pipelines oftext to image synthesis and image captioningto help maximize the information flow throughthese pipelines and improve the quality of gen-erated images. In our experiments on CUBand Oxford-102 datasets, we demonstrate theeffectiveness of our method on generating im-ages of specific categories from unstructuredtext descriptions.

1 Introduction

Recently with the help of deep neural net-works, impressive progress has been made on textto photo-realistic image synthesis (Kingma andWelling, 2013; Goodfellow et al., 2014; Gregoret al., 2015; Denton et al., 2015). Among allthe state-of-art networks for text-to-image syn-thesis, Generative Adversarial Networks (GAN)shows its advantages in generating sharper images.Plenty of techniques have been proposed to furtherimprove GAN for more stable training process andhigher resolution images (Salimans et al., 2016;Reed et al., 2016b; Yang et al., 2017; Zhang et al.,2017a,b; Chen et al., 2018).

However, their main focus is to generate im-ages with higher resolution and photo-realistic,and none of them ensures that the information inthe input text is expressed in the generated imagein a complete manner, which means it may be hardto infer the original text from the generated image.Instead of mainly focusing on improving the reso-lution, the completeness of info from the input text

StackGAN

StackGAN+Info

this bird has a white breast and side with a gray nape, crown, and wing.

a rust colored bird with darker wings, with bright red eyes.

Figure 1: Comparison of Stack GAN with and withoutour proposed information flow on 64× 64 images. Thetop two images are generated from Stack GAN and thebottom two images are generated from our proposedmethod. Our method better conserves the informationfrom the input sentence.

should be a more fundamental task that needs tobe guaranteed first. Let’s describe this problem ininformation theory. Given an input sentence withwords describing some images, we use a genera-tor to obtain a fake image, which is the essentialcomponent of the text to image synthesis pipeline.To encourage the fake image to express the infor-mation of the input sentence as much as possible,we want to maximize the mutual information ofthe input sentence and the fake image. However,directly computing the mutual information is in-tractable, so we need to propose another new neu-ral network to approximate this metric and opti-mize with it so as to produce high-quality imageswith more complete info from the text.

In this paper, we combine the pipelines of textto image synthesis and image captioning to maxi-mize the information flow based on GAN network,

such that more complete info is encouraged in thetraining pipelines. Compared to traditional text toimage synthesis, after adding the image captioningpipeline, we can generate a fake sentence from thefake image and feed it back to the generator. Moreconcretely, we design three components in our net-work. Firstly, we use a Generator to generate thefake image from the input text. Secondly, we use aDiscriminator to judge whether the generated im-age is real or fake. Thirdly, we use a Captionercomponent to generate the text from the generatedimage. The first two components are the standardGAN framework (Goodfellow et al., 2014). Whilethe third component Captioner is used to recoverthe text given the generated image, which providesanother signal to train the Generator. The majoradvantage of the third component is that we canrequire the generated image to contain as much in-formation of the input text as possible and this canfurther improve the quality of the generated imagegiven extra signals iteratively. That is, with thisnew Captioner component, we can approximatelymaximize the mutual information of the input sen-tence and the fake image, such that more completeinformation is encouraged in the pipelines whichwill produce higher quality images. For example,in Figure 1, we want to generate bird figures asthe text captions described on the top. However,the top two figures generated from StackGAN donot show some primitive properties like gray nape,darker wings, etc. While after adding our mutualinfo optimization design, the bottom two figuresgive a better illustration of the texts.

The contribution of this paper is threefold. First,it’s novel to propose the idea of maximizing infor-mation flow in the task of text to image synthesis,and we use an approximation method to solve thisproblem. Second, we design a new network com-ponent in GAN architecture to complete the infor-mation flow and implement four variants of objec-tive functions Third, qualitative analysis and userstudy of our experiments prove the effectivenessand usefulness of our design.

The rest of this paper is arranged as follows. InSec.2, we describe related works on text to imagesynthesis. In Sec.3, we introduce our methodol-ogy, which includes the background of GAN andour model architecture. In Sec.4, experimentalanalysis and user study demonstrate the effective-ness of our method. Finally, we conclude our workin Sec.5.

2 Related Works

Tons of recent success of deep neural network hasbeen achieved in various tasks, typically in NLPand Computer Vision areas. Prior works on text toimage synthesis are mostly based on recent pop-ular algorithms on language modeling and deepgenerative models.

With the emergence of deep learning, Varia-tional Autoencoders (VAE) (Kingma and Welling,2013) first formulated this problem with prob-abilistic graphical model and continuous latentvariables whose goal was to maximize the lowerbound of data likelihood. Later another approachnamed DRAW (Gregor et al., 2015) built a DeepRecurrent Attention Writer which further incorpo-rated a novel differentiable attention mechanism.Text modality and text feature representation arechallenging subproblems. AlignDRAW (Mansi-mov et al., 2015) used an attention model to iter-atively generate patches on canvas. They have theadvantage that focuses on different phrases eachtime step. However, their generated images are farfrom photo-realistic and have low resolution.

Recently, Generative Adversarial Networks(GAN) (Goodfellow et al., 2014) shows its advan-tages in generating sharper image, and many mod-ified and extended works have been proposed forfurther optimization. Given the unstable trainingdynamics of GAN, several techniques (Salimanset al., 2016; Arjovsky and Bottou, 2017) havebeen proposed to stabilize the training process andgenerate compelling results. Conditional GANs(cGANs) use conditional information for the dis-criminator and generator and they have drawnpromising results for image generation from text(Mirza and Osindero, 2014; Denton et al., 2015;Reed et al., 2016a). Impressively (Reed et al.,2016b) is able to generate photo-realistic imageswith the help of GAN. Also, they proposed sev-eral variants by using different objective functionsto enforce smoothness on the language manifoldand reduce overfitting. Based on that, StackGAN(Zhang et al., 2017a,b) is the first model to useconditional loss for text to image synthesis, and itis one of the latest works that can synthesize high-quality images from coarse to fine in 256 × 256from text descriptions.

Our design is based on GAN architecture andwe choose StackGAN as baseline method. In-stead of focusing on generating high-resolutionand photo-realistic images, we add another com-

ponent to maximize the mutual information of theinput text and the generated image to express asmuch information as possible.

3 Methodology

In this section, we first briefly describe some pre-liminaries of GAN which we built upon, and thenintroduce our new network architecture design.

3.1 Background

Generative Adversarial Networks (GAN) is theclass of unsupervised learning, which is composedof two models: the generator modelG and the dis-criminator model D. These two models are op-timized alternatively. The discriminator D is op-timized to distinguish whether an image is a realimage or fake image that generated from G, whilethe generator G is optimized to generate imagesthat are close to the true distribution of the realimages such that D can’t distinguish.

More concretely, we optimize the following ob-jective function which is similar to a two-playermin-max game:

minG

maxD

V (D,G) = Ex∼pdata [logD(x)]

−Ez∼pz [log(1−D(G(z)))](1)

where x is sampled from the true data distributionpdata and z is a noise vector sampled from Gaus-sian distribution pz . It is proved in (Goodfellowet al., 2014) that this min-max game has globaloptimum when pg converges to pdata.

3.2 Information Flow

The task of text to image is to generate the imagefrom the text description and we want to maximizethe information of the input text to be expressed inthe generated image. The GAN model is able togenerate photo-realistic images, however, we arenot sure whether the information from the inputtext is expressed in the generated image as muchas possible, since the information is learned in aimplicit way. Also, it’s well known that GANmodels suffer from mode collapse, which cast fur-ther concerns about whether the information iswell reserved.

To solve this limitation, we introduce the idea ofinformation flow. That is, given the input text, wewant the information from the text to be reservedand expressed as much as possible in the generatedimage. In other words, we want to maximize the

mutual information of the input text and the gener-ated images. However, it’s intractable to directlycalculate the mutual information of the text andthe images. To approximate this problem, we canmake an assumption that if the generated imagereserves the information from the text, then fromthe generated image, we should also be able to re-cover the original text from the generated image.This can be expressed in the chain manner:

text→ image→ text.

In addition to text to image pipeline, the generatedimage can be decoded back to text description orembedding vector and use this info to maximizethe mutual info in objective function. In this way,we realize combining both pipelines of text to im-age synthesis and image captioning in the infor-mation flow.

3.3 Model Architecture

The network architecture are shown in Fig. 2.There are 3 main components in our network ar-chitecture (best view in colors):

• Generator G: the generator takes the text de-scription and a random noise z from Gaus-sian distribution as input and generate a fakeimage.

• Discriminator D: the discriminator takes afake or real image as input along with thecorresponding text embedding ψ, then judgewhether the input image is fake or real.

• Captioner C: the Captioner will take the fakeor real image as input and try to decode intoa feature vector that matches the size of thetext embedding.

The Generator and Discriminator can be sum-marized into a traditional text to image synthesiswork (Mansimov et al., 2015; Reed et al., 2016b;Zhang et al., 2017a) using GAN techniques. In theexperiment, we use the implementation of Stack-GAN as our baseline. While the Captioner isused to enforce the information flow introduced inSec.3.2. The naming comes from the similarity tothe traditional Image Captioning works (Karpathyand Fei-Fei, 2017; Collobert and Collobert, 2015)that map the images into text descriptions.

amostlybrownbirdwithawhitefaceandabrownbeak.

FCLayer

fakeorreal?

ImageCaptioning

Fake Image

Text Embedding ψ

z

μ

σ

ε

c

GeneratorDiscriminator

Captioner

Text Description s

Upsampling

RealImageDown

sampling

Figure 2: Network Architecture. There are 3 main components: the Generator G in blue, the Discriminator D inpurple and the Captioner C in orange. G and D are the standard components of GAN, while C is our proposedcomponents to implement the information flow, which takes the fake or real image as input and output a featurevector.

3.3.1 Text to Image SynthesisIn this subsection, we will introduce the networkframework that we used for text to image syn-thesis. Basically, any framework on text to im-age synthesis can be extended by our algorithm.Here, we follow the ideas from VAE (Kingmaand Welling, 2013) and GAN (Goodfellow et al.,2014).

More concretely, for a given text descriptions, we obtain its corresponding text embeddingψ using pretrained text encoding model. Thenwe use a fully-connected layer to generate twovectors µ and σ, following the reparameteriza-tion trick introduced in Variational Auto-encoder(VAE) (Kingma and Welling, 2013). Finally weuse the following element-wise formula to get thelatent text vector c:

c = µ+ σ ∗ ε (2)

where ε is a noise vector sampled from the normaldistribution N(0, 1). The motivation of introduc-ing ε is to enforce smoothness in the latent textspace. z is also a random noise vector sampledfrom the normal distribution, which is a standardsetting used in GAN. By varying z, we can gener-ate different images based on the input text. Thenc is concatenated with noise vector z and generatefake images by upsampling blocks. For the dis-criminator D, the text embedding ψ is fed withdownsampled real image vectors. Finally, the Dis-criminator can produce the decision, fake or real.

3.3.2 Captioner

In the information flow chain text → image →text introduced in Sec.3.2, the text to image syn-thesis serves as the first half text→ image, whilethe Captioner component serves the image →text part. Basically, any framework of Image Cap-tioning works can be extended with our algorithmdesign.

The Captioner used in our experiments sharesthe same architecture with the Discriminator. Ittakes a fake or real image as input and use a basicConvolutional Network to encode the image into afeature vector ψ′. So our target goal is to make theimaged decoded text feature vector ψ′ similar tothe original text embedding vector ψ. The optimalψ′ should contain the same information as the textembedding ψ. More concretely, we want to maxi-mize the mutual information between ψ and ψ′ asshown below:

I(ψ;C(G(ψ))) (3)

However, we cannot optimize the mutual infor-mation directly since it is intractable in our prob-lem. Thus in Sec.3.4, we introduce a new objectivefunction to approximate it.

Note that, to save computation, we also shareweights between the Captioner and the Discrim-inator. However in our theoretical analysis, theDiscriminator D and Captioner C are consideredindependently.

3.4 Objective FunctionsDirectly maximizing the mutual information is in-tractable, instead we approximate this problemby minimizing the Euclidean distance betweenthe text embedding ψ and ψ′. Other approxima-tion methods can be further implemented in futureworks.

The three components in Sec.3.3 can be opti-mized alternatively. Specifically speaking, we de-note (It, t) as the real image and the input sen-tence pair sampled from the real data distributionpdata, with ψt as the corresponding text embed-ding. Following the formulas of GAN, we al-ternatively train the Generator, the Discriminatorand the Captioner by optimizing the following for-mula:

minG,C

maxD

LG + LD + LC (4)

where LG, LD, LC represents for the loss functionof the Generator, the Discriminator and the Cap-tioner respectively. To be exact,

LG = Ez∼pz ,t∼pdata [log(1−D(G(z, c)), φt)]

+ λDKL(N(µ(φt),Σ(φt)||N(0, I))(5)

LD = E(It,t)∼pdata [logD(It, φt)]

+ Ez∼pz ,t∼pdata [log(1−D(G(z, c)), φt)](6)

LC = Ez∼pz ,t∼pdata [e(C(G(z, c)), φt)]

+ E(It,t)∼pdata [e(C(It), φt)](7)

in which the e function stands for the mean squareerror between C(G(z, c)) and the text embedding.DKL is the KL divergence regularization termused to enforce the distribution of c to be a nor-mal distribution from VAE.

Overall, G is trained to generate fake imagesfrom the text description that are closed to the realdata distribution, and it should get to the mini-mum of the Euclidean distance between ψt andC(G(ψt)). D is trained to distinguish the fakeand real images. And C is trained to minimize theEuclidean distance between ψt and C(G(ψt)) andthe distance between ψt and C(It). In this way,our new network cooperating with three compo-nents can learn to reserve and express more mutualinformation between text descriptions and images.

4 Experiments

To validate our approach, we first analyze the qual-itative results with figures generated by five meth-ods for different primitive properties, and thenconduct human evaluation as well. The baselinemodel is generated by the first stage of Stack-GAN using the existing code they released online.We compare four variant models listed in Sec.4.3with it and demonstrate that our models producehigher-quality samples than StackGAN and ex-press more correct info in the generated figures.

4.1 Dataset

To evaluate our method, we conduct the exper-iments on two dataset: CAltech-UCSD Birds(CUB) (Wah et al., 2011) dataset and Oxford-102Flowers (Nilsback and Zisserman, 2008) dataset.The CUB dataset contains 200 bird categories with11,788 images. Since each image is associatedwith a bounding box, we preprocess the image bycropping its bounding box to ensure the object-image size ratio greater than 0.75. While theOxford-102 dataset consists of 102 flower cate-gories with 8,189 images.

4.2 Experimental Setup

Our network architecture design is based on Stack-GAN (Zhang et al., 2017a). In StackGAN,their Stage-I sketches the primitive properties likeshape and color, and Stage-II improves its reso-lution and refines with details. Since in currentstage, we just want to purely demonstrate the ef-fectiveness of our new proposed third componentCaptioner in our model design based on GAN. Forthe sake of brevity, we only adopt the Stage-I ofthe Stack GAN as the baseline. In the followingcontents, we use StackGAN to refer to the Stage-Iof StackGAN paper. Stack GAN is based on GANtechniques. In our method, we mix the StackGANwith our proposed Captioner component.

The size of text embedding ψ is 1024. Andwe use Adam (Kingma and Ba, 2014) optimizerthroughout the experiments with batch size of 64and the initial learning rate is 0.0002. For every100 epochs, the learning rate is decayed by 0.5times. During test phase, for each sentence, wesample 4 different Gaussian noise z to get 4 dif-ferent fake images.

StackGAN StageInfo_dfake

StageInfo_dfake_embed

StageInfo_dfake_dreal_embed

Groundtruth

The bird is almost dark gray with long skinny

wings and white head.

Description StageInfo

A bird has a large hooked bill, long neck, brown and white feathers and white

crown.

A black bird with long tail feathers and large curved

beak.

The bird is completely back which has tiny brown

bill and skinny body.

Figure 3: Example results on CUB dataset by StackGAN StageI and our improved methods, i.e. StageInfo,StageInfo dfake, StageInfo dfake embedded and StageInfo dfake dreal embed.

4.3 InfoStack Variants

To evaluate the effectiveness of our proposedmethod, we compare 4 different variants with dif-ferent LC :

• First, StageInfo method directly use the fea-ture vector of D as the ψ′ and compute itsdistance to the text embedding ψ:

• Second, StageInfo dfake method is our firsttry to add Captioner model to StackGANwith the fake image info generated by G.Basically, we change the objective functionwith Captioner Loss function LC on the la-tent code c:

LC = Ez∼pz ,t∼pdata [e(C(G(z, c)), c)]

• Third, StageInfo dfake embed method cal-culates the loss of embeddings rather than thelatent code c:

LC = Ez∼pz ,t∼pdata [e(C(G(z, c)), φt)]

• Finally, the StageInfo dfake dreal embedmethod further add the real image info basedon StageInfo dfake embed method:

LC = Ez∼pz ,t∼pdata [e(C(G(z, c)), φt)] +E(It,t)∼pdata [e(C(It), φt)]

4.4 Evaluation Metrics

It’s hard to compare the quality of generated im-ages by some handcrafted rules. To analyze theresults quantitatively, we ask people outside theproject to visually evaluate images generated fromthe baseline and from our proposed method.

4.5 Experimental Results

We compare our 4 variants with StackGAN base-line method on CUB and Oxford datasets. In gen-eral, our variants outperform StackGAN and showtheir advantages in different aspects. The betteraveraged results indicate our proposed model isable to produce higher quality samples and con-tain more complete info.

As shown in Figure 3, the 64 × 64 images aregenerated by StackGAN and our InfoGAN vari-ants. We choose bird examples in different cate-gories, colors and postures. Each line of figuresare generated by the same text description listedin the first column. The first column of figures areground-truth. The second column of figures arethe baseline StackGAN results. And the followingfour columns are our four variants’ results. Wewill first analyze them in different primitive prop-erties like shape and color of the birds, and thencompare the advantages and disadvantages of ourfour variants in details.

this flower has a brown center surrounded by layers of long yellow petals with

rounded tips.

this flower has long white petals and a large yellow

stigma in the middle

this flower is pink and white in color, with petals that are

pointed upward.

the flower is big and white with soft, smooth and big

petals that has stamen in the center.

Ground truth Description Stack GAN StageInfo_dfake_embed

StageInfo_dfake_dreal_embed

Figure 4: Example results on Oxford Dataset by StackGAN StageI and our improved methods, i.e. Stage-Info dfake embedded and StageInfo dfake dreal embed.

Qualitative comparison: First, let’s take alook at the first example figures in the first rowof Figure 3. The text description is to generatea bird that is almost dark gray with long skinnywings and white head. For color property, thekey words are “dark grey” and “white”. We cansee that most of the results have correct dark greycolor but only StageInfo presents the “white head”obviously. This shows that our Captioner compo-nent successfully completes more info in the gen-erated result. As for the shape property, the keywords are “skinny wings”. The baseline figure iseven hard for us to distinguish the wings and alsono “skinny” feature presented. In our InfoStackvariants, we can see the shape of flying birds andStageInfo outperforms best. It also expressed theskinny shape property.

Let’s take a further look to the second row sam-ples. This time we need to generate a brown‘duck-like’ bird with hooked bill, long neck andwhite crown. Because this ‘duck-like’ shape is notcommon in the training set, the results are not asgood as others. But we still shows the improve-ments from our models. We can see StackGANand our StageInfo methods are fail to present anyshape of birds. For our later variants, we canclearly see the brown feathers in StageInfo dfakemethod result, and white colors in all the four vari-ants, and the long neck in StageInfo dfake andStageInfo dfake dreal embed methods.

This is a relatively difficult case, let’s see someeasier examples. This time we focus on one prim-itive property, shape, in the third and fourth ex-amples. The third row aims to draw a back birdwith long tail and large curved beak. We can seethe third variant StageInfo dfake embed methodgives the best fit result. Long downward tail andlarge curved beak, which is exactly the same as theground truth figure. Furthermore, the last row isan example to draw a bird that is completely backwhich has tiny brown bill and skinny body. Ex-cept for the posture and shape, this time we haveanother new property “skinny body”, which is alsobetter expressed in the last four results.

The above analysis are based on the CUBdataset for birds. However, we didn’t see big ad-vantages in the Oxford dataset for flowers. Be-cause flowers have less variance in shapes andposes, the StackGAN method has already beenable to capture enough information. The addedCaptioner component may be helpful but it’s notobvious.

Component analysis: We just list some typi-cal results above to demonstrate the effectivenessof our approaches. Here we can analyze differ-ent variants of InfoStack on CUB bird dataset withbaseline StackGAN. Generally speaking, com-pared to StackGAN, the StageInfo method canmaintain almost all main color features. The shapefeature gives either extremely good or terribly bad

CUB Oxfordrealistic relevance realistic relevance mean

StackGAN 1.91 1.78 1.62 2.04 1.84StageInfo dfake embed 2.07 2.15 2.27 1.95 2.11

StageInfo dfake dreal embed 2.02 2.05 2.11 2.02 2.05

Table 1: Average Human Rating

results. So this indicates that our Captioner designdoes help maintain main more information of textdescriptions to image, but it’s hard to capture thetrue relationship among the features. Furthermore,adding fake image info into our model reduces thefailure rate and makes the generated figure morerobust. The either extremely good or terribly badissue does not happen in StageInfo dfake results.One more improvement is that, we find that takinguse of embedding info rather than the real imagedata gives better performance. No matter for thecolor, shape, smoothness or even image resolution,the resulted figure of StageInfo dfake embed havebetter quality. Finally, we also add real image info,which seems not to help much in primitive prop-erties. But we find that normally the results fromStageInfo dfake dreal embed method present thesimilar contexts with the ground-truth figure, likebirds flying in the sky, sitting on a stick, swimmingin a late, etc.

All in all, we can say that generally speak-ing, our proposed InfoStack approaches outper-form StackGAN in primitive properties compari-son. But among the four variants, we cannot saywhich one is the best. Each variants has its prosand cons and different improvements expressed ondifferent samples.

User Study: To better present a objecti ari-son on the visual results, we conduct a user studyby questionnaire survey. In the questionnaire, theusers are asked to rank images generated from 3methods, StackGan, StageInfo dfake embed andStageInfo dfake dreal embed, by their degree ofrealistic without text descriptions or by their de-gree of relevance to the given text description. Forboth CUB dataset and Oxford dataset, we ran-domly sample 10 groups of images with the sametext description for each group. During the survey,the identifier of each image is hidden and the im-age orders within each group are randomly shuf-fled. When ranking 1st, 2nd or 3rd, 3, 2 or 1 cred-its will be assigned respectively. The results ofaverage human rating are shown in Tab. 1.

From the user study, we can see that in the CUBdataset, the StageInfo dfake embed method andthe StageInfo dfake dreal embed method havesimilar performance and they both outperformthe StackGAN method in terms of realisticand relevance. In the Oxford dataset, theStageInfo dfake embed method and the Stage-Info dfake dreal embed method outperform theStackGAN method in degree of realistic, but notin degree of relevance. Overall we can see that ourproposed methods with information flow can gen-erate more visually plausible and more relevant re-sults than the baseline method StackGAN withoutinformation flow.

5 Conclusions

In this paper, we proposed a new neural networkwith information flow which adds a third compo-nent ‘Captioner’ to GAN structor. The key ideais to utilize the recovered original text from thegenerated image and feed it back to our networkarchitecture, which is better expressed in the chainmanner: text→ image→ text. Based on the exist-ing framework StackGAN, we compare it with ourfour variant implementations. Our experimentsshow that the Captioner component can keep moreprimitive features and express more genuine infoin the generated images. Furthermore, taking theembeddings in error calculation helps to improvethe image info quality. In addition, generated fakeimage info and real image info are also helpfulto improve the reliability of quality. Our workis promising to be further improved by combin-ing these variants of implements and show its ad-vantage in more advanced GAN architectures withmore sophisticated datasets.

Acknowledgments

This work is our final project in course CS388NLP, UT Austin, Spring 2018. We are grateful toProf. Raymond J. Mooney and TA Ghufran Baigfor their insightful comments and help.

ReferencesMartın Arjovsky and Leon Bottou. 2017. Towards

principled methods for training generative adversar-ial networks. CoRR, abs/1701.04862.

Xinyuan Chen, Chang Xu, Xiaokang Yang, andDacheng Tao. 2018. Attention-gan for object trans-figuration in wild images. CoRR, abs/1803.06798.

Ronan Collobert and Ronan Collobert. 2015. Phrase-based image captioning. pages 2085–2094.

Emily L. Denton, Soumith Chintala, Arthur Szlam, andRob Fergus. 2015. Deep generative image modelsusing a laplacian pyramid of adversarial networks.In NIPS.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza,Bing Xu, David Warde-Farley, Sherjil Ozair, AaronCourville, and Yoshua Bengio. 2014. Generative ad-versarial nets. In Advances in neural informationprocessing systems, pages 2672–2680.

Karol Gregor, Ivo Danihelka, Alex Graves,Danilo Jimenez Rezende, and Daan Wierstra.2015. Draw: A recurrent neural network for imagegeneration. In ICML.

A Karpathy and L Fei-Fei. 2017. Deep visual-semanticalignments for generating image descriptions. IEEETransactions on Pattern Analysis & Machine Intelli-gence, 39(4):664–676.

Diederik P. Kingma and Jimmy Ba. 2014. Adam:A method for stochastic optimization. CoRR,abs/1412.6980.

Diederik P. Kingma and Max Welling. 2013. Auto-encoding variational bayes. CoRR, abs/1312.6114.

Elman Mansimov, Emilio Parisotto, Jimmy Ba,and Ruslan Salakhutdinov. 2015. Generatingimages from captions with attention. CoRR,abs/1511.02793.

Mehdi Mirza and Simon Osindero. 2014. Conditionalgenerative adversarial nets. CoRR, abs/1411.1784.

Maria-Elena Nilsback and Andrew Zisserman. 2008.Automated flower classification over a large numberof classes. 2008 Sixth Indian Conference on Com-puter Vision, Graphics & Image Processing, pages722–729.

Scott E. Reed, Zeynep Akata, Honglak Lee, and BerntSchiele. 2016a. Learning deep representations offine-grained visual descriptions. 2016 IEEE Confer-ence on Computer Vision and Pattern Recognition(CVPR), pages 49–58.

Scott E. Reed, Zeynep Akata, Xinchen Yan, Lajanu-gen Logeswaran, Bernt Schiele, and Honglak Lee.2016b. Generative adversarial text to image synthe-sis. In ICML.

Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba,Vicki Cheung, Alec Radford, and Xi Chen. 2016.Improved techniques for training gans. In NIPS.

C. Wah, S. Branson, P. Welinder, P. Perona, and S. Be-longie. 2011. The Caltech-UCSD Birds-200-2011Dataset. Technical Report CNS-TR-2011-001, Cal-ifornia Institute of Technology.

Jianwei Yang, Anitha Kannan, Dhruv Batra, and DeviParikh. 2017. Lr-gan: Layered recursive generativeadversarial networks for image generation. CoRR,abs/1703.01560.

Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang,Xiaolei Huang, Xiaogang Wang, and DimitrisMetaxas. 2017a. Stackgan: Text to photo-realisticimage synthesis with stacked generative adversar-ial networks. In IEEE Int. Conf. Comput. Vision(ICCV), pages 5907–5915.

Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang,Xiaogang Wang, Xiaolei Huang, and Dimitris N.Metaxas. 2017b. Stackgan++: Realistic image syn-thesis with stacked generative adversarial networks.CoRR, abs/1710.10916.