Disentangled Representation Learning for Text Style TransferVineet John

University of Waterloovineet.john@uwaterloo.ca

Lili MouAdeptMind Research

doublepower.mou@gmail.comlili@adeptmind.ai

Hareesh BahuleyanUniversity of Waterloo

hpallika@uwaterloo.ca

Olga VechtomovaUniversity of Waterloo

ovechtom@uwaterloo.ca

Abstract

This paper tackles the problem of disentangling the latentvariables of style and content in language models. We pro-pose a simple, yet effective approach, which incorporatesauxiliary objectives: a multi-task classification objective, anddual adversarial objectives for label prediction and bag-of-words prediction, respectively. We show, both qualitativelyand quantitatively, that the style and content are indeed dis-entangled in the latent space, using this approach. This dis-entangled latent representation learning method is appliedto attribute (e.g. style) transfer on non-parallel corpora.We achieve similar content preservation scores compared toprevious state-of-the-art approaches, and significantly betterstyle-transfer strength scores. Our code is made publiclyavailable for replicability and extension purposes 1.

1 IntroductionThe neural network has been a successful learning machineduring the past decade due to its highly expressive modelingcapability, which is a consequence of multiple layers of non-linear transformations of input features. Such transforma-tions, however, make intermediate features ‘latent’, in thatthey do not have explicit meaning and are not explainable.Therefore, neural networks are usually treated as black-boxmachinery.

Disentangling the latent space of neural networks has be-come an increasingly important research topic. In the im-age domain, for example, Chen et al. (2016) use adversarialand information maximization objectives to produce inter-pretable latent representations that can be tweaked to adjustwriting style for handwritten digits, as well as lighting andorientation for face models. Mathieu et al. (2016) utilizea convolutional autoencoder to achieve the same objective.However, this problem is not well explored in natural lan-guage processing.

In this paper, we address the problem of disentangling thelatent space of neural networks for text generation. Ourmodel is built on an autoencoder that encodes the latentspace (vector representation) of a sentence by learning to re-construct the sentence itself. We would like the latent space

Copyright c© 2015, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

1https://github.com/vineetjohn/linguistic-style-transfer

to be disentangled with respect to different features, namely,style and content.

To accomplish this, we propose a simple approach thatcombines multi-task and adversarial objectives. We artifi-cially divide the latent representation into two parts: stylespace and content space. We learn model parameters thatproduce separate style and content latent spaces from theencoded sentence representation. The multi-task objectiveoperates on the style space to ensure it encodes style infor-mation. The adversarial style objective, on the contrary, op-erates on the content space to minimize the predictability ofstyle. In addition, the bag-of-words adversary minimizes thepredictability of original words of the sentence in the stylespace. In this way, the style and content variables can bedisentangled from each other.

These representation learning objectives can be directlyused for style-transfer in sentence generation (Hu et al.2017; Shen et al. 2017; Fu et al. 2018), in which a modelcan generate a sentence with the same content, but a differ-ent style. We simply use an autoencoder to encode the con-tent vector of a sentence, but ignore its encoded style vector.Then we infer from the training data, an empirical embed-ding of the style that we want to transfer. The encoded con-tent vector and the inferred style vector are concatenated andfed to the decoder. Using this grafting technique, we gener-ate a new sentence similar in content but different in style.

We conducted experiments on two customer reviewbenchmark datasets. Both qualitative and quantitative re-sults show that the style vector does contain most style in-formation, whereas the content vector contains little (if any).In the empirical style-transfer evaluations, we achieve sig-nificantly better style-transfer strength scores than previousresults, while obtaining better or comparable content preser-vation scores. We also show, using ablation tests, that theauxiliary losses can be combined well, each playing its ownrole in disentangling the latent space.

2 Related WorkDisentanglement of latent spaces has been explored inthe image processing domain in the recent years, and re-searchers have successfully disentangled rotation features,color features, etc. (Chen et al. 2016; Luan et al. 2017).Some image characteristics (e.g. artistic style) can be cap-tured well by certain statistics, like minimizing the L2

arX

iv:1

808.

0433

9v1

[cs

.CL

] 1

3 A

ug 2

018

GRURNN

GRURNN

the productis great }the

productis great

Figure 1: Model Training Overview

GRURNN

GRURNN

the book

is good

the book

is boring

Figure 2: Model Generation Overview

loss between a noise image and a pair of style and con-tent images (Gatys, Ecker, and Bethge 2016). In otherworks, researchers adopt data augmentation techniques tolearn a disentangled latent space (Kulkarni et al. 2015;Champandard 2016).

In natural language processing, the definition of ‘style’itself is vague, and as a convenient starting point, NLP re-searchers typically treat sentiment as a salient style of text.Hu et al. (2017) manage to control the sentiment by us-ing discriminators to reconstruct sentiment and content fromgenerated sentences. However, there is no evidence thatthe latent space would be disentangled by this reconstruc-tion. Shen et al. (2017) use a pair of adversarial discrim-inators to align the recurrent hidden decoder states of (a)original sentences with a given style and (b) sentences trans-ferred to the given style, thereby performing style-transfer.Fu et al. (2018) propose two approaches, (a) using a styleembedding matrix, and (b) using style-specific decoders forstyle-transfer. They apply an adversarial loss on the encodedspace to discourage encoding style in the latent space of anautoencoding model.

Our paper differs from the previous work in that both ourstyle space and content space are encoded from the input.We apply three auxiliary losses to ensure that each space en-codes its own information, including a novel bag-of-wordsadversary that helps disentangle the style space from sen-tence content. Our disentangled representation learningfunction can then be directly applied to text style-transfertasks, as in the aforementioned studies.

3 ApproachIn this section, we describe our approach in detail. Ourmodel is built upon an autoencoder with a sequence-to-sequence (Seq2Seq) neural network (Sutskever, Vinyals, andLe 2014), as described in Subsection 3.1. Then, we intro-duce three auxiliary losses: a multi-task classification loss,

a style adversarial loss, and a content adversarial loss inSubsections 3.2, 3.3 and 3.4, respectively. Subsection 3.6presents the approach to transfer style in the context of nat-ural language generation. Figure 1 and Figure 2 depict thetraining and style-transferred sentence generation processesof our approach, respectively.

3.1 AutoencoderAn autoencoder encodes an input to a latent vector space,which is usually of a much lower dimensionality than theinput data. From this latent space, it decodes the input itself.By doing so, the autoencoder learns salient and compact rep-resentations of data. This serves as our primary learning ob-jective. Besides, we also use the autoencoder for text gener-ation in the style-transfer application.

Let x = (x1, x2, · · ·xn) be an input sequence with n to-kens. The encoder recurrent neural network (RNN) withgated recurrent units (GRU) (Cho et al. 2014) encodes x andobtains a hidden state h. Then a decoder RNN generates asentence, which ideally should be x itself. Suppose at a timestep t, the decoder RNN predicts the word xt with probabil-ity p(xt|h, x1 · · ·xt−1), then the deterministic autoencoder(DAE) is trained using a sequence-aggregated cross-entropyloss, given by

Jrec(θE ,θD) = −n∑

t=1

log p(xt|h, x1 · · ·xt−1) (1)

where θE and θD are the parameters of the encoder and de-coder, respectively.

Variational Autoencoder In addition to the determinis-tic auto-encoding objective presented above, we also imple-ment a variational autoencoder (VAE) (Kingma and Welling2014). The variational sampling and Kullback-Leibler (KL)divergence (Kullback and Leibler 1951) losses are applica-ble to both the style space s and the content space c, sepa-rately. The weights applied to each KL divergence loss aretuned independently as model hyperparameters.

Equation 2 shows the reconstruction objective that is min-imized in the VAE variant of our model.

Jrec(θE ,θD) =− EqE(s|x)qE(c|x)[log p(x|s, c)]+ λsklKL(qE(s|x)||p(z))+ λcklKL(qE(c|x)||p(z)) (2)

where λskl and λckl are the weights for each of the KL di-vergence terms, and p(z) is the standard normal distributionN (0, I) that both the encoded style and content distributionsare aligned to.

The motivation for using a variational autoencoder vari-ant to compare the base deterministic autoencoder model to,is the property of VAEs that enable them to learn smoothand continuous latent spaces, without many dead zones inthe latent space that cannot be decoded from. Bowman etal. (2016) show this empirically by using the latent spaceof a text variational autoencoder to interpolate and generatenovel sentences from the latent spaces between two validsentences from a corpus.

(a) Style Space (b) Content SpaceEpoch 1

(a) Style Space (b) Content SpaceEpoch 10

(a) Style Space (b) Content SpaceEpoch 20

Figure 3: t-SNE Plots: Deterministic Autoencoder

Since this objective is used to train the model to recon-struct x, it is also called the reconstruction loss. Besides theabove reconstruction loss, we design three auxiliary losses todisentangle the latent space h. In particular, we hope that hcan be separated into two spaces s and c, representing styleand content respectively, i.e., h = [s; c], where [·; ·] denotesconcatenation. This is accomplished by the auxiliary lossesdescribed in the subsequent sections.

3.2 Multi-Task Classification LossOur first auxiliary loss ensures the style space does containstyle information. We build a classifier on the style space swith the objective of predicting the true style label y′, whichis part of the training data.

This loss can be viewed as a multi-task loss, which incen-tivizes the model to not only decode the sentence, but alsopredict its sentiment from a fixed subset of the latent vari-ables. Similar multi-task losses are used in previous workfor sequence-to-sequence learning (Luong et al. 2015), sen-tence representation learning (Jernite, Bowman, and Sontag2017) and sentiment analysis (Balikas, Moura, and Amini2017), among others.

In our application, we follow previous work (Hu et al.2017; Shen et al. 2017; Fu et al. 2018) and treat the sen-timent as the style of interest. We introduce a multi-labelclassifier

y = σ(w>mults+ bmult) (3)

where θmult = [wmult; bmult] are the classifier’s parametersfor multi-task learning, and y is the predicted label distribu-tion.

(a) Style Space (b) Content SpaceEpoch 1

(a) Style Space (b) Content SpaceEpoch 10

(a) Style Space (b) Content SpaceEpoch 20

Figure 4: t-SNE Plots: Variational Autoencoder

This is trained using a simple cross-entropy loss

Jmult(θE ;θmult) = −E[log p(y′|s;θmult)] (4)

where θE are the encoder’s parameters, and y′ is the truelabel distribution.

3.3 Adversarial Style DiscriminationThe above multi-task loss only operates on the style space,but does not have an effect on the content space c.

We therefore apply an adversarial loss to disentangle thecontent space from style information, inspired by adver-sarial generation (Goodfellow et al. 2014), adversarial do-main adaptation (Liu, Qiu, and Huang 2017), and adversar-ial style-transfer (Fu et al. 2018).

The idea of adversarial loss is to introduce an objectivethat deliberately discriminates the true style label using thecontent vector c. Then the autoencoder is trained to learn acontent vector space that its adversary cannot predict styleinformation from.

Concretely, the adversarial discriminator predicts style bycomputing a softmax distribution over the possible class la-bels

y = σ(w>disc+ bdis) (5)

where θdis = [wdis; bdis] are the parameters of the adversary,and y is the predicted label distribution.

It is trained by minimizing the following objective, usinga cross-entropy loss.

Jdis(θdis) = −E[log p(y′|c;θdis)] (6)

where y′ is the true label distribution.

Model Transfer Content Word LanguageStrength Preservation Overlap Fluency

Cross-Alignment (Shen et al. 2017) 0.809 0.892 0.209 -23.39Style Embedding (Fu et al. 2018) 0.182 0.959 0.666 -16.17Ours (DAE) 0.843 0.892 0.255 -16.48Ours (VAE) 0.890 0.882 0.211 -14.41

Table 1: Yelp Dataset - Comparison

Model Transfer Content Word LanguageStrength Preservation Overlap Fluency

Cross-Alignment (Shen et al. 2017) 0.606 0.893 0.024 -26.31Style Embedding (Fu et al. 2018) 0.417 0.933 0.359 -28.13Ours (DAE) 0.703 0.918 0.131 -32.42Ours (VAE) 0.726 0.909 0.081 -28.50

Table 2: Amazon Dataset - Comparison

The adversarial loss appears similar to the multi-task lossas in Equation 4. However, it should be emphasized that,for the adversary, the gradients are not propagated back tothe autoencoder, i.e. the variables in c are treated as shallowfeatures.

Having trained an adversary, we would like the autoen-coder to be tuned in such an ad hoc fashion, that c is not dis-criminative in style. In other words, we penalize the Shan-non entropy of the adversary’s prediction, given by

Jadv(θE) = H(y|c;θdis) (7)

where H(p) = −∑

i∈labels pi log pi is the entropy and y isthe predicted distribution over the style labels. The adver-sarial objective is maximized, in this phase, with respect tothe encoder. This objective helps the autoencoder maximizethe uncertainty of the discriminator’s predicted probabilitydistribution.

3.4 Adversarial Bag-of-Words DiscriminatorIn addition to the auxiliary losses used above, we also pro-pose a bag-of-words discriminator on the style space tomake our approach complete.

The motivation is to emulate the adversarial signal pro-vided by the style discriminator, and do the same for thecontent. Here, we define the content of the sentence as thewords from the original sentence without any words that arediscriminative of style.

The input sentences x are represented as vectors of thesame size as the corpus vocabulary, with each index of thevector denoting the discrete probability of a word’s presencein the sentence. Therefore, this bag-of-words representationis comprised of only 0s and 1s.

The bag-of-words discriminator uses the style vector sproduced by the autoencoder model and tries to predict thetrue bag-of-words distribution using a set of parameters thatare distinct from those of the autoencoder. The discriminatoruses a logistic regression to predict the probability of eachword’s occurrence in the original sentence, between 0 and 1.

b = σ(w>bows+ bbow) (8)

where θbow = [wbow; bbow] are the classifier’s parameters forbag-of-words prediction, and b is the predicted word distri-bution.

This objective is trained in a similar method to the styleadversary, using a cross-entropy loss

Jbow(θbow) = −E[log p(b′|s;θbow)] (9)

where b′ is the true word distribution.We also refrain from propagating the effects of this dis-

criminator loss Jbow to the encoder parameters, ensuring thatthe parameters that each adversary and the autoencoder canupdate are mutually exclusive.

Similar to the style adversary, the empirical Shannon en-tropy of the predicted distribution is provided as a trainingsignal for the autoencoder model to maximize.

Jbadv(θE) = H(b|s;θbow) (10)

where b is the predicted word distribution.The motivation for this adversarial loss is similar to the

one used in the context of the style discriminator. We wantto encourage the encoder to learn a representation of style sthat the bag-of-words discriminator cannot predict most ofthe original words from.

3.5 Training ProcessThe overall loss Jovr, used for the autoencoder, is thus com-prised of four distinct objectives: the reconstruction objec-tive, the multi-task objective, and the adversarial objectivesfor style and content.

Jovr =Jrec(θE ,θD)− λadvJadv(θE)+

λmultJmult(θE ,θmult)− λbadvJbadv(θE)

where λmult, λadv and λbadv balance the model’s auxiliarylosses.

To put it all together, the model training consists of a loopinvolving the processes as shown in Algorithm 1.

We use the Adam optimizer (Kingma and Ba 2014) forthe autoencoder and the RMSProp optimizer (Tieleman and

Objectives Transfer Content Word LanguageStrength Preservation Overlap Fluency

Jrec 0.144 0.915 0.329 -14.28Jrec, Jadv 0.727 0.880 0.204 -14.16Jrec, Jmult 0.789 0.898 0.259 -14.56Jrec, Jbadv 0.168 0.915 0.328 -14.45Jrec, Jadv, Jmult 0.890 0.882 0.211 -14.41Jrec, Jadv, Jbadv 0.749 0.883 0.202 -14.36Jrec, Jmult, Jbadv 0.783 0.896 0.257 -14.34Jrec, Jadv, Jmult, Jbadv 0.885 0.878 0.197 -14.05

Table 3: Ablation Tests

1 while epochs remaining do2 minimize Jdis(θdis) w.r.t. θdis;3 minimize Jbow(θbow) w.r.t. θbow;4 minimize Jovr w.r.t. θE ,θD,θmult;5 end

Algorithm 1: Training Process

Hinton 2012) for the discriminators, each with an initiallearning rate of 10−3, and train the model for 20 epochs.Both the autoencoder and the discriminators are trained onceper epoch with λmult = 1, λadv = 0.3 and λbadv = 0.001.The recurrent unit size is 256, the style vector size is 8, andthe content vector size is 128. We append the latent vector hto the hidden state at every time step of the decoder (Bahu-leyan et al. 2018b). For the VAE model, we set λskl = 0.03and λckl = 0.03 and use the sigmoid KL-weight annealingschedule proposed by Bahuleyan et al. (2018a).

3.6 Generating Style-Transferred SentencesA direct application of our disentangled latent space is style-transfer for natural language generation. For example, wecan generate a sentence with generally the same meaning(content) but a different style (e.g. sentiment).

Let x∗ be an input sentence with s∗ and c∗ being the en-coded, disentangled style and content vectors, respectively.If we would like to transfer its content to a different style, wecompute an empirical estimate of the target style’s vector sby

s =

∑i∈target style si

# target style samples

The inferred target style s is concatenated with the encodedcontent c∗ for decoding style-transferred sentences (Fig-ure 2).

4 Experiments4.1 DatasetsWe conduct experiments on two datasets, the details forwhich are given below. Both of these datasets are comprisedof sentences accompanied by binary sentiment labels (pos-itive, negative) and are, therefore, used to evaluate the taskof sentiment transfer.

Yelp Service Reviews We use a Yelp review dataset (Chal-lenge 2013), which has been sourced from the code reposi-tory accompanying the implementation of the paper by Shenet al. (2017) 2. It contains 444101, 63483 and 126670 sen-tences for train, validation, and test, respectively. The maxi-mum sentence length is 15, and the vocabulary size is about9200.

Amazon Product Reviews We also use an Amazon prod-uct reviews dataset 3, following Fu et al. (2018). The re-views were sourced from the code repository accompanyingthe paper 4. It contains 559142, 2000 and 2000 sentences fortrain, validation, and test, respectively. The maximum sen-tence length is 20, and the vocabulary size is about 58000.

4.2 Evaluation MetricsWe evaluate our method using four metrics: style-transferstrength, content preservation, word overlap, and languagefluency.

Style-Transfer Strength We train a convolutional neuralnetwork (CNN) style classifier (Kim 2014) and predict thestyle of the generated sentences. While the style classi-fier itself may not be perfect, it provides a quantitative wayof evaluating the strength of style-transfer (Hu et al. 2017;Shen et al. 2017; Fu et al. 2018). The validation accuraciesof the sentiment classifier trained on the Yelp and Amazondatasets are ≈ 0.97 and ≈ 0.82, respectively. The classi-fier accuracy on the style-transferred sentences, consideringthe target style to be the true label, is reported as the style-transfer strength.

Content Preservation We compute a sentence embeddingby min, max, and average pooling its constituent word em-beddings. Then, the cosine similarity between the sourceand generated sentence embedding is computed to evaluatehow close they are in meaning. Here, sentiment words froma stop list (Hu and Liu 2004) are removed (Fu et al. 2018).The cosine similarity is reported as the content preservationscore.

2https://github.com/shentianxiao/language-style-transfer

3http://jmcauley.ucsd.edu/data/amazon/4https://github.com/fuzhenxin/text_style_

transfer

Original (Positive) DAE Transferred VAE Transferred(Negative) (Negative)

i would recommend a visit here i would not recommend this placeagain

i would not recommend this place formy experience

the restaurant itself is romantic andquiet

the restaurant itself is soooo quiet the restaurant itself was dirty

my experience was brief but very good my experience was very loud and veryexpensive

my experience was ok but not verymuch

the food is excellent and the service isexceptional

the food is by the worst part is the hor-rible costumer service

the food was bland and i am notthrilled with this

the food is very very amazing like beefand fish

the food is very horrible i have everhad mostly fish

the food is very bland and just notfresh

we will definitely come back here we will not come back here again we will never come back hereboth were very good everything was very bland both were very bad

Original (Negative) DAE Transferred VAE Transferred(Positive) (Positive)

so nasty so helpful so fabulousconsistently slow consistently awesome fast servicecrap fries hard hamburger bunsburger tasted like crap

cheap and yummy sandwiches reallysomething different

yummy hamburgers and blue cheesebagels are classic italian

oh and terrible tea oh and awesome tea oh and great teathe interior is old and generallyfalling apart

the interior is clean and orderly as en-tertaining

the interior is old and noble

front office customer service does notexist here

front office is very professional doesyou

kudos to customer service is very pro-fessional

the crust was kinda gooey like the crust is kinda traditional the crust is soooooo worth it

Table 4: Examples of Style-Transfer Generation

Word Overlap We also utilize a simpler unigram over-lap metric to evaluate the similarity between the source andgenerated sentences. Given a source sentence x and an at-tribute style-transferred sentence y, let wx and wy be the setof unique words present in x and y respectively, while ex-cluding sentiment words (Hu and Liu 2004) and stopwords(Bird and Loper 2004). Then, the word overlap score can becalculated using

word-overlap =count(wx ∩ wy)

count(wx ∪ wy)

which is simply a normalized measure of overlapping uni-grams in the source and target sentences.

Language Fluency We use a trigram Kneser-Neysmoothed language model (Kneser and Ney 1995) as a quan-tifiable and automated scoring metric by which to assessthe quality of generated sentences. It calculates the prob-ability distribution of trigrams in a corpus, based on theiroccurrence statistics, to build a language model. We trainthis language model on the complete corpus that we evalu-ate our style-transfer models with. The log-likelihood scorefor a generated sentence, as predicted by the Kneser-Neylanguage model, is reported as the indicator of language flu-ency.

Manual Evaluation of Language Quality We performmanual evaluations by randomly sampling sentences gen-

erated from each model trained on the Yelp dataset, and re-questing 5 human annotators to rate them on a 1-5 Likertscale 5 (Stent, Marge, and Singhai 2005) based on their syn-tax, grammar and fluency. The aggregate score of all theevaluators is reported as a measure of language quality.

5 Discussion5.1 Disentangling Latent SpaceWe first analyze how the style (sentiment) and content ofthe latent space are disentangled. We train classifiers on thedifferent latent spaces, and report their inference-time clas-sification accuracies in Table 5. These are results from theexperiments on the Yelp dataset.

Random/Majority guess 0.6018

Latent Space DAE VAEContent space (c) 0.6137 0.6567Style space (s) 0.7927 0.7911Complete space ([s; c]) 0.7918 0.7918

Table 5: Style Classification Accuracy

We see that the 128-dimensional content vector c is notparticularly discriminative for style. It achieves accuracies

55=Flawless, 4=Good, 3=Adequate, 2=Poor, 1=Incomprehen-sible

that are only slightly better than random/majority guess.However, the 8-dimensional style vector s, despite its lowdimensionality, achieves significantly higher style classifi-cation accuracy. When combining content and style vectors,we achieve no further improvement. These results verifythe effectiveness of our disentangling approach, because thestyle space does contain style information, whereas the con-tent space does not.

We show t-SNE plots of both the deterministic autoen-coder (DAE) and the variational autoencoder (VAE) modelsin Figure 3 and Figure 4, respectively. As can be seen fromthe t-SNE plots, sentences with different styles are notice-ably separated in a cleaner manner in the style space (LHS),but are indistinguishable in the content space (RHS). It isalso evident that the latent space learned by the variationalautoencoder is considerably smoother and continuous com-pared to the one learned by the deterministic autoencoder.

5.2 Style-Transfer Sentence GenerationWe apply the disentangled latent space to a style-transfersentence generation task, where the goal is to generate a sen-tence with different sentiment. We compare our approachwith previous state-of-the-art work in Table 1 and Table 2.We replicated the experiments with their publicly availablecode and data.

We observe that the style embedding model (Fu et al.2018) performs poorly on the style-transfer objective, result-ing in inflated content preservation and word overlap scores.A qualitative analysis indicates that this model resorts tosimply reconstructing most of the source sentences, and isnot very effective at transferring style.

Results show that, our approach achieves comparablecontent preservation and word overlap scores to previouswork (Shen et al. 2017), and significantly better style-transfer strength scores than either of the models com-pared to, showing that our disentangled latent space can beused for better style-transfer sentence generation. Our VAEmodel also produces the most fluent sentences for the Yelpdataset task, which is corroborated by the manual evaluationresults.

Table 3 presents the results of an ablation test. We seethat both the style adversarial loss and multi-task classifi-cation loss play a role in the strength of style-transfer, andthat they can be combined to further improve performance.Also, the combination of all the losses described yields thebest language fluency score. We observe that the bag-of-words adversary described in Section 3.4 does not providemuch of an improvement in terms of our evaluation metrics.However, this idea can be improved upon in future work.

Model LanguageQuality

Cross-Alignment (Shen et al. 2017) 3.188Style Embedding (Fu et al. 2018) 3.784Ours (DAE) 3.460Ours (VAE) 3.824

Table 6: Results - Manual Evaluation

The manual evaluation results are presented in Table 6.Our VAE model attains the best score for generated sentencequality amongst all the evaluated models. We also observethat the ranking of these models is positively correlated withthe automated ‘Language Fluency’ metric presented in Table1.

Some examples of style-transfer sentence generation arepresented in Table 4. We see that, with the empirically esti-mated style vector, we can reliably control the sentiment ofgenerated sentences.

6 Conclusion and Future WorkIn this paper, we propose a simple yet effective approachfor disentangling the latent space of neural networks usingmulti-task and adversarial objectives. Our learned disentan-glement approach can be applied to text style-transfer tasks.It achieves similar content preservation scores, and signifi-cantly better style-transfer strength scores compared to pre-vious state-of-the-art work.

For future work, we intend to evaluate the effects of dis-entangling the style space for datasets with greater than twodistinct styles. We would also like to explore the possibilityof aligning each encoded style distribution to a unique prior,which could be sampled from at inference time for style-transfer, as opposed to using the empirical mean of training-time style embeddings.

7 AcknowledgementsWe acknowledge the support of the Natural Sciences and En-gineering Research Council of Canada (NSERC) [261439-2013-RGPIN], and Amazon Research Award. The Titan XpGPU used for this research was donated by the NVIDIACorporation.

References[Bahuleyan et al. 2018a] Bahuleyan, H.; Mou, L.; Vamaraju,K.; Zhou, H.; and Vechtomova, O. 2018a. Probabilisticnatural language generation with wasserstein autoencoders.arXiv preprint arXiv:1806.08462.

[Bahuleyan et al. 2018b] Bahuleyan, H.; Mou, L.; Vechto-mova, O.; and Poupart, P. 2018b. Variational atten-tion for sequence-to-sequence models. Proceedings of the27th International Conference on Computational Linguis-tics (COLING).

[Balikas, Moura, and Amini 2017] Balikas, G.; Moura, S.;and Amini, M.-R. 2017. Multitask learning for fine-grainedtwitter sentiment analysis. In SIGIR, 1005–1008.

[Bird and Loper 2004] Bird, S., and Loper, E. 2004. Nltk:the natural language toolkit. In Proceedings of the ACL 2004on Interactive poster and demonstration sessions, 31. As-sociation for Computational Linguistics.

[Bowman et al. 2016] Bowman, S. R.; Vilnis, L.; Vinyals,O.; Dai, A.; Jozefowicz, R.; and Bengio, S. 2016. Generat-ing sentences from a continuous space. In CoNLL, 10–21.

[Challenge 2013] Challenge, Y. D. 2013. Yelp dataset chal-lenge.

[Champandard 2016] Champandard, A. J. 2016. Semanticstyle transfer and turning two-bit doodles into fine artworks.arXiv preprint arXiv:1603.01768.

[Chen et al. 2016] Chen, X.; Duan, Y.; Houthooft, R.; Schul-man, J.; Sutskever, I.; and Abbeel, P. 2016. Infogan: Inter-pretable representation learning by information maximizinggenerative adversarial nets. In NIPS, 2172–2180.

[Cho et al. 2014] Cho, K.; van Merrienboer, B.; Gulcehre,C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; and Ben-gio, Y. 2014. Learning phrase representations using rnnencoder–decoder for statistical machine translation. In Pro-ceedings of the 2014 Conference on Empirical Methods inNatural Language Processing (EMNLP), 1724–1734.

[Fu et al. 2018] Fu, Z.; Tan, X.; Peng, N.; Zhao, D.; and Yan,R. 2018. Style transfer in text: Exploration and evaluation.In AAAI, 663–670.

[Gatys, Ecker, and Bethge 2016] Gatys, L. A.; Ecker, A. S.;and Bethge, M. 2016. Image style transfer using convolu-tional neural networks. In CVPR, 2414–2423.

[Goodfellow et al. 2014] Goodfellow, I.; Pouget-Abadie, J.;Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville,A.; and Bengio, Y. 2014. Generative adversarial nets. InNIPS, 2672–2680.

[Hu and Liu 2004] Hu, M., and Liu, B. 2004. Mining andsummarizing customer reviews. In KDD, 168–177.

[Hu et al. 2017] Hu, Z.; Yang, Z.; Liang, X.; Salakhutdinov,R.; and Xing, E. P. 2017. Toward controlled generation oftext. In ICML, 1587–1596.

[Jernite, Bowman, and Sontag 2017] Jernite, Y.; Bowman,S. R.; and Sontag, D. 2017. Discourse-based objectives forfast unsupervised sentence representation learning. arXivpreprint arXiv:1705.00557.

[Kim 2014] Kim, Y. 2014. Convolutional neural networksfor sentence classification. In EMNLP, 1746–1751.

[Kingma and Ba 2014] Kingma, D. P., and Ba, J. 2014.Adam: A method for stochastic optimization. arXiv preprintarXiv:1412.6980.

[Kingma and Welling 2014] Kingma, D. P., and Welling, M.2014. Auto-encoding variational bayes. International Con-ference on Learning Representations.

[Kneser and Ney 1995] Kneser, R., and Ney, H. 1995. Im-proved backing-off for m-gram language modeling. Inicassp, volume 1, 181e4.

[Kulkarni et al. 2015] Kulkarni, T. D.; Whitney, W. F.; Kohli,P.; and Tenenbaum, J. 2015. Deep convolutional inversegraphics network. In Advances in Neural Information Pro-cessing Systems, 2539–2547.

[Kullback and Leibler 1951] Kullback, S., and Leibler, R. A.1951. On information and sufficiency. The annals of math-ematical statistics 22(1):79–86.

[Liu, Qiu, and Huang 2017] Liu, P.; Qiu, X.; and Huang, X.2017. Adversarial multi-task learning for text classification.In Proceedings of the 55th Annual Meeting of the Associ-ation for Computational Linguistics (Volume 1: Long Pa-pers), volume 1, 1–10.

[Luan et al. 2017] Luan, F.; Paris, S.; Shechtman, E.; andBala, K. 2017. Deep photo style transfer. In Proceed-ings of the IEEE Conference on Computer Vision and Pat-tern Recognition, 4990–4998.

[Luong et al. 2015] Luong, M.-T.; Le, Q. V.; Sutskever, I.;Vinyals, O.; and Kaiser, L. 2015. Multi-task sequence tosequence learning. arXiv preprint arXiv:1511.06114.

[Mathieu et al. 2016] Mathieu, M. F.; Zhao, J. J.; Zhao, J.;Ramesh, A.; Sprechmann, P.; and LeCun, Y. 2016. Dis-entangling factors of variation in deep representation usingadversarial training. In Advances in Neural Information Pro-cessing Systems, 5040–5048.

[Shen et al. 2017] Shen, T.; Lei, T.; Barzilay, R.; andJaakkola, T. 2017. Style transfer from non-parallel text bycross-alignment. In NIPS, 6833–6844.

[Stent, Marge, and Singhai 2005] Stent, A.; Marge, M.; andSinghai, M. 2005. Evaluating evaluation methods for gener-ation in the presence of variation. In International Confer-ence on Intelligent Text Processing and Computational Lin-guistics, 341–351. Springer.

[Sutskever, Vinyals, and Le 2014] Sutskever, I.; Vinyals, O.;and Le, Q. V. 2014. Sequence to sequence learning withneural networks. In NIPS, 3104–3112.

[Tieleman and Hinton 2012] Tieleman, T., and Hinton, G.2012. Lecture 6.5-rmsprop: Divide the gradient by a run-ning average of its recent magnitude. COURSERA: Neuralnetworks for machine learning 4(2):26–31.