Generative Bridging Network for Neural Sequence Prediction

Wenhu Chen,1 Guanlin Li,2 Shuo Ren,5 Shujie Liu,3 Zhirui Zhang,4 Mu Li,3 Ming Zhou3

University of California, Santa Barbara1

Harbin Institute of Technology2

Microsoft Research Asia3

University of Science and Technology of China4

Beijing University of Aeronautics and Astronautics5

wenhuchen@cs.ucsb.edu epsilonlee.green@gmail.com {v-shure, shujliu, v-zhirzh, muli, mingzhou}@microsoft.com

Abstract

In order to alleviate data sparsity and over-fitting problems in maximum likelihood esti-mation (MLE) for sequence prediction tasks,we propose the Generative Bridging Network(GBN), in which a novel bridge module is in-troduced to assist the training of the sequenceprediction model (the generator network). Un-like MLE directly maximizing the conditionallikelihood, the bridge extends the point-wiseground truth to a bridge distribution condi-tioned on it, and the generator is optimized tominimize their KL-divergence. Three differentGBNs, namely uniform GBN, language-modelGBN and coaching GBN, are proposed to pe-nalize confidence, enhance language smooth-ness and relieve learning burden. Experimentsconducted on two recognized sequence predic-tion tasks (machine translation and abstractivetext summarization) show that our proposedGBNs can yield significant improvements overstrong baselines. Furthermore, by analyz-ing samples drawn from different bridges, ex-pected influences on the generator are verified.

1 Introduction

Sequence prediction has been widely used in taskswhere the outputs are sequentially structured andmutually dependent. Recently, massive explo-rations in this area have been made to solve prac-tical problems, such as machine translation (Bah-danau et al., 2014; Ma et al., 2017; Norouzi et al.,2016), syntactic parsing (Vinyals et al., 2015),spelling correction (Bahdanau et al., 2014), imagecaptioning (Xu et al., 2015) and speech recogni-tion (Chorowski et al., 2015). Armed with mod-ern computation power, deep LSTM (Hochreiterand Schmidhuber, 1997) or GRU (Chung et al.,2014) based neural sequence prediction modelshave achieved the state-of-the-art performance.

The typical training algorithm for sequenceprediction is Maximum Likelihood Estimation

𝑥1 𝑥2

𝐴𝑡𝑡

𝑥𝑇

𝑦1𝑗

𝑦2𝑗

𝑦𝑇𝑗𝑗

𝑠 𝑦1𝑗

𝑦𝑇𝑗−1

𝑗

𝑿

𝒀𝒋𝑦1∗ 𝑦2

∗ 𝑦𝑇′∗

𝒀∗

𝒀𝒋

𝑃𝜂(𝑌|𝑌∗)

𝑦1𝑗

𝑦2𝑗

𝑦𝑇𝑗𝑗

𝐵𝑟𝑖𝑑𝑔𝑒 𝑚𝑜𝑑𝑢𝑙𝑒

𝐺𝑒𝑛𝑒𝑟𝑎𝑡𝑜𝑟 𝑛𝑒𝑡𝑤𝑜𝑟𝑘

𝒀𝟏 𝒀𝑲… …

𝑃𝜃(𝑌|𝑋)

𝐵𝑟𝑖𝑑𝑔𝑒 𝑠𝑎𝑚𝑝𝑙𝑒𝑠

…

…

…

…

…

……

Figure 1: The overall architecture of our novel Gen-erative Bridging Network (GBN). Two main compo-nents, namely the generator network and the bridgemodule, are connected through samples (Y 1 . . . Y K inred) from the bridge module during training time. (Wesometimes call them generator and bridge in brief re-spectively in the following discussion.) The generatoris implemented through an attentive encoder-decoder,where in the figure Att represents the attention module.

(MLE), which maximizes the likelihood of the tar-get sequences conditioned on the source ones:

θ∗ = argmaxθ

E(X,Y ∗)∼D

log pθ(Y∗|X) (1)

Despite the popularity of MLE or teacher forc-ing (Doya, 1992) in neural sequence predictiontasks, two general issues are always haunting: 1).data sparsity and 2). tendency for overfitting, withwhich can both harm model generalization.

To combat data sparsity, different strategieshave been proposed. Most of them try to takeadvantage of monolingual data (Sennrich et al.,2015; Zhang and Zong, 2016; Cheng et al., 2016).Others try to modify the ground truth target basedon derived rules to get more similar examples fortraining (Norouzi et al., 2016; Ma et al., 2017).To alleviate overfitting, regularization techniques,

arX

iv:1

706.

0915

2v5

[cs

.AI]

17

Mar

201

8

such as confidence penalization (Pereyra et al.,2017) and posterior regularization (Zhang et al.,2017), are proposed recently.

As shown in Figure 1, we propose a novel learn-ing architecture, titled Generative Bridging Net-work (GBN), to combine both of the benefits fromsynthetic data and regularization. Within the ar-chitecture, the bridge module (bridge) first trans-forms the point-wise ground truth into a bridgedistribution, which can be viewed as a target pro-poser from whom more target examples are drawnto train the generator. By introducing differentconstraints, the bridge can be set or trained to pos-sess specific property, with which the drawn sam-ples can augment target-side data (alleviate datasparsity) while regularizing the training (avoidoverfitting) of the generator network (generator).

In this paper, we introduce three different con-straints to build three bridge modules. Togetherwith the generator network, three GBN systemsare constructed: 1). a uniform GBN, instantiatingthe constraint as a uniform distribution to penal-ize confidence; 2). a language-model GBN, in-stantiating the constraint as a pre-trained neurallanguage model to increase language smoothness;3). a coaching GBN, instantiating the constraint asthe generator’s output distribution to seek a close-to-generator distribution, which enables the bridgeto draw easy-to-learn samples for the generator tolearn. Without any constraint, our GBN degradesto MLE. The uniform GBN is proved to minimizeKL-divergence with a so-called payoff distributionas in reward augmented maximum likelihood orRAML (Norouzi et al., 2016).

Experiments are conducted on two sequenceprediction tasks, namely machine translation andabstractive text summarization. On both of them,our proposed GBNs can significantly improvetask performance, compared with strong base-lines. Among them, the coaching GBN achievesthe best. Samples from these three differentbridges are demonstrated to confirm the expectedimpacts they have on the training of the generator.In summary, our contributions are:

• A novel GBN architecture is proposed for se-quence prediction to alleviate the data spar-sity and overfitting problems, where thebridge module and the generator network areintegrated and jointly trained.

• Different constraints are introduced to buildGBN variants: uniform GBN, language-

model GBN and coaching GBN. Our GBNarchitecture is proved to be a generalizedform of both MLE and RAML.

• All proposed GBN variants outperform theMLE baselines on machine translation andabstractive text summarization. Similar rela-tive improvements are achieved compared torecent state-of-the-art methods in the trans-lation task. We also demonstrate the advan-tage of our GBNs qualitatively by comparingground truth and samples from bridges.

𝐶𝑜𝑛𝑠𝑡𝑟𝑎𝑖𝑛𝑡

𝐺𝑒𝑛𝑒𝑟𝑎𝑡𝑜𝑟

𝐵𝑟𝑖𝑑𝑔𝑒

Regularization-by-synthetic-samples

𝑌∗

𝑝𝐶(𝑌)

𝑆(𝑌, 𝑌∗)

𝐶(𝑝𝜂, 𝑝𝑐)

Knowledge injection

𝑃𝜂(𝑌|𝑌∗)

𝐾𝐿(𝑝𝜂, 𝑝𝜃) 𝑝𝜃(𝑌|𝑋)𝑌∗

𝑝𝐶(𝑌)

Figure 2: Conceptual interpretation of our GenerativeBridging Network (GBN). See detailed discussion inthe beginning of Sec. 2.

2 Generative Bridging Network

In this section, we first give a conceptual interpre-tation of our novel learning architecture which issketched in Figure 2. Since data augmentation andregularization are two golden solutions for tack-ling data sparsity and overfitting issues. We arewilling to design an architecture which can inte-grate both of their benefits. The basic idea is to usea so-called bridge which transforms Y ∗ to an easy-to-sample distribution, and then use this distribu-tion (samples) to train and meanwhile regularizethe sequence prediction model (the generator).

The bridge is viewed as a conditional distribu-tion1 pη(Y |Y ∗) to get more target Y s given Y ∗

so as to construct more training pairs (X,Y ). Inthe meantime, we could inject (empirical) priorknowledge into the bridge through its optimiza-tion objective which is inspired by the design ofthe payoff distribution in RAML. We formulatethe optimization objective with two parts in Equa-tion (2): a) an expected similarity score com-puted through a similarity score function S(·, Y ∗)interpolated with b) a knowledge injection con-straint2 C(pη(Y |Y ∗), pc(Y )) where α controls the

1η should be treated as an index of the bridge distribution,so it is not necessarily the parameters to be learned.

2Note that, in our paper, we specify C to be KL-divergencebetween the bridge distribution pη and certain constraint dis-

strength of the regularization, formally, we writethe objective function LB(η) as follows:

LB(η) =

EY∼pη(Y |Y ∗)

[−S(Y, Y ∗)] + αC(pη(Y |Y ∗), pc(Y ))

(2)

Minimizing it empowers the bridge distributionnot only to concentrate its mass around the groundtruth Y ∗ but also to adopt certain hope propertyfrom pc(Y ). With the constructed bridge distribu-tion, we optimize the generator network Pθ(Y |X)to match its output distribution towards the bridgedistribution by minimizing their KL-divergence:

LG(θ) = KL(pη(Y |Y ∗)||pθ(Y |X)) (3)

In practice, the KL-divergence is approximatedthrough sampling process detailed in Sec. 2.3.As a matter of fact, the bridge is the crux of theintegration: it synthesizes new targets to allevi-ate data sparsity and then uses the synthetic dataas regularization to overcome overfitting. Thusa regularization-by-synthetic-example approach,which is very similar to the prior-incorporation-by-virtual-example method (Niyogi et al., 1998).

2.1 Generator NetworkOur generator network is parameterized withthe commonly used encoder-decoder architec-ture (Bahdanau et al., 2014; Cho et al., 2014). Theencoder is used to encode the input sequence Xto a sequence of hidden states, based on whichan attention mechanism is leveraged to computecontext vectors at the decoding stage. The con-text vector together with previous decoder’s hid-den state and previously predicted label are used,at each time step, to compute the next hidden stateand predict an output label.

As claimed in Equation (3), the generator net-work is not trained to maximize the likelihood ofthe ground truth but tries best to match the bridgedistribution, which is a delegate of the groundtruth. We use gradient descent to optimize the KL-divergence with respect to the generator:

∇LG(θ) = EY∼pη(Y |Y ∗)

log∇pθ(Y |X) (4)

The optimization process can be viewed as thegenerator maximizing the likelihood of samples

tribution pc, however, we believe mathematical form of C isnot restricted, which could motivate further development.

drawn from the bridge. This may alleviate datasparsity and overfitting by posing more unseenscenarios to the generator and may help the gen-erator generalize better in test time.

2.2 Bridge Module3

Our bridge module is designed to transform asingle target example Y ∗ to a bridge distribu-tion pη(Y |Y ∗). We design its optimization tar-get in Equation (2) to consist of two terms,namely, a concentration requirement and a con-straint. The constraint is instantiated as KL-divergence between the bridge and a contraint dis-tribution pc(Y ). We transform Equation (2) as fol-lows, which is convenient for mathematical ma-nipulation later:

LB(η) =

EY∼pη(Y |Y ∗)

[−S(Y, Y∗)

τ] +KL(pη(Y |Y ∗)||pc(Y ))

(5)

S(Y, Y ∗) is a predefined score function whichmeasures similarity between Y and Y ∗ and peakswhen Y = Y ∗, while pc(Y ) reshapes the bridgedistribution. More specifically, the first term en-sures that the bridge should concentrate around theground truth Y ∗, and the second introduces willingproperty which can help regularize the generator.The hyperparameter τ can be interpreted as a tem-perature which scales the score function. In thefollowing bridge specifications, the score functionS(Y, Y ∗) is instantiated according to Sec. 3.1.

1. Delta Bridge The delta bridge can be seenas the simplest case where α = 0 or no con-straint is imposed. The bridge seeks to minimize

EY∼pη(Y |Y ∗)

[−S(Y,Y ∗)τ ]. The optimal solution is

when the bridge only samples Y ∗, thus the Diracdelta distribution is described as follows:

pη(Y |Y ∗) = δY ∗(Y ) (6)

This exactly corresponds to MLE, where only ex-amples in the dataset are used to train the genera-tor. We regard this case as our baseline.

2. Uniform Bridge The uniform bridge adoptsa uniform distribution U(Y ) as constraint. This

3Although we name it bridge module, we explicitly learnit with the generator when a closed-form static solution ex-ists in terms of Equation (5). Otherwise, we will adopt anencoder-decoder to construct a dynamic bridge network.

bridge motivates to include noise into target exam-ple, which is similar to label smoothing (Szegedyet al., 2016). The loss function can be written as:

LB(η) =

EY∼pη(Y |Y ∗)

[−S(Y, Y∗)

τ] +KL(pη(Y |Y ∗)||U(Y ))

(7)

We can re-write it as follows by adding a constantto not change the optimization result:

LB(η) + C = KL(pη(Y |Y ∗)||exp S(Y,Y ∗)

τ

Z)

(8)

This bridge is static for having a closed-form so-lution:

pη(Y |Y ∗) =exp S(Y,Y ∗)

τ

Z(9)

where Z is the partition function. Note that ouruniform bridge corresponds to the payoff distribu-tion described in RAML (Norouzi et al., 2016).

3. Language-model (LM) Bridge The LMbridge utilizes a pretrained neural language modelpLM (Y ) as constraint, which motivates to proposetarget examples conforming to language fluency.

LB(η) =

EY∼pη(Y |Y ∗)

[−S(Y, Y∗)

τ] +KL(pη(Y |Y ∗)||pLM )

(10)

Similar to the uniform bridge case, we can re-writethe loss function to a KL-divergence:

LB(η) + C

=KL(pη(Y |Y ∗)||pLM (Y ) · exp S(Y,Y ∗)

τ

Z)

(11)

Thus, the LM bridge is also static and can be seenas an extension of the uniform bridge, where theexponentiated similarity score is re-weighted by apretrained LM score, and renormalized:

p(Y |Y ∗) =pLM (Y ) exp S(Y,Y ∗)

τ

Z(12)

where Z is the partition function. The above equa-tion looks just like the payoff distribution, whereasan additional factor is considered.

4. Coaching Bridge The coaching bridge uti-lizes the generator’s output distribution as con-straint, which motivates to generate training sam-ples which are easy to be understood by thegenerator, so as to relieve its learning burden.The coaching bridge follows the same spirit asthe coach proposed in Imitation-via-Coaching (Heet al., 2012), which, in reinforcement learning vo-cabulary, advocates to guide the policy (genera-tor) with easy-to-learn action trajectories and letit gradually approach the oracle when the optimalaction is hard to achieve.

LB(η) =

EY∼pη(Y |Y ∗)

[−S(Y, Y∗)

τ] +KL(pθ(Y |X)||pη(Y |Y ∗))

(13)

Since the KL constraint is a moving target whenthe generator is updated, the coaching bridgeshould not remain static. Therefore, we performiterative optimization to train the bridge and thegenerator jointly. Formally, the derivatives for thecoaching bridge are written as follows:

∇LB(η) = EY∼pη(Y |Y ∗)

[−S(Y, Y∗)

τ∇ log pη(Y |Y ∗)]

+ EY∼pθ(Y |X)

∇ log pη(Y |Y ∗)

(14)

The first term corresponds to the policy gradientalgorithm described in REINFORCE (Williams,1992), where the coefficient −S(Y, Y ∗)/τ corre-sponds to reward function. Due to the mutual de-pendence between bridge module and generatornetwork, we design an iterative training strategy,i.e. the two networks take turns to update theirown parameters treating the other as fixed.

2.3 TrainingThe training of the above three variants is illus-trated in Figure 3. Since the proposed bridges canbe divided into static ones, which only require pre-training, and dynamic ones, which require contin-ual training with the generator, we describe theirtraining process in details respectively.

2.3.1 Stratified-Sampled TrainingSince closed-formed optimal distributions can befound for uniform/LM GBNs, we only need todraw samples from the static bridge distributionsto train our sequence generator. Unfortunately,

Generator𝑝𝜃(𝑌|𝑋)

𝑝𝑑𝑎𝑡𝑎(𝑌∗)

LM Bridge

𝑝𝜂(𝑌|𝑌∗)

Coach Bridge

𝑝𝜂(𝑌|𝑌∗)

𝑈(𝑌)

Iterative Training

Stratified-sampled Training

Uniform Bridge

𝑝𝜂(𝑌|𝑌∗)

𝑝𝐿𝑀(𝑌)

Pre-trained

Figure 3: The training processes of the three differentvariants of our GBN architecture (Sec. 2.3).

due to the intractability of these bridge distribu-tions, direct sampling is infeasible. Therefore, wefollow Norouzi et al. (2016); Ma et al. (2017) andadopt stratified sampling to approximate the directsampling process. Given a sentence Y ∗, we firstsample an edit distance m, and then randomly se-lectm positions to replace the original tokens. Thedifference between the uniform and the LM bridgelies in that the uniform bridge replaces labels bydrawing substitutions from a uniform distribution,while LM bridge takes the history as condition anddraws substitutions from its step-wise distribution.

2.3.2 Iterative Training

Since the KL-constraint is a moving target for thecoaching bridge, an iterative training strategy isdesigned to alternately update both the generatorand the bridge (Algorithm 1). We first pre-trainboth the generator and the bridge and then start toalternately update their parameters. Figure 4 intu-itively demonstrates the intertwined optimizationeffects over the coaching bridge and the generator.We hypothesize that iterative training with easy-to-learn guidance could benefit gradient update,thus result in better local minimum.

3 Experiment

We select machine translation and abstractive textsummarization as benchmarks to verify our GBNframework.

3.1 Similarity Score Function

In our experiments, instead of directly usingBLEU or ROUGE as reward to guide the bridgenetwork’s policy search, we design a simple sur-

𝑷𝜼 𝑷𝜽

𝑌2. Update learner 𝑃𝜃

𝑌

4. Update learner 𝑃𝜃

1. Update coach 𝑃𝜂𝑌

3. Update coach 𝑃𝜂

𝑌

𝜹(𝒀)

Figure 4: Four iterative updates of the coaching bridgeand the generator. In an early stage, the pre-trainedgenerator Pθ may not put mass on some ground truthtarget points within the output space, shown by δ(Y ).The coaching bridge is first updated with Equation (14)to locate in between the Dirac delta distribution andthe generator’s output distribution. Then, by samplingfrom the coaching bridge for approximating Equation(4), target samples which demonstrate easy-to-learn se-quence segments facilitate the generator to be opti-mized to achieve closeness with the coaching bridge.Then this process repeats until the generator converges.

rogate n-gram matching reward as follows:

S(Y, Y ∗) = 0.4∗N4+0.3∗N3+0.2∗N2+0.1∗N1

(15)Nn represents the n-gram matching score betweenY and Y ∗. In order to alleviate reward sparsity atsequence level, we further decompose the globalreward S(Y, Y ∗) as a series of local rewards at ev-ery time step. Formally, we write the step-wisereward s(yt|y1:t−1, Y ∗) as follows:

s(yt|y1:t−1, Y∗) =

1.0;N(y1:t, yt−3:t) ≤ N(Y ∗, yt−3:t)

0.6;N(y1:t, yt−2:t) ≤ N(Y ∗, yt−2:t)

0.3;N(y1:t, yt−1:t) ≤ N(Y ∗, yt−1:t)

0.1;N(y1:t, yt) ≤ N(Y ∗, yt)

0.0; otherwise

(16)

where N(Y, Y ) represents the occurrence of sub-sequence Y in whole sequence Y . Specifically, if

Algorithm 1 Training Coaching GBNprocedure PRE-TRAINING

Initialize pθ(Y |X) and pη(Y |Y ∗) with ran-dom weights θ and η

Pre-train pθ(Y |X) to predict Y ∗ given XUse pre-trained pθ(Y |X) to generate Y

given XPre-train pη(Y |Y ∗) to predict Y given Y ∗

end procedureprocedure ITERATIVE-TRAINING

while Not Converged doReceive a random example (X,Y ∗)if Bridge-step then

Draw samples Y from pθ(Y |X)Update bridge via Equation (14)

else if Generator-step thenDraw samples Y from pη(Y |Y ∗)Update generator via Equation (4)

end ifend while

end procedure

a certain sub-sequence yt−n+1:t from Y appearsless times than in the reference Y ∗, yt receives re-ward. Formally, we rewrite the step-level gradientfor each sampled Y as follows:

− S(Y, Y ∗)

τ∇ log pη(Y |Y ∗)

=∑t

−s(yt|y1:t−1, Y∗)

τ· ∇ log pη(yt|y1:t−1, Y ∗)

(17)

3.2 Machine TranslationDataset We follow Ranzato et al. (2015); Bah-danau et al. (2016) and select German-English ma-chine translation track of the IWSLT 2014 eval-uation campaign. The corpus contains sentence-wise aligned subtitles of TED and TEDx talks. Weuse Moses toolkit (Koehn et al., 2007) and removesentences longer than 50 words as well as lower-casing. The evaluation metric is BLEU (Papineniet al., 2002) computed via the multi-bleu.perl.

System Setting We use a unified GRU-basedRNN (Chung et al., 2014) for both the generatorand the coaching bridge. In order to compare withexisting papers, we use a similar system settingwith 512 RNN hidden units and 256 as embed-ding size. We use attentive encoder-decoder tobuild our system (Bahdanau et al., 2014). Dur-ing training, we apply ADADELTA (Zeiler, 2012)

Methods Baseline ModelMIXER 20.10 21.81 +1.71

BSO 24.03 26.36 +2.33

AC 27.56 28.53 +0.97

Softmax-Q 27.66 28.77 +1.11

Uniform GBN(τ = 0.8)

29.10

29.80 +0.70

LM GBN(τ = 0.8)

29.90 +0.80

Coaching GBN(τ = 0.8)

29.98 +0.88

Coaching GBN(τ = 1.2)

30.15 +1.05

Coaching GBN(τ = 1.0)

30.18 +1.08

Table 1: Comparison with existing works on IWSLT-2014 German-English Machine Translation Task.

70

75

80

85

90

95

100

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

BLE

U

Epoch (Bridge)

Coaching GBN Learning Curve

31.5

31.6

31.7

31.8

31.9

32

32.1

32.2

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

BLE

U

Epoch (Generator)

Figure 5: Coaching GBN’s learning curve on IWSLTGerman-English Dev set.

with ε = 10−6 and ρ = 0.95 to optimize pa-rameters of the generator and the coaching bridge.During decoding, a beam size of 8 is used to ap-proximate the full search space. An importanthyper-parameter for our experiments is the tem-perature τ . For the uniform/LM bridge, we fol-low Norouzi et al. (2016) to adopt an optimal tem-perature τ = 0.8. And for the coaching bridge,we test hyper-parameters from τ ∈ {0.8, 1.0, 1.2}.Besides comparing with our fine-tuned baseline,other systems for comparison of relative BLEUimprovement are: MIXER (Ranzato et al., 2015),BSO (Wiseman and Rush, 2016), AC (Bahdanauet al., 2016), Softmax-Q (Ma et al., 2017).

Results The experimental results are summa-rized in Table 1. We can observe that ourfine-tuned MLE baseline (29.10) is already over-

Methods RG-1 RG-2 RG-LABS 29.55 11.32 26.42ABS+ 29.76 11.88 26.96Luong-NMT 33.10 14.45 30.71SAEASS 36.15 17.54 33.63seq2seq+att 34.04 15.95 31.68Uniform GBN

(τ = 0.8)34.10 16.70 31.75

LM GBN(τ = 0.8)

34.32 16.88 31.89

Coaching GBN(τ = 0.8)

34.49 16.70 31.95

Coaching GBN(τ = 1.2)

34.83 16.83 32.25

Coaching GBN(τ = 1.0)

35.26 17.22 32.67

Table 2: Full length ROUGE F1 evaluation results onthe English Gigaword test set used by (Rush et al.,2015). RG in the Table denotes ROUGE. Resultsfor comparison are taken from SAEASS (Zhou et al.,2017).

competing other systems and our proposed GBNcan yield a further improvement. We also ob-serve that LM GBN and coaching GBN have bothachieved better performance than Uniform GBN,which confirms that better regularization effectsare achieved, and the generators become more ro-bust and generalize better. We draw the learningcurve of both the bridge and the generator in Fig-ure 5 to demonstrate how they cooperate duringtraining. We can easily observe the interactionbetween them: as the generator makes progress,the coaching bridge also improves itself to proposeharsher targets for the generator to learn.

3.3 Abstractive Text Summarization

Dataset We follow the previous works by Rushet al. (2015); Zhou et al. (2017) and use thesame corpus from Annotated English Gigaworddataset (Napoles et al., 2012). In order to be com-parable, we use the same script 4 released by Rushet al. (2015) to pre-process and extract the train-ing and validation sets. For the test set, we use theEnglish Gigaword, released by Rush et al. (2015),and evaluate our system through ROUGE (Lin,2004). Following previous works, we employROUGE-1, ROUGE-2, and ROUGE-L as the eval-uation metrics in the reported experimental results.

4https://github.com/facebookarchive/NAMAS

81

81.5

82

82.5

83

83.5

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

RO

UG

E-2

Epoch (Bridge)

Coaching GBN Learning Curve

21.7

21.9

22.1

22.3

22.5

22.7

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

RO

UG

E-2

Epoch (Generator)

Figure 6: Coaching GBN’s learning curve on Abstrac-tive Text Summarization Dev set.

System Setting We follow Zhou et al. (2017);Rush et al. (2015) to set input and output vo-cabularies to 119,504 and 68,883 respectively,and we also set the word embedding size to300 and all GRU hidden state size to 512.Then we adopt dropout (Srivastava et al., 2014)with probability p = 0.5 strategy in our out-put layer. We use attention-based sequence-to-sequence model (Bahdanau et al., 2014; Cho et al.,2014) as our baseline and reproduce the results ofthe baseline reported in Zhou et al. (2017). Asstated, the attentive encoder-decode architecturecan already outperform existing ABS/ABS+ sys-tems (Rush et al., 2015). In coaching GBN, due tothe fact that the input of abstractive summarizationX contains more information than the summarytarget Y ∗, directly training the bridge pη(Y |Y ∗)to understand the generator pθ(Y |X) is infeasible.Therefore, we re-design the coaching bridge to re-ceive both source and target input X,Y and weenlarge its vocabulary size to 88,883 to encom-pass more information about the source side. InUniform/LM GBN experiments, we also fix thehyper-parameter τ = 0.8 as the optimal setting.

Results The experimental results are summa-rized in Table 2. We can observe a significantimprovement via our GBN systems. Similarly,the coaching GBN system achieves the strongestperformance among all, which again reflects ourassumption that more sophisticated regularizationcan benefit generator’s training. We draw thelearning curve of the coaching GBN in Figure 6to demonstrate how the bridge and the generatorpromote each other.

4 Analysis

By introducing different constraints into the bridgemodule, the bridge distribution will propose dif-ferent training samples for the generator to learn.From Table 3, we can observe that most samplesstill reserve their original meaning. The uniformbridge simply performs random replacement with-out considering any linguistic constraint. The LMbridge strives to smooth reference sentence withhigh-frequent words. And the coaching bridgesimplifies difficult expressions to relieve genera-tor’s learning burden. From our experimental re-sults, the more rational and aggressive diversifica-tion from the coaching GBN clearly benefits gen-erator the most and helps the generator generalizeto more unseen scenarios.

5 Related Literature

5.1 Data Augmentation and Self-trainingIn order to resolve the data sparsity problem inNeural Machine Translation (NMT), many workshave been conducted to augment the dataset. Themost popular strategy is via self-learning, whichincorporates the self-generated data directly intotraining. Zhang and Zong (2016) and Sennrichet al. (2015) both use self-learning to leveragemassive monolingual data for NMT training. Ourbridge can take advantage of the parallel trainingdata only, instead of external monolingual ones tosynthesize new training data.

5.2 Reward Augmented MaximumLikelihood

Reward augmented maximum likelihood orRAML (Norouzi et al., 2016) proposes to in-tegrate task-level reward into MLE training byusing an exponentiated payoff distribution. KLdivergence between the payoff distribution and thegenerator’s output distribution are minimized toachieve an optimal task-level reward. Followingthis work, Ma et al. (2017) introduces softmaxQ-Distribution to interpret RAML and reveals itsrelation with Bayesian decision theory. Thesetwo works both alleviate data sparsity problem byaugmenting target examples based on the groundtruth. Our method draws inspiration from thembut seeks to propose the more general GenerativeBridging Network, which can transform theground truth into different bridge distributions,from where samples are drawn will account fordifferent interpretable factors.

System Uniform GBNProperty Random ReplacementReference the question is , is it worth it ?

Bridge the question lemon , was it worth it ?

System Language-model GBNProperty Word ReplacementReference now how can this help us ?

Bridge so how can this help us ?

System Coaching GBNProperty ReorderingReference i need to have a health care lexicon .

Bridge i need a lexicon for health care .

Property Simplification

Referencethis is the way that most of us were taught

to tie our shoes .

Bridge most of us learned to bind our shoes .

Table 3: Qualitative analysis for three different bridgedistributions.

5.3 Coaching

Our coaching GBN system is inspired by imita-tion learning by coaching (He et al., 2012). In-stead of directly behavior cloning the oracle, theyadvocate learning hope actions as targets from acoach which is interpolated between learner’s pol-icy and the environment loss. As the learner makesprogress, the targets provided by the coach willbecome harsher to gradually improve the learner.Similarly, our proposed coaching GBN is moti-vated to construct an easy-to-learn bridge distri-bution which lies in between the ground truth andthe generator. Our experimental results confirm itseffectiveness to relieve the learning burden.

6 Conclusion

In this paper, we present the Generative Bridg-ing Network (GBN) to overcome data sparsity andoverfitting issues with Maximum Likelihood Esti-mation in neural sequence prediction. Our imple-mented systems prove to significantly improve theperformance, compared with strong baselines. Webelieve the concept of bridge distribution can beapplicable to a wide range of distribution matchingtasks in probabilistic learning. In the future, we in-tend to explore more about GBN’s applications aswell as its provable computational and statisticalguarantees.

ReferencesDzmitry Bahdanau, Philemon Brakel, Kelvin Xu,

Anirudh Goyal, Ryan Lowe, Joelle Pineau, AaronCourville, and Yoshua Bengio. 2016. An actor-criticalgorithm for sequence prediction. arXiv preprintarXiv:1607.07086 .

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-gio. 2014. Neural machine translation by jointlylearning to align and translate. arXiv preprintarXiv:1409.0473 .

Yong Cheng, Wei Xu, Zhongjun He, Wei He, HuaWu, Maosong Sun, and Yang Liu. 2016. Semi-supervised learning for neural machine translation.arXiv preprint arXiv:1606.04596 .

Kyunghyun Cho, Bart Van Merrienboer, Caglar Gul-cehre, Dzmitry Bahdanau, Fethi Bougares, HolgerSchwenk, and Yoshua Bengio. 2014. Learningphrase representations using rnn encoder-decoderfor statistical machine translation. arXiv preprintarXiv:1406.1078 .

Jan K Chorowski, Dzmitry Bahdanau, DmitriySerdyuk, Kyunghyun Cho, and Yoshua Bengio.2015. Attention-based models for speech recogni-tion. In Advances in Neural Information ProcessingSystems. pages 577–585.

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho,and Yoshua Bengio. 2014. Empirical evaluation ofgated recurrent neural networks on sequence model-ing. arXiv preprint arXiv:1412.3555 .

Kenji Doya. 1992. Bifurcations in the learning of re-current neural networks. In Circuits and Systems,1992. ISCAS’92. Proceedings., 1992 IEEE Interna-tional Symposium on. IEEE, volume 6, pages 2777–2780.

He He, Jason Eisner, and Hal Daume. 2012. Imitationlearning by coaching. In Advances in Neural Infor-mation Processing Systems. pages 3149–3157.

Sepp Hochreiter and Jurgen Schmidhuber. 1997.Long short-term memory. Neural computation9(8):1735–1780.

Philipp Koehn, Hieu Hoang, Alexandra Birch, ChrisCallison-Burch, Marcello Federico, Nicola Bertoldi,Brooke Cowan, Wade Shen, Christine Moran,Richard Zens, et al. 2007. Moses: Open sourcetoolkit for statistical machine translation. In Pro-ceedings of the 45th annual meeting of the ACL oninteractive poster and demonstration sessions. As-sociation for Computational Linguistics, pages 177–180.

Chin-Yew Lin. 2004. Rouge: A package for auto-matic evaluation of summaries. In Text summariza-tion branches out: Proceedings of the ACL-04 work-shop. Barcelona, Spain, volume 8.

Xuezhe Ma, Pengcheng Yin, Jingzhou Liu, GrahamNeubig, and Eduard Hovy. 2017. Softmax q-distribution estimation for structured prediction: Atheoretical interpretation for raml. arXiv preprintarXiv:1705.07136 .

Courtney Napoles, Matthew Gormley, and BenjaminVan Durme. 2012. Annotated gigaword. In Pro-ceedings of the Joint Workshop on Automatic Knowl-edge Base Construction and Web-scale KnowledgeExtraction. Association for Computational Linguis-tics, pages 95–100.

Partha Niyogi, Federico Girosi, and Tomaso Poggio.1998. Incorporating prior information in machinelearning by creating virtual examples. Proceedingsof the IEEE 86(11):2196–2209.

Mohammad Norouzi, Samy Bengio, Navdeep Jaitly,Mike Schuster, Yonghui Wu, Dale Schuurmans,et al. 2016. Reward augmented maximum likeli-hood for neural structured prediction. In AdvancesIn Neural Information Processing Systems. pages1723–1731.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic eval-uation of machine translation. In Proceedings ofthe 40th annual meeting on association for compu-tational linguistics. Association for ComputationalLinguistics, pages 311–318.

Gabriel Pereyra, George Tucker, Jan Chorowski,Łukasz Kaiser, and Geoffrey Hinton. 2017. Regular-izing neural networks by penalizing confident outputdistributions. arXiv preprint arXiv:1701.06548 .

Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli,and Wojciech Zaremba. 2015. Sequence level train-ing with recurrent neural networks. arXiv preprintarXiv:1511.06732 .

Alexander M Rush, Sumit Chopra, and Jason We-ston. 2015. A neural attention model for ab-stractive sentence summarization. arXiv preprintarXiv:1509.00685 .

Rico Sennrich, Barry Haddow, and Alexandra Birch.2015. Improving neural machine translationmodels with monolingual data. arXiv preprintarXiv:1511.06709 .

Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky,Ilya Sutskever, and Ruslan Salakhutdinov. 2014.Dropout: a simple way to prevent neural networksfrom overfitting. Journal of Machine Learning Re-search 15(1):1929–1958.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe,Jon Shlens, and Zbigniew Wojna. 2016. Rethinkingthe inception architecture for computer vision. InProceedings of the IEEE Conference on ComputerVision and Pattern Recognition. pages 2818–2826.

Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov,Ilya Sutskever, and Geoffrey Hinton. 2015. Gram-mar as a foreign language. In Advances in NeuralInformation Processing Systems. pages 2773–2781.

Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforce-ment learning. Machine learning 8(3-4):229–256.

Sam Wiseman and Alexander M Rush. 2016.Sequence-to-sequence learning as beam-search op-timization. arXiv preprint arXiv:1606.02960 .

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho,Aaron Courville, Ruslan Salakhudinov, Rich Zemel,and Yoshua Bengio. 2015. Show, attend and tell:Neural image caption generation with visual at-tention. In International Conference on MachineLearning. pages 2048–2057.

Matthew D Zeiler. 2012. Adadelta: an adaptive learn-ing rate method. arXiv preprint arXiv:1212.5701 .

Jiacheng Zhang, Yang Liu, Huanbo Luan, Jingfang Xu,and Maosong Sun. 2017. Prior knowledge integra-tion for neural machine translation using posteriorregularization. In Proceedings of the 55th AnnualMeeting of the Association for Computational Lin-guistics (Volume 1: Long Papers). volume 1, pages1514–1523.

Jiajun Zhang and Chengqing Zong. 2016. Exploit-ing source-side monolingual data in neural machinetranslation. In EMNLP. pages 1535–1545.

Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou.2017. Selective encoding for abstractive sentencesummarization. arXiv preprint arXiv:1704.07073 .

A Supplemental Material

This part first provides detailed derivation ofEquation (8) and (11) from Equation (7) and(10), since our uniform bridge distribution andlanguage-model bridge distribution have closed-form solutions given a fixed uniform distributionand a language model as constraints. Then, wegive explanation of Equation (13), the objectivefunction of coaching bridge, where the constraintis the inverse KL compared with previous twobridges and then give detailed derivation of thegradient update Equation (14).

Derivation of Equation (8)

LB(η)

= EY∼pη

− S(Y, Y ∗)

τ+KL(pη(Y |Y ∗)||U(Y ))

=

∫Y−pη(Y |Y ∗) log exp(

S(Y, Y ∗)

τ)

+

∫Ypη(Y |Y ∗) log

pη(Y |Y ∗)U(Y )

=

∫Ypη(Y |Y ∗) log

pη(Y |Y ∗)exp(S(Y,Y

∗)τ ) · U(Y )

=

∫Ypη(Y |Y ∗) log

pη(Y |Y ∗)exp(S(Y,Y

∗)τ ) · 1

|Y|

=

∫Ypη(Y |Y ∗) log

pη(Y |Y ∗)exp S(Y,Y ∗)

τ

+ log |Y|∫Ypη(Y |Y ∗)

=

∫Ypη(Y |Y ∗) log

pη(Y |Y ∗)exp S(Y,Y ∗)

τ

+ Const

=

∫Ypη(Y |Y ∗) log

pη(Y |Y ∗)exp

S(Y,Y ∗)τ

Z

+ Const′

= KL(pη(Y |Y ∗)||exp S(Y,Y ∗)

τ

Z) + Const′

(18)

Here, the Y ∗ related constant Z is needed totransform a unnormalized similarity score to aprobability:

Z(Y ∗) =

∫Yexp

S(Y, Y ∗)

τ(19)

Derivation of Equation (11)

LB(η)

= EY∼pη

− S(Y, Y ∗)

τ+KL(pη(Y |Y ∗)||pLM (Y ))

=

∫Y−pη(Y |Y ∗) log exp(

S(Y, Y ∗)

τ)

+

∫Ypη(Y |Y ∗) log

pη(Y |Y ∗)pLM (Y )

=

∫Ypη(Y |Y ∗) log

pη(Y |Y ∗)exp(S(Y,Y

∗)τ ) · pLM (Y )

=

∫Ypη(Y |Y ∗) log

pη(Y |Y ∗)exp

S(Y,Y ∗)τ

·PLM (Y )

Z

+ Const

=KL(pη(Y |Y ∗)||exp S(Y,Y ∗)

τ · PLM (Y )

Z) + Const′

(20)

Here, the Y ∗ related constant Z is neededto transform a unnormalized weighted similarityscore to a probability:

Z(Y ∗) =

∫Yexp

S(Y, Y ∗)

τ· PLM (Y ) (21)

Explanation of Equation (13) This equationis the objective function of our coaching bridge,which uses an inverse KL term5 as part of its ob-jective. The use of inverse KL is out of the consid-eration of computational stability. The reasons aretwo-fold: 1). the inverse KL will do not changethe effect of the constraint; 2). the inverse KL re-quires sampling from the generator and uses thosesamples as the target to train the bridge, which hasthe same gradient update ad MLE, so we do notneed to consider baseline tricks in ReinforcementLearning implementation.

Gradient derivation of Equation (13)

∇LB(η)

=∇η EY∼pη(Y |Y ∗)

− S(Y, Y ∗)

τ+∇ηKL(pθ(Y |X)||pη(Y |Y ∗))

= EY∼pη(Y |Y ∗)

− S(Y, Y ∗)

τ∇η log pη(Y |Y ∗)

+∇η EY∼pθ(Y |X)

log pη(Y |Y ∗)

= EY∼pη(Y |Y ∗)

− S(Y, Y ∗)

τ∇ log pη(Y |Y ∗)

+ EY∼pθ(Y |X)

∇ log pη(Y |Y ∗)

(22)

5That is the use of KL(pθ||pη) instead of KL(pη||pθ).