A Syntactic Neural Model for General-Purpose Code Generation

Pengcheng YinLanguage Technologies Institute

Carnegie Mellon Universitypcyin@cs.cmu.edu

Graham NeubigLanguage Technologies Institute

Carnegie Mellon Universitygneubig@cs.cmu.edu

Abstract

We consider the problem of parsing natu-ral language descriptions into source codewritten in a general-purpose programminglanguage like Python. Existing data-driven methods treat this problem as a lan-guage generation task without consideringthe underlying syntax of the target pro-gramming language. Informed by previ-ous work in semantic parsing, in this pa-per we propose a novel neural architecturepowered by a grammar model to explicitlycapture the target syntax as prior knowl-edge. Experiments find this an effectiveway to scale up to generation of complexprograms from natural language descrip-tions, achieving state-of-the-art results thatwell outperform previous code generationand semantic parsing approaches.

1 Introduction

Every programmer has experienced the situationwhere they know what they want to do, but donot have the ability to turn it into a concrete im-plementation. For example, a Python programmermay want to “sort my list in descending order,”but not be able to come up with the proper syn-tax sorted(my list, reverse=True) to real-ize his intention. To resolve this impasse, it iscommon for programmers to search the web innatural language (NL), find an answer, and mod-ify it into the desired form (Brandt et al., 2009,2010). However, this is time-consuming, andthus the software engineering literature is ripewith methods to directly generate code from NLdescriptions, mostly with hand-engineered meth-ods highly tailored to specific programming lan-guages (Balzer, 1985; Little and Miller, 2009;Gvero and Kuncak, 2015).

In parallel, the NLP community has developedmethods for data-driven semantic parsing, whichattempt to map NL to structured logical forms ex-ecutable by computers. These logical forms can begeneral-purpose meaning representations (Clarkand Curran, 2007; Banarescu et al., 2013), for-malisms for querying knowledge bases (Tang andMooney, 2001; Zettlemoyer and Collins, 2005;Berant et al., 2013) and instructions for robots orpersonal assistants (Artzi and Zettlemoyer, 2013;Quirk et al., 2015), among others. While thesemethods have the advantage of being learnablefrom data, compared to the programming lan-guages (PLs) in use by programmers, the domain-specific languages targeted by these works have aschema and syntax that is relatively simple.

Recently, Ling et al. (2016) have proposed adata-driven code generation method for high-level,general-purpose PLs like Python and Java. Thiswork treats code generation as a sequence-to-sequence modeling problem, and introduce meth-ods to generate words from character-level mod-els, and copy variable names from input descrip-tions. However, unlike most work in semanticparsing, it does not consider the fact that code hasto be well-defined programs in the target syntax.

In this work, we propose a data-driven syntax-based neural network model tailored for genera-tion of general-purpose PLs like Python. In or-der to capture the strong underlying syntax of thePL, we define a model that transduces an NL state-ment into an Abstract Syntax Tree (AST; Fig. 1(a),§ 2) for the target PL. ASTs can be deterministi-cally generated for all well-formed programs us-ing standard parsers provided by the PL, and thusgive us a way to obtain syntax information withminimal engineering. Once we generate an AST,we can use deterministic generation tools to con-vert the AST into surface code. We hypothesizethat such a structured approach has two benefits.

arX

iv:1

704.

0169

6v1

[cs

.CL

] 6

Apr

201

7

Production Rule Role ExplanationCall 7→ expr[func] expr*[args] keyword*[keywords] Function Call . func: the function to be invoked . args: arguments list

. keywords: keyword arguments listIf 7→ expr[test] stmt*[body] stmt*[orelse] If Statement . test: condition expression . body: statements inside

the If clause . orelse: elif or else statementsFor 7→ expr[target] expr*[iter] stmt*[body] For Loop . target: iteration variable . iter: enumerable to iterate

over . body: loop body . orelse: else statementsstmt*[orelse]FunctionDef 7→ identifier[name] arguments*[args] Function Def. . name: function name . args: function arguments

. body: function bodystmt*[body]

Table 1: Example production rules for common Python statements (Python Software Foundation, 2016)

First, we hypothesize that structure can be usedto constrain our search space, ensuring generationof well-formed code. To this end, we propose asyntax-driven neural code generation model. Thebackbone of our approach is a grammar model(§ 3) which formalizes the generation story of aderivation AST into sequential application of ac-tions that either apply production rules (§ 3.1), oremit terminal tokens (§ 3.2). The underlying syn-tax of the PL is therefore encoded in the grammarmodel a priori as the set of possible actions. Ourapproach frees the model from recovering the un-derlying grammar from limited training data, andinstead enables the system to focus on learning thecompositionality among existing grammar rules.Xiao et al. (2016) have noted that this impositionof structure on neural models is useful for seman-tic parsing, and we expect this to be even more im-portant for general-purpose PLs where the syntaxtrees are larger and more complex.

Second, we hypothesize that structural informa-tion helps to model information flow within theneural network, which naturally reflects the recur-sive structure of PLs. To test this, we extend astandard recurrent neural network (RNN) decoderto allow for additional neural connections whichreflect the recursive structure of an AST (§ 4.2).As an example, when expanding the node ? inFig. 1(a), we make use of the information fromboth its parent and left sibling (the dashed rectan-gle). This enables us to locally pass informationof relevant code segments via neural network con-nections, resulting in more confident predictions.

Experiments (§ 5) on two Python code gener-ation tasks show 11.7% and 9.3% absolute im-provements in accuracy against the state-of-the-artsystem (Ling et al., 2016). Our model also givescompetitive performance on a standard semanticparsing benchmark.

2 The Code Generation ProblemGiven an NL description x, our task is to generatethe code snippet c in a modern PL based on the in-

tent of x. We attack this problem by first generat-ing the underlying AST. We define a probabilisticgrammar model of generating an AST y given x:p(y|x). The best-possible AST y is then given by

y = argmaxy

p(y|x). (1)

y is then deterministically converted to the corre-sponding surface code c.1 While this paper usesexamples from Python code, our method is PL-agnostic.

Before detailing our approach, we first presenta brief introduction of the Python AST and itsunderlying grammar. The Python abstract gram-mar contains a set of production rules, and anAST is generated by applying several productionrules composed of a head node and multiple childnodes. For instance, the first rule in Tab. 1 isused to generate the function call sorted(·) inFig. 1(a). It consists of a head node of type Call,and three child nodes of type expr, expr* andkeyword*, respectively. Labels of each node arenoted within brackets. In an AST, non-terminalnodes sketch the general structure of the targetcode, while terminal nodes can be categorized intotwo types: operation terminals and variable ter-minals. Operation terminals correspond to basicarithmetic operations like AddOp.Variable termi-nal nodes store values for variables and constantsof built-in data types2. For instance, all terminalnodes in Fig. 1(a) are variable terminal nodes.

3 Grammar ModelBefore detailing our neural code generationmethod, we first introduce the grammar model atits core. Our probabilistic grammar model definesthe generative story of a derivation AST. We fac-torize the generation process of an AST into se-quential application of actions of two types:

• APPLYRULE[r] applies a production rule r tothe current derivation tree;

1We use astor library to convert ASTs into Python code.2bool, float, int, str.

Expr

root

expr[value]

Call

expr*[args] keyword*[keywords]

Name

str(sorted)

expr[func]

expr

Name

str(my_list)

keyword

str(reverse) expr[value]

Name

str(True)

������(��_����, �������=����)

⋆

t1

t2

t3

t4

t5

t6

t7

t8

t9

t10

t11

t12

t14

root ↦ Expr

Expr ↦ expr[value]

expr ↦ Call

expr ↦ Name

Name ↦ str

��������[sorted]

��������[</n>]

expr* ↦ expr

expr ↦ Name

Name ↦ str

��������[my_list]

keyword* ↦ keyword

Call ↦ expr[func]  expr*[args]  keyword*[keywords]

Action FlowParent Feeding

ti Apply Rule

ti Generate Token

ti GenToken with Copyt13 ��������[</n>]

t1

t2

t3

t4

t5

t6t7t8

t4

t9

t10

t11 t12 t13

t4

t14

t15 t16 t17 t16

t18

t19 t20 t21

t0

(a) (b)

���� ��_���� �� ���������� �����Input: Code:

. . .

Figure 1: (a) the Abstract Syntax Tree (AST) for the given example code. Dashed nodes denote terminals. Nodes are labeledwith time steps during which they are generated. (b) the action sequence (up to t14) used to generate the AST in (a)

• GENTOKEN[v] populates a variable terminalnode by appending a terminal token v.

Fig. 1(b) shows the generation process of the tar-get AST in Fig. 1(a). Each node in Fig. 1(b) in-dicates an action. Action nodes are connected bysolid arrows which depict the chronological orderof the action flow. The generation proceeds indepth-first, left-to-right order (dotted arrows rep-resent parent feeding, explained in § 4.2.1).

Formally, under our grammar model, the prob-ability of generating an AST y is factorized as:

p(y|x) =T∏t=1

p(at|x, a<t), (2)

where at is the action taken at time step t, and a<t

is the sequence of actions before t. We will explainhow to compute Eq. (2) in § 4. Put simply, thegeneration process begins from a root node at t0,and proceeds by the model choosing APPLYRULE

actions to generate the overall program structurefrom a closed set of grammar rules, then at leavesof the tree corresponding to variable terminals, themodel switches to GENTOKEN actions to gener-ate variables or constants from the open set. Wedescribe this process in detail below.

3.1 APPLYRULE ActionsAPPLYRULE actions generate program structure,expanding the current node (the frontier node attime step t: nft) in a depth-first, left-to-righttraversal of the tree. Given a fixed set of produc-tion rules, APPLYRULE chooses a rule r from thesubset that has a head matching the type of nft ,and uses r to expand nft by appending all childnodes specified by the selected production. As an

example, in Fig. 1(b), the rule Call 7→ expr. . .expands the frontier node Call at time step t4, andits three child nodes expr, expr* and keyword*

are added to the derivation.APPLYRULE actions grow the derivation AST

by appending nodes. When a variable terminalnode (e.g., str) is added to the derivation and be-comes the frontier node, the grammar model thenswitches to GENTOKEN actions to populate thevariable terminal with tokens.

Unary Closure Sometimes, generating an ASTrequires applying a chain of unary productions.For instance, it takes three time steps (t9 − t11)to generate the sub-structure expr* 7→ expr 7→Name 7→ str in Fig. 1(a). This can be effectivelyreduced to one step of APPLYRULE action by tak-ing the closure of the chain of unary productionsand merging them into a single rule: expr* 7→∗str. Unary closures reduce the number of actionsneeded, but would potentially increase the size ofthe grammar. In our experiments we tested ourmodel both with and without unary closures (§ 5).

3.2 GENTOKEN Actions

Once we reach a frontier node nft that correspondsto a variable type (e.g., str), GENTOKEN actionsare used to fill this node with values. For general-purpose PLs like Python, variables and constantshave values with one or multiple tokens. For in-stance, a node that stores the name of a function(e.g., sorted) has a single token, while a nodethat denotes a string constant (e.g., a=‘hello

world’) could have multiple tokens. Our modelcopes with both scenarios by firing GENTOKEN

actions at one or more time steps. At each time

step, GENTOKEN appends one terminal token tothe current frontier variable node. A special </n>token is used to “close” the node. The grammarmodel then proceeds to the new frontier node.

Terminal tokens can be generated from a pre-defined vocabulary, or be directly copied from theinput NL. This is motivated by the observationthat the input description often contains out-of-vocabulary (OOV) variable names or literal valuesthat are directly used in the target code. For in-stance, in our running example the variable namemy list can be directly copied from the the inputat t12. We give implementation details in § 4.2.2.

4 Estimating Action ProbabilitiesWe estimate action probabilities in Eq. (2) usingattentional neural encoder-decoder models with aninformation flow structured by the syntax trees.

4.1 EncoderFor an NL description x consisting of n words{wi}ni=1, the encoder computes a context sen-sitive embedding hi for each wi using a bidi-rectional Long Short-Term Memory (LSTM) net-work (Hochreiter and Schmidhuber, 1997), simi-lar to the setting in (Bahdanau et al., 2014). Seesupplementary materials for detailed equations.

4.2 DecoderThe decoder uses an RNN to model the sequentialgeneration process of an AST defined as Eq. (2).Each action step in the grammar model naturallygrounds to a time step in the decoder RNN. There-fore, the action sequence in Fig. 1(b) can be in-terpreted as unrolling RNN time steps, with solidarrows indicating RNN connections. The RNNmaintains an internal state to track the generationprocess (§ 4.2.1), which will then be used to com-pute action probabilities p(at|x, a<t) (§ 4.2.2).

4.2.1 Tracking Generation StatesOur implementation of the decoder resembles avanilla LSTM, with additional neural connections(parent feeding, Fig. 1(b)) to reflect the topologicalstructure of an AST. The decoder’s internal hiddenstate at time step t, st, is given by:

st = fLSTM([at−1 : ct : pt : nft ], st−1), (3)

where fLSTM(·) is the LSTM update function.[:] denotes vector concatenation. st will then beused to compute action probabilities p(at|x, a<t)in Eq. (2). Here, at−1 is the embedding of the pre-vious action. ct is a context vector retrieved from

h1 h2

sort my_list

h3 h4 h5

in descending

s9p9

a8

nf9

GenToken[</n>]

ApplyRule[Call] Parent State

s4a4 +

expr*

s8

ApplyRule GenToken

type of       ?

order

... ...

nf9

non­terminal variable terminal

embedding of node type

embedding of

c9

Figure 2: Illustration of a decoder time step (t = 9)

input encodings {hi} via soft attention. pt is avector that encodes the information of the parentaction. nft denotes the node type embedding ofthe current frontier node nft

3. Intuitively, feedingthe decoder the information of nft helps the modelto keep track of the frontier node to expand.Action Embedding at We maintain two actionembedding matrices, WR and WG. Each row inWR (WG) corresponds to an embedding vectorfor an action APPLYRULE[r] (GENTOKEN[v]).Context Vector ct The decoder RNN uses soft at-tention to retrieve a context vector ct from the in-put encodings {hi} pertain to the prediction of thecurrent action. We follow Bahdanau et al. (2014)and use a Deep Neural Network (DNN) with a sin-gle hidden layer to compute attention weights.Parent Feeding pt Our decoder RNN uses ad-ditional neural connections to directly pass infor-mation from parent actions. For instance, whencomputing s9, the information from its parent ac-tion step t4 will be used. Formally, we define theparent action step pt as the time step at whichthe frontier node nft is generated. As an exam-ple, for t9, its parent action step p9 is t4, sincenf9 is the node ?, which is generated at t4 by theAPPLYRULE[Call7→. . .] action.

We model parent information pt from twosources: (1) the hidden state of parent action spt ,and (2) the embedding of parent action apt . pt isthe concatenation. The parent feeding schema en-ables the model to utilize the information of par-ent code segments to make more confident predic-tions. Similar approaches of injecting parent in-formation were also explored in the SEQ2TREE

model in Dong and Lapata (2016)4.3We maintain an embedding for each node type.4SEQ2TREE generates tree-structured outputs by condi-

4.2.2 Calculating Action Probabilities

In this section we explain how action probabilitiesp(at|x, a<t) are computed based on st.APPLYRULE The probability of applying rule ras the current action at is given by a softmax5:

p(at = APPLYRULE[r]|x, a<t) =

softmax(WR · g(st))ᵀ · e(r) (4)

where g(·) is a non-linearity tanh(W ·st+b), ande(r) the one-hot vector for rule r.GENTOKEN As in § 3.2, a token v can be gener-ated from a predefined vocabulary or copied fromthe input, defined as the marginal probability:

p(at = GENTOKEN[v]|x, a<t) =

p(gen|x, a<t)p(v|gen, x, a<t)

+ p(copy|x, a<t)p(v|copy, x, a<t).

The selection probabilities p(gen|·) and p(copy|·)are given by softmax(WS · st). The prob-ability of generating v from the vocabulary,p(v|gen, x, a<t), is defined similarly as Eq. (4),except that we use the GENTOKEN embeddingmatrix WG, and we concatenate the context vectorct with st as input. To model the copy probability,we follow recent advances in modeling copyingmechanism in neural networks (Gu et al., 2016;Jia and Liang, 2016; Ling et al., 2016), and use apointer network (Vinyals et al., 2015) to computethe probability of copying the i-th word from theinput by attending to input representations {hi}:

p(wi|copy, x, a<t) =exp(ω(hi, st, ct))∑n

i′=1 exp(ω(hi′ , st, ct)),

where ω(·) is a DNN with a single hidden layer.Specifically, if wi is an OOV word (e.g., my list,which is represented by a special <unk> token inencoding), we directly copy the actual word wi tothe derivation.

4.3 Training and InferenceGiven a dataset of pairs of NL descriptions xi andcode snippets ci, we parse ci into its AST yi anddecompose yi into a sequence of oracle actions un-der the grammar model. The model is then op-timized by maximizing the log-likelihood of theoracle action sequence. At inference time, weuse beam search to approximate the best AST yin Eq. (1). See supplementary materials for thepseudo-code of the inference algorithm.tioning on the hidden states of parent non-terminals, whileour parent feeding uses the states of parent actions.

5We do not show bias terms for all softmax equations.

Dataset HS DJANGO IFTTT

Train 533 16,000 77,495Development 66 1,000 5,171Test 66 1,805 758

Avg. tokens in description 39.1 14.3 7.4Avg. characters in code 360.3 41.1 62.2Avg. size of AST (# nodes) 136.6 17.2 7.0

Statistics of Grammarw/o unary closure# productions 100 222 1009# node types 61 96 828terminal vocabulary size 1361 6733 0Avg. # actions per example 173.4 20.3 5.0

w/ unary closure# productions 100 237 –# node types 57 92 –Avg. # actions per example 141.7 16.4 –

Table 2: Statistics of datasets and associated grammars

5 Experimental Evaluation5.1 Datasets and Metrics

HEARTHSTONE (HS) dataset (Ling et al., 2016)is a collection of Python classes that implementcards for the card game HearthStone. Each cardcomes with a set of fields (e.g., name, cost, anddescription), which we concatenate to create theinput sequence. This dataset is relatively difficult:input descriptions are short, while the target codeis in complex class structures, with each AST hav-ing 137 nodes on average.DJANGO dataset (Oda et al., 2015) is a collectionof lines of code from the Django web framework,each with a manually annotated NL description.Compared with the HS dataset where card imple-mentations are somewhat homogenous, examplesin DJANGO are more diverse, spanning a wide va-riety of real-world use cases like string manipula-tion, IO operations, and exception handling.IFTTT dataset (Quirk et al., 2015) is a domain-specific benchmark that provides an interest-ing side comparison. Different from HS andDJANGO which are in a general-purpose PL, pro-grams in IFTTT are written in a domain-specificlanguage used by the IFTTT task automationApp. Users of the App write simple instruc-tions (e.g., If Instagram.AnyNewPhotoByYou

Then Dropbox.AddFileFromURL) with NL de-scriptions (e.g., “Autosave your Instagram photosto Dropbox”). Each statement inside the If orThen clause consists of a channel (e.g., Dropbox)and a function (e.g., AddFileFromURL)6. This

6Like Beltagy and Quirk (2016), we strip function param-

simple structure results in much more conciseASTs (7 nodes on average). Because all examplesare created by ordinary Apps users, the datasetis highly noisy, with input NL very loosely con-nected to target ASTs. The authors thus provide ahigh-quality filtered test set, where each exampleis verified by at least three annotators. We use thisset for evaluation. Also note IFTTT’s grammar hasmore productions (Tab. 2), but this does not implythat its grammar is more complex. This is becausefor HS and DJANGO terminal tokens are generatedby GENTOKEN actions, but for IFTTT, all the codeis generated directly by APPLYRULE actions.Metrics As is standard in semantic parsing, wemeasure accuracy, the fraction of correctly gen-erated examples. However, because generating anexact match for complex code structures is non-trivial, we follow Ling et al. (2016), and use token-level BLEU-4 with as a secondary metric, definedas the averaged BLEU scores over all examples.7

5.2 SetupPreprocessing All input descriptions are tok-enized using NLTK. We perform simple canoni-calization for DJANGO, such as replacing quotedstrings in the inputs with place holders. See sup-plementary materials for details. We extract unaryclosures whose frequency is larger than a thresh-old k (k = 30 for HS and 50 for DJANGO).Configuration The size of all embeddings is 128,except for node type embeddings, which is 64.The dimensions of RNN states and hidden layersare 256 and 50, respectively. Since our datasets arerelatively small for a data-hungry neural model,we impose strong regularization using recurrentdropouts (Gal and Ghahramani, 2016), togetherwith standard dropout layers added to the inputsand outputs of the decoder RNN. We validate thedropout probability from {0, 0.2, 0.3, 0.4}. Fordecoding, we use a beam size of 15.

5.3 ResultsEvaluation results for Python code generationtasks are listed in Tab. 3. Numbers for our sys-

eters since they are mostly specific to users.7These two metrics are not ideal: accuracy only measures

exact match and thus lacks the ability to give credit to seman-tically correct code that is different from the reference, whileit is not clear whether BLEU provides an appropriate proxyfor measuring semantics in the code generation task. A moreintriguing metric would be directly measuring semantic/func-tional code equivalence, for which we present a pilot studyat the end of this section (cf. Error Analysis). We leave ex-ploring more sophisticated metrics (e.g. based on static codeanalysis) as future work.

HS DJANGO

ACC BLEU ACC BLEU

Retrieval System† 0.0 62.5 14.7 18.6Phrasal Statistical MT† 0.0 34.1 31.5 47.6Hierarchical Statistical MT† 0.0 43.2 9.5 35.9

NMT 1.5 60.4 45.1 63.4SEQ2TREE 1.5 53.4 28.9 44.6SEQ2TREE–UNK 13.6 62.8 39.4 58.2LPN† 4.5 65.6 62.3 77.6

Our system 16.2 75.8 71.6 84.5

Ablation Study– frontier embed. 16.7 75.8 70.7 83.8– parent feed. 10.6 75.7 71.5 84.3– copy terminals 3.0 65.7 32.3 61.7+ unary closure – 70.3 83.3– unary closure 10.1 74.8 –

Table 3: Results on two Python code generation tasks.†Results previously reported in Ling et al. (2016).

tems are averaged over three runs. We compareprimarily with two approaches: (1) Latent Pre-dictor Network (LPN), a state-of-the-art sequence-to-sequence code generation model (Ling et al.,2016), and (2) SEQ2TREE, a neural semantic pars-ing model (Dong and Lapata, 2016). SEQ2TREE

generates trees one node at a time, and the tar-get grammar is not explicitly modeled a priori,but implicitly learned from data. We test boththe original SEQ2TREE model released by the au-thors and our revised one (SEQ2TREE–UNK) thatuses unknown word replacement to handle rarewords (Luong et al., 2015). For completeness,we also compare with a strong neural machinetranslation (NMT) system (Neubig, 2015) using astandard encoder-decoder architecture with atten-tion and unknown word replacement8, and includenumbers from other baselines used in Ling et al.(2016). On the HS dataset, which has relativelylarge ASTs, we use unary closure for our modeland SEQ2TREE, and for DJANGO we do not.System Comparison As in Tab. 3, our modelregisters 11.7% and 9.3% absolute improvementsover LPN in accuracy on HS and DJANGO. Thisboost in performance strongly indicates the impor-tance of modeling grammar in code generation.For the baselines, we find LPN outperforms oth-ers in most cases. We also note that SEQ2TREE

achieves a decent accuracy of 13.6% on HS, whichis due to the effect of unknown word replacement,since we only achieved 1.5% without it. A closer

8For NMT, we also attempted to find the best-scoring syn-tactically correct predictions in the size-5 beam, but this didnot yield a significant improvement over the NMT results inTab. 3.

0 10 20 30 40 50Reference AST Size (# nodes)

0.0

0.2

0.4

0.6

0.8

1.0

Perf

orm

ance

BLEU acc

Figure 3: Performance w.r.t reference AST size on DJANGO

50 100 150 200 250Reference AST Size (# nodes)

0.0

0.2

0.4

0.6

0.8

1.0

Perf

orm

ance

BLEU acc

Figure 4: Performance w.r.t reference AST size on HS

comparison with SEQ2TREE is insightful for un-derstanding the advantage of our syntax-driven ap-proach, since both SEQ2TREE and our system out-put ASTs: (1) SEQ2TREE predicts one node eachtime step, and requires additional “dummy” nodesto mark the boundary of a subtree. The sheer num-ber of nodes in target ASTs makes the predictionprocess error-prone. In contrast, the APPLYRULE

actions of our grammar model allows for gener-ating multiple nodes at a single time step. Em-pirically, we found that in HS, SEQ2TREE takesmore than 300 time steps on average to generate atarget AST, while our model takes only 170 steps.(2) SEQ2TREE does not directly use productionsin the grammar, which possibly leads to grammat-ically incorrect ASTs and thus empty code out-puts. We observe that the ratio of grammaticallyincorrect ASTs predicted by SEQ2TREE on HSand DJANGO are 21.2% and 10.9%, respectively,while our system guarantees grammaticality.Ablation Study We also ablated our best-performing models to analyze the contribution ofeach component. “–frontier embed.” removes thefrontier node embedding nft from the decoderRNN inputs (Eq. (3)). This yields worse results onDJANGO while gives slight improvements in ac-curacy on HS. This is probably because that thegrammar of HS has fewer node types, and thusthe RNN is able to keep track of nft without de-pending on its embedding. Next, “–parent feed.”removes the parent feeding mechanism. The ac-curacy drops significantly on HS, with a marginaldeterioration on DJANGO. This result is interest-ing because it suggests that parent feeding is moreimportant when the ASTs are larger, which willbe the case when handling more complicated codegeneration tasks like HS. Finally, removing thepointer network (“–copy terminals”) in GENTO-

CHANNEL FULL TREE

Classical Methodsposclass (Quirk et al., 2015) 81.4 71.0LR (Beltagy and Quirk, 2016) 88.8 82.5

Neural Network MethodsNMT 87.7 77.7NN (Beltagy and Quirk, 2016) 88.0 74.3SEQ2TREE (Dong and Lapata, 2016) 89.7 78.4Doubly-Recurrent NN 90.1 78.2(Alvarez-Melis and Jaakkola, 2017)

Our system 90.0 82.0– parent feed. 89.9 81.1– frontier embed. 90.1 78.7

Table 4: Results on the noise-filtered IFTTT test set of “>3agree with gold annotations” (averaged over three runs), ourmodel performs competitively among neural models.

KEN actions gives poor results, indicating that itis important to directly copy variable names andvalues from the input.

The results with and without unary closuredemonstrate that, interestingly, it is effective onHS but not on DJANGO. We conjecture that this isbecause on HS it significantly reduces the numberof actions from 173 to 142 (c.f., Tab. 2), with thenumber of productions in the grammar remainingunchanged. In contrast, DJANGO has a broaderdomain, and thus unary closure results in moreproductions in the grammar (237 for DJANGO

vs. 100 for HS), increasing sparsity.Performance by the size of AST We further in-vestigate our model’s performance w.r.t. the sizeof the gold-standard ASTs in Figs. 3 and 4. Notsurprisingly, the performance drops when the sizeof the reference ASTs increases. Additionally, onthe HS dataset, the BLEU score still remains ataround 50 even when the size of ASTs grows to200, indicating that our proposed syntax-drivenapproach is robust for long code segments.Domain Specific Code Generation Although thisis not the focus of our work, evaluation on IFTTT

brings us closer to a standard semantic parsing set-ting, which helps to investigate similarities anddifferences between generation of more compli-cated general-purpose code and and more limited-domain simpler code. Tab. 4 shows the results,following the evaluation protocol in (Beltagy andQuirk, 2016) for accuracies at both channel andfull parse tree (channel + function) levels. Ourfull model performs on par with existing neu-ral network-based methods, while outperformingother neural models in full tree accuracy (82.0%).This score is close to the best classical method(LR), which is based on a logistic regression

input <name> Brawl </name> <cost> 5 </cost> <desc>Destroy all minions except one (chosen randomly)</desc> <rarity> Epic </rarity> ...

pred. class Brawl(SpellCard):def init (self):super(). init (’Brawl’, 5, CHARACTER CLASS.

WARRIOR, CARD RARITY.EPIC)def use(self, player, game):super().use(player, game)targets = copy.copy(game.other player.minions)targets.extend(player.minions)for minion in targets:minion.die(self) A

ref. minions = copy.copy(player.minions)minions.extend(game.other player.minions)if len(minions) > 1:survivor = game.random choice(minions)for minion in minions:if minion is not survivor: minion.die(self)

B

input join app config.path and string ’locale’ into a filepath, substitute it for localedir.

pred. localedir = os.path.join(app config.path, ’locale’) 3

input self.plural is an lambda function with an argumentn, which returns result of boolean expression n notequal to integer 1

pred. self.plural = lambda n: len(n) 7ref. self.plural = lambda n: int(n!=1)

Table 5: Predicted examples from HS (1st) and DJANGO.Copied contents (copy probability > 0.9) are highlighted.

model with rich hand-engineered features (e.g.,brown clusters and paraphrase). Also note that theperformance between NMT and other neural mod-els is much closer compared with the results inTab. 3. This suggests that general-purpose codegeneration is more challenging than the simplerIFTTT setting, and therefore modeling structuralinformation is more helpful.Case Studies We present output examples inTab. 5. On HS, we observe that most of thetime our model gives correct predictions by fillinglearned code templates from training data with ar-guments (e.g., cost) copied from input. However,we do find interesting examples indicating that themodel learns to generalize beyond trivial copy-ing. For instance, the first example is one that ourmodel predicted wrong — it generated code blockA instead of the gold B (it also missed a functiondefinition not shown here). However, we find thatthe block A actually conveys part of the input in-tent by destroying all, not some, of the minions.Since we are unable to find code block A in thetraining data, it is clear that the model has learnedto generalize to some extent from multiple trainingcard examples with similar semantics or structure.

The next two examples are from DJANGO. Thefirst one shows that the model learns the usageof common API calls (e.g., os.path.join), and

how to populate the arguments by copying frominputs. The second example illustrates the dif-ficulty of generating code with complex nestedstructures like lambda functions, a scenario worthfurther investigation in future studies. More exam-ples are attached in supplementary materials.Error Analysis To understand the sources of er-rors and how good our evaluation metric (exactmatch) is, we randomly sampled and labeled 100and 50 failed examples (with accuracy=0) fromDJANGO and HS, resp. We found that around 2%of these examples in the two datasets are actuallysemantically equivalent. These examples include:(1) using different parameter names when defininga function; (2) omitting (or adding) default valuesof parameters in function calls. While the rarity ofsuch examples suggests that our exact match met-ric is reasonable, more advanced evaluation met-rics based on statistical code analysis are definitelyintriguing future work.

For DJANGO, we found that 30% of failedcases were due to errors where the pointer net-work failed to appropriately copy a variable nameinto the correct position. 25% were because thegenerated code only partially implementated therequired functionality. 10% and 5% of errorswere due to malformed English inputs and pre-processing errors, respectively. The remaining30% of examples were errors stemming from mul-tiple sources, or errors that could not be easily cat-egorized into the above. For HS, we found thatall failed card examples were due to partial imple-mentation errors, such as the one shown in Table 5.

6 Related Work

Code Generation and Analysis Most existingworks on code generation focus on generatingcode for domain specific languages (DSLs) (Kush-man and Barzilay, 2013; Raza et al., 2015; Man-shadi et al., 2013), with neural network-based ap-proaches recently explored (Parisotto et al., 2016;Balog et al., 2016). For general-purpose code gen-eration, besides the general framework of Linget al. (2016), existing methods often use languageand task-specific rules and strategies (Lei et al.,2013; Raghothaman et al., 2016). A similar lineis to use NL queries for code retrieval (Wei et al.,2015; Allamanis et al., 2015). The reverse task ofgenerating NL summaries from source code hasalso been explored (Oda et al., 2015; Iyer et al.,2016). Finally, there are probabilistic models of

source code (Maddison and Tarlow, 2014; Nguyenet al., 2013). The most relevant work is Allama-nis et al. (2015), which uses a factorized modelto measure semantic relatedness between NL andASTs for code retrieval, while our model tacklesthe more challenging generation task.Semantic Parsing Our work is related to thegeneral topic of semantic parsing, where the tar-get logical forms can be viewed as DSLs. Theparsing process is often guided by grammaticalformalisms like combinatory categorical gram-mars (Kwiatkowski et al., 2013; Artzi et al.,2015), dependency-based syntax (Liang et al.,2011; Pasupat and Liang, 2015) or task-specificformalisms (Clarke et al., 2010; Yih et al., 2015;Krishnamurthy et al., 2016; Misra et al., 2015; Meiet al., 2016). Recently, there are efforts in design-ing neural network-based semantic parsers (Misraand Artzi, 2016; Dong and Lapata, 2016; Nee-lakantan et al., 2016; Yin et al., 2016). Severalapproaches have be proposed to utilize grammarknowledge in a neural parser, such as augmentingthe training data by generating examples guidedby the grammar (Kocisky et al., 2016; Jia andLiang, 2016). Liang et al. (2016) used a neu-ral decoder which constrains the space of nextvalid tokens in the query language for questionanswering. Finally, the structured prediction ap-proach proposed by Xiao et al. (2016) is closelyrelated to our model in using the underlying gram-mar as prior knowledge to constrain the genera-tion process of derivation trees, while our methodis based on a unified grammar model which jointlycaptures production rule application and terminalsymbol generation, and scales to general purposecode generation tasks.

7 Conclusion

This paper proposes a syntax-driven neural codegeneration approach that generates an abstractsyntax tree by sequentially applying actions froma grammar model. Experiments on both code gen-eration and semantic parsing tasks demonstrate theeffectiveness of our proposed approach.

Acknowledgment

We are grateful to Wang Ling for his generous helpwith LPN and setting up the benchmark. We alsothank Li Dong for helping with SEQ2TREE andinsightful discussions.

ReferencesMiltiadis Allamanis, Daniel Tarlow, Andrew D. Gor-

don, and Yi Wei. 2015. Bimodal modelling ofsource code and natural language. In Proceedingsof ICML. volume 37.

David Alvarez-Melis and Tommi S. Jaakkola. 2017.Tree-structured decoding with doubly recurrent neu-ral networks. In Proceedings of ICLR.

Yoav Artzi, Kenton Lee, and Luke Zettlemoyer. 2015.Broad-coverage CCG semantic parsing with AMR.In Proceedings of EMNLP.

Yoav Artzi and Luke Zettlemoyer. 2013. Weakly su-pervised learning of semantic parsers for mappinginstructions to actions. Transaction of ACL 1(1).

Dzmitry Bahdanau, Kyunghyun Cho, and YoshuaBengio. 2014. Neural machine translation byjointly learning to align and translate. CoRRabs/1409.0473.

Matej Balog, Alexander L. Gaunt, Marc Brockschmidt,Sebastian Nowozin, and Daniel Tarlow. 2016.Deepcoder: Learning to write programs. CoRRabs/1611.01989.

Robert Balzer. 1985. A 15 year perspective on au-tomatic programming. IEEE Trans. Software Eng.11(11).

Laura Banarescu, Claire Bonial, Shu Cai, MadalinaGeorgescu, Kira Griffitt, Ulf Hermjakob, KevinKnight, Philipp Koehn, Martha Palmer, and NathanSchneider. 2013. Abstract meaning representationfor sembanking. In Proceedings of the 7th Linguis-tic Annotation Workshop and Interoperability withDiscourse, LAW-ID@ACL.

I. Beltagy and Chris Quirk. 2016. Improved seman-tic parsers for if-then statements. In Proceedings ofACL.

Jonathan Berant, Andrew Chou, Roy Frostig, and PercyLiang. 2013. Semantic parsing on freebase fromquestion-answer pairs. In Proceedings of EMNLP.

Joel Brandt, Mira Dontcheva, Marcos Weskamp, andScott R. Klemmer. 2010. Example-centric program-ming: integrating web search into the developmentenvironment. In Proceedings of CHI.

Joel Brandt, Philip J. Guo, Joel Lewenstein, MiraDontcheva, and Scott R. Klemmer. 2009. Two stud-ies of opportunistic programming: interleaving webforaging, learning, and writing code. In Proceedingsof CHI.

Stephen Clark and James R. Curran. 2007. Wide-coverage efficient statistical parsing with CCG andlog-linear models. Computational Linguistics 33(4).

James Clarke, Dan Goldwasser, Ming-Wei Chang, andDan Roth. 2010. Driving semantic parsing from theworld’s response. In Proceedings of CoNLL.

Li Dong and Mirella Lapata. 2016. Language to logicalform with neural attention. In Proceedings of ACL.

Yarin Gal and Zoubin Ghahramani. 2016. A theoret-ically grounded application of dropout in recurrentneural networks. In Proceedings of NIPS.

Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K.Li. 2016. Incorporating copying mechanism insequence-to-sequence learning. In Proceedings ofACL.

Tihomir Gvero and Viktor Kuncak. 2015. Interactivesynthesis using free-form queries. In Proceedingsof ICSE.

Sepp Hochreiter and Jurgen Schmidhuber. 1997. Longshort-term memory. Neural Computation 9(8).

Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, andLuke Zettlemoyer. 2016. Summarizing source codeusing a neural attention model. In Proceedings ofACL.

Robin Jia and Percy Liang. 2016. Data recombinationfor neural semantic parsing. In Proceedings of ACL.

Tomas Kocisky, Gabor Melis, Edward Grefenstette,Chris Dyer, Wang Ling, Phil Blunsom, andKarl Moritz Hermann. 2016. Semantic parsing withsemi-supervised sequential autoencoders. In Pro-ceedings of EMNLP.

Jayant Krishnamurthy, Oyvind Tafjord, and AniruddhaKembhavi. 2016. Semantic parsing to probabilisticprograms for situated question answering. In Pro-ceedings of EMNLP.

Nate Kushman and Regina Barzilay. 2013. Using se-mantic unification to generate regular expressionsfrom natural language. In Proceedings of NAACL.

Tom Kwiatkowski, Eunsol Choi, Yoav Artzi, andLuke S. Zettlemoyer. 2013. Scaling semanticparsers with on-the-fly ontology matching. In Pro-ceedings of the EMNLP.

Tao Lei, Fan Long, Regina Barzilay, and Martin C. Ri-nard. 2013. From natural language specifications toprogram input parsers. In Proceedings of ACL.

Chen Liang, Jonathan Berant, Quoc Le, Kenneth D.Forbus, and Ni Lao. 2016. Neural symbolic ma-chines: Learning semantic parsers on freebase withweak supervision. CoRR abs/1611.00020.

Percy Liang, Michael I. Jordan, and Dan Klein. 2011.Learning dependency-based compositional seman-tics. In Proceedings of ACL.

Wang Ling, Phil Blunsom, Edward Grefenstette,Karl Moritz Hermann, Tomas Kocisky, FuminWang, and Andrew Senior. 2016. Latent predictornetworks for code generation. In Proceedings ofACL.

Greg Little and Robert C. Miller. 2009. Keyword pro-gramming in java. Autom. Softw. Eng. 16(1).

Thang Luong, Ilya Sutskever, Quoc V. Le, OriolVinyals, and Wojciech Zaremba. 2015. Addressingthe rare word problem in neural machine translation.In Proceedings of ACL.

Chris J. Maddison and Daniel Tarlow. 2014. Structuredgenerative models of natural source code. In Pro-ceedings of ICML. volume 32.

Mehdi Hafezi Manshadi, Daniel Gildea, and James F.Allen. 2013. Integrating programming by exampleand natural language programming. In Proceedingsof AAAI.

Hongyuan Mei, Mohit Bansal, and Matthew R. Wal-ter. 2016. Listen, attend, and walk: Neural mappingof navigational instructions to action sequences. InProceedings of AAAI.

Dipendra K. Misra and Yoav Artzi. 2016. Neural shift-reduce CCG semantic parsing. In Proceedings ofEMNLP.

Dipendra Kumar Misra, Kejia Tao, Percy Liang, andAshutosh Saxena. 2015. Environment-driven lex-icon induction for high-level instructions. In Pro-ceedings of ACL.

Arvind Neelakantan, Quoc V. Le, and Ilya Sutskever.2016. Neural programmer: Inducing latent pro-grams with gradient descent. In Proceedings ofICLR.

Graham Neubig. 2015. lamtram: A toolkit for lan-guage and translation modeling using neural net-works. http://www.github.com/neubig/lamtram.

Tung Thanh Nguyen, Anh Tuan Nguyen, Hoan AnhNguyen, and Tien N. Nguyen. 2013. A statisticalsemantic language model for source code. In Pro-ceedings of ACM SIGSOFT .

Yusuke Oda, Hiroyuki Fudaba, Graham Neubig,Hideaki Hata, Sakriani Sakti, Tomoki Toda, andSatoshi Nakamura. 2015. Learning to generatepseudo-code from source code using statistical ma-chine translation (T). In Proceedings of ASE.

Emilio Parisotto, Abdel-rahman Mohamed, RishabhSingh, Lihong Li, Dengyong Zhou, and PushmeetKohli. 2016. Neuro-symbolic program synthesis.CoRR abs/1611.01855.

Panupong Pasupat and Percy Liang. 2015. Composi-tional semantic parsing on semi-structured tables. InProceedings of ACL.

Python Software Foundation. 2016. Python abstractgrammar. https://docs.python.org/2/library/ast.html.

Chris Quirk, Raymond J. Mooney, and Michel Galley.2015. Language to code: Learning semantic parsersfor if-this-then-that recipes. In Proceedings of ACL.

Mukund Raghothaman, Yi Wei, and Youssef Hamadi.2016. SWIM: synthesizing what i mean: codesearch and idiomatic snippet synthesis. In Proceed-ings of ICSE.

Mohammad Raza, Sumit Gulwani, and Natasa Milic-Frayling. 2015. Compositional program synthesisfrom natural language and examples. In Proceed-ings of IJCAI.

Lappoon R. Tang and Raymond J. Mooney. 2001. Us-ing multiple clause constructors in inductive logicprogramming for semantic parsing. In Proceedingsof ECML.

Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly.2015. Pointer networks. In Proceedings of NIPS.

Yi Wei, Nirupama Chandrasekaran, Sumit Gul-wani, and Youssef Hamadi. 2015. Build-ing bing developer assistant. Techni-cal report. https://www.microsoft.com/en-us/research/publication/building-bing-developer-assistant/.

Chunyang Xiao, Marc Dymetman, and Claire Gardent.2016. Sequence-based structured prediction for se-mantic parsing. In Proceedings of ACL.

Wen-tau Yih, Ming-Wei Chang, Xiaodong He, andJianfeng Gao. 2015. Semantic parsing via stagedquery graph generation: Question answering withknowledge base. In Proceedings of ACL.

Pengcheng Yin, Zhengdong Lu, Hang Li, and Ben Kao.2016. Neural enquirer: Learning to query tables innatural language. In Proceedings of IJCAI.

Luke Zettlemoyer and Michael Collins. 2005. Learn-ing to map sentences to logical form structured clas-sification with probabilistic categorial grammars. InProceedings of UAI.

Supplementary Materials

A Encoder LSTM Equations

Suppose the input natural language description xconsists of n words {wi}ni=1. Let wi denote theembedding of wi. We use two LSTMs to processx in forward and backward order, and get the se-quence of hidden states {~hi}ni=1 and { ~hi}ni=1 inthe two directions:

~hi = f→LSTM(wi, ~hi−1)

~hi = f←LSTM(wi, ~hi+1),

where f→LSTM and f←LSTM are standard LSTM up-date functions. The representation of the i-thword, hi, is given by concatenating ~hi and ~hi.

B Inference Algorithm

Given an NL description, we approximate the bestAST y in Eq. 1 using beam search. The inferenceprocedure is listed in Algorithm 1.

We maintain a beam of size K. The beam isinitialized with one hypothesis AST with a singleroot node (line 2). At each time step, the decoderenumerates over all hypotheses in the beam. Foreach hypothesis AST, we first find its frontier nodenft (line 6). If nft is a non-terminal node, we col-lect all syntax rules r with nft as the head nodeto the actions set (line 10). If nft is a variableterminal node, we add all terminal tokens in thevocabulary and the input description as candidateactions (line 13). We apply each candidate actionon the current hypothesis AST to generate a newhypothesis (line 15). We then rank all newly gen-erated hypotheses and keep the top-K scored onesin the beam. A complete hypothesis AST is gener-ated when it has no frontier node. We then convertthe top-scored complete AST into the surface code(lines 18-19).

We remark that our inference algorithm canbe implemented efficiently by expanding multi-ple hypotheses (lines 5-16) simultaneously usingmini-batching on GPU.

C Dataset Preprocessing

Infrequent Words We replace word types whosefrequency is lower than d with a special <unk>token (d = 3 for DJANGO, 3 for HS and 2 forIFTTT).Canonicalization We perform simple canonical-ization for the DJANGO dataset: (1) We ob-serve that input descriptions often come with

quoted string literals (e.g., verbose name is astring ‘cache entry’). We therefore replace quotedstrings with indexed placeholders using regularexpression. After decoding, we run a post-processing step to replace all placeholders withtheir actual values. (2) For descriptions withcascading variable reference (e.g., call methodself.makekey), we append after the whole variablename with tokens separated by ‘.’ (e.g., appendself and makekey after self.makekey). This givesthe pointer network flexibility to copy either par-tial or whole variable names.Generate Oracle Action Sequence To train ourmodel, we generate the gold-standard action se-quence from reference code. For IFTTT, we sim-ply parse the officially provided ASTs into se-quences of APPLYRULE actions. For HS andDJANGO, we first convert the Python code intoASTs using the standard ast module. Valuesinside variable terminal nodes are tokenized byspace and camel case (e.g., ClassName is tok-enized to Class and Name). We then traverse theAST in pre-order to generate the reference actionsequence according to the grammar model.

D Additional Decoding Examples

We provide extra decoding examples from theDJANGO and HS datasets, listed in Table 6 and Ta-ble 7, respectively. The model heavily relies on thepointer network to copy variable names and con-stants from input descriptions. We find the sourceof errors in DJANGO is more diverse, with mostincorrect examples resulting from missing argu-ments and incorrect words copied by the pointernetwork. Errors in HS are mostly due to partiallyor incorrectly implemented effects. Also note thatthe first example in Table 6 is semantically cor-rect, although it was considered incorrect underour exact-match metric. This suggests more ad-vanced evaluation metric that takes into accountthe execution results in future studies.

Algorithm 1: Inference AlgorithmInput : NL description xOutput: code snippet c

1 call Encoder to encode x2 Q = {y0 (root)} . Initialize a beam of size K3 for time step t do4 Q′ = ∅5 foreach hypothesis yt ∈ Q do6 nft = FrontierNode(yt)7 A = ∅ . Initialize the set of candidate actions8 if nft is non-terminal then9 foreach production rule r with nft as the head node do

10 A = A ∪ {APPLYRULE[r]} . APPLYRULE actions for non-terminal nodes11 else12 foreach terminal token v do13 A = A ∪ {GENTOKEN[v]} . GENTOKEN actions for variable terminal nodes14 foreach action at ∈ A do15 y′

t = ApplyAction(yt, at)16 Q′ = Q′ ∪ {y ′

t}17 Q = top-K scored hypotheses in Q′

18 y = top-scored complete hypothesis AST19 convert y to surface code c20 return c

input for every i in range of integers from 0 to length of result, not includedpred. for i in range(0, len(result)): 3 ref. for i in range(len(result)):

input call the function blankout with 2 arguments: t.contents and ’B’, write the result to out.pred. out.write(blankout(t.contents, ’B’)) 3 ref. out.write(blankout(t.contents, ’B’))

pred. code list.append(foreground[v]) 3 ref. code list.append(foreground[v])

input zip elements of inner result and inner args into a list of tuples, for every i item and i args in the resultpred. for i item, i args in zip(inner result,

inner args): 3ref. for i item, i args in zip(inner result,

inner args):

input activate is a lambda function which returns None for any argument x.pred. activate = lambda x: None 3 ref. activate = lambda x: None

input if elt is an instance of Choice or NonCapture classespred. if isinstance(elt, Choice): 7 ref. if isinstance(elt, (Choice, NonCapture)):

input get translation function attribute of the object t, call the result with an argument eol message, substitute the result forresult.

pred. translation function = getattr(t,translation function) 7

ref. result = getattr(t, translation function)(eol message)

input for every s in strings, call the function force text with an argument s, join the results in a string, return the result.pred. return ’’.join(force text(s)) 7 ref. return ’’.join(force text(s) for s in strings)

input for every p in parts without the first elementpred. for p in p[1:]: 7 ref. for p in parts[1:]:

input call the function get language, split the result by ’-’, substitute the first element of the result for base lang.pred. base lang = get language().split()[0] 7 ref. base lang = get language().split(’−’)[0]

Table 6: Predicted examples from DJANGO dataset. Copied contents (copy probability > 0.9) are highlighted

input <name> Burly Rockjaw Trogg </name> <cost> 5 </cost> <attack> 3 </attack> <defense> 5 </defense><desc> Whenever your opponent casts a spell, gain 2 Attack. </desc> <rarity> Common </rarity> ...

pred. class BurlyRockjawTrogg(MinionCard):def init (self):

super(). init (’Burly Rockjaw Trogg’, 4, CHARACTER CLASS.ALL, CARD RARITY.COMMON)def create minion(self, player):

return Minion(3, 5, effects=[Effect(SpellCast(player=EnemyPlayer()),ActionTag(Give(ChangeAttack(2)), SelfSelector()))]) 3

input <name> Maexxna </name> <cost> 6 </cost> <attack> 2 </attack> <defense> 8 </defense> <desc> Destroyany minion damaged by this minion. </desc> <rarity> Legendary </rarity> ...

pred. class Maexxna(MinionCard):def init (self):

super(). init (’Maexxna’, 6, CHARACTER CLASS.ALL, CARD RARITY.LEGENDARY,minion type=MINION TYPE.BEAST)

def create minion(self, player):return Minion(2, 8, effects=[Effect(DidDamage(), ActionTag(Kill(),

TargetSelector(IsMinion())))]) 3

input <name> Hellfire </name> <cost> 4 </cost> <attack> -1 </attack> <defense> -1 </defense> <desc> Deal 3damage to ALL characters. </desc> <rarity> Free </rarity> ...

pred. class Hellfire(SpellCard):def init (self):

super(). init (’Hellfire’, 4, CHARACTER CLASS.WARLOCK, CARD RARITY.FREE)

def use(self, player, game):super().use(player, game)for minion in copy.copy(game.other player.minions):

minion.damage(player.effective spell damage(3), self) 7

ref. class Hellfire(SpellCard):def init (self):

super(). init (’Hellfire’, 4, CHARACTER CLASS.WARLOCK, CARD RARITY.FREE)

def use(self, player, game):super().use(player, game)targets = copy.copy(game.other player.minions)targets.extend(game.current player.minions)targets.append(game.other player.hero)targets.append(game.current player.hero)for minion in targets:

minion.damage(player.effective spell damage(3), self)

reason Partially implemented effect: only deal 3 damage to opponent’s characters

input <name> Darkscale Healer </name> <cost> 5 </cost> <attack> 4 </attack> <defense> 5 </defense> <desc>Battlecry: Restore 2 Health to all friendly characters. </desc> <rarity> Common </rarity> ...

pred. class DarkscaleHealer(MinionCard):def init (self):

super(). init (’Darkscale Healer’, 5, CHARACTER CLASS.ALL,CARD RARITY.COMMON, battlecry=Battlecry(Damage(2),CharacterSelector(players=BothPlayer(), picker=UserPicker())))

def create minion(self, player):return Minion(4, 5) 7

ref. class DarkscaleHealer(MinionCard):def init (self):

super(). init (’Darkscale Healer’, 5, CHARACTER CLASS.ALL,CARD RARITY.COMMON, battlecry=Battlecry(Heal(2), CharacterSelector()))

def create minion(self, player):return Minion(4, 5)

reason Incorrect effect: damage 2 health instead of restoring. Cast effect to all players instead of friendly players only.

Table 7: Predicted card examples from HS dataset. Copied contents (copy probability > 0.9) are highlighted.