learning neurosymbolic generative models via program...

Learning Neurosymbolic Generative Models via Program Synthesis

Halley Young 1 Osbert Bastani 1 Mayur Naik 1

AbstractGenerative models have become significantlymore powerful in recent years. However, thesemodels continue to have difficulty capturingglobal structure in data. For example, images ofbuildings typically contain spatial patterns such aswindows repeating at regular intervals, but state-of-the-art models have difficulty generating thesepatterns. We propose to address this problem byincorporating programs representing global struc-ture into generative models—e.g., a 2D for-loopmay represent a repeating pattern of windows—along with a framework for learning these modelsby leveraging program synthesis to obtain trainingdata. On both synthetic and real-world data, wedemonstrate that our approach substantially out-performs state-of-the-art at both generating andcompleting images with global structure.

1. IntroductionThere has been much interest recently in generative models,following the introduction of both variational autoencoders(VAEs) (Kingma & Welling, 2014) and generative adver-sarial networks (GANs) (Goodfellow et al., 2014). Thesemodels have successfully been applied to a range of tasks,including image generation (Radford et al., 2015), imagecompletion (IIzuka et al., 2017), texture synthesis (Jetchevet al., 2017; Xian et al., 2018), sketch generation (Ha & Eck,2017), and music generation (Dieleman et al., 2018).

Despite their successes, generative models still have diffi-culty capturing global structure. For example, consider theimage completion task in Figure 1. The original image (left)is of a building, for which the global structure is a 2D repeat-ing pattern of windows. Given a partial image (middle left),the goal is to predict the completion of the image. As can beseen, a state-of-the-art image completion algorithm has trou-

1University of Pennsylvania, USA. Correspondence to: HalleyYoung <[email protected]>.

Proceedings of the 36 th International Conference on MachineLearning, Long Beach, California, PMLR 97, 2019. Copyright2019 by the author(s).

ble reconstructing the original image (right) (IIzuka et al.,2017). Real-world data often contains such global structure,including repetitions, reflectional or rotational symmetry, oreven more complex patterns.

In recent years, program synthesis (Solar-Lezama et al.,2006) has emerged as a promising approach to capturingpatterns in data (Ellis et al., 2015; 2018; Valkov et al., 2018).The idea is that simple programs can capture global struc-ture that evades state-of-the-art deep neural networks. A keybenefit of using program synthesis is that we can design thespace of programs to capture different kinds of structure—e.g., repeating patterns (Ellis et al., 2018), symmetries, orspatial structure (Deng et al., 2018)—depending on the ap-plication domain. The challenge is that for the most part,existing approaches have synthesized programs that oper-ate directly over raw data. Since programs have difficultyoperating over perceptual data, existing approaches havelargely been limited to very simple data—e.g., detecting 2Drepeating patterns of simple shapes (Ellis et al., 2018).

We propose to address these shortcomings by synthesizingprograms that represent the underlying structure of high-dimensional data. In particular, we decompose programsinto two parts: (i) a sketch s 2 S that represents the skeletalstructure of the program (Solar-Lezama et al., 2006), withholes that are left unimplemented, and (ii) components c 2C that can be used to fill these holes. We consider perceptualcomponents—i.e., holes in the sketch are filled with rawperceptual data. For example, the program

represents part of the structure in the original image x⇤ inFigure 1 (left). The code is the sketch, and the componentis a sub-image from the given partial image. Together, wecall such a program a neurosymbolic program.

Building on these ideas, we propose an approach calledSynthesis-guided Generative Models (SGM) that combinesneurosymbolic programs representing global structure withstate-of-the-art deep generative models. By incorporatingprogrammatic structure, SGM substantially improves thequality of these models. As can be seen, the completionproduced using SGM (middle right of Figure 1) substantiallyoutperforms state-of-the-art.


original image x⇤ partial image xpart completion x (ours) completion x (baseline)

Figure 1. The task is to complete the partial image xpart (middle left) into an image that is close to the original image x

⇤ (left). Byincorporating programmatic structure, our approach (middle right) substantially outperforms state-of-the-art (IIzuka et al., 2017) (right).

SGM can be used for both generation and completion. Thegeneration pipeline is shown in Figure 2. At a high level,SGM for generation operates in two phases:

• First, it generates a program that represents the globalstructure in the image to be generated. In particular, itgenerates a program P = (s, c) representing the latentglobal structure in the image (left in Figure 2), wheres is a sketch and c is a perceptual component.

• Second, our algorithm executes P to obtain a struc-ture rendering xstruct representing the program as animage (middle of Figure 2). Then, our algorithm usesa deep generative model to complete xstruct into a fullimage (right of Figure 2). The structure in xstruct helpsguide the deep generative model towards images thatpreserve the global structure.

The image-completion pipeline (see Figure 3) is similar.

Training these models end-to-end is challenging, since a pri-ori, ground truth global structure is unavailable. To addressthis shortcoming, we leverage domain-specific program syn-thesis algorithms to produce examples of programs that rep-resent global structure of the training data. In particular, wepropose a synthesis algorithm tailored to the image domain,which extracts programs with nested for-loops that can rep-resent multiple 2D repeating patterns in images. Then, weuse these example programs as supervised training data.

Our programs can capture rich spatial structure in the train-ing data. For example, in Figure 2, the program structureencodes a repeating structure of 0’s and 2’s on the wholeimage, and a separate repeating structure of 3’s on the right-hand side of the image. Furthermore, in Figure 1, the gen-erated image captures the idea that the repeating pattern ofwindows does not extend to the bottom portion of the image.

Contributions. We propose an architecture of generativemodels that incorporates programmatic structure, as wellas an algorithm for training these models (Section 2). Ourlearning algorithm depends on a domain-specific programsynthesis algorithm for extracting global structure from thetraining data; we propose such an algorithm for the imagedomain (Section 3). Finally, we evaluate our approach on

synthetic data and on a real-world dataset of building fa-cades (Tylecek & Sara, 2013), both on the task of generationfrom scratch and on generation from a partial image. Weshow that our approach substantially outperforms severalstate-of-the-art deep generative models (Section 4).

Related work. There has been growing interest in apply-ing program synthesis to machine learning—e.g., for smalldata (Liang et al., 2010), interpretability (Wang & Rudin,2015; Verma et al., 2018), safety (Bastani et al., 2018), andlifelong learning (Valkov et al., 2018). Most relevantly, therehas been interest in using programs to capture structure thatdeep learning models have difficulty representing (Lakeet al., 2015; Ellis et al., 2015; 2018; Pu et al., 2018). Forinstance, Ellis et al. (2015) proposes an unsupervised learn-ing algorithm for capturing repeating patterns in simple linedrawings; however, not only are their domains simple, butthey can only handle a very small amount of noise. Simi-larly, Ellis et al. (2018) captures 2D repeating patterns ofsimple circles and polygons; however, rather than synthe-sizing programs with perceptual components, they learn asimple mapping from images to symbols as a preprocess-ing step. The closest work we are aware of is Valkov et al.(2018), which synthesizes programs with neural compo-nents (i.e., components implemented as neural networks);however, their application is to lifelong learning, not gener-ation, and to learning with supervision (labels) rather thanto unsupervised learning of structure.

There has been related work on synthesizing probabilis-tic programs (Hwang et al., 2011; Perov & Wood, 2014),including applications to learning structured ranking func-tions (Nori et al., 2015) and for learning design patterns (Tal-ton et al., 2012). More recently, DeepProbLog (Manhaeveet al., 2018) has extended the probabilistic logic program-ming language ProbLog (De Raedt et al., 2007) to includelearned neural components.

Additionally, there has been work extending neural modulenetworks (Andreas et al., 2016) to generative models (Denget al., 2018). These algorithms essentially learn a collectionof neural components that can be composed together basedon hierarchical structure. However, they require that the


sampled program P single for loop rendering structure rendering xstruct generated image x

Figure 2. SGM generation pipeline: (i) Sample a latent vector z ⇠ p(z), and sample a program P = (s, c) ⇠ p

�

(s, c | z) (left: a singlesampled for loop). (ii) Execute P to obtain a structure rendering (middle left: the rendering of the single for loop shown on the left,middle right: the structure rendering). (iii) Sample a completion x ⇠ p

✓

(x | s, c) of xstruct into a full image (right).

structure be available (albeit in natural language form) bothfor training the model and for generating new images.

Finally, there has been work incorporating spatial structureinto models for generating textures (Jetchev et al., 2017);however, their work only handles a single infinite repeating2D pattern. In contrast, we can capture a rich variety ofspatial patterns parameterized by a space of programs—e.g.,the image in Figure 1 generated by our approach containsdifferent repeating patterns in different parts of the image.

2. Generative Models with Latent StructureWe describe our proposed architecture for generative mod-els that incorporate programmatic structure. For most ofthis section, we focus on generation; we discuss how weadapt these techniques to image completion at the end. Weillustrate our generation pipeline in Figure 2.

Let p✓,�

(x) be a distribution over a space X with unknownparameters ✓,� that we want to estimate. We study the set-ting where x is generated based on some latent structure,which consists of a program sketch s 2 S and a percep-tual component c 2 C, and where the structure is in turngenerated conditioned on a latent vector z 2 Z—i.e.,

p✓,�

(x) =

Z

Z

Z

C

X

s2Sp✓

(x | s, c)p�

(s, c | z)p(z)dcdz.

Figure 2 shows an example of a sampled program P =

(s, c) ⇠ p�

(s, c | z) (left) and a sampled completion x ⇠p✓

(x | s, c) (right). To sample a completion, our modelexecutes P to obtain a structure rendering xstruct = eval(P )

(middle), and then uses p✓

(x | s, c) = p✓

(x | xstruct).

We now describe our algorithm for learning the parameters✓,� of p

✓,�

, followed by a description of our choices ofarchitecture for p

�

(s, c | z) and p✓

(x | s, c).

Learning algorithm. Given training data {x(i)}ni=1 ✓ X ,

where x(i) ⇠ p✓,�

(x), the maximum likelihood estimate is

✓⇤MLE,�⇤MLE = argmax

✓,�

nX

i=1

log p✓,�

(x(i)).

Since log p✓,�

(x) is intractable to optimize, we use an ap-proach based on the variational autoencoder (VAE). In par-ticular, we use a variational distribution

q�

(s, c, z | x) = q�

(z | s, c)q(s, c | x),

which has parameters ˜�. Then, we optimize ˜� while simulta-neously optimizing ✓,�. Using q

�

(s, c, z | x), the evidencelower bound on the log-likelihood is

log p✓,�

(x) � Eq(s,c,z|x)[log p✓(x | s, c)]�DKL(q(s, c, z | x) k p

�

(s, c | z)p(z))= E

q(s,c|x)[log p✓(x | s, c)] (1)+ E

q(s,c|x),q�

(z|s,c)[log p�(s, c | z)]

� Eq(s,c|x)[DKL(q

�

(z | s, c) k p(z))]�H(q(s, c | x)),

where DKL is the KL divergence and H is information en-tropy. Thus, we can approximate ✓⇤,�⇤ by optimizing thelower bound (1) instead of log p

✓,�

(x). However, (1) re-mains intractable since we are integrating over all programsketches s 2 S and perceptual components c 2 C. As wedescribe next, our approach is to synthesize a single pointestimate s

x

2 S and cx

2 C for each x 2 X .

Synthesizing structure. For a given x 2 X , we use pro-gram synthesis to infer a single likely choice s

x

2 S andcx

2 C of the latent structure. The program synthesis al-gorithm must be tailored to a specific domain; we proposean algorithm for inferring for-loop structure in images inSection 3. Then, we use these point estimates in place ofthe integrals over S and C—i.e., we assume that

q(s, c | x) = �(s� sx

)�(c� cx

),

where � is the Dirac delta function. Plugging into (1) gives

log p✓,�

(x) � log p✓

(x | sx

, cx

) (2)+ E

q

�

(z|sx

,c

x

)[log p�(sx, cx | z)]

�DKL(q�

(z | sx

, cx

) k p(z)).


partial image xpart synthesized program Ppart extrapolated program ˆP

structure rendering xstruct completion x (ours) original image x⇤

Figure 3. SGM image completion pipeline: (i) Given a partial image xpart (top left), use our program synthesis algorithm (Section 3) tosynthesize a program Ppart representing the structure in the partial image (top middle). (ii) Extrapolate Ppart to a program P = f

(Ppart)representing the structure of the full image. (iii) Execute P to obtain a rendering of the program structure xstruct (bottom left). (iv)Complete xstruct into an image x (bottom middle), which resembles the original image x

⇤ (bottom right).

where we have dropped the degenerate terms log �(s� sx

)

and log �(c � cx

) (which are constant with respect to theparameters ✓,�, ˜�). As a consequence, (1) decomposes intotwo parts that can be straightforwardly optimized—i.e.,

log p✓,�

(x) � L(✓;x) + L(�, ˜�;x)L(✓;x) = log p

✓

(x | sx

, cx

)

L(�, ˜�;x) = Eq

�

(z|sx

,c

x

)[log p�(sx, cx | z)]

�DKL(q�

(z | sx

, cx

) k p(z)),

where we can optimize ✓ independently from �, ˜�:

✓⇤ = argmax

✓

nX

i=1

L(✓;x(i))

�⇤, ˜�⇤= argmax

�,�

nX

i=1

L(�, ˜�;x(i)).

Latent structure VAE. Note that L(�, ˜�;x) is exactlyequal to the objective of a VAE, where q

�

(z | s, c) is theencoder and p

�

(s, c | z) is the decoder—i.e., learning thedistribution over latent structure is equivalent to learning theparameters of a VAE. The architecture of this VAE dependson the representation of s and c. In the case of for-loopstructure in images, we use a sequence-to-sequence VAE.

Generating data with structure. The term L(✓;x) cor-responds to learning a probability distribution (conditioned

on the latent structure s and c)—e.g., we can estimate thisdistribution using another VAE. As before, the architectureof this VAE depends on the representation of s and c. Ratherthan directly predicting x based on s and c, we can leveragethe program structure more directly by first executing theprogram P = (s, c) to obtain its output xstruct = eval(P ),which we call a structure rendering. In particular, xstruct isa more direct representation of the global structure repre-sented by P , so it is often more suitable to use as input to aneural network. The middle of Figure 2 shows an exampleof a structure rendering for the program on the left. Then,we can train a model p

✓

(x | s, c) = p✓

(x | xstruct).

In the case of images, we use a VAE with convolutionallayers for the encoder q

�

and transpose convolutional lay-ers for the decoder p

✓

. Furthermore, instead of estimatingthe entire distribution p

✓

(x | s, c), we also consider twonon-probabilistic approaches that directly predict x fromxstruct, which is an image completion problem. We cansolve this problem using GLCIC, a state-of-the-art imagecompletion model (IIzuka et al., 2017). We can also use Cy-cleGAN (Zhu et al., 2017), which solves the more generalproblem of mapping a training set of structured renderings{xstruct} to a training set of completed images {x}.

Image completion. In image completion, we are givena set of training pairs (xpart, x

⇤), and the goal is to learn a

model that predicts the complete image x⇤ given a partial


image xpart. Compared to generation, our likelihood is nowconditioned on xpart—i.e., p

✓,�

(x | xpart). Now, we describehow we modify each of our two models p

✓

(x | s, c) andp�

(s, c | z) to incorporate this extra information.

First, the programmatic structure is no longer fully latent,since we can observe the structure in xpart. In particular, weleverage our program synthesis algorithm to help performcompletion. We first synthesize programs P ⇤ and Ppart rep-resenting the global structure in x⇤ and xpart, respectively.Then, we train a model f

that predicts P ⇤ given Ppart—i.e.,it extrapolates Ppart to a program ˆP = f

(Ppart) represent-ing the structure of the full image. Thus, unlike generation,where we sample a program ˆP = (s, c) ⇠ p

�

(s, c | z), weuse the extrapolated program ˆP = f

(Ppart). Second, whenwe execute ˆP = (s, c) to obtain a structure rendering xstruct,we render it on top of the given partial image xpart. Finally,we sample a completion x ⇠ p

✓

(x | xstruct) as before. Ourimage completion pipeline is shown in Figure 3.

3. Synthesizing Programmatic StructureImage representation. Since the images we work withare very high dimensional, for tractability, we assume thateach image x 2 RNM⇥NM is divided into a grid containingN rows and N columns, where each grid cell has size M ⇥M pixels (where M 2 N is a hyperparameter). For example,this grid structure is apparent in Figure 3 (top right), whereN = 15, M = 17 and N = 9, M = 16 for the facade andsynthetic datasets respectively. For t, u 2 [N ] = {1, ..., N},we let x

tu

2 RM⇥M denote the sub-image at the (t, u)position in the N ⇥N grid.

Program grammar. Given this structure, we considerprograms that draw 2D repeating patterns of M ⇥M sub-images on the grid. More precisely, we consider programs

P = ((s1, c1), ..., (sk, ck)) 2 (S ⇥ C)

k

that are length k lists of pairs consisting of a sketch s 2 Sand a perceptual component c 2 C; here, k 2 N is ahyperparameter. 1 A sketch s 2 S has form

s = for (i, j) 2 {1, ..., n}⇥ {1, ..., n0} dodraw(a · i+ b, a0 · j + b0, ??)

end for

where n, a, b, n0, a0, b0 2 N are undetermined parametersthat must satisfy a · n+ b N and a0 · n0

+ b0 N , andwhere ?? is a hole to be filled by a perceptual component,which is an M ⇥M sub-image c 2 RM⇥M . 2 Then, upon

1So far, we have assumed that a program is a single pair P =(s, c), but the generalization to a list of pairs is straightforward.

2For colored images, we have I 2 RM⇥M⇥3.

executing the (i, j) iteration of the for-loop, the programrenders sub-image I at position (t, u) = (a · i+b, a0 · j+b0)in the N ⇥N grid. Figure 3 (top middle) shows an exampleof a sketch s where its hole is filled with a sub-image c,and Figure 3 (bottom left) shows the image rendered uponexecuting P = (s, c). Figure 2 shows another such example.

Program synthesis problem. Given a training image x 2RNM⇥NM , our program synthesis algorithm outputs theparameters n

h

, ah

, bh

, n0h

, a0h

, b0h

of each sketch sh

in theprogram (for h 2 [k]), along with a perceptual componentch

to fill the hole in sketch sh

. Together, these parametersdefine a program P = ((s1, c1), ..., (sk, ck)).

The goal is to synthesize a program that faithfully representsthe global structure in x. We capture this structure using aboolean tensor B(x) 2 {0, 1}N⇥N⇥N⇥N , where

B(x)t,u,t

0,u

0 =

(1 if d(x

tu

, xt

0u

0) ✏

0 otherwise,

where ✏ 2 R+ is a hyperparameter, and d(I, I 0) is a dis-tance metric between on the space of sub-images. In ourimplementation, we use a weighted sum of earthmover’sdistance between the color histograms of I and I 0, and thenumber of SIFT correspondences between I and I 0.

Additionally, we associate a boolean tensor with a givenprogram P = ((s1, c1), ..., (sk, ck)). First, for a sketchs 2 S with parameters a, b, n, a0, b0, n0, we define

cover(s) = {(a · i+ b, a0 · j + b0) | i 2 [n], j 2 [n0]},

i.e., the set of grid cells where sketch renders a sub-imageupon execution. Then, we have

B(s)t,u,t

0,u

0 =

(1 if (t, u), (t0, u0

) 2 cover(s)0 otherwise,

i.e., B(s)t,u,t

0,u

0 indicates whether the sketch s renders a sub-image at both of the grid cells (t, u) and (t0, u0

). Then,

B(P )= B(s1) _ ... _B(s

k

),

where the disjunction of boolean tensors is defined element-wise. Intuitively, B(P ) identifies the set of pairs of grid cells(t, u) and (t0, u0

) that are equal in the image rendered uponexecuting each pair (s, c) in P . 3

Finally, our program synthesis algorithm aims to solve thefollowing optimization problem:

P ⇤= argmax

P

`(P ;x) (3)

`(P ;x) = kB(x) ^B(P )k1 + �k¬B(x) ^ ¬B(P )k1,3Note that the covers of different sketches in P can overlap;

ignoring this overlap does not significantly impact our results.


Algorithm 1 Synthesizes a program P representing theglobal structure of a given image x 2 RNM⇥NM .

Input: X = {x} ✓ RNM⇥NM

ˆC {xtu

| t, u 2 [N ]}P ?for h 2 {1, ..., k} dosh

, ch

= argmax(s,c)2S⇥C

`(Ph�1 [ {(s, c)};x)

P P [ {(sh

, ch

)}end forOutput: P

where ^ and ¬ are applied elementwise, and � 2 R+ isa hyperparameter. In other words, the objective of (3) isthe number of true positives (i.e., entries where B(P )

=

B(x)= 1), and the number of false negatives (i.e., entries

where B(P )= B(x)

= 0), and computes their weightedsum. Thus, the objective of (3) measures for how well Prepresents the global structure of x. For tractability, werestrict the search space in (3) to programs of the form

P = ((s1, c1), ..., (sk, ck)) 2 (S ⇥ ˆC)

k

ˆC = {xtu

| t, u 2 [N ]}.

In other words, rather than searching over all possible sub-images c 2 RM⇥M , we only search over the sub-imagesthat actually occur in the training image x. This may leadto a slightly sub-optimal solution, for example, in caseswhere the optimal sub-image to be rendered is in fact aninterpolation between two similar but distinct sub-images inthe training image. However, we found that in practice thissimplifying assumption still produced viable results.

Program synthesis algorithm. Exactly optimizing (3) isin general an NP-complete problem. Thus, our programsynthesis algorithm uses a partially greedy heuristic. Inparticular, we initialize the program to P = ?. Then,on each iteration, we enumerate all pairs (s, c) 2 S ⇥ˆC and determine the pair (s

h

, ch

) that most increases theobjective in (3), where ˆC is the set of all sub-images x

tu

fort, u 2 [N ]. Finally, we add (s

h

, ch

) to P . We show the fullalgorithm in Algorithm 1.Theorem 3.1. If � = 0, then `( ˆP ;x) � (1� e�1

)`(P ⇤;x),

where ˆP is returned by Algorithm 1 and P ⇤ solves (3).

Proof. If � = 0, then optimizing `(P ;x) is equivalent toset cover, where the items are tuples

{(t, u, t0, u0) 2 [N ]

4 | B(x)t,u,t

0,u

0 = 1},

and the sets are (s, c) 2 S ⇥ ˆC. The theorem followsfrom (Hochbaum, 1997).

In general, (3) is not submodular, but we find that the greedyheuristic still works well in practice.

4. ExperimentsWe perform two experiments—one for generation fromscratch and one for image completion. We find substan-tial improvement in both tasks. Details on neural networkarchitectures are in Appendix A, and additional examplesfor image completion are in Appendix B.

4.1. Datasets

Synthetic dataset. We developed a synthetic datasetbased on MNIST. Each image consists of a 9 ⇥ 9 grid,where each grid cell is 16⇥ 16 pixels. Each grid cell is ei-ther filled with a colored MNIST digit or a solid color back-ground. The program structure is a 2D repeating pattern ofan MNIST digit; to add natural noise, we each iteration ofthe for-loop in a sketch s

h

renders different MNIST digits,but with the same MNIST label and color. Additionally,we chose the program structure to contain correlations char-acteristic of real-world images—e.g., correlations betweendifferent parts of the program, correlations between the pro-gram and the background, and noise in renderings of thesame component. Examples are shown in Figure 4. We givedetails of how we constructed this dataset in Appendix A.This dataset contains 10,000 training and 500 test images.

Facades dataset. Our second dataset consists of 1855images (1755 training, 100 testing) of building facades.4These images were all scaled to a size of 256 ⇥ 256 ⇥ 3

pixels, and were divided into a grid of 15 ⇥ 15 cells eachof size 17 or 18 pixels. These images contain repeatingpatterns of objects such as windows and doors.

4.2. Generation from Scratch

Experimental setup. First, we evaluate our approachSGM applied to generation from scratch. We focus onthe synthetic dataset—we found that our facades datasetwas too small to produce meaningful results. For the firststage of SGM (i.e., generating the program P = (s, c)), weuse a LSTM architecture for the encoder p

�

(s, c | z) anda feedforward architecture for the decoder q

�

(z | s, c). Asdescribed in Section 2, we use Algorithm 1 to synthesizeprograms P

x

= (sx

, cx

) representing each training imagex 2 Xtrain. Then, we train p

�

and q�

on the training set ofprograms {P

x

| x 2 X}.

For the second stage of SGM (i.e., completing the struc-ture rendering xstruct into an image x), we use a variationalencoder-decoder (VED)

p✓

(x | s, c) =Z

p✓

(x | w) · q✓

(w | xstruct)dw,

where q✓

(w | xstruct) encodes a structure rendering xstruct

4We chose a large training set since our dataset is so small.


Model ScoreSGM (CycleGAN) 85.51BL (SpatialGAN) 258.68SGM (VED) 59414.7BL (VAE) 60368.4

SGM (VED Stage 1 p�

(s, c | z)) 32.0SGM (VED Stage 2 p

✓

(x | s, c)) 59382.6

Table 1. Performance of our approach SGM versus the baseline(BL) for generation from scratch. We report Frechet inceptiondistance for GAN-based models, and negative log-likelihood forthe VAE-based models

into a latent vector w, and p✓

(x | w) decodes the latentvector to a whole image We train p

✓

and q✓

using the re-construction error kx � x⇤k. Additionally, we trained aCycle-GAN model to map structure renderings to completeimages, by giving the CycleGAN model unaligned pairs ofxstruct and x⇤ as training data. We compare our VED modelto a VAE (Kingma & Welling, 2014), and compare ourCycleGAN model to a SpatialGAN (Jetchev et al., 2017).

Results. We measure performance for SGM with the VEDand the baseline VAE using the variational lower bound onthe negative log-likelihood (NLL) (Zhao et al., 2017) on aheld-out test set. For our approach, we use the lower bound(2),5 which is the sum of the NLLs of the first and secondstages; we report these NLLs separately as well. Figure 4shows examples of generated images. For SGM and Spa-tialGAN, we use Frechet inception distance (Heusel et al.,2017). Table 1 shows these metrics of both our approachand the baseline.

Discussion. The models based on our approach quantita-tively improve over the respective baselines. The examplesof images generated using our approach with VED comple-tion appear to contain more structure than those generatedusing the baseline VAE. Similarly, the images generatedusing our approach with CycleGAN clearly capture morecomplex structure than the unbounded 2D repeating texturepatterns captured by SpatialGAN.

4.3. Image Completion

Experimental setup. Second, we evaluated our approachSGM for image completion, on both our synthetic and the fa-cades dataset. For this task, we compare using three imagecompletion models: GLCIC (IIzuka et al., 2017), Cycle-GAN (Zhu et al., 2017), and the VED architecture describedin Section 4.2. GLCIC is a state-of-the-art image completionmodel. CycleGAN is a generic image-to-image transformer.

5Technically, p✓

(x | sx

, c

x

) is lower bounded by the loss ofthe variational encoder-decoder).

Original Images

SGM (CycleGAN)

Baseline (SpatialGAN)

SGM (VED)

Baseline (VAE)

Figure 4. Examples of synthetic images generated using our ap-proach, SGM (with VED and CycleGan), and the baseline (a VAEand a SpatialGAN). Images in different rows are unrelated sincethe task is generation from scratch.

It uses unpaired training data, but we found that for ourtask, it outperforms approaches such as Pix2Pix (Isola et al.,2017) that take paired training data. For each model, wetrained two versions:

• Our approach (SGM): As described in Section 2 (forimage completion), given a partial image xpart, weuse Algorithm 1 to synthesize a program Ppart. Weextrapolate Ppart to ˆP = f

(Ppart), and execute ˆP toobtain a structure rendering xstruct. Finally, we trainthe image completion model (GLCIC, CycleGAN, orVED) to complete xstruct to the original image x⇤.

• Baseline: Given a partial image xpart, we train the im-age completion model (GLCIC, CycleGAN, or VED)to directly complete xpart to the original image x⇤.


Original Image (Synthetic) Original Image (Facades)

SGM (GLCIC, Synthetic) SGM (GLCIC, Facades)

Baseline (GLCIC, Synthetic) Baseline (GLCIC, Facades)

Figure 5. Examples of images generated using our approach (SGM) and the baseline, using GLCIC for image completion.

Model Synthetic FacadesSGM BL SGM BL

GLCIC 106.8 163.66 141.8 195.9CycleGAN 91.8 218.7 124.4 251.4VED 44570.4 52442.9 8755.4 8636.3

Table 2. Performance of our approach SGM versus the baseline(BL) for image completion. We report Frechet distance for GAN-based models, and negative log-likelihood (NLL) for the VED.

Results. As in Section 4.2, we measure performance usingFrechet inception distance for GLCIC and CycleGAN, andnegative log-likelihood (NLL) for the VED, reported on aheld-out test set. We show these results in Table 2. We showexamples of completed image using GLCIC in Figure 5. Weshow additional examples of completed images, includingthose completed using CycleGAN and VED, in Appendix B.

Discussion. Our approach SGM outperforms the baselinein every case except the VED on the facades dataset. Webelieve the last result is since both VEDs failed to learn anymeaningful structure (see Figure 7 in Appendix B).

A key reason why the baselines perform so poorly on thefacades dataset is that the dataset is very small. Nevertheless,SGM substantially outperforms the baselines even on thelarger synthetic dataset. Finally, generative models such asGLCIC are known to perform poorly away from the edges ofthe given partial image (IIzuka et al., 2017). A benefit of ourapproach is that it provides global context for models suchas GLCIC that are good at performing local completion.

5. ConclusionWe have proposed a new approach to generation that incor-porates programmatic structure into state-of-the-art deeplearning models. In our experiments, we have demonstratedthe promise of our approach to improve generation of high-dimensional data with global structure that current state-of-the-art deep generative models have difficulty capturing.

There are a number of directions for future work that couldimprove the quality of the images generated using our ap-proach. Most importantly, we have relied on a relativelysimple grammar of programs. Designing more expressiveprogram grammars that can more accurately capture globalstructure could substantially improve our results. Examplesof possible extensions include if-then-else statements andvariable grids. Furthermore, it may be useful to incorporatespatial transformations so we can capture patterns that aredistorted due to camera projection.

Correspondingly, more sophisticated synthesis algorithmsmay be needed for these domains. In particular, learning-based program synthesizers may be necessary to infermore complex global structure. Devising new learningalgorithms—e.g., based on reinforcement learning—wouldbe needed to learn these synthesizers in conjunction withthe parameters of the SGM model.

AcknowledgementsWe thank the anonymous reviewers for insightful feedback.This research was supported by NSF awards #1737858 and#1836936.


ReferencesAndreas, J., Rohrbach, M., Darrell, T., and Klein, D. Neural

module networks. In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition, pp. 39–48,2016.

Bastani, O., Pu, Y., and Solar-Lezama, A. Verifiable rein-forcement learning via policy extraction. In NIPS, 2018.

De Raedt, L., Kimmig, A., and Toivonen, H. Problog: Aprobabilistic prolog and its application in link discovery.In Proceedings of the 20th International Joint Confer-ence on Artifical Intelligence, IJCAI’07, pp. 2468–2473,San Francisco, CA, USA, 2007. Morgan Kaufmann Pub-lishers Inc. URL http://dl.acm.org/citation.cfm?id=1625275.1625673.

Deng, Z., Chen, J., Fu, Y., and Mori, G. Probabilisticneural programmed networks for scene generation. InAdvances in Neural Information Processing Systems, pp.4032–4042, 2018.

Dieleman, S., van den Oord, A., and Simonyan, K. The chal-lenge of realistic music generation: modelling raw audioat scale. In Advances in Neural Information ProcessingSystems, pp. 8000–8010, 2018.

Ellis, K., Solar-Lezama, A., and Tenenbaum, J. Unsuper-vised learning by program synthesis. In Advances in neu-ral information processing systems, pp. 973–981, 2015.

Ellis, K., Ritchie, D., Solar-Lezama, A., and Tenenbaum,J. Learning to infer graphics programs from hand-drawnimages. In Advances in Neural Information ProcessingSystems, pp. 6060–6069, 2018.

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B.,Warde-Farley, D., Ozair, S., Courville, A., and Bengio,Y. Generative adversarial nets. In Advances in neuralinformation processing systems, pp. 2672–2680, 2014.

Ha, D. and Eck, D. A neural representation of sketch draw-ings. arXiv preprint arXiv:1704.03477, 2017.

Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., andHochreiter, S. Gans trained by a two time-scale updaterule converge to a local nash equilibrium. In Advances inNeural Information Processing Systems, pp. 6626–6637,2017.

Hochbaum, D. S. Approximating covering and packingproblems: set cover, vertex cover, independent set, andrelated problems. Approximation Algorithms for NP-Hard Problem, pp. 94–143, 1997.

Hwang, I., Stuhlmuller, A., and Goodman, N. D. Induc-ing probabilistic programs by bayesian program merging.arXiv preprint arXiv:1110.5667, 2011.

IIzuka, S., Simo-Serra, E., and Ishikawa, H. Globally and lo-cally consistent image completion. In ACM Trans. Graph.,2017.

Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A. Image-to-image translation with conditional adversarial networks.In 2017 IEEE Conference on Computer Vision and Pat-tern Recognition (CVPR), pp. 5967–5976. IEEE, 2017.

Jetchev, N., Bergmann, U., and Vollgraf, R. Texture synthe-sis with spatial generative adversarial networks. 2017.

Kingma, D. P. and Welling, M. Auto-encoding variationalbayes. In ICLR, 2014.

Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B.Human-level concept learning through probabilistic pro-gram induction. Science, 350(6266):1332–1338, 2015.

Liang, P., Jordan, M. I., and Klein, D. Learning programs:A hierarchical bayesian approach. In Proceedings ofthe 27th International Conference on Machine Learning(ICML-10), pp. 639–646, 2010.

Manhaeve, R., Dumancic, S., Kimmig, A., Demeester, T.,and Raedt, L. D. Deepproblog: Neural probabilisticlogic programming. CoRR, abs/1805.10872, 2018. URLhttp://arxiv.org/abs/1805.10872.

Nori, A. V., Ozair, S., Rajamani, S. K., and Vijaykeerthy, D.Efficient synthesis of probabilistic programs. In PLDI,volume 50, pp. 208–217. ACM, 2015.

Perov, Y. N. and Wood, F. Learning probabilistic programs.In NIPS Probabilistic Programming Workshop, 2014.

Pu, Y., Miranda, Z., Solar-Lezama, A., and Kaelbling, L.Selecting representative examples for program synthesis.In International Conference on Machine Learning, pp.4158–4167, 2018.

Radford, A., Metz, L., and Chintala, S. Unsupervised rep-resentation learning with deep convolutional generativeadversarial networks. In ICLR, 2015.

Solar-Lezama, A., Tancau, L., Bodik, R., Seshia, S., andSaraswat, V. Combinatorial sketching for finite programs.In ASPLOS, volume 41, pp. 404–415. ACM, 2006.

Talton, J., Yang, L., Kumar, R., Lim, M., Goodman, N., andMech, R. Learning design patterns with bayesian gram-mar induction. In Proceedings of the 25th annual ACMsymposium on User interface software and technology,pp. 63–74. ACM, 2012.

Tylecek, R. and Sara, R. Spatial pattern templates for recog-nition of objects with regular structure. In Proc. GCPR,Saarbrucken, Germany, 2013.


Valkov, L., Chaudhari, D., Srivastava, A., Sutton, C., andChaudhuri, S. Houdini: Lifelong learning as programsynthesis. In Advances in Neural Information ProcessingSystems, pp. 8701–8712, 2018.

Verma, A., Murali, V., Singh, R., Kohli, P., and Chaudhuri,S. Programmatically interpretable reinforcement learning.In ICML, 2018.

Wang, F. and Rudin, C. Falling rule lists. In ArtificialIntelligence and Statistics, pp. 1013–1022, 2015.

Xian, W., Sangkloy, P., Agrawal, V., Raj, A., Lu, J., Fang,C., Yu, F., and Hays, J. Texturegan: Controlling deepimage synthesis with texture patches. In Proceedings ofthe IEEE Conference on Computer Vision and PatternRecognition, pp. 8456–8465, 2018.

Zhao, S., Song, J., and Ermon, S. Towards deeper un-derstanding of variational autoencoding models. arXivpreprint arXiv:1702.08658, 2017.

Zhu, J.-Y., Park, T., and Efros, A. Unpaired image-to-imagetranslation using cycle-consistent adversarial networks.In ICCV., 2017.

learning neurosymbolic generative models via program...

Documents