Checking Beliefs in Dynamic NetworksNuno P. Lopes

Microsoft ResearchNikolaj Bjørner

Microsoft ResearchPatrice Godefroid

Microsoft ResearchKarthick Jayaraman

Microsoft Azure

George VargheseMicrosoft Research

Abstract Network Optimized Datalog (NoD) is atool for checking beliefs about network reachability poli-cies in dynamic networks. A belief is a high-level invari-ant (e.g., “Internal controllers cannot be accessed fromthe Internet”) that a network operator thinks is true. Be-liefs may not hold, but checking them can uncover bugsor policy exceptions with little manual effort. Refutedbeliefs can be used as a basis for revised beliefs. Further,in real networks, machines are added and links fail; ona longer term, packet formats and even forwarding be-haviors can change, enabled by OpenFlow and P4. NoDallows the analyst to model such dynamic networks byadding new rules.

By contrast, existing network verification tools (e.g.,VeriFlow) lack the ability to model policies at a highlevel of abstraction and cannot handle dynamic net-works; even a simple packet format change requireschanges to internals. Standard verification tools (e.g.,model checkers) easily model dynamic networks butdo not scale to large header spaces. NoD has the ex-pressiveness of Datalog while scaling to large headerspaces because of a new filter-project operator and asymbolic header representation. NoD has been releasedas part of the publicly available Z3 SMT solver. We de-scribe experiments using NoD for policy template check-ing at scale on two large production data centers. Fora large Singapore data center with 820K rules, NoDchecks whether any guest VM can access any controller(the equivalent of 5K specific reachability invariants) in12 minutes. NoD checks for loops in an experimentalSWAN backbone network with new headers in a fractionof a second. The NoD checker generalizes a specializedsystem, SecGuru, we currently use in production to catchhundreds of configuration bugs over the past year.

1 IntroductionManually discovering any significant numberof rules a system must satisfy is a dispiritingadventure – Engler et al. [13].

Our goal is to catch as many latent bugs as possi-ble by static inspection of router forwarding tables andACLs without waiting for the bugs to trigger expensivelive site incidents. In our operational network, we seeroughly one customer visible operational outage of morethan one hour every quarter across our major properties;

these live site incidents are expensive to troubleshoot, re-duce revenue and customer satisfaction. As businessesdeploy services, bug finding using static verification willbecome increasingly essential in a competitive world.

We have already deployed an early version of ourchecker in our public cloud; it has regularly found bugs(§ 7). Operators find our checker to be indispensable es-pecially when rapidly building out new clusters. How-ever, we wish to go deeper, and design a more usefulverification engine that tackles two well-known [2, 15]obstacles to network verification at scale.

O1. Lack of knowledge: As Engler and others havepointed out [12, 14], a major impediment is determiningwhat specification to check. Reachability policies can bethought of intuitively as “who reaches who”. These poli-cies evolve organically and are in the minds of networkoperators [2], some of whom leave. How can one use ex-isting network verification techniques [1, 21–23, 25, 33]when one does not know the pairs of stations and headersthat are allowed to communicate?

O2. Network Churn: Existing network verificationtechniques assume the network is static and operate ona static snapshot of the forwarding state. But in our ex-perience many bugs occur in buildout when the networkis first being rolled out. Another particularly insidiousset of bugs only gets triggered when failures occur; forexample, a failure could trigger using a backup routerthat is not configured with the right Drop rules. Be-yond such short term dynamism, the constant need forcost reduction and the availability of new mechanismslike SDNs, keeps resulting in new packet formats andforwarding behaviors. For example, in recent years, wehave added VXLAN [24] and transitioned to softwareload balancers in our VM Switches [28]. The high-levelpoint is that verification tools must be capable of mod-eling such dynamism and provide insight into the effectof such changes. However, all existing tools we know ofincluding Anteater [25], VeriFlow [23], Hassel [22] andNetPlumber [21] assume fixed forwarding rules and fixedpacket headers, with little or no ability to model faults oreven header changes.

Our approach to tackling these obstacles is twofold,and is embodied in a new engine called Network Opti-mized Datalog (NoD).

A1. Specify beliefs: We allow support for specify-ing and checking higher-level abstract policy specifica-

1

tions that one can think of as operator beliefs [13]. Be-liefs by and large hold, but may fail because of bugsor exceptions to a policy we wish to discover. For ex-ample, a common belief in our network is “fabric con-trollers, BGP and management ports should not be reach-able from customer VMs or external Internet addresses”.This is a specific instance of what we call a Protection Setpolicy template: “Stations in Set A cannot reach Stationsin Set B”. For this paper, a belief is a Boolean combi-nation of reachability predicates that is easily expressedusing Datalog definitions. Rather than the operator enu-merate the crossproduct of all specific reachability predi-cates between specific customer VM prefixes and all pos-sible controllers, it takes less manual effort to state suchbeliefs. If the engine is armed with a mapping from thepredicate “customer VM”, “Internet”, etc. to a list ofprefixes, then the tool can unroll this more abstract pol-icy specification into specific invariants between pairs ofaddress prefixes. Of course, checks can lead to false pos-itives (if there are exceptions which then lead to refiningour beliefs) or false negatives (if our list is incomplete)but we can get started without waiting for perfect knowl-edge.

We have found five abstract policy templates (Table 1)cover every policy our operators have enforced and oursecurity auditors check for: protection sets, reachabil-ity sets, reachability consistency, middlebox processing,and locality. An example of a reachability consistencybelief and bug is shown in Figure 2. We will describethese in detail in the next section but Table 1 summarizesexamples. Most are present in earlier work except forlocality and reachability consistency. Despite this, ear-lier work in network verification with the exception ofNetPlumber [21] does not allow reachability invariantsto be specified at this higher level of abstraction. Notethat we make no attempt to learn these abstract policiesas in [2,13]. Instead, we glean these templates from talk-ing to operators, encode them using NoD, and determinewhether violations are failures or bugs by further discus-sion. Perhaps surprisingly, checking for a general beliefcan often be faster than checking for all implied specificreachability invariants separately because of wider scopefor optimization.

A2. Model dynamism: Our tool offers all the fea-tures of Datalog augmented with support for large headerspaces. Our Datalog engine has support for bit vectors.Thus it is easy for users to add support for MPLS or anynew packet header such as VXLAN without changinginternals. Second, we can model new forwarding behav-iors enabled by programmable router languages such asP4 [4]. Third, we can model failure at several levels. Theeasiest level is not to model the routing protocol. Forexample, our links and devices are divided into avail-ability zones that share components such as wires and

Policy Example DatalogTemplate Feature

NeededProtection Customer VMs cannot DefinitionsSets access controllers of setsReachable Customer VMs can Definitions,Sets can access VMs NegationReachability ECMP/Backup routes Negation,Consistency should have identical Non-

reachability/ determinism,same path length Bit vectors

Middleboxprocessing

Forward path connec-tions through a mid-dlebox should reverse

Negation

Locality

Packets between twostations in the samecluster should staywithin the cluster

Booleancombinationsof reachabilitypredicates

Table 1: 5 common policy templates.

power supplies that can fail together. Will an availabilityzone failure disconnect the network? This can be mod-eled by adding a predicate to each component that mod-els its availability zone and state, and changing forward-ing rules to drop if a component is unavailable.

At a second level, we can easily model failure responsewhere the backup routes are predetermined as in MPLSfast reroute: when one tunnel in a set of equal cost tun-nels fails, the traffic should be redistributed among thelive tunnels in proportion to their weights. The next levelof depth is to model the effect of route protocols likeOSPF and BGP as has been done by Fogel et al. [15]for a university network using an early version of NoD.All of these failure scenarios can be modeled as Data-log rules, with routing protocol modeling [15] requiringthat rules be run to a fixed point indicating routing proto-col convergence. Existing tools for network verificationlike VeriFlow and NetPlumber cannot model dynamismwithout changing internals. Our tool can be used to askanalogous “what if” questions for reliability or securityof network paths.

In summary, we need a verification engine that canspecify beliefs and has the ability to model dynamismsuch as new packet headers or failures. Existing ver-ification network tools scale to large networks at highspeeds but with the exception of NetPlumber do not alsoadd a policy language front-end to specify beliefs. Net-Plumber’s regular expression language for reachabilitypredicates, however, is less rich than Datalog; for ex-ample, it cannot model reachability consistency acrossECMP routes as in Figure 2. More importantly, none ofthe existing tools including NetPlumber can be easily ex-tended to model failures without modifying the internals.

2

On the other hand, the verification community has pro-duced an arsenal of tools such as model checkers, Data-log, and SAT Solvers that are extremely extensible, andcan easily model dynamism and beliefs. The catch is thatthey tend to work well only with small state spaces. Forexample, several Datalog implementations use relationalbackends that use tables as data structures. If the solu-tions are sets of headers and the headers can be 100 bitslong, the standard implementations scale horribly. Eventools such as Margrave [26] while offering the ability toask “what if” questions for firewalls do not scale to enu-merating large header spaces. Thus, our contributionsare:

1. Modeling Beliefs and Dynamism (§ 3): We showhow to encode higher-level beliefs that can catch con-crete bugs using succinct Datalog queries.

2. Network Optimized Datalog (§ 4): We modifythe Z3 Datalog implementation by adding new optimiza-tions such as symbolic representation of packets and acombined Filter-Project operator. While we will attemptto convey the high-level idea, the main point is that theseoptimizations are crucial to scale to large data centerswith say 100,000 rules. Our code is publicly released aspart of Z3 version 3.4.2 so that others can build networkverification tools on top of our engine.

3. Evaluation (§ 6): We show that the ability tomodel dynamism of Network Optimized Datalog comesat reasonable speeds by running belief checks on existingbenchmarks, synthetic benchmarks where we can varyparameters, and our own operational networks. We alsoreport briefly on the performance of other tools such asmodel checkers and SAT solvers.

4. Experience (§ 7): We describe our experiences dur-ing the last year with an existing checker called SecGuru.We also describe our experience checking for beliefs ona large data center. Finally, we allude to Batfish [15] thatchecks for bugs in the face of failures using our tool as asubstrate.

In addition, § 2 describes our Datalog model, and § 5describes our benchmarks.

2 Datalog Model2.1 Why Datalog

Five key features an ideal modeling language/tool fornetwork verification should possess are:

1. All Solutions: We want to find all packet head-ers from A that can reach B. In other words, we needall solutions for a reachability query. Most classicalverification tools such as model checkers [20] and SATsolvers [3] only provide single solutions; the naive ap-proach of adding the negation of the solution and iterat-ing is too slow.

2. Packet Rewrites: Among classical verification log-ics, Datalog does provide a native way to model routersas relations over input and output packets. Packet reach-ability is modeled as a recursive relation. Rewriting fewselected bits or copying a range is also simple to modelin this framework.

3. Large Header Spaces: Both Features 1 and 2 arechallenging when the header space is very large; for ex-ample with packet headers of 80 bytes, the header spaceis of up to 2640 bits. Without a way to compress headers,naive solutions will scale poorly.

4. Higher-Level Constructs: Encoding beliefs min-imally requires a language with Boolean operators forcombining reachability sets, and negation to express dif-ferential queries.

5. Dynamism: Modeling failure response to protocolssuch as BGP and OSPF requires the ability to model re-cursion and running solutions to a fixed point as has beendone by [15]. The language must also allow modelingsimple failure scenarios such as availability zone failuresand packet format changes.

Recent work in network verification including Veri-Flow [23] and NetPlumber [21] provide all solutions,while allowing packet rewriting and scaling to largeheader spaces (Features 1-3). However, none of thesedomain-specific languages support Feature 5 (modelingany changes to network forwarding requires a changeto tool internals) or Feature 4 (queries are expressed ata fairly low level of abstraction). While the FlowExpreachability language in NetPlumber [21] allows combi-nation of reachability sets using regular expressions, it isfairly limited (e.g., it cannot do differential queries acrosspaths) and is missing other higher-level features (e.g., re-cursion). By contrast, FlowExp queries can be encodedusing Datalog, similar to the way finite automata are en-coded in Datalog.

Thus, existing network verification languages [21–23,25] fail to provide Features 4 and 5 while existing ver-ification languages fail to provide Features 1, 2, and 3.Out of the box Datalog is the only existing verificationlanguage that provides a solution to Features 1, 4, and5. Natively, Datalog implementations struggle with Fea-tures 2 and 3. We deal with these challenges by over-hauling the underlying Datalog engine (§ 4) to create atool we call Network Optimized Datalog (NoD).

2.2 Modeling Reachability in NoDNoD is a Datalog implementation optimized for largeheader spaces. At the language level, NoD is just Dat-alog. We model reachability in NoD as follows.

Figure 1 shows a network with three routers R1, R2,and R3, and three end-point nodes A, B and D. The rout-ing tables are shown below the picture. For this simple

3

A -R1 - R2 -BJJJJJ

�

R3 - Din dst src rewrite outR1 10? 01? R2R1 1?? ??? R3R2 10? ??? BR3 ??? 1?? DR3 1?? ??? dst[1] := 0 R2

Figure 1: R1 has a QoS routing rule that routes packets from a videosource on the short route and other packets to destination 1?? along alonger path that traverses R3. R3 has an ACL that drops packets from1 ? ?. R3 also rewrites the middle bit in dst to 0. This ensures thatre-routed packets reach B regardless of the value of dst[1].

example, assume packets have only two fields dst andsrc, each a bit vector of 3 bits. When there are multiplerules in a router, the first matching rule applies. The lastrule of Figure 1 includes a packet rewrite operation. Weuse dst[1] := 0 to indicate that position dst[1] is set to 0.(Position 0 corresponds to the right-most bit.)

The goal is to compute the set of packets that can reachfrom A to B. For this example, the answer is easy tocompute by hand and is the set of 6-bit vectors

10?01? ∪ (10????\???1??)

where each packet is a 6-bit vector defined by a 3-bitvalue for dst followed by a 3-bit value for src, ? denoteseither 0 or 1, and \ denotes set difference.

For reachability, we only model a single (symbolic)packet starting at the source. The current location of apacket is modeled by a location predicate. For example,the predicate R1(dst,src) is true when a packet with des-tination dst and source src is at router R1.

Forwarding changes the location of a packet, andrewriting changes packet fields. A Datalog rule consistsof two main parts separated by the :- symbol. The part tothe left of this symbol is the head, while the part to theright is the body of the rule. A rule is read (and can beintuitively understood) as “head holds if it is known thatbody holds”. The initial state/location of a packet is afact, i.e., a rule without a body. For example, A(dst,src)states that the packet starts at location A with destinationaddress dst and source address src.

We use a shorthand for predicates that represent thematching condition in a router rule called a guard andfor packet updates. The relevant guards and updates fromFig. 1 are in equation (1). Notice that G13 includes thenegation of G12 to model the fact that the rule forwarding

G12 := dst = 10?∧ src = 01? (1)G13 := ¬G12 ∧ dst = 1??G2B := dst = 10?G3D := src = 1??G32 := ¬G3D ∧ dst = 1??

Id := src′ = src ∧ dst′ = dst

Set0 := src′ = src ∧ dst′ = dst[2] 0 dst[0]

B(dst,src) (2)R1(dst,src) :− G12∧ Id∧R2(dst′,src′)

R1(dst,src) :− G13∧ Id∧R3(dst′,src′)

R2(dst,src) :− G2B∧ Id∧B(dst′,src′)

R3(dst,src) :− G3D∧ Id∧D(dst′,src′)

R3(dst,src) :− G32∧Set0∧R2(dst′,src′)

A(dst,src) :− R1(dst,src)

? A(dst,src)

packets from R1 to R3 has lower priority than the one for-warding packets from R1 to R2. The update from the lastrule (Set0) sets dst′ to the concatenation of dst[2] 0 dst[0].

Armed with this shorthand, the network of Fig. 1 cannow be modeled as equation (2). To find all the packetsleaving A that could reach B, we pose the Datalog query?A(dst,src) at the end of all the router rules. The symbol? specifies that this is a query.

Router FIBs and ACLs can be modeled by Datalogrules similarly. similar way. A router that can forwarda packet to either R1 or R2 (load balancing) will have aseparate (non-deterministic) rule for each possible nexthop. We model bounded encapsulation using additionalfields that unused when the packet is decapsulated. Dat-alog queries can also check for cycles and forwardingloops. A loop detection query for an MPLS network isdescribed below.

3 Beliefs and Dynamism in NoDWe now describe how to encode beliefs in NoD/Datalog,which either cannot be expressed succinctly, or at all,by previous work [22, 23, 25] without changing internaldata structures or code. While the FlowExp reachabilitylanguage in NetPlumber [21] allows modification of areachability query within a single path, it cannot expressqueries across paths as Datalog can. We now show howto write Datalog queries for the belief templates alludedto in the introduction.

4

3.1 Protection setsConsider the following belief: Fabric managers are notreachable from guest virtual machines. While this canbe encoded in existing tools such as Hassel [22] and Ver-iFlow [23], since the guest VMs are set of size 5000,and the fabric manager is a set of around 12 addresses,and the naive way to express this query is to explode thequery to around 60,000 separate queries. While this canbe reduced by aggregating across routers, it is still likelyto take many queries using existing tools [22, 23]. Whilethis could be fixed by adding a way to define sets in sayHassel, this requires subtle changes to the internals. Ourpoint is that Datalog allows this to be stated succinctlyin a single query by taking advantage of the power ofdefinitions.

The compact encoding in Datalog is as follows. LetV M(dst,src) denote the fact that a packet is at one of theguest virtual machines and destined to a address dst thatbelongs to the set of fabric managers. We now query(based on the rules of the network, encoded for exampleas in Figure 2) for a violation of the belief: whether thefabric manager can be reached (encoded by the predicateFM) from a customer VM.

V M(dst,src) :− AddrOfVM(src),AddrOfFM(dst).

? FM(dst,src).

Datalog Features: definitions for sets of addresses.

3.2 Reachability setsConsider the following belief: All Fabric managers arereachable from jump boxes (internal managementdevices).

As before we check for the corresponding violation ofthe belief (a bug). Namely we can query for addressesinjected from jump boxes J, destined for fabric managerFM that nevertheless do not reach FM.

J(dst,src) :− AddrOfJ(src),AddrOfFM(dst).

? J(dst,src)∧¬FM(dst,src).

Datalog Features: Definitions, negation.

3.3 Equivalence of Load Balanced PathsConsider the following bug:

Load Balancer ACL Bug: In Figure 2, an operatormay set up two routers R2 and R3 that are load balancingtraffic from a source S to destination D. R2 has an ACLentry that specifies that packets to the SQL port shouldbe dropped but R3 does not. Assume that ECMP (equal

Drop DNS

M N D

NAT BoxManagement

Allow SQLDeny SQL

S

R2 R3

R5

R4 R1

Internet

Figure 2: Bugs encountered in a cloud setting.

cost multipath routing) [31] currently uses a hash func-tion that routes SQL traffic via R2 where it is (correctly)dropped. However, the hash function can change to routepackets via R3 and now SQL packets will (incorrectly) belet through.

In general, this bug violates a belief that reachabil-ity across load balanced paths must be identical regard-less of other variables such as hash functions. We cancheck whether this belief is true by encoding a differen-tial query encoded in Datalog. Is it possible that somepacket reaches a destination under one hash function butnot another? Such a query is impossible to answer usingcurrent network verification tools [21–23].

Datalog has the power needed for expressing reacha-bility over different hashing schemes. We encode a hash-ing scheme as a bit vector h that determines what hashingchoices (e.g., should R1 forward packets to D via R2 orR3 in Figure 2) are made at every router. We assume wehave defined a built-in predicate Select that selectivelyenables a rule. We can then augment load balancingrules by adding Select as an extra guard in addition tothe guards modeling the match predicate from the FIB.For example, if there are rules for routing from R1 to R2and R3, guarded by G12, G13, then the modified rulestake the form:

R2(dst,h) :− G12∧R1(dst,h)∧Select(h,dst).

R3(dst,h) :− G13∧R1(dst,h)∧Select(h,dst).

To check for inconsistent hashing, we pose a query thatasks if there exists an intermediate node A at which pack-ets to destination dst arrive using one hash assignment h1but are dropped using a different hash assignment h2:

? A(dst,h1)∧¬A(dst,h2). (3)

By adding an ordering across routers, the size of h can

5

be encoded to grow linearly, not exponentially, with thepath length.

Datalog Features: Negation, bit vectors, non-determinism, Boolean combinations of reachabilitypredicates.

3.4 LocalityConsider the following bug:

Cluster reachability: In our Hong Kong data center,we found a locality violation. For example, it would beodd if packets from S to M in Figure 2 flowed throughR2.

Putting aside the abstruse details of the wiring issuesduring buildout that caused this bug, the bug violates abelief that routing preserves traffic locality. For exam-ple, traffic within one rack must not leave the top-of-rackswitch. Datalog definitions let us formulate such queriescompactly.

Consider packets that flow between S and M in Fig-ure 2. Observe that it would be odd if these packetsflowed through R2, or R3, or R5 for that matter. The abil-ity to define entire sub-scopes in Datalog comes in handyin this example. For example, we can define a predicateDSP (for Data center SPine) to summarize packets arriv-ing at these routers:

DSP(dst) :− R2(dst).

DSP(dst) :− R3(dst).

DSP(dst) :− R5(dst).

Conversely, local addresses like S and M that can bereached via R1 can be summarized using another pred-icate LR1 (Local R1 addresses), that we assume is givenas an explicit set of IP address ranges.

LR1(dst) :− dst = 125.55.10.0/24.

Assume a packet originates at S and sends to such a lo-cal address; we ask if DSP is reached indicating that thepacket has (incorrectly) reached the spine.

S(dst) :− LR1(dst).

? DSP(dst). (4)

Note the power of Datalog definitions. The query canalso be abstracted further to check whether traffic be-tween N and D (that sit on different racks and could bein a different cluster) have the same property in a singlequery, without writing separate queries for each cluster.For example, we could add rules, such as:

LR4(dst) :− dst = 125.75.10.0/24.D(dst) :− LR4(dst).

N(dst) :− LR4(dst).

This query can be abstracted further by defining lo-cality sets and querying whether any two stations in thesame locality set take a route outside their locality set.

Datalog Features: Scoping via predicates.

3.5 Dynamic Packet Headers

Going beyond encoding beliefs, we describe an exampleof dynamism. Network verification tools such as Has-sel [22] and NetPlumber [21] support IP formats but donot yet support MPLS [29].

However, in Datalog one does not require a priori defi-nitions of all needed protocols headers before starting ananalysis. One can easily define new headers post factoas part of a query. More importantly, one can also de-fine new forwarding behaviors as part of the query. Thisallows modeling flexible routers whose forwarding be-havior can be ”metamorphosed” at run-time [4, 5]

To illustrate this power, assume that the Datalog en-gine has no support for MPLS or the forwarding behav-ior of label stacking. A bounded stack can be encodedusing indexed predicates. For example, if R1 is a router,then R13 encodes a forwarding state with a stack of 3MPLS labels and R10 encodes a forwarding state with-out any labels. Using one predicate per control state wecan encode a forwarding rule from R5 to R2 that pushesthe label 2016 on the stack when the guard G holds as:

R21(dst,src,2016) :− G,R50(dst,src).

R22(dst,src, l1,2016) :− G,R51(dst,src, l1).

R23(dst,src, l1, l2,2016) :− G,R52(dst,src, l1, l2).

Ovfl(dst,src, l1, l2, l3) :− G,R53(dst,src, l1, l2, l3).

We assume that l1, l2, l3 are eight bit vectors that modelMPLS labels. The first three rules model both MPLSlabel stacking procedure and format. Predicates, suchas R13, model the stack size. The last rule checks for amisconfiguration that causes label stack overflow.

SWAN [19] uses an MPLS network updated dynami-cally by an SDN controller. To check for the belief thatthe SDN controller does not create any loops, we usestandard methods [35]. We use a field to encode a partialhistory of a previously visited router. For every routingrule, we create two copies. The first copy of the rule setsthe history variable h to the name of the router. The othercopy of the rule just forwards the history variable. Thereis a loop (from R5 to R5) if R5 is visited again where thehistory variable holds R5. We omit details.

Further, router states [19] may be in flux during up-dates. Nevertheless, a reasonable belief is even duringupdates ‘rules don’t overlap”, to avoid forwarding con-flicts. Similarly, another reasonable belief is that any rule

6

that sends a packets out of the MPLS network should popthe last element from the MPLS label stack.

3.6 Middleboxes and Backup Routers

While ensuring that all traffic between a source and des-tination goes through a middlebox M is a well-studiedinvariant in past work [21], we found a more subtle bugacross paths:

Incorrect Middlebox traversal: Once again, refer toFigure 2. A management box M in a newer data cen-ter in Brazil attempted to start a TCP connection to alocal host D. Newer data centers use a private addressspace and internal hosts must go through a Network Ad-dress Translator (NAT) before going to external publicservices. The box M sent the TCP connection request tothe private address of D, but D sends the packet to M viathe NAT which translates D’s source address. TCP at Mreset the connection because the SYN-ACK arrived withan unexpected source.

This bug violates a belief that packets should gothrough the same set of middleboxes in the forward andreverse path. This is a second example of an invariantthat relates reachability across two different paths, in thiscase the forward and reverse paths, as was reachabilityacross load balanced paths. This is easily encoded inDatalog by adding fictitious bit to packets that is set whenthe packet passes through a middlebox. We omit detailsto save space.

Consider next the following bug:Backup Non-equivalence: Two data centers D1 and

in the same region are directly connected through a bor-der network by a pair of backup routers at the border ofD1 and D2. The border routers are also connected to thecore network. The intent is that if a single failure oc-curs D1 and D2 remain directly connected. However, wefound after a single failure, because of an incorrect BGPconfiguration, the route from D1 to D2 went through alonger path through the core. While this is an inefficiencybug, if it is not fixed and then if the route to the core sub-sequently fails, then D1 and D2 lose connectivity evenwhen there is a physical path.

This bug suggests the belief that all paths between asource and destination pair passing through any one ofa set of backup routers should have the same number ofhops. We omit the encoding except to note that we en-code path lengths in Datalog as a small set of control bitsin a packet, and query whether a destination is reachedfrom the same source across one of the set of backuprouters, but using two different path lengths. Once againthis query appears impossible to state with existing net-work verification tools except NetPlumber [21]. Net-Plumber, however, cannot handle the range of queriesNoD can, especially allowing dynamic networks.

4 Network Optimized DatalogWhile Datalog can express higher-level beliefs andmodel dynamism (Features 4 and 5 in Section 2.1) andcomputes all solutions (Feature 1), naive Datalog imple-mentations struggle with Features 2 and 3 (scalably ex-pressing large header spaces and packet rewrites). Whilewe describe our experience with modifying µZ (the Dat-alog framework in Z3), there are two general lessons thatmay apply to other Datalog tools: the need for a newtable data structure to compactly encode large headerspaces, and a new Select-Project operator. We will tryto convey the high-level ideas for a networking reader.

4.1 Compact Data Structures

One can pose reachability queries as in Figure 2 to µZ(our Datalog framework) to compute the set of packetsthat flow from A to B. Think of the Datalog networkmodel as expressing relations between input and outputpackets at each router: the router relation models the for-warding behavior of the router including all forwardingrules and ACLs. The router is modeled not as a func-tion but as a relation, to allow multicast and load balanc-ing, where several output packets can be produced for thesame input packet. Whenever the set of input packets ata router change, the corresponding set of output packetsare recomputed using the router relation. Thus for a net-work, eventually sets of packets will flow from the inputsof the network to the endpoints of the network.

The main abstract data structure to encode a relationin Datalog is a table. For example, a table is used to storethe set of input packets at a router, and a table is usedto store output packets. To update the relationship be-tween input and output tables at a router, under the cov-ers, µZ executes Datalog queries by converting them intorelational algebra, as described elsewhere [9]. Network-ing readers can think of µZ as providing a standard suiteof database operators such as select, project and join tomanipulate tables representing sets of packet headers inorder to compute reachability sets. Figure 3 – which rep-resents a single router that drops HTTP packets using anACL that drops Port 80 packets – makes this clearer.

In our toy example, Figure 3, assume that the setof packets that reach this router have source addresseswhose first bit is 1 because of some earlier ACLs. Thusthe set of packet headers that leave the router are thosethat have first bit 1 and whose TCP destination port isnot 80. As we have seen, the natural way for Datalog torepresent a set of packet headers is as a table. Represent-ing all source addresses that start with a 1 would require2127 rows if 128-bit packet headers are represented by ar-rays. None of the existing table data structures (encapsu-lated in what are called backends) in µZ performed wellfor this reason. Hence, we implemented two new table

7

Drop HTTP Packets

***

Packets with

Source starting

with 1?

JOIN SELECT (hard to compress) PROJECT

SELECT-PROJECT

100 X Performance Lift

1** 1** 1** 1**\10* 1**\10*

All 1** packets non-HTTP

packets

project on

output

Figure 3: Manipulating tables of packet headers with acombined select-project operator.

backends.The first backend uses BDDs (Binary Decision Dia-

grams [7]) for Datalog tables. BDDs are a classic datastructure to compactly represent a Boolean function, andwidely used in hardware verification to represent cir-cuits [8]. A classic program analysis paper also aug-ments Datalog with BDDs [32] for program analysis, soour use of BDDs for Datalog tables is natural.

The second backend is based on ternary bit vectors,inspired by Header Space Analysis (HSA) [22, 23], butplaced in a much more general setting by adding a newdata structure to Datalog. This data structure we addedwas what we call difference of cubes or DoC. DoC rep-resents sets of packets as a difference of ternary strings.For example, 1 ? ? \ 10? succinctly represents all pack-ets that start with 1 other than packets that start with 10.Clearly, the output of Figure 3 can be compactly repre-sented as the set difference of all 1 ? ? packets and allpackets whose destination port is 80.

More precisely, for ternary bit vectors vi and v j, a dif-ference of cubes represents a set

⋃i

(vi \

⋃j

v j

)The difference of cubes representation is particularly

efficient at representing router rules that have dependen-cies. For example, the second rule in Figure 1 takes ef-fect only if the first rule does not match. More precisely,difference of cubes is particularly efficient at represent-ing formulas of the form ϕ ∧¬ϕ1 ∧ ·· · ∧ ¬ϕn, with ϕ

and ϕi of the form∧

i φi and φi having no Boolean op-erators. This form is precisely what we obtain in thetransfer functions of routing rules, with ϕ being the routematching formula, and the ¬ϕi being the negation of thematching formula of the dependencies of the rule. Again,in networking terminology dependencies of a forwardingrule or ACL are all higher priority matching rules.

Code: The Datalog backends added to Z3 were imple-mented in C++. The BDD backend takes 1,300 LoC, and

the difference of cubes backend takes almost 2,000 LoC.

4.2 Combining Select and Project

We needed to go beyond table compression in order tospeedup Datalog’s computation of reachability sets asfollows. Returning to Figure 3, µZ computes the set ofoutput packets by finding a relation between input pack-ets and corresponding output packets. The relation iscomputed in two steps: first, µZ joins the set of inputpackets I to the set of all possible output packets A tocreate a relation (I,A). Next, it selects the output packets(rows) that meet the matching and rewrite conditions tocreate a pruned relation (I,O). Finally, it projects awayinput packets and produces the set of output packets O.

Thus, in Figure 3, the output of the first join is the setof all possible input packets with source addresses thatstart with 1, together with all possible output packets.While this sounds like a very indirect and inefficient pathto the goal, this is the natural procedure in µZ. Joinsare the only way to create a new relation, and to availof the powerful set of operators that work on relations.While the join with all possible output packets A appearsexpensive, A is compactly represented as a single ternarystring/cube and so its cost is small.

Next, after the select, the relation is “trimmed” to beall possible input packets with source bit equal to 1, to-gether with all output packets with source bit equal to 1and destination port not equal to 80. Finally, in the laststep, the project step removes all columns correspondingto the input packets, resulting in the correct set of outputpackets (Figure 3) as desired. Observe that the output ofthe join is easily compressible (a single ternary string)and the output of the final project is also compressible(difference of two ternary strings).

The elephant in the room is the output of the selectwhich is extremely inefficient to represent as a BDD oras a difference of ternary strings. But the output of theselect is merely a way station on the path to the output; sowe do not need to explicitly materialize this intermediateresult. Thus, we define a new combined select-projectoperator whose inputs and outputs are both compressible.This is the key insight, but making it work in the presenceof packet rewriting requires more intricacy.

4.3 Handling Packet Rewriting

The example in Figure 3 does not include any packetrewrites. In Figure 1, however, rule R3 rewrites the firstdestination address bit to a 0. Intuitively, all bits exceptthe first are to be copied from the input packet to the out-put. While this can be done by a brute force enumerationof all allowed bit vectors for this 6-bit toy example, such

8

an approach does not scale to 128 bit headers. We need,instead, an efficient way to represent copying constraints.

Back to the toy example, consider an input packet 1????? at router R3 that is forwarded to router R2. Recallthat the first 3 bits in this toy example are the destinationaddress, the next 3 are the source address. We first jointhe table representing input packets with a full table (allpossible output packets), obtaining a table with the row1???????????, where the first six bits correspond tothe input packet at R3, and the remaining six bits belongto the output destined to R2.

Then we apply the guard and the rewrite formulas andthe negation of all of the rule’s dependencies using a gen-eralized select). Since we know that the 5th rule in Fig-ure 1 can only apply if 4th rule does not match, we knowthat in addition to the first destination address bit being1, the “negation of the dependencies” requires that bit 2of the source address should also be 0.

One might be tempted to believe that 1?????10?0?? compactly represents the input-output relation as “Allinput packets whose second destination address bit is 1”(the first six bits) together with “All output packets forwhich bit 2 of destination address bit is 1, bit 1 of thedestination address has been set to 0, and for which bit2 of the source address bit is 0”. But this incorrectlyrepresents the copying relation! For example, it allowsinput packets where bit 1 of the source address of theinput packet is a 0, but bit 1 of the source address of theoutput packet is a 1. Fortunately, we can rule out theseexceptions fairly compactly using set differences whichare allowed in difference of cubes notation. We obtainthe following expression (in difference of cubes notation;the expression in union of cubes notation would be evenlarger):

1?????10?0?? \(??0?????1???∪??1?????0???∪????0?????1?∪????1?????0?∪?????0?????1∪?????1?????0)

While this looks complicated, the key idea is effi-ciently representing copying using a difference of ternarystrings. The unions in the difference are ruling out caseswhere the “don’t care” ? bits are not copied correctly.The first term in the difference states that we cannot havebit 0 of destination address bit be a 0 in the input packetand bit 0 of destination address bit in the output packetbe a 1; the next term disallows the bits being 1 and 0 re-spectively. And so on for all the bit positions in dst andsrc whose bits are being copied.

After the select operation, we perform a projection toremove the columns corresponding to the input packet(the first 6 bits) and therefore obtain a table with onlythe output packets. Again, in difference of cubes repre-

sentation, we obtain 10?0??. The final result is signifi-cantly smaller than the intermediate result, and this effectis much more pronounced when we use 128 bit headers!

Generalizing: To make select-project efficient, weneed to compute the projection implicitly without explic-itly materializing intermediate results. We did this usinga standard union-find data structure to represent equiva-lence classes (copying) between columns. When estab-lishing the equality of two columns, if both columns (saybit 3 of the Destination address in both input and out-put packets) contain “don’t care” values and one of them(bit 3 in the input packet) will be projected out, we ag-gregate the two columns in the same equivalence class.While this suffices for networking, we added two morerules to generalize this construction soundly to other do-mains besides networking. In verification terminology,this operation corresponds to computing the strongestpost-condition of the transition relation. However, ittakes some delicacy to implement such an operator in ageneral verification engine such as Z3 so it can be usedin other domains besides network verification.

Code: Besides select-project, we made several addi-tional improvements to the Datalog solver itself, whichwere released with Z3 4.3.2, reducing Z3’s memory us-age in our benchmarks by up to 40% .

5 BenchmarksWe use four benchmarks:

Stanford: This is a publicly available [30] snapshotof the routing tables of the Stanford backbone, and a setof network reachability and loop detection queries. Thecore has 16 routers. The total number of rules acrossall routers is 12,978, and includes extensive NAT andVLAN support.

Generic Cloud Provider: We use a parameterizablemodel of a cloud provider network with multiple datacenters, as used by say Azure or Bing. We use a fat treeas the backbone topology within a data center and sin-gle top-of-rack routers as the leaves. Data centers areinterconnected by an all-to-all one-hop mesh network.Parameters include replication factor, router ports, datacenters, machines per data center, VMs per machine, andnumber of services. These parameters can be set to in-stantiate small, medium, or large clouds. This bench-mark is publicly available [27].

Production Cloud: Our second source comes fromtwo newer live data centers located in Hong Kong and inSingapore whose topology is shown in Figure 4. Theyconsist of a hundred routers each ranging over borderleaves, data center and cluster spines, and finally top-of-rack switches. In the Hong Kong data center, eachrouter has roughly 2000 ECMP forwarding rules addingup to a combined 200K rules. In the Singapore data cen-ter there are about 820K combined forwarding rules. The

9

text files for these rules take from 25MB to 120MB forthe respective data-centers.

This infrastructure services three clusters and thou-sands of machines. We extracted routing tables from theArista and Cisco devices using a show ip route com-mand. This produced a set of routing tables includingthe ECMP routing options in Datalog format. To handlelongest prefix match semantics, the translation into Dat-alog uses a trie to cluster ranges with common prefixes,avoiding redundancy in the Datalog rules. The range ofprivate and public IP addresses assigned to each clusterwas extracted from a separate management device.

Experimental Backbone: We check an SDN back-bone based on the SWAN design [19]. To maximizebandwidth utilization, the SDN controller periodicallyrecomputes routing rules that encapsulate packets withan MPLS labels stack encoding tunnels. Our tool takesthe output from the controller: a set of routing tables,a network topology and configurations that map IP ad-dresses to end-points. We check selected beliefs, such asloop freedom and absence of stack overflow.

Experimental Toolkit: In addition to the publiclyavailable Hassel C code [16], for comparison we usedtwo classic model checking algorithms (BMC and PDR),a Datalog framework µZ [18], and a state-of-the-artSAT/SMT solver, all implemented in the Z3 [11] en-gine. Our engines are publicly available [27] so otherresearchers can extend our results.

6 EvaluationWe first describe the results for checking beliefs ona production cloud IP network, followed by examplesof checking for loops in an experimental MPLS back-bone. Finally, for completeness, we compare the perfor-mance of our tool to the performance of existing toolsin network verification and standard verification (modelcheckers, SAT solvers).

6.1 Protection Sets in a Production CloudWe checked whether two policies based on the Protec-tion sets template (see Table 1) hold in the Singapore datacenter. The two queries were to verify that neither Inter-net addresses or customer VMs can access the protectedfabric controllers for security reasons.

Both experiments took around 12 minutes. While thismay seem slow, the sets of addresses are very large. Forthe power of exploring a very general belief, the per-formance seems acceptable and easily incorporated in achecker that runs every hour.

Both queries failed, thus the beliefs of the network de-signers and operators were not being realized in practice.Closer inspection showed that these were not bugs in thenetwork, but actually incorrect beliefs. There are two

ranges of IPs of fabric controllers that are supposed to bereachable (and they are), and certain ICMP packets arealso allowed to flow to fabric controllers. Although nobugs were found in the network, these queries allowednetwork operators to refine their beliefs about the net-work and its correct operation.

6.2 Reachable Sets on a Production Cloud

As in the previous experiment, we also checked whethertwo policies from the reachable sets template hold in thesame Singapore data center. The first query checked ifall of “utility boxes” can reach all “fabric controllers”.The second query is similar, but checks whether “serviceboxes” can reach “fabric controllers”.

The first query took around 4 minutes to execute,while the second took 6 minutes. Both queries passedsuccessfully, guaranteeing that the beliefs of the networkdesigners hold in practice.

6.3 Locality on a Production Cloud

Figure 4 shows our public cloud topology which hasmore levels of hierarchy than the simple example of Fig-ure 2. These levels motivate more general queries thanthe simple locality query (4) used in [§3.4].

Figure 4 also shows the boundaries of traffic local-ity in our public cloud. Based on these boundaries, weformulate the following queries to check for traffic lo-cality. First, C2C requires that traffic within a clusternot reach beyond the designated cluster spines. Next,B2DSP, B2DSP, B2CSP, and B2CSP require that traffictargeting public addresses in a cluster must reach onlydesignated data center spines, not reach other data cen-ter spines belonging to a different DC, must reach onlydesignated cluster spines, and not reach cluster spines be-longing to other clusters, respectively.

Table 2 shows the time spent for these five differentkinds of queries over three clusters. They check correct-ness of routing configurations in all network devices.

For the Hong Kong cluster we identified some vio-lated locality queries due to bugs during buildout. Anexample output for a C2C query was 100.79.126.0/23suggesting that an address range got routed outside thecluster it was said to be part of. On the other hand, ourtool produced public address ranges from a B2CSP querywhich were supposed to reach the cluster spine but didnot: 192.114.2.62/32 ∪ . . .∪ 192.114.3.48/29 .

6.4 An Experimental MPLS backbone

We checked loop-freedom, absence of black holes, andrule disjointness on configurations for an experimental

10

Data Center Router

Border Leaves

Data Center Spines

Cluster Spines

Host Leaves

Virtual Machines

Inter Cluster

Traffic

Inter Data Center

Traffic

Figure 4: Production Cloud Layout

Query Cluster 1 Cluster 2 Cluster 3C2C 17 (3) 18 (3) 15 (3)B2DSP 15 (3) 16 (3) 14 (3)B2DSP 7 (1) 7 (1) 7 (1)B2CSP 15 (3) 16 (3) 14 (3)B2CSP 15 (3) 16 (3) 14 (3)

Table 2: Query times are in seconds, times in parenthe-ses are for a snapshot where only one of the availableECMP options is part of the model (20% the size of thefull benchmarks).

backbone based on the SWAN design [19]. Note the ver-satility of Datalog. Both the packet formats and forward-ing behaviors are completely different from those in theproduction cloud queries we used earlier. We report ona selected experiment with a configuration comprising of80 routers, for a total of 448 forwarding rules. To modellabel stacks, this resulted in 3400 Datalog rules over aheader-space of 153 bits (64 for IP and remaining forthe label stack). The loop check is very fast and takes afraction of a second for the configurations we checked.Checking for black holes takes under 5 seconds, iden-tifying 56 flows out of 448 as black holes. The sameconfiguration had 96 pairs of overlapping rules. All wereenumerated in less than 1 second. This experience bodeswell for scaling this validation throughout the develop-ment lifecycle of the backbone and to larger backbonesizes with more checks.

6.5 Comparison with existing Tools

We ran the benchmarks with multiple tools to benchmarkour speeds against existing work. Besides the differ-ence of cubes backend described in § 4, we also used aBDD-based Datalog backend, a bounded model checker(BMC) [10], an SMT solver that uses an unrolled rep-resentation of the network as in [35], and a state-of-the-art solver based on the IC3 [6, 17] algorithm. The

benchmarks were run on a machine with an Intel XeonE5620 (2.4 GHz) CPU. We also used an SMT algorithmthat was modified to return all solutions efficiently. Themodel checkers, however, return only one solution so thespeed comparison is not as meaningful.

Table 3 is a small sampling of extensive test resultsthat can be found in [27]. It shows time (in seconds) torun multiple tools on a representative subset of the Stan-ford benchmarks, including reachable and unreachablequeries, and a loop detection query. The tests were givena timeout of 5 minutes and a memory limit of 8 GBs.

Note that the model checkers only compute satisfiabil-ity answers (i.e., “is a node reachable or not?”), whileDatalog computes all reachable packets at the destina-tion. For the SMT technique, we provide results for bothtype of queries. All the SMT experiments were run withthe minimum TTL (i.e., an unrolling) for each test; forexample, the TTL for Stanford was 3 for reachability and4 for loop detection. Higher TTLs significantly increasethe running time. We do not provide the running timefor reachability sets with SMT for the cloud benchmarks,since our prototype AllSAT algorithm does not supportnon-determinism. We were unable to run the Hassel Ctool [16] on the cloud benchmarks, since Hassel C has afew hardwired assumptions, such as router port numbersfollowing a specific naming policy.

The first takeaway is that Z3’s NoD implementationof Datalog is faster at computing reachability sets (allsolutions) than its model checkers or SAT solvers are atcomputing a single solution. This is not surprising asthe model checkers make multiple copies of each routertransfer function and stitch them together which is quitewasteful. The performance of model checkers also seemsto degrade exponentially with path length (see row 3 ver-sus row 2 where the model checkers run out of mem-ory). Similarly, unrolling seems to exact a price for SMTsolvers. Even our efficient AllSAT generalization algo-rithm is around 200× slower than Datalog (row 7). Dat-alog with difference of cubes is the most competitive im-plementation we have tested. Datalog with a BDD back-end shows good performance as well.

Clearly, the Hassel C tool takes under a secondfor Stanford, faster than Datalog. Tools like Net-Plumber [21] and VeriFlow [23] can be even faster forincremental analysis. Further, Yang and Lam [34] haveused predicate abstraction to speed up reachability test-ing even further. However, none of these tools have theability to model higher-level beliefs and model dynamicnetworks as NoD can. We believe the speed penaltyis worth the greatly increased expressiveness. Further,NoD speeds are still acceptable for an offline networkchecker, which is what the vast majority of customersneed in their networks today.

11

Test Model Checkers SMT Datalog Hassel CBMC PDR Reach. All sols. BDDs DoCSmall Cloud 0.3 0.3 0.1 – 0.2 0.2 –Medium Cloud T/O 10.0 0.2 – 1.8 1.7 –Medium Cloud Long M/O M/O 4.8 – 7.4 7.2 –Cloud More Services 7.2 8.5 12.5 – 5.3 4.8 –Large Cloud T/O M/O 2.8 – 16.1 15.7 –Large Cloud Unreach. T/O M/O 1.1 n/a 16.1 15.7 –Stanford 56.2 13.7 11.5 1,121 6.6 5.9 0.9Stanford Unreach. T/O 12.2 0.1 n/a 2.6 2.1 0.1Stanford Loop 20.4 11.7 11.2 290.2 6.1 3.9 0.2

Table 3: Time (in seconds) taken by multiple tools to solve network benchmarks. Model checkers only check forsatisfiability, while Datalog produces reachability sets. T/O and M/O are used for time- and memory-out.

7 ExperienceSecGuru: The SecGuru tool has been actively used inour production cloud in the past years for continuousmonitoring and validation, and for maintaining legacyedge ACLs. In continuous validation, SecGuru checkspolicies over ACLs on every router update as well asonce a day. This is more than 40,000 checks per month,where each check takes 150-600 ms. It uses a databaseof predefined common beliefs. For example, there is apolicy that says that SSH ports on fabric devices shouldnot be open to guest virtual machines. While these poli-cies themselves rarely change, IP addresses do changefrequently, which makes using SecGuru as a regressiontest useful. SecGuru had a measurable positive impactin prohibiting policy misconfigurations during build-out,raising in average of one alert per day; each identifies 3-5 buggy address ranges in the /20 range, e.g., 16K faultyaddresses. We also used SecGuru to reduce our legacycorporate ACL from roughly 3000 rules to 1000 withoutany derived outages or business impact.

Belief Refinement: As described in § 6.1, our opera-tor’s belief in a protection set policy (customer VMs can-not reach fabric controllers) was subtly incorrect. Thecorrect belief required new reachability exceptions. Wecurrently code this at the level of Datalog in lieu of a GUIinterface to the 5 policy templates from Table 1.

Batfish: Fogel et al. [15] developed a tool called Bat-fish that models routing protocols like BGP and OSPFin the face of link failures. For example, Batfish can beused to find whether two stations remain reachable acrossany set of single link failures. Batfish models the effectsof OSI and BGP as a set of Datalog rules; Datalog re-cursion is used to model routing convergence. Batfishspecifically uses NoD for reachability analysis becauseit was more expressive than other tools such as Hassel Cor VeriFlow. The final stage of the Batfish pipeline [15]also uses Z3 to find concrete packet headers. This con-firms our intuition that supplying NOD in a general toolsetting allows unexpected uses, in this case, of the Z3

constraint solver. Batfish has found several consistencybugs in a university network.

8 Conclusion

Network Optimized Datalog (NoD) is more expressivethan existing verification tools in its ability to encode be-liefs to uncover true specifications, and to model networkdynamism without modifying internals. NoD is muchfaster than existing (but equally expressive) verificationtools such as model checkers and SMT solvers. Key tothe efficiency are ternary encoding of tables and a Select-Project operator.

By contrast, current verification network verificationtools operate at a low level of abstraction. Propertiessuch as cluster scoping (Figure 4) expressible in a singleDatalog query would require iterating across all source-destination pairs. Further, existing work cannot easilymodel new packet formats, new forwarding behaviors,or failures [15]. We were pleased to find we could modelMPLS and label stacking succinctly. It took a few hoursto write a query to find loops in SWAN.

As in Engler et al.’s work [13], our work shows howfragile the understanding of the true network specifica-tion is. Even when working with an experienced networkoperator, we found that simple beliefs (e.g., no Internetaddresses can reach internal controllers) had subtle ex-ceptions. Our belief templates in Table 1 abstract a vastmajority of specific checks in our network and probablyother networks. A GUI interface for belief templates willgreatly assist operators.

If network verification is to mature into a network-ing CAD industry, its tools must evolve from a collec-tion of ad hoc software into principled and extensibletechniques, built upon common foundations that are con-stantly being improved. We suggest NoD as a candidatefor such a foundation.

12

References[1] E. Al-Shaer and S. Al-Haj. FlowChecker: configu-

ration analysis and verification of federated Open-Flow infrastructures. In SafeConfig, 2010.

[2] T. Benson, A. Akella, and D. A. Maltz. Miningpolicies from enterprise network configuration. InIMC, 2009.

[3] A. Biere, A. Biere, M. Heule, H. van Maaren, andT. Walsh. Handbook of Satisfiability. IOS Press,2009.

[4] P. Bosshart, D. Daly, G. Gibb, M. Izzard, N. McK-eown, J. Rexford, C. Schlesinger, D. Talayco,A. Vahdat, G. Varghese, and D. Walker. P4: Pro-gramming protocol-independent packet processors.SIGCOMM Comput. Commun. Rev., 44(3):87–95,July 2014.

[5] P. Bosshart, G. Gibb, H.-S. Kim, G. Vargh-ese, N. McKeown, M. Izzard, F. Mujica, andM. Horowitz. Forwarding metamorphosis: Fastprogrammable match-action processing in hard-ware for SDN. In SIGCOMM, 2013.

[6] A. R. Bradley. SAT-based model checking withoutunrolling. In VMCAI, 2011.

[7] R. E. Bryant. Graph-based algorithms for booleanfunction manipulation. IEEE Trans. Comput.,35(8):677–691, Aug. 1986.

[8] J. R. Burch, E. M. Clarke, K. L. McMillan, D. L.Dill, and L. J. Hwang. Symbolic model checking:1020 states and beyond. In LICS, 1990.

[9] S. Ceri, G. Gottlob, and L. Tanca. What you alwayswanted to know about Datalog (and never daredto ask). IEEE Trans. on Knowl. and Data Eng.,1(1):146–166, Mar. 1989.

[10] E. Clarke, A. Biere, R. Raimi, and Y. Zhu. Boundedmodel checking using satisfiability solving. Form.Methods Syst. Des., 19(1):7–34, 2001.

[11] L. de Moura and N. Bjørner. Z3: an efficient SMTsolver. In TACAS, 2008.

[12] D. Engler, B. Chelf, A. Chou, and S. Hallem.Checking system rules using system-specific,programmer-written compiler extensions. In OSDI,2000.

[13] D. R. Engler, D. Y. Chen, and A. Chou. Bugs asdeviant behavior: A general approach to inferringerrors in systems code. In SOSP, 2001.

[14] C. Flanagan, K. R. M. Leino, M. Lillibridge,G. Nelson, J. B. Saxe, and R. Stata. Extended staticchecking for java. In PLDI, 2002.

[15] A. Fogel, S. Fung, L. Pedrosa, M. Walraed-Sullivan, R. Govindan, R. Mahajan, and T. Mill-stein. A general approach to static analysis of net-work configurations. Submitted, Sept. 2014.

[16] Hassel C. http://goo.gl/esnZrA.

[17] K. Hoder and N. Bjørner. Generalized property di-rected reachability. In SAT, 2012.

[18] K. Hoder, N. Bjørner, and L. De Moura. µZ: anefficient engine for fixed points with constraints. InCAV, 2011.

[19] C.-Y. Hong, S. Kandula, R. Mahajan, M. Zhang,V. Gill, M. Nanduri, and R. Wattenhofer. Achiev-ing high utilization with software-driven WAN. InSIGCOMM, 2013.

[20] R. Jhala and R. Majumdar. Software model check-ing. ACM Comput. Surv., 41(4):21:1–21:54, Oct.2009.

[21] P. Kazemian, M. Chang, H. Zeng, G. Varghese,N. McKeown, and S. Whyte. Real time networkpolicy checking using header space analysis. InNSDI, 2013.

[22] P. Kazemian, G. Varghese, and N. McKeown.Header space analysis: static checking for net-works. In NSDI, 2012.

[23] A. Khurshid, X. Zou, W. Zhou, M. Caesar, and P. B.Godfrey. VeriFlow: verifying network-wide invari-ants in real time. In NSDI, 2013.

[24] M. Mahalingam, D. Dutt, K. Duda, P. Agarwal,L. Kreeger, T. Sridhar, M. Bursell, and C. Wright.Virtual extensible local area network (VXLAN): Aframework for overlaying virtualized layer 2 net-works over layer 3 networks. RFC 7348, Aug.2014.

[25] H. Mai, A. Khurshid, R. Agarwal, M. Caesar, P. B.Godfrey, and S. T. King. Debugging the data planewith Anteater. In SIGCOMM, 2011.

[26] T. Nelson, C. Barratt, D. J. Dougherty, K. Fisler,and S. Krishnamurthi. The Margrave tool for fire-wall analysis. In LISA, 2010.

[27] Network verification website. http://goo.gl/

UK0rKS.

13

[28] P. Patel, D. Bansal, L. Yuan, A. Murthy, A. Green-berg, D. A. Maltz, R. Kern, H. Kumar, M. Zikos,H. Wu, C. Kim, and N. Karri. Ananta: Cloud scaleload balancing. In SIGCOMM, 2013.

[29] E. Rosen, A. Viswanathan, and R. Callon. Multi-protocol label switching architecture. RFC 3031,Jan. 2001.

[30] Stanford benchmark. http://goo.gl/FtzxRr.

[31] D. Thaler and C. Hopps. Multipath issues in unicastand multicast next-hop selection. RFC 2991, Nov.2000.

[32] J. Whaley and M. S. Lam. Cloning-based context-sensitive pointer alias analysis using binary deci-sion diagrams. In PLDI, 2004.

[33] G. G. Xie, J. Zhan, D. A. Maltz, H. Zhang, A. G.Greenberg, G. Hjalmtysson, and J. Rexford. Onstatic reachability analysis of IP networks. In IN-FOCOM, 2005.

[34] H. Yang and S. Lam. Real-time verification of net-work properties using atomic predicates. In ICNP,2013.

[35] S. Zhang, S. Malik, and R. McGeer. Verificationof computer switching networks: An overview. InATVA, 2012.

14