interface grammars for modular software verification tevfik bultan department of computer science...
Post on 21-Dec-2015
221 views
TRANSCRIPT
Interface Grammars for Modular Software Verification
Tevfik BultanDepartment of Computer Science
University of California, Santa Barbara
This talk is based on:
• Graham Hughes and Tevfik Bultan. "Interface Grammars for Modular Software Model Checking." Proceedings of the 2007 ACM/SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2007).
• Graham Hughes and Tevfik Bultan. "Extended Interface Grammars for Automated Stub Generation." Proceedings of the Automated Formal Methods Workshop (AFM 2007).
• Graham Hughes, Tevfik Bultan and Muath Alkhalaf. "Client and Server Verification for Web Services Using Interface Grammars." To appear in the Proceedings of the Workshop on Testing, Analysis and Verification of Web Software (TAV-WEB 2008).
• Graham Hughes and Tevfik Bultan. "Interface Grammars for Modular Software Model Checking." To appear in IEEE Transactions on Software Engineering.
Outline
• Motivation• Interface Grammars• Interface Compiler• A Case Study• Interface Grammars for Web Services• Another Case Study• Conclusions
Model Checking Software
• Model checking– An automated software verification technique– Exhaustive exploration of the state space of a program
to find bugs
• Systematically explore all possible behaviors of a program– look for violations of the properties of interest
• assertion violations, deadlock
• Software model checkers: Verisoft, Java PathFinder (JPF), SLAM, BLAST, CBMC
Two Challenges in Model Checking
• State space explosion– Exponential increase in the state space with increasing
number of variables and threads• State space includes everything: threads, variables,
control stack, heap
• Environment generation– Finding models for parts of software that are
• either not available for analysis, or• are outside the scope of the model checker
Modular Verification
• Modularity is key to scalability of any verification technique– Moreover, it can help isolating the behavior you wish
to focus on, removing the parts that are beyond the scope of your verification technique
• Modularity is also a key concept for successful software design– The question is finding effective ways of exploiting the
modularity in software during verification
Interfaces for Modularity
• How do we do modular verification?– Divide the software to a set of modules– Check each module in isolation
• How do we isolate a module during verification/testing?– Provide stubs representing other modules (environment)
• How do we get the stubs representing other modules?– Write interfaces
• Interfaces specify the behavior of a module from the viewpoint of other modules
• Generate stubs from the interfaces
Our Approach: Interface Grammars
Here is the basic use case for our approach:
1. User writes an interface grammar specifying the interface
2. Our interface compiler automatically generates a stub from the interface grammar
3. Automatically generated stub provides the environment during modular verification
Interface Grammars
Component A
Component B
Inte
rfac
e G
ram
mar
InterfaceCompiler
Component B Stub
Component A
ModelChecker
Inte
rfac
e G
ram
mar
An Example
• An interface grammar for transactions– Specifies the appropriate ordering for method calls to a
transaction manager
– Method calls are the terminal symbols of the interface grammar
SStarttart →→ BaseBaseBaseBase →→ beginbegin TailTail BaseBase
|| εεTailTail →→ commitcommit
|| rollbackrollback
An Example
• Consider the call sequence
begin rollback begin commit
• Here is a derivation:
Start Base begin Tail Base begin rollback Base begin rollback begin Tail Base begin rollback begin commit Base begin rollback begin commit
SStarttart →→ BaseBaseBaseBase →→ beginbegin TailTail BaseBase
|| εεTailTail →→ commitcommit
|| rollbackrollback
Another Example
• This example can also be specified as a Finite State Machine (FSM)
• However, the following grammar which specifies nested transactions cannot be specified as a FSM
StartStart →→ BaseBaseBaseBase →→ beginbegin BaseBase TailTail BaseBase
|| εεTailTail →→ commitcommit
|| rollbackrollback
begin
commit rollback
Yet Another Example
• Let’s add another method called setsetrollbackonlyrollbackonly which forces all the pending transactions to finish with which forces all the pending transactions to finish with rollbackrollback instead of instead of commitcommit
• We achieve this by extending the interface grammars with We achieve this by extending the interface grammars with semantic predicates and semantic actionssemantic predicates and semantic actions
StartStart →→ «r:=false; l:=0»«r:=false; l:=0» BaseBaseBaseBase →→ beginbegin «l:=l+1»«l:=l+1» BaseBase Tail Tail «l:=l-1; if l=0 then r:=false»«l:=l-1; if l=0 then r:=false» BaseBase
|| setsetrollbackonly rollbackonly «r:=true»«r:=true» BaseBase || εε
TailTail →→ «r=false»«r=false» commitcommit || rollbackrollback
Interface Grammar Translation
• Our interface compiler translates interface grammars to executable code: – the generated code is the stub for the component
• The generated code is a parser that – parses the incoming method calls– while making sure that the incoming method calls
conform to the interface grammar specification
Verification with Interface GrammarsIn
terf
ace
Gra
mm
ar
InterfaceCompiler
Program
ModelChecker
Top-downparser
parsetable
semanticpredicates
andsemanticactions
pars
er s
tack
Com
pone
nt S
tub
method invocation(lookahead)
A Case Study
• We wrote an interface grammar for the EJB 3.0 Persistence API
– This is an API specification for mapping Java object graphs to a relational database
– Hibernate is an implementation of this API
• Hibernate distribution contains several example EJB clients that are designed to fail, and test exceptional behavior by violating the interface specification
A Case Study, Continued
• We used these simple clients to check the the fidelity of the stub generated from our interface specification
– We used the JPF software model checker
• None of these examples can run under JPF directly
• Time taken to develop the interface was dominated by the need to understand EJB Persistence first
– about a couple of hours
Experiments: Falsification
sec. sec. sec. sec. # obj. # iter.
1.9 1.8 1.8 2.1 1 1
1.9 1.8 1.8 2.1 1 5
1.9 1.8 1.8 2.1 5 1
1.9 1.8 1.8 2.1 5 5
Client 1 Client 2 Client 3 Client 4
Experiments: Verification
sec. sec. sec. sec. # obj. # iter.
3.1 1.9 1.8 2.5 1 1
26.9 11.1 9.9 17.1 1 5
10.1 5.9 4.2 11.8 5 1
126.3 63.9 43.5 138.5 5 5
Client 1 Client 2 Client 3 Client 4
A Case Study, Continued
• For these simple clients, interface violations can be detected by JPF in a couple of seconds using the EJB stub generated by our interface compiler– Falsification time does not increase with the number of
operations executed or the number of objects created by the clients
• When we fix the errors, JPF is able to verify the absence of interface violations– Verification time increases with the number of
operations executed or the number of objects created by the clients
Interface Grammars: Uni/Bidirectional
• Interface grammars can be – Unidirectional: No callbacks
– Bidirectional: Need to handle Callbacks
CallerCallee
Inte
rfac
e
Comp AComp B
Inte
rfac
e
Interface Grammars: Client/Server
• Interface grammars can be used for – Client verification: Generate a stub for the server– Server verification: Generate a driver for the server
InterfaceCompiler
Inte
rfac
e
ClientStub
ServerDriver
Interface Grammars and Data
• A crucial part of the interface specification is specifying the allowable values for the method arguments and generating allowable return values
• Approach 1: These can be specified in the semantic actions and semantic predicates of the interface grammars
• Approach 2: Can we specify the constraints about the arguments and return values using the grammar rules?– Yes, grammar productions can be used to specify the
structure of most recursive data structures.
Modular Verification of Web Services
• We applied our modular verification approach based on interface grammars to client and server side verification of Web services
Interface Grammars and Web Services
Our approach:
1. A WSDL-to-interface grammar translator automatically generates grammar productions that generate and/or validate XML arguments and return values
2. User adds control flow constraints by modifying the grammar
3. Interface compiler automatically generates a stub for client side verification and a driver for server-side verification
Interface Grammars for Web Services
Another Case Study: AWS-ECS
• We tested the Amazon E-Commerce Web Service (AWS-ECS) using our approach
• AWS-ECS WSDL specification lists 40 operations – that provide differing ways of searching Amazon’s
product database
• We focus on the core operations:– ItemSearch, CartCreate, CartAdd, CartModify, CartGet,
CartClear
Client-side Verification
• For client verification we used a demonstration client provided by Amazon
• This client does not check any constraints such as– You should not try to insert an item to a shopping cart
before creating a shopping cart• When such requests are sent to AWS-ECS they would
return an error message• Using our approach we can easily check if the client allows
such erroneous requests• Falsification time changes with the type of faults we are
looking for (data or control errors), changes from 10 to 60 seconds
AWS-ECS: Server Verification
• Our interface compiler automatically generates a driver that sends sequences of requests to AWS-ECS server and checks that the return values conform to the interface specification
• The driver is a sentence generator– It generates sequences of SOAP requests based on the
interface specification
• We used two algorithms for sentence generation:– A random sentence generation algorithm– Purdom’s algorithm: A directed sentence generation
algorithm that guarantees production coverage
Directed Sentence Generation
• Number of sentences generated: 5
• Average derivation length: 24
• Average number of SOAP requests/responses: 3.8
• Verification time: 20 seconds
Random Sentence Algorithm
0 5 10 15 20 25 30 35 40 45 500%
20%
40%
60%
80%
100%
Percentage of covered productions
Percentage of covered nonterminals
Number of exceutions0 20 40 60 80 100 120
0
50
100
150
200
250
300
350
time in seconds
Number of executions
• Number of sentences generated: 100
• Average derivation length: 17.5
• Average number of SOAP requests/responses: 3.2
Server-side verification
• We found two errors during server side verification– Errors were discovered within 10 seconds
• These errors indicate a mismatch between the interface specification and the server implementation
• It may mean that we misunderstood the description of the Web service interface
• It may also mean that there is an error in the service implementation
Conclusions
• Modular verification is a necessity
• Interfaces are crucial for modular verification
• Interface grammars provide a new specification mechanism for interfaces
• Interface grammars can be used for automated stub generation leading to modular verification
THE END
Related Work: Interfaces
• L. de Alfaro and T. A. Henzinger. Interface automata.
• O. Tkachuk, M. B. Dwyer, and C. Pasareanu. Automated environment generation for software model checking.
• A. Betin-Can and T. Bultan. Verifiable concurrent programming using concurrency controllers.
• T. Ball and S. K. Rajamani. SLAM interface specification language.
• G. T. Leavens et al.: JML
Related: Grammar-based Testing
• A. G. Duncan, J. S. Hurchinson: Using attributed grammars to test designs and implementations
• P. M. Maurer: Generating test data with enhanced context free grammars
• P. M. Maurer: The design and implementation of a grammar-based data generator
• E. G. Sirer and B. N. Bershad: Using production grammars in software testing