unitesk: model based testing in industrial practice victor kuliamin alexander petrenko alexander...
TRANSCRIPT
UniTesK:Model Based Testing in Industrial Practice
Victor KuliaminAlexander PetrenkoAlexander KossatchevIgor Burdonov
Institute for System Programming
Moscow, Russia
Outline
Origin of UniTesK Method UniTesK Manifesto UniTesK Solutions Case Studies
Origin of UniTesK Method
1994 – 1996ISP RAS – Nortel Networks project onfunctional test suite development for Switch Operating System kernel Few hundreds of bugs found in the OS kernel, which
had been in use for 10 years KVEST technology
About 600K lines of Nortel code tested by 2000
BUT: Failed to be introduced in Nortel processes
UniTesK Test Construction
System under Test
Behavior Model Testing Model
Coverage Model
Test InputBehavior Correctness Checking
Specification Development
Test Development Process
Test Quality Criteria
Requirements
Interface Definition
Scenario Development
System under Test
Mediator Development
Test Execution
Behavior Specifications
Test Scenarios
Mediators
Test Reports
Design
Test Results Analysis
Engineering Problems
How to simplify transformation of requirements into formal specifications?
How to minimize human involvement in test development, but make it possible in necessary points?
How to support a large variety of problem domains and test completeness criteria?
How to decouple tests and implementation, but keep them working together?
How to measure testing quality without implementation?
To make easier their usage for ordinary software engineers
To make relation between them more clear
To increase productivity, but not to loose flexibility
To make possible early start of test development
To make specifications and tests more reusable
UniTesK Manifesto
Maximum simplification of MBT techniques Automation where possible Accommodation to current industrial
practice Integration with widely used tools Integration with widely used languages
Tools
J@T CTesK Ch@se
UniTesK Test Architecture
Oracle
Test action construction
Test action construction
System under Test
Test Engine
Test Action Iterator
Specification
✕ Mediator
Test Scenario
Example
Client Data Management System: A number of clients Clients can be related as parent-subsidiary
Client can have only one parent, but may have several subsidiaries
No cycles along parent-subsidiary linksWhen parent is removed, all its subsidiaries
should be removed
Example : Interface
class ClientManager{ Client addClient ( String name ); boolean removeClient ( Client client );}
class Client{ boolean addSubsidiary ( Client client );}
Example : Client Dataclass Client{ String name; ClientSpecification parent = null; HashSet children = new HashSet(); invariant ParentChildLink() { if(parent != null && !parent.children.contains(this)) return false; Iterator j = children.iterator(); while(j.hasNext()) if(((Client)j.next()).parent != this) return false; return true; }
invariant NoCyclesAlongLinks() { Client r = this; do { r = r.parent; if(r == this) return false; } while(r != null); return true; }}
Example : addClient() Methodclass ClientManager{ specification Client addClient ( String name ) updates clients.? { post { if(name == null || clients.containsKey(name)) { // Client cannot be created return addClient == null && clients.equals(@clients.clone()); } else { // Client can be created HashMap oldClients = (HashMap)clients.clone(); oldClients.remove(name); return addClient != null && addClient.name.equals(name) && addClient.parent == null && addClient.children.isEmpty() && clients.get(name) == addClient && oldClients.equals(@clients.clone()); } } }}
Test Coverage Criteria
branch “Single branch”
return
post {
}
if(a || b) branches “Single branch”
predicates ( a || b ) !( a || b )
disjuncts a !a && b !a && !bbranch “Single branch”
branches “Single branch”
disjuncts a !a && !b
predicates ( a || b ) !( a || b )
!a && b
if(a || b) if(a || b)
class ClientManager{ specification Client addClient ( String name ) updates clients.? { post { if(name == null || clients.containsKey(name)) { branch "Client cannot be created"; return addClient == null && clients.equals(@clients.clone()); } else { branch "Client can be created"; HashMap oldClients = (HashMap)clients.clone(); oldClients.remove(name); return addClient != null && addClient.name.equals(name) && addClient.parent == null && addClient.children.isEmpty() && clients.get(name) == addClient && oldClients.equals(@clients.clone()); } } }}
Example : addClient() Method
Automaton from Coverage Goals
states
parameters operation domain
1
2
3coverage goals
Implicit Automata Description
Implicit specifications cannot be resolved To decrease effort they should not be resolved Huge automata can be described shortly
Test Scenarios
User can describe an automatonimplicitly in the form of Test Scenario
Functions: Provide the current state identifier Provide the next admissible stimulus in
this state
Test Scenario : Statescenario class ClientManagerTestScenario{ ClientManager target;
AbstractGenState getGenState() { IntToIntMapGenState genstate = new IntToIntMapGenState(); int n = 0, m = 0; Iterator j = objectUnderTest.clients.keySet().iterator(); while(j.hasNext()) { String s = (String)j.next(); n = Integer.parseInt(s); if(((Client)target.clients.get(s)).parent == null) genstate.setValue(n, 0); else { m = Integer.parseInt(((Client)target.clients.get(s)).parent.name); genstate.setValue(n, m); } } return genstate; }}
Test Scenario : Stimuliscenario class ClientManagerTestScenario{ scenario boolean addClient() { iterate(int i = 0; i < numberOfClients; i++; ) { String name = (i == 0)?(null):("" + i); if( target.precondition_addClient( name ) ) target.addClient( name ); } return true; }
scenario boolean removeClient() { iterate(int i = -1; i < numberOfClients; i++; ) { Client client = (i < 0)? ClientMediator.create(new ClientImpl("1")) : (Client)target.clients.get(("" + i)); if( target.precondition_removeClient( client ) ) target.removeClient( client ); } return true; }}
Test Execution : First StageTest
Engine
Test Scenario
Factorization
Testing Concurrency : Modeling
?a
?{a,b}
?b
?{a,a}?{b,b}?{a,a,a}?{a,a,b}
How to model responses on all possible multistimuli?
Plain concurrency axiom :reaction on multistimulus is the same as on some sequence of its stimuli
Asynchronous Reaction Specification
specification reaction UDPPacket UDPResponse ( ){ pre { return !ModelUDPPackets.isEmpty ( ); } post { return (@ModelUDPPackets.clone()).contains ( UDPResponse ) && !ModelUDPPackets.contains ( UDPResponse ); }}
Providing Concurrent Inputs
s11
System under Test
s21
s31
s12
s32
s13
s22s23
s14
s24
s33
Multisequence is used instead of sequence of stimuli
Collecting Asynchronous Reactions
Reactions form a partially ordered set
System under Test
r33r32
r12
r23r22
r11
r21
r31
Time
11
12
21 31
22 32
23
33
Evaluation of Reactions
Stimuli Reactions
Plain concurrency axiom
✕
UniTesK Test Architecture, 2
Oracle
Test input construction
System under Test
Test Engine
Test Action Iterator
Mediator
Synchronization Manager
Reaction Collector
Serializer
✕
✕
UniTesK Tools
J@T (C++ Link)Java (C++) / NetBeans, Eclipse (plan)
CTesKC / Visual Studio 6.0, gcc
Ch@seC# / Visual Studio .NET 7.1
OTKSpecialized tool for compiler testing
Technology Transfer
Forms Training Pilot projects Joint projects Project supervising
Successful transfers Lanit-Tercom
CTesK, Feb 2002 Luxoft (IBS group)
J@T, Jul 2003 Intel
OTK, Nov 2003
Case Studies
IPv6 implementations - 2001-2003 Microsoft Research Mobile IPv6 (in Windows CE 4.1) Oktet
Intel compilers - 2001-2003 Web-based client management system in bank Enterprise application development framework Billing system
References
1. V. Kuliamin, A. Petrenko, N. Pakoulin, I. Bourdonov, and A. Kossatchev. Integration of Functional and Timed Testing of Real-time and Concurrent Systems. Proc. of PSI 2003. LNCS, Springer-Verlag, 2003.
2. V. Kuliamin, A. Petrenko, I. Bourdonov, and A. Kossatchev. UniTesK Test Suite Architecture. Proc. of FME 2002. LNCS 2391, pp. 77-88, Springer-Verlag, 2002.
3. A. K. Petrenko, I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin. Experiences in using testing tools and technology in real-life applications. Proceedings of SETT’01, India, Pune, 2001
4. I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin. Using Finite State Machines in Program Testing. "Programmirovanije", 2000, No. 2 (in Russian). Programming and Computer Software, Vol. 26, No. 2, 2000, pp. 61-73 (English version)
5. I. Bourdonov, A. Kossatchev, A. Petrenko, and D. Galter. KVEST: Automated Generation of Test Suites from Formal Specifications. Proceedings of World Congress of Formal Methods, Toulouse, France, LNCS, No. 1708, 1999, pp. 608-621
6. http://www.ispras.ru/groups/rv/rv.html
Contacts
Victor V. Kuliamin Alexander K. Petrenko
E-mail: [email protected], [email protected], B. Kommunisticheskaya, 25Moscow, RussiaWeb: http://www.ispras.ru/groups/rv/rv.htmlPhone: 007-095-9125317Fax: 007-095-9121524
Thank you!