theory and practice of co-verification process: unitesk story

Theory and Practice ofCo-verification Process: UniTesK Story

RedVerst group of ISP RASAlexander K. Petrenko

Victor V. Kuliaminhttp://www.ispras.ru

Overview

• Introduction : Why co-verification?

• Main part : What is UniTesK?Solving Engineering Problems

• Case studies

What these numbers mean?

180 109 $ 21 109 $

total revenue of US software development

companies

loss caused by inadequate testing

infrastructure for US economy1

1. The Economic Impacts of Inadequate Infrastructure for Software Testing,NIST Report, May 2002

9 : 1

Waterflow Process Model

Requirements

Design

Implementation

Testing

Deployment

Iterative Process Model

Requirements

Design

Implementation

Testing

Deployment

Inception Elaboration Construction Transition

Agile Development Methods Expansion

According to “The Decision is in: Agile versus Heavy Methodologies” by Robert Charette,Cutter IT Journal, vol. 2, No. 19, 2002http://www.cutter.com/freestuff/apmupdate.html

Percent of organizations using modern development methods

0%

10%

20%

30%

40%

50%

60%

Agile Development Methods

RUP

CMM-related Methods

ISO 9000-based Methods

Agile Development Methods Expansion

0%

10%

20%

30%

40%

50%

60%

2001

2002

2003

According to “The Decision is in: Agile versus Heavy Methodologies” by Robert Charette,Cutter IT Journal, vol. 2, No. 19, 2002http://www.cutter.com/freestuff/apmupdate.html

Percent of organizations recognizing more than 50% of projects as agile

Inadequate Quality of Software

Loss Potential Cost Reduction

Total Sales

Software Vendors

21.2109 $ 10.6109 $ 180109 $

Software Users

38.3109 $ 11.7109 $

Total 59.5109 $ 22.2109 $

The Economic Impacts of Inadequate Infrastructure for Software Testing,NIST Report, May 2002

Evolution of Testing

• Localization of errors• Demonstration of errors

– Testing is the process of executing a program or system with the intent of finding errors [Myers, 1979]

– The purpose of testing is to show that a program has bugs [Hetzel, 1988]

• Evaluation of quality– Testing is the process of operating a system or component

under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component [IEEE 90]

Co-verification

Verification confirms that activity products properly reflect the requirements specified for them at the beginning of the activity.

Co-verification process

• Perform verification of activity before proceeding to the dependent activities

• Prepare all the artifacts needed for verification concurrently with main activity products

Co-verification Development Process

Business Modeling

Requirements

Architecture Design

Component Design

Implementation

Integration

Deployment

Main Part Overview

• Traditional approaches to verification

• UniTesK approach

• Example

Traditional Software Development Process

• Requirements Analysis• Design• Implementation-

debugging• Requirements

Elicitation• Test Case Design• Test Implementation• Test Execution• Test Result Analysis

Design and development Verification

Co-verification Process

• Requirements Analysis

• Design• Implementation-

debugging

• Requirements Elicitation

• Test Case Design• Test Implementation• Test Execution• Test Result Analysis

Design and development Verification

Uniform Test

Architecture

UniTesK Approach

Uniform Specification

Extension

Integration with Development Environments

CT

esK

J@T

.N@

T

VD

M+

+T

esK

Fou

ndat

ions

Too

ls

UniTesK

Model Based Testing

Requirements Formalization

Requirements FormalSpecifications

Requirements

Co-verification Support

FormalSpecificationsRequirements

Ambiguity? Incompleteness?

Inconsistency?

Engineering Problems (1)

• Specification technique should support– Easy transformation of requirements into

specifications– Easy automation of further test development– Functional test coverage definition– High reusability of specifications

• Specification notation should not require special education and skills

Specification Techniques

• ExecutableImperative state based specifications

• ConstraintsState based data type constraints, pre- and postconditions, internal invariants

• AxiomaticAction based axioms

Executable Specifications

• Are very close to some implementation – Are easy to use in the industry– Can be transformed into prototypes

• Are not close to requirements( √¯ = e½ln = lim(xn+1 = ½(xn +x/xn)) )– Unsuitable for test coverage measurement– Can cause problems with conformance checking

How to compare the results?

• Are highly reusable as executable code• But are low reusable as criteria of correctness

Constraint Specifications

• Have the structure similar with implementation– Are easy to use in the industry

• But have different form• Are close to requirements in most cases

– Are easy to construct from requirements

– Suitable for test coverage measurement

– Counterexample: memory management subsystem

• Can be directly used in conformance checking• Special constructs enabling reuse can be added

Axiomatic Specifications

• Are far from common implementations and have greatly different structure– Can hardly be introduced in the industry

• Are usually far from requirements – Are hard to develop in real-life projects– Can hardly be used for coverage measurement– But sometimes are the only appropriate form

• Can be used for conformance checking• But provide poor error localization• Reusability is a problem

Comparison of Specification Techniques

0 1 2 3 4 5 6

Reusability

Ease of oracle generation

Source for coverage models

Closeness to requirements

Decoupling with implementation

Ease of use in the industry

Constraints

Executable

Algebraic

0 1 2 3 4

Average

Constraints

Executable

Algebraic

Specification Notation

• Specification language– Suitable for capture abstract properties– Has formal semantics– Requires complex mediator layer for implementation– Requires special education, mastering is enduring

• Extension of programming language– Abstract concepts can be added by libraries– Ambiguous parts of language can be excluded– Complex mediators are not required– Mastering can be made more effective

– Facilitates co-verification process

• Constraint Specifications– Preconditions and postconditions of operations

– Data type constraints

• Functional coverage description based on specification structure

UniTesK Specification Technique

Uniform Test

ArchitectureUniform

Specification Extension


CT

esK

J@T

.N@

T

VD

M+

+T

esK

UniTesK

Model Based Testing

UniTesK Specification Technique

• Constraint Specifications– Preconditions and postconditions of operations

– Data type constraintsspecification Operation()pre block, returning Boolean value

post block, returning Boolean value use @ to refer to pre-value of expressions

invariant Invariant() block, returning Boolean value

• void enq (Object obj, int priority)priority [Min_Priority, Max_Priority]

• Object deq ()

• int size ()

Priority Queue Example

5

deq () enq ()

size ()

5 4 3 1 1

enq ()enq ()

05 2

J@va : Specification Extension of Java

specification package pqueue;

public class PQueueSpecification{ specification public void enq(Object obj, int prty) reads obj, prty updates items.?, priorities.? { pre { return obj != null; } post { int i = 0; for(i = 0; i < items.size() && ((Integer)priorities.elementAt(i)).intValue() > prty; i++ ); ...

Uniform Test

ArchitectureUniform



CT

esK

J@T

.N@

T

VD

M+

+T

esK

UniTesK

Model Based Testing

Model State Definition

specification package;

public class PQueueSpecification

{

// List of elements of the queue

public List items = new List();

// Accompanying list of their priorities

public IntList priorities = new IntList();

}

Data Integrity Constraints : Lists are not Null

public class PQueueSpecification{ public List items = new List(); public IntList priorities = new IntList();

invariant ListsAreNotNull() { return items != null && priorities != null; }}

Data Integrity Constraints : Lists’ Sizes are Equal

public class PQueueSpecification{ invariant ListsSizesAreEqual() { return items.size() == priorities.size(); }}

Data Integrity Constraints : Priorities Lie in the Range

public class PQueueSpecification{ public static int Min_Priority; public static int Max_Priority;

invariant PrioritiesLieInTheRange() { for(int i = 0; i < priorities.size(); i++) { if( priorities.get(i) < Min_Priority || priorities.get(i) > Max_Priority ) return false; } return true; }}

Data Integrity Constraints : Priorities do not Increase

public class PQueueSpecification{ invariant PrioritiesDoNotIncrease() { for(int i = 1; i < priorities.size(); i++) { if( priorities.get(i-1) < priorities.get(i) ) return false; }

return true; }}

Operation Specification : size() Method


{

specification public int size ()

reads this.?

{

pre { return true; }

post

{

return size == items.size();

}

}

}

Operation Specification : enq() Method

public class PQueueSpecification{ specification public void enq (Object obj, int prty) reads obj, prty updates this.? { pre { return obj != null && Min_Priority <= prty && prty <= Max_Priority; } post { int i = 0; for(i = 0; i < items.size() && priorities.get(i) >= prty; i++ );

PQueueSpecification new_state = (PQueueSpecification)@clone(); new_state.items.add(i, obj); new_state.priorities.add(i, prty); return this.equals(new_state); } }}

Auxiliary Methods

public class PQueueSpecification{ public boolean equals (PQueueSpecification other) { return items.equals(other.items) && priorities.equals(other.priorities); } public Object clone () { PQueueSpecification result = new PQueueSpecification(); result.items = (List)items.clone(); result.priorities = (IntList)priorities.clone(); return result; }}

Operation Specification : deq() Method


{

specification public Object deq ()

updates this.?

{

post

{

if(items.isEmpty())

return deq == null && items.isEmpty();

else

{

PQueueSpecification new_state = (PQueueSpecification)@clone();

Object result = new_state.items.first();

new_state.items.removeFirst();

new_state.priorities.removeFirst();

return this.equals(new_state) && deq == result;

}

}

}

}

Coverage Goals Definition

Requirements

Formal Specificationspre ---------------------------post --------------------------- -------------------------------------------------- ---------------------------------------------- ------------ ------------------------------------- ---------------------------------------------- --------------------------------------------- -------------------------------------------------- ---- -------------------------------------

Test Case 1--------------------------------------

Testable Specifications

pre ---------------------------post ---------------------------

-------------------------------------------------- ---------------------------------------------- ------------ ------------------------------------- ---------------------------------------------- ---------------------------------------------- --------------------------------------------------

---- -------------------------------------

123


Formal Specificationspre ---------------------------post --------------------------- -------------------------------------------------- ---------------------------------------------- ------------ ------------------------------------- ---------------------------------------------- --------------------------------------------- -------------------------------------------------- ---- -------------------------------------


pre ---------------------------post ---------------------------

-------------------------------------------------- ---------------------------------------------- ------------ ------------------------------------- ---------------------------------------------- ---------------------------------------------- --------------------------------------------------

---- -------------------------------------

Software --------------------------------------------------- --------------------------------------------------- --------------------------------------------------- --------------------------------------------------- --------------------------------------------------- --------------------------------------------------- --------------------------------------------------- --------------------------------------------------- ---------------------------------------------------

UniTesK supports this transformation

with techniques and tools


• More than one coverage metric is needed

• Automatic extraction of coverage goals

• Enabling test designer to introduce additional goals

Several Levels of Coverage Metrics

Formal Specifications

pre ---------------------------post ---------------------------

-------------------------------------------------- ---------------------------------------------- ------------ ------------------------------------- ---------------------------------------------- ---------------------------------------------- --------------------------------------------------

---- -------------------------------------

123

1.11.21.3

2.12.2

3.13.2

1.2.11.2.2

1.3.11.3.21.3.3

3.1.13.1.23.1.3

Definition of Coverage Goals : Functionality Branches

• Functional coverage description based on specification structurepost if(a || b) ... branch “Case 1”; ... else if(!c && d) ... branch “Case 2”; ... else ... branch “Case 3”; ...

Branches Disjuncts

Case 1a

!a b

Case 2 !a !b !c d

Case 3!a !b c

!a !b !c !d


{

specification public int size ()

reads this.?

{

post

{

branch “Single branch”;

return size == items.size();

}

}

}

Functionality Branches in size() Method

specification public void enq (Object obj, int prty) reads obj, prty updates this.?{ pre { return obj != null; } post { int i = 0; for(i = 0; i < items.size() && priorities.get(i) >= prty; i++ );

branch “Single branch”;

PQueueSpecification new_state = (PQueueSpecification)@clone(); new_state.items.add(i, obj); new_state.priorities.add(i, prty); return this.equals(new_state); }}

Functionality Branches in enq() Method

specification public Object deq ()

updates this.?

{

post

{

if(items.isEmpty())

{

branch “Empty queue”;

return deq == null && items.isEmpty();

}

else

{

branch “Nonempty queue”;

PQueueSpecification new_state = (PQueueSpecification)@clone();

Object result = new_state.items.firstElement();

new_state.items.removeFirstElement();

new_state.priorities.removeFirstElement();

return this.equals(new_state) && deq == result;

}

}

}

Functionality Branches in deq() Method

Additional Coverage Goals : enq()

5 5 4 3 1 1

05 27 4 1

1 1

1

1

Additional Coverage Goals : deq()

5 5 4 3 1 1

5 4 3 1

Definition of Coverage Goals : Marked Paths

post if(a || b || c) {

Branches Marked Paths Disjuncts

Case 1

“a holds”; “Case 1” a

“Case 1”!a b

!a !b c

branch “Case 1”; ... }

if(a) mark “a holds”;

Marked Paths in enq() Method

post{ int i = 0; for(i = 0; i < items.size() && priorities.get(i) >= prty; i++ );

if( items.isEmpty() ) mark "Insertion in the empty queue"; else if( prty > priorities.maximum() ) mark "Insertion in the head"; else if( prty == priorities.maximum() && prty == priorities.minimum() ) mark "Insertion with single existing priority"; else if( prty == priorities.maximum() ) mark "Insertion with maximum priority"; else if( prty < priorities.minimum() ) mark "Insertion in the tail with new priority"; else if( prty == priorities.minimum() ) mark "Insertion with minimum priority"; else if( !priorities.contains(prty) ) mark "Insertion in the middle with new priority"; else mark "Insertion in the middle with existing priority";

branch “Single branch”; ...}

post

{

if(items.isEmpty())

{


...

}

else

{

if( items.size() == 1 )

mark "Single element in the queue";

else if( !(priorities.maximum() == priorities.get(1)) )

mark "Single element with maximum priority, several elements in the queue";

else if( priorities.toSet().size() > 1 )

mark "Several elements with maximum priority, there are other priorities";

else

mark "Several elements in the queue with the same priority";

branch “Nonempty queue”;

...

}

}

Marked Paths in deq() Method

Test Implementation

Test Case 1--------------------------------------


pre ---------------------------post ---------------------------

-------------------------------------------------- ---------------------------------------------- ------------ ------------------------------------- ---------------------------------------------- ---------------------------------------------- --------------------------------------------------

---- -------------------------------------

Test Program

Test Scenario



pre ---------------------------post --------------------------- -------------------------------------------------- ---------------------------------------------- ------------ ------------------------------------- ---------------------------------------------- --------------------------------------------- -------------------------------------------------- ---- -------------------------------------


pre ---------------------------post ---------------------------

-------------------------------------------------- ---------------------------------------------- ------------ ------------------------------------- ---------------------------------------------- ---------------------------------------------- --------------------------------------------------

---- -------------------------------------

Software --------------------------------------------------- --------------------------------------------------- --------------------------------------------------- --------------------------------------------------- --------------------------------------------------- --------------------------------------------------- --------------------------------------------------- --------------------------------------------------- ---------------------------------------------------

UniTesK supports this transformations with techniques and tools

Test Scenario


• Test construction technique should ensure coverage goals achievement

• Enabling test designer to introduce additional tests

• Tests should be decoupled with implementation

Overview of UniTesK Approach to Test Implementation

• Test construction technique : traversal of FSM– FSM is constructed in so far that its traversal ensures

coverage

• FSM represented implicitly as test scenario– Implicit representation saves a lot of work

– Test scenarios can be• Generated on the base of templates and specifications

• Developed manually

• Developed in mixed way

• Mediators are used to decouple test scenarios and implementation– The same three ways to develop mediators

UniTesK Test Architecture

Uniform Test

ArchitectureUniform



CT

esK

J@T

.N@

T

VD

M+

+T

esK

UniTesK

Model Based Testing

• From specification we can generate oracle to check the conformance of system behaviour

• The entire test is constructed as a test sequence intended to achieve some coverage– Test sequence required is produced on-the-fly

during test execution as a transition tour of FSM model

Test sequence construction

UniTesK Test Architecture


Oracle

Target system

Test Engine

Test Action Iterator

Specification

Test Sequence Construction

Test Engine


Engineering Problems: FSM Construction (4)

• Implicit specifications are hard to resolve• Nondeterminism• Huge numbers of states and transitions

Factorization Based on Coverage Goals

I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin. Using Finite State Machines in Program Testing. Programming and Computer Software, Vol. 26, No. 2, 2000, pp. 61-73

Helps to cope with state explosion and nondeterminism

Implicit State Machine Description

Allows not to resolve implicit constraintsMakes factorization easier

State Machine Construction Based on Coverage Goals

states

parametersoperation domain

defined by precondition

1

2

3coverage goals

Test Sequence Construction for Implicit FSM

Test Engine


Test Scenario


Oracle

Target system

Test Engine


Specification

Test Scenario

Priority Queue Example : enq() Coverage Goals

if( items.isEmpty() ) mark "Insertion in the empty queue";else if( prty > priorities.maximum() ) mark "Insertion in the head";else if( prty == priorities.maximum() && prty == priorities.minimum() ) mark "Insertion with single existing priority";else if( prty == priorities.maximum() ) mark "Insertion with maximum priority";else if( prty < priorities.minimum() ) mark "Insertion in the tail with new priority";else if( prty == priorities.minimum() ) mark "Insertion with minimum priority";else if( !priorities.contains(prty) ) mark "Insertion in the middle with new priority";else mark "Insertion in the middle with existing priority";

States:

Arguments:

Empty queueSingle priority is usedAt least two priorities are usedAt least three priorities are used

At least three different priorities should be provided

Priority Queue Example : deq() Coverage Goals

if(items.isEmpty())


else

if( items.size() == 1 )

mark "Single element in the queue";

else if( !(priorities.maximum() == priorities.get(1)) )

mark "Single element with maximum priority, several elements in the queue";

else if( priorities.toSet().size() > 1 )

mark "Several elements with maximum priority, there are other priorities";

else

mark "Several elements in the queue with the same priority";

States: Empty queueSingle element in the queueSingle element with maximum priority, several prioritiesSeveral elements with maximum priority, several priorities

Several elements with the single used priority

Priority Queue Example : Analysis of Determinism

0 : Empty queue

1 : Single element in the queue

2 : Several elements with single used priority

0 1 2

deq()

To make the behavior deterministic we should split this state according to the number of elements

Priority Queue Example : Analysis of Determinism

0,1,2, … : Number of elements with maximum priority

A : Single element with maximum priority, several priorities

0 1 2 3

deq()

To make the behavior deterministic we should split this state according to the number of elements with the next priority

A

…

And so on, for all priorities (by induction)

Priority Queue Example : Resulting State

Resulting state S : int → int ismap from integers to integers, wherekey is priorityvalue is the number of queue elements having such priority

Reference to Specification Object

scenario public class PQueueTestScenario

{

// Reference to the object under test

public PQueueSpecification queue;

}

State Construction Method

scenario public class PQueueTestScenario{ public AbstractGenState getGenState() { IntToIntMapGenState state = new IntToIntMapGenState(); for(int p = PQueueSpecification.Min_Priority; p < queue.priorities.toSet().size(); p++ ) { int n = queue.priorities.numberOf(p);

if( n != 0 ) state.put(p, n); } return state; }}

Definition of Test Actions : size() method

scenario public class PQueueTestScenario{ scenario public boolean size() { queue.size(); return true; }}

Definition of Test Actions : enq() method


{

scenario public boolean enq()

{

Object arg = new Object();

iterate( int priority = PQueueSpecification.Min_Priority;

priority <= PQueueSpecification.Max_Priority;

priority++;

queue.priorities.toSet().size() <= 3

|| queue.priorities.contains(priority)

&& queue.priorities.numberOf(priority) < 2

)

{

queue.enq(param, priority);

}

return true;

}

}

Definition of Test Actions : deq() method

scenario public class PQueueTestScenario{ scenario public boolean deq() { queue.deq(); return true; }}

Using Nondeterministic FSMs : Bounded Queue

addEmpty

Single element

Intermediate

Almost full

Fullremove

Empty

Single element

Intermediate

Almost full

Full

fill

empty

Testing Concurrency

• Fair Concurrency Axiom :Concurrent execution of several operations is correct if and only if it is equivalent to somehow ordered one

• So, to test the concurrent execution of calls we should check that it behaves like some sequence of the same calls– The system can be modeled as an ordinary FSM– Test engine generates pairs, triples, and so on, of

concurrent actions

• System not satisfying Fair Concurrency Axiom should be modeled as asynchronous FSM

Possible behavior:input: abaouput: yyx xxyxyyyx xyyyxyall possibleoutputs: { x*yx*yy*x, x*yx*yy*xx*yy*x, x*yy*xx*yy*x }

Asynchronous FSM

• Some transitions are fired by a stimulus• Some transitions are fired along with a reaction• Some transitions are empty, modeling internal operation

a/

/x /y

b/a/

/y/x

b/

Co-verification Support : Changes


pre ---------------------post --------------------- ----------------------------------------- ------------------------------------- ------------ ---------------------------- ------------------------------------


pre ---------------------post --------------------- ----------------------------------------- ------------------------------------- ------------ ---------------------------- ------------------------------------- ----------------------------------

Prototype ------------------------------------------- ------------------------------------------- ------------------------------------------- -----------------------------------------

Test Scenario

Software ------------------------------------------- ------------------------------------------- ------------------------------------------- -----------------------------------------

Interface changed

???


• Tests and implementation should be decoupled– Tests can be reused many times– Easy regression testing

• Specification should be as abstract as possible

• Specification should be even more reusable than tests


Oracle

Mediator

Mediator decouples specification and implementation

Mediator

Target system

Mediator Construction

Mediator

Target system

Oracle

Mediatorin Extended Language

Specification

Test Engine


Test Scenario

Open State Testing

• Mediator constructs new model state only on the base of new implementation state

• Requires: Implementation state should be accessible by public fields or reliable observers

• Implies: Current model state actually corresponds to implementation one– Detected failure demonstrates a fault in the last operation– Transition tour is sufficient to ensure reliability

MediatorTarget system

call callresponse

get state data

construct new model state

state data

Hidden State Testing

• Mediator constructs new model state on the base of old one and implementation call results

• Implies: Current model state is actually a hypothesis– Detected failure can be manifestation of a fault in any

previously called operation

– Transition tour is insufficient to ensure reliability

MediatorTarget system

call callresponse

construct new model state

Mediator for Priority Queue

mediator public class PQueueMediator // Extends specification class extends PQueueSpecification{ // Reference to an implementation object implementation public PriorityQueue target;}

Model State Synchronization Method (Open State Testing)

mediator public class PQueueMediator{ public void mapStateUp() { items.clear(); priorities.clear(); if( target != null ) { for(int i = target._clusters.size()- 1; i >= 0; i--) { Vector v = (Vector)target._clusters.elementAt(i); for(int j = 0; j < v.size(); j++) { items.add(v.elementAt(j)); priorities.add(i); } } } }}

Mediator Methods (Open State Testing)

mediator public class PQueueMediator{ public int size() { return target.size(); } public void enq(Object obj, int prty) { target.enq(obj, prty); } public Object deq() { return target.deq(); }}

Mediator Methods (Hidden State Testing)

mediator public class PQueueMediator{ public int size() { return target.size(); } public void enq(Object obj, int prty) { target.enq(obj, prty);

items.add(obj); priorities.add(prty); } public Object deq() { Object result = target.deq(); items.removeFirst(); priorities.removeFirst();

return result; }}

Test Scenario Initialization


{

public PQueueTestScenario()

{

setTestEngine ( new DeterministicFSM_TestEngine() );

PQueueMediator.initClassState();

queue = PQueueMediator.create(new PriorityQueue());

queue.attachOracle();

}

}

UniTesK Practice Overview

• UniTesK tools

• UniTesK history

• Training courses

• QUTILUI project

• Other case studies

UniTesK Tools

Uniform Test

ArchitectureUniform



CT

esK

J@T

.N@

T

VD

M+

+T

esK

UniTesK

Model Based Testing

UniTesK History

pre-UniTesK• 1994 – 1996

ISP RAS – Nortel Networks contract onfunctional test suite development for Switch Operating System kernel– Few hundreds of bugs found in the OS kernel, which had

been 10 years in use

• About 600K lines of Nortel code tested by 2001But failed to be introduced in Nortel processes

UniTesK Tools History

• 1999-2000– Conception

• 2001– J@T Prototype– CTesK Lite

• 2002– VDM++ TesK– J@T Product

• 2003– J@T Suite 1.2 (Java and C++ testing)– CTesK Full– .N@T

postcondition

class SqrtSpecification{ specification static double sqrt(double x) reads x { pre { return x >= 0; } post { if(x == 0) { branch "Zero argument"; return sqrt == 0; } else { branch "Positive argument"; return sqrt >= 0 && Math.abs((sqrt*sqrt-x)/x < epsilon; } } }}

specification double SQRT(double x) reads (double)x{ pre { return x >= 0.; } coverage ZP { if(x == 0.) return (ZERO, "Zero argument"); else return (POS, "Positive argument"); } post { if(coverage(ZP, ZERO)) return SQRT == 0.; else return SQRT >= 0. && abs((SQRT*SQRT - x)/x) < epsilon; } }}

operation signature declaration

preconditionaccess constraints

Sqrt Specification in J@va and Extended C

functional branches definition

Training Courses

CTesK training was conducted for Tercom (St.-Petersburg, Russia),Saarland University (Germany);J@T training was conducted for Systematic Software Engineering (Denmark)

QUTILUI Project

• GoalRedesign of SOS Queue Manipulation Utilities Subsystem

• Activities– Requirements elicitation

– Specification development

– Specification review

– Architecture design

– Component design

– Implementation

– Testing and debugging

QUTILUI Project Results

• Bugs removed– No bugs found after the project end!

• 25% decrease of implementation size

• Design documentation appeared– And it became actual!

• Tests demonstrated conformance with requirements

--------------------------------------------

--------------------

- - - - - - -- - - - - - -- - - - - - -- - - - - - -

Other Examples of UniTesK Usage

• IPv6 implementations– Microsoft Research– Mobile IPv6 (in Windows CE 4.1)

• mpC Workshop– Debug API– mpC expression static and dynamic semantics

• Optimization units in Intel compilers• UniTesK tools

– J@T test system runtime support– Components of J@va translator– CTesK abstract types library

• Lanit-Tercom– IPv6 implementation– DSP software

References

1. B. Bourdonov, A.S. Kossatchev, V.V. Kuliamin, A.K. Petrenko,. UniTesK Test Suite Architecture// Proceedings of FME’2002 conference, Copenhagen, Denmark, LNCS, No. 2391, 2002, pp. 77-88.

2. I.B.Burdonov, A.V.Demakov, A.A.Jarov, A.S.Kossatchev, V.V.Kuliamin, A.K.Petrenko, S.V.Zelenov. J@va : extension of Java for real-life specification and testing // Proc. of Andrei Ershov Fourth International Conference PCI’01, Novosibirsk, LNCS, Vol. 2244, 2001, pp.301-308.

3. A.K. Petrenko, V.V. Kuliamin, I.B. Bourdonov, A.S. Kossatchev. Experiences in using testing tools and technology in real-life applications// Proceedings of SETT’01, India, Pune, 2001

4. A.K.Petrenko, V.V.Kuliamin, I.B.Burdonov, A.V.Demakov, A.A.Jarov, A.S.Kossatchev, S.V.Zelenov. J@va : extension of Java for real-life specification and testing// Proceedings of PSI’01, Novosibirsk, LNCS 2244, 2001, pp.301-308.

5. A.K.Petrenko. Specification Based Testing: Towards Practice. Proceedings of PSI’01, Novosibirsk, LNCS 2244, 2001, pp.157-162.

6. A.Petrenko, A.Vorobiev. Industrial Experience in Using Formal Methods for Software Development in Nortel Networks// Proceedings of Testing Computer Software Conference TCS 2000, Washington, June, 2000.

7. V.V. Kuliamin, I.B. Bourdonov, A.S. Kossatchev. Using Finite State Machines in Program Testing// Programming and Computer Software, Vol. 26, No. 2, 2000, pp. 61-73.

8. I. Bourdonov, A. Kossatchev, A. Petrenko, and D. Galter. KVEST: Automated Generation of Test Suites from Formal Specifications// Proceedings of World Congress of Formal Methods, Toulouse, France, LNCS, No. 1708, 1999, pp. 608-621

theory and practice of co-verification process: unitesk story

Documents

verification of activity

based methods26

software testing

nist report

robert charette

heavy methodologies

purpose of testing

cmmrelated methods27