testing interfaces in multi-program systems ecen5043 software engineering of multiprogram systems...

Testing Interfaces in Multi-Program Systems

ECEN5043 Software Engineering of Multiprogram Systems

University of Colorado, Boulder

April 17, 2007 ECEN5043 University of Colorado Testing Interfaces

2

Topics• Computational Models• Implications of nondeterminism• Model Testing

– Class Testing extended– UML for state diagrams, concurrency,

components– Timing and Time-outs -- UML and testing– Threads & synchronization


3

Topics (cont. 2)

• Dynamic Testing– Definition of component– Component Interactions

• Identifying -- nonprimitive classes• Specifying

– Testing Interactions• Collections • Collaborations• One-way pass along• Interaction between Testing & Design


4

Topics (cont. 3)

• Sampling Test Cases -- OATS revisited• Testing Off-The-Shelf Components• Protocol Testing• Test Patterns -- McGregor & Sykes; Binder• Testing Interactions at the System Level• Testing Exceptions


5

Computational Models

• Sequential processing -- default– Model testing: straightforward inspections– Dynamic testing: see course #1

• Concurrency– Multiple things that appear to be happening at the

same time.– Central issue -- nondeterminism of interleaved

statement execution– Preliminary thoughts on implications for testing?


6

Computational Models (cont.)

• Distributed – Characterized by: multiple processes to support

a flexible architecture in which the number of participating objects can change

– Can be distributed across multiple processes on the same machine -- we’re postponing this to course #3 when we focus on distributed architectures (on one or more processors).


7

Implications of Nondeterminism

• Nondeterminism– Difficult (impossible) to replicate a test run when

the software contains multiple threads or concurrent processes

– Ordering determined by the scheduler– Changes in programs NOT associated with the

system under test can affect the order in which threads of the system under test are executed

– If a failure is encountered and the defect is isolated and repaired, the fact that the program passes test execution is ... ?


8

Nondeterminism and Static Testing

• More thorough model testing at the class level

– Design (and review) should investigate whether there is

• a need

• an appropriate provision for synchronization

• During model testing,

– Ensure test cases require execution of critical sections

– Consider impact of different orderings with respect to

• starvation

• mutual exclusion

• deadlock

• synchronization


9

But how??

• Three kinds of model testing

1. Is the code for one process correct by itself (are the sequential statements correct) assuming its interactions are correct

2. If a propos, inspect with multiple copies of same process in parallel

3. For shared data, different processes, have copies of each

• For the critical sections

– “What would starvation require in this context?” Is it possible?

– Similarly for mutual exclusion, deadlock, and necessary synchronization


10

Testing Multi-program Systems

• Methods -- review the class testing described in previous course– state transition coverage– code coverage– pre & post condition coverage

• Extend notion of state diagram to that of interacting components (programs/processes)

• Extend state transition coverage similarly– Recall notion of state transition pairs coverage


11

Sample State Transition Diagram - Gomaa


12

Time-Outs for Testing

• It’s possible that a process/program will make a request of another process that never responds

• The requester must be able to detect that and abandon the request

• Design should explicitly define correct behavior for – when the request is answered– and also when it is not – the behavior may be very different in those two cases.

• Model testing should verify these• Dynamic testing should also.


13

Expressing Time on Interaction Diagrams

• Interaction diagrams are collaboration diagrams and sequence diagrams

• time expression -- resolves to a relative or absolute value of time when evaluated

• timing mark -- time-related name or label on a message

• timing constraint -- condition that must be satisfied with regard to time. – usually contains a time expression– may also contain a timing mark


14

Sample from Collaboration Diagram

Reminder: A sequence or collaboration diagram focuses on the time ordering of the messages that go back and forth between objects.

;Login Page ;Accountv:validateLogin(userID, password)

{v.executionTime < 3 sec}

v is the timing mark. The timing mark makes it easier to refer to that message elsewhere on the sequence diagram.

The expression between { } is the constraint.


15

Types of Timing

• balking message -- synchronous message but the sending object gives up on the message if the receiving object is not ready to accept it.– timing constraint {wait = 0}

• timeout message -- synchronous message but the sender waits only for a specified period for the receiver to get ready to accept the message– timing constraint {wait = 50 ms}


16

Threads & Concurrent Processes

• Working definition: a thread is a unit of computation that can be scheduled

• During design and design review, should explicitly consider the number of threads– Increasing it

• Can simplify certain algorithms/techniques• Increase the risk of sequencing problems

– Reducing it• Reduces sequencing problems• Makes software more rigid and maybe

inefficient


17

Concurrent Processes, Synchronization, & Notation

• During design inspections, verify presence of the helpful clauses in the UML state diagram

• Some languages provide a language keyword to automatically add the mechanism to prevent simultaneous access -- be sure it works as expected– In Java, the keyword synchronized is used on the

signature of a method to specify the need for a synchronization mechanism (see later slide)

• C++ requires explicit structures that each individual developer must construct

• In C++, creation of an instance of a monitor object indicates the location at which synchronization is needed


18

Java sample

public synchronized int get( ) {

while (available == false) {

try {

wait ( );

} catch (InterruptedException e) { }

}

available = false;

notifyAll( );

return contents;

}

public synchronized void put (int value) {

while (available == true) {

try {

wait ( );

} catch (InterruptedException e) { }

}

contents = value;

available = true;

notify ( );

}

Example from The Java Tutorial, Addison Wesley, Lesson 15, Threads of Control, pp. 285-325.


19

Dynamic TestingConcerns


20

Collections & Collaborators

Collections:Classes that maintain

associations with instances of other classes but never actually interact with those instances

Hold the objects they are handed and return them in specific orders or find them based on specific criteria

Test with techniques for primitive classes

Collaborators:

may be addressed directly (name)

or by a pointer or a reference

dynamic type of object may be different from static type associated

Pointers & references are polymorphic

Bound to an instance of any number of classes

pre & post conditions refer to states or values of collaborating objects


21

Testing Collection Classes• Test driver

– create instances passed as parameters in messages to collection being tested

– Ensure those instances are correctly built into and removed from the collection.

• Address capacity limitations– Test behavior under circumstances where it cannot

allocate memory to add the new item to itself• If defensive, use negative tests also

– If the collection class uses a structure (array, etc.), include test cases to fill it completely and attempt to add one more.

• Test sequences of operations


22

Collaborators at Program Interfaces

• Collaborating class– Post condition in a class’ interface refers to the

state of an instance and/or specifies that some attribute of that object is used or modified

• We assume operations are specified by pre-, post condition and class invariants

• Need to know if defensive design or design by contract– Changes how senders and receivers interact


23

Nondeterminism & Dynamic Testing

• There is no perfect way to do this but the following will help:• Begin dynamic testing with as “clean” a machine as possible

– document the environment– identify apps that must run while items-under-test are running– add basic set of typical unrelated apps

• For each test case (set of programs/ thread-generating processes)– Describe any modifications made to this “standard

environment”– Note the order in which processes are started– Include a debugger that allows verification of order in which

threads are created, executed, and deleted


24

Orthogonal array testing applied to OO

• Orthogonal array testing is a sampling technique to limit the explosion of test cases

• Define pair-wise combinations of a set of interacting objects because most faults result from interactions due to two-way interactions

• An orthogonal array is an array of values in which each column represents a factor.

• A factor is a variable in an experiment; in our case, a specific class family in the software sys

• Each variable takes on a certain set of values called levels; in our case, a specific class

• Each cell has an instance of the class, that is, a specific state of an object


25

OATS – orthogonal array testing system

• The factors (columns) are combined pair-wise rather than representing all possible combinations of the levels for the factors.

• Suppose we have three factors – A, B, C – and each has three levels – 1, 2, and 3

• How many possible combinations of these values are there?

• How many pair-wise combinations? That is, how many combinations where a given level appears exactly twice?

factor=class family; level=class; cell=instance


26

array factor:

class family

factor:

class family

level:

specific class

an instance of the specific class


level:

specific class



level:

specific class



a specific state

a specific state

a specific state


27

Excerpt from Fig. 6.16, M&S, 21 x 37

1 2 3 4 5 6

1 1 1 1 1 1 1

2 1 1 2 2 2 2

3 1 1 3 3 3 3

4 1 2 1 1 2 2

5 1 2 2 2 3 3

6 1 2 3 3 1 1

7 1 3 1 2 1 3

8 1 3 2 3 2 1

9 1 3 3 1 3 2

10 2 1 1 3 3 2

11 2 1 2 1 1 3


28

OATS – uses a balanced design

• Every level of a factor appears exactly the same number of times as every other level of that factor

• Think of the rows of a table as test cases (In example, 18 of the possible 27 are not being conducted.

• This is a systematic way of reducing the number of test cases.

• If we decide to add more later, we know which ones have not been run.

• Also logical – most errors are between pairs of objects rather than among several objects


29

OATS adequacy criteria

• Ability to vary how completely the software under test is covered

• Exhaustive – all combinations of all factors

• Minimal – only interactions between the base classes from each hierarchy are tested

• Random – tester haphazardly selects cases from several of the classes; not statistically random

• Representative – uniform sample ensures every class is tested to some level

• Weighted representative – adds cases to the representative approach based on relative importance or risk associated with the class


30

Testing off-the-shelf components –

• Investigate the limits of whatever specification for it that is available– Create one if none is available– Developers will need it in order to properly use the

component• Begin with extreme boundary values -- even incorrect• Do stress tests next• If worth continuing

– Treat as a class interacting with rest of your product– There is usually a main class that presents itself to

the user or other products. Base tests on that class.


31

Protocol testing –

• Protocols: – Sequences of method invocations*– Identify them by combining a method whose post

condition enables the precondition of another method

– Easier to see from a state diagram than from written pre- and post conditions.

• Investigates whether the implementation of a class satisfies its specification.

• Abstract to state diagram with interacting components


32

Protocol Interaction Test Suite

• Consider as the “class under test” the component’s class that presents an interface to the other components

• Each protocol represents a life cycle for objects from that class in combination with instances of classes in the interacting components.– Begin with initial states of a given two objects as

denoted in the state diagrams of the two corresponding classes

– A sequence of states for each object – Ending with the terminal states– A test case takes the two objects through one complete

sequence of methods


33

Protocol between two components

• The sequence of messages exchanged between them.

• A component should advertise – the services it provides (through its interface)– what it requires (through its preconditions)

• Should also describe the sequence in which these interactions are expected

• One set of test cases defined from the protocols should exercise the integration of two components through the complete protocol.


34

Example of protocol between two components

• Note: can be described using sequence diagram• “user” component sends asynchronous message to the

“hardware” component to start the action• Eventually hw that services multiple clients sends an

asynchronous message back that the action has been started• “User” wants hw to stop -- similar exchange• No notational convention but these four occur as a group

although 2 for one component and 2 for the other– Tester class plays role of the other component in a protocol– Java -- Tester object can pass itself as a parameter to the

component under test and then invoke the services specified in the protocol


35

Example: Timer Protocol – diagram on next slide• Trace through the state diagram to create a life cycle test.• Create the object• Send one or more attach (...) messages • Followed by enable ( ), disable ( ), and• delete ( )• This is a simple example.• A life cycle test case provides a test of the object in the

ways that it will interact with its client objects• When the object is one that interfaces with client objects

in another program, this provides a realistic test sequence.


36

McGregor & Sykes Timer

Disabled Enabled

Timing

notify( )

attach(TimerObserver)

enable( )

disable( )

detach(TimerObserver)

isEnabled( ) limit( ) size( )

notify()/observers->forAll(notify())

delete ( )


37

Test Patterns –

• Test patterns are design patterns for test software.– The pattern description explains the context

within which the pattern should be considered– Provides a set of forces that guide a trade-off

analysis– Explains how to construct the objects.

• When developer uses a certain design pattern, tester can use a certain test pattern to structure the test code.

• Binder gives several examples in some detail.


38

Test Design Patterns Presented in Binder

• Test Design Patterns are for designing test suites• Robert Binder presents test design patterns for over

30 architectures


39

Exceptions

• An alternative route for returning from a method that does not necessarily move to the next statement after the method is invoked.– The exceptional return value is an object and can

be arbitrarily complex– Where the exception is thrown varies based on

the depth of the aggregation hierarchy


40

Testing Exceptions -- class• Class level testing: verify that each of the methods

does throw the exceptions it claims to throw in the appropriate circumstance.– Coverage criteria – class throws every exception

contained in the specifications of the methods– At least one test case per exception– Handled in normal class testing because each

potential throw should be a postcondition clause


41

Role of test driver of exceptions -- class

• Test driver sets up conditions to create exception event.– Provides try block

• inside that is a stimulus that invokes the method that throws the specific exception.

– Exception caught by test driver• verifies correct exception

• Exceptions are objects and belong to classes– catch statements can use typing system to

verify that it is of the correct typeS


42

Testing Exceptions – integration

• Integration level -- test they are being – thrown at correct time – caught at the correct place

• During inspection of system-level design model– every user-defined exception instantiated

• should be traced to an object that will catch the exception

• using the sequence diagram for the scenario that throws the exception

• Exception handling can be model tested early


43

Testing interactions at system level

• Components can be so complex it’s easier to test them in the context of the application itself

• Which interactions are tested at system level?– Those that can also be verified at the system level– That is, can see the direct results of the tests– There must be a direct relationship between user

interface and the ability to view test results• By these criteria, many multi-program interactions

must be tested as class vs. program or class vs. class because program vs. program interactions may not be visible at the system level.


44

Books

• McGregor & Sykes, A Practical Guide to Testing Object-Oriented Software, Addison Wesley, 2001. ISBN 0-201-32564-0

• Scott, Kendall, UML Explained, Addison Wesley, 2001. ISBN 0-201-72182-1

• Binder, Robert V., Testing Object-Oriented Systems: Models, Patterns, and Tools, Addison Wesley 2000. ISBN 0-201-80938-9


45


• Orthogonal array testing is a sampling technique to limit the explosion of test cases

• Define pair-wise combinations of a set of interacting objects because most faults result from interactions due to two-way interactions



46



• A factor is – a variable in an experiment; in our case, a

specific class family in the software system– Or states of that class family

• Each variable takes on a certain set of values called levels (rows); in our case, specific classes or states of those classes

• Each cell is an instance of the class, that is, a specific object or a particular state


47

OATS – orthogonal array testing system

• The factors (columns) are combined pair-wise rather than representing all possible combinations of the levels for the factors.

• Suppose we have three factors – A, B, C – and each has three levels

• How many possible combinations of these values are there? 3 x 3 x 3

• How many pair-wise combinations? That is, how many combinations where a given level appears exactly twice?

factor=class family; level=class; cell=instance


48

How many pair-wise combinations?

• A B C• 1 1 1 3• 2 1 2 2• 3 1 3 1• 4 2 1 2• 5 2 2 1• 6 2 3 3• 7 3 1 1• 8 3 2 3• 9 3 3 2


49

Mapping the terms of OATS to OO sw

array factor:

class family

factor:

class family state

level:

specific class


A state of that instance

level:

specific class



level:

specific class



A state of that nstance




50

OATS – uses a balanced design

• Every level of a factor appears exactly the same number of times as every other level of that factor

• Think of the rows of a table as test cases • This is a systematic way of reducing the number of

test cases.• If we decide to add more later, we know which ones

have not been run.• Also logical – most errors are between pairs of

objects rather than among several objects


51

Look at elevator example (M&S, p. 230)

• Want to test interactions between elevator controller, elevator, and wall button-panel

• Suppose:

– Elevator can be basic, hi-speed, or express

– Elevator Controller can control basic, hi-speed, or express

– Wall button-panel can be one-button panel or two-button panel

• Elevator states: Moving to a floor, stopped, idle

• Elevator Controller states: idle, scheduling, dispatching

• Wall button-panel states?


52

Steps from M & S, p. 230-232

1. Identify all factors (class families in this case)

2. Determine levels for each factor (states)

3. Locate a standard orthogonal array that fits the problem

1. We have 2 class families that each have 3 levels and three states

2. We have one class family that has _a_ levels and _b_ states

That’s how many possible test cases? 3 x 3 x 3 x 3 x a x b


53

Excerpt from Fig. 6.16, M&S, 21 x 37

1 2 3 4 5 6

1 1 1 1 1 1 1

2 1 1 2 2 2 2

3 1 1 3 3 3 3

4 1 2 1 1 2 2

5 1 2 2 2 3 3

6 1 2 3 3 1 1

7 1 3 1 2 1 3

8 1 3 2 3 2 1

9 1 3 3 1 3 2

10 2 1 1 3 3 2

11 2 1 2 1 1 3


54

Need a way to map Fig 6.16 to this

Elevator ElevatorState

Elev Ctrl Elev Ctrl State

Wall Button Panl

Wall Button PnlStte

1

2

3

4

5

6

7

8


55

Mapping classes and states

• The class family with two levels (Wall Button Panel) maps to Column 1

• Its states map to Column 2• Column 3 can have a class family with three levels;

either elevator or elevator controller will do;– Map Elevator Controller family to column 3– Map its states to column 4

• What’s left?– Map Elevator family to column 5– Map its states to column 6


56

Mapping levels – Wall Button Panel

• For Wall Button Panel– 1 = 1-button panel – 2 = 2-button panel

• One-button Panel states (column 2)– 1 = off– 2 = on

• For two-button panel (column 2)– 1 = both off– 2 = 1 requesting– 3 = both requesting


57

Mapping levels – Elevator Controller

• Column 3 – elevator controller class family– 1 = basic– 2 = hi-speed– 3 = express

• Column 4 – elevator controller class states– 1 = idle– 2 = scheduling– 3 = dispatching


58

Mapping levels -- Elevator

• Column 5 = elevator class family– 1 = basic– 2 = hi-speed– 3 = express

• Column 6 = elevator class states– 1 = Moving to a floor– 2 = stopped– 3 = idle


59

Label Fig 6.16 with headers that “fit”

Wall-Button Panel

Wall-Button Pnl stte

Elev Ctrlr

Elev Ctrlr State

Elevator ElevatorState

1 1 1 1 1 1 1

2 1 1 2 2 2 2

3 1 1 3 3 3 3

4 1 2 1 1 2 2

5 1 2 2 2 3 3

6 1 2 3 3 1 1

7 1 3 1 2 1 3

8 1 3 2 3 2 1

9 1 3 3 1 3 2


60

When mapped to all of Fig 6.16

• Each row becomes a test case• Decode the level numbers for a row in the array

back to the individual lists for each factor• In Fig 6.16, there are actually two more columns.

– We can ignore them because we don’t need them.– Or we may decide we can include one more class

family and its states in the future


61

What about mismatches?

• When there is a difference in the number of levels, we can use a column that matches or exceeds the maximum

• What about the Wall-Button Panel at level 1 where there is only one button?

– That only has two states but there can be a 3 in column 2 associated with the 1 in the column 1

– We interpret the 3 as if it were either 1 or 2. What do you want to emphasize in the testing?

• Arbitrary assignment

• High frequency

• High risk


62

Other what-ifs?

• See M & S, p. 232


63

OATS adequacy criteria

• Ability to vary how completely the software under test is covered

• Exhaustive – all combinations of all factors

• Minimal – only interactions between the base classes from each hierarchy are tested

• Random – tester haphazardly selects cases from several of the classes; not statistically random

• Representative – uniform sample ensures every class is tested to some level

• Weighted representative – adds cases to the representative approach based on relative importance or risk associated with the class

testing interfaces in multi-program systems ecen5043 software engineering of multiprogram systems...

Documents

static testing

testing design slide

default model testing

thorough model testing

kinds of model testing

test execution

boulder slide

multiple processes