january 27, 2002 ecen5033 university of colorado -- class testing 1 adequacy of a class test suite...

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

1

Adequacy of a Class Test Suite Ideally – exhaustively test each class Practically – impossible or too hard Worth it to exhaustively test some classes with

high risk Measures of adequacy to increase confidence that

we have tested enough state-based coverage constraint-based coverage

Constraints are pre and post conditions and the class invariant

code-based coverage


2

State-based coverage How many of the transitions in a state transition diagram

are covered by the test suite? “Covered” = touched at least once May reveal each transition was covered but test values

do not adequately cover value ranges If test cases were generated correctly from a state

transition diagram with typical values and good boundary values, the test cases will achieve adequate state-based coverage

If test cases were generated from pre- and post conditions, then it is useful to check them against the state transition diagram to ensure each transition is covered.


3

State-based coverage: object interaction Note how operations interact w.r.t. transitions

Current State

Input or event

Action Output Next State

State A Event 1 Output 1

State C

State B Event 2 Output 2

State C

State C Event 3 State D

Test cases for the transition from State C to State D may work if State C was reached from State A but not if State C was reached from State B. “State” is a function of history.


4

State-based coverage: transition pairs Concerning problem on previous page:

Check that the test cases cover all pairs of transitions in the state transition diagram.

In previous table, create test cases to test: Transition from State A to State C and

transition from State C to State D Transition from State B to State C and

transition from State C to State D In other words, the event sequences:

Event 1 followed by Event 3Event 2 followed by Event 3


5

Statechart for Elevator Control


6

Hierarchical statechart for Elevator Control


7

Portion enlarged


8

Constraint-based coverage How many pairs of pre- and post conditions have

been covered? Using the technique described earlier for

generating test cases from pre and post conditions, if one test case is generated to satisfy each requirement, then the test suite meets this measure


9

Constraint-Based coverage: object interaction

For each operation opn that is not an accessor operation, identify the operations op1, op2, etc. for which

their preconditions are met when the post conditions for opn hold.

That is, post condition(opn) satisfies (>=) precondition(op1), precondition(op2), etc.

Then execute test cases for operation sequences opn-op1, opn-op2, etc.


10

Class Diagram

Elevator Elevator Doors

Timer

Elevator Button

Up Floor Button

Floor

1 1 1 1

1

11

11

m

0…n

0…n

Travels-to

Travels-from

Opens-Closes Controls

Controls

Contains

velocity: ±m/sposition: meters

floor: integeron: Boolean

open: Booleanclosed: Booleanjammed: Boolean

time: seconds

floor: integerposition: meterson: Boolean

Down Floor Button

1

1

Contains

on: Boolean

Light

1on: Boolean

1

Turns-on

1

Turns-on 1Turns-on

1

1

Elevator Controller

1

0…n

ControlsRequests-elevator-of

Indicates-arrival-to


11

How do we complete the class test? States do not correspond 1-1 to methods or to

classes so the state transition diagram gives a different perspective

We want to adequately test the class Look at the class diagram and see which classes

send messages to it (“it” is the class under test) It could be difficult to create a driver to simulate

each class sending msgs to the class-under-test

If you only test each association in this application, it may be a weak level of testing for that class

Test each state transition, better yet, the pairs


12

Constraint coverage completes testing it

If you test class A calling every msg it can in class B, that’s more coverage but …

If you use existing classes to do that, it’s hard to get the right combination set up to make it happen

Alternative: Write a driver with sequences of messages to

access sequences of methods in the class you are testing.

Which sequences?


13

Which sequences??

Look at the post conditions of each method in the class you want to test

Look at preconditions of each Find sequences where

the Post(m1) >= pre(m2) the Post(m2) >= pre(m3) Send msgs to m1; m2; m3

Do this for all possible combinations


14

Completion check

See which associations were tested by comparing to the class diagram

See which state transitions were covered this way by comparing to the statechart

If some state transitions were missed or if some post/pre condition pairs or associations were missed, add a few tests


15

Where do the pre and post conditions come from?

Use cases?? The methods themselves? The class invariant?

Class TestingTesting Interactions, OATS

(Lecture 3 on Class Testing)

Software Engineering of

Standalone Programs

University of Colorado, Boulder


17

Overview Class Testing Testing Interactions between objects Testing Class Hierarchies


18

Testing Interactions Class Testing Testing Interactions between objects

Identifying & specifying object interactions Testing object interactions

Collection classes Collaborator classes Testing and Design Approach

Sampling COTS testing Patterns Testing exceptions

Testing Class Hierarchies


19

Testing Interactions with other classes An OO program is made of a collection of objects

that collaborate to solve a problem. Correct collaboration is necessary for program

correctness. In this section, we assume interactions are

sequential, not concurrent (that’s a subject for next semester)

Focus of interaction testing: Ensure that messaging occurs correctly between objects whose classes have already been tested separately.

Can happen embedded in application or in special environment using a test harness


20

What is an object interaction? Object interaction is:

request by a sender object to a receiver object to perform one of the receiver’s operations all of the processing performed by the receiver

to complete the request including a possible reply

Includes messages between an object and its components an object and other objects with which it is

associated


21

Object interaction impact During the processing of any single method invocation

on a receiving object Multiple object interactions can occur Want to consider impact of these on

the internal state of the receiving object those objects with which it has an association

Impact may be “no change” changes in attribute values in 1 or more objects

involved state changes in 1 or more objects including state changes of creation or deletion of

objects


22

Primitive and Non-Primitive Classes Interactions implied in a class spec where there

are references to other objects Primitive class can be instantiated and used

without needing to create any other instance Relatively few primitive classes in a program Since an OO program should model the objects in

a problem and all the relationships between them, non-primitive classes are common & essential

Non-primitive classes require the use of other objects in some or all of their operations

Identify the classes of those objects


23

Identifying Interactions Identify interactions by association relationships in the

class diagram regular associations aggregation composition

“Includes messages between an object and its components an object and other objects with which it is

associated “


24

Recognizing collaborators Collaborators may be addressed

directly, e.g. using a variable name by a pointer or a reference

dynamic type of the object may be different from the static type associated with the pointer

pointers and references are polymorphic, bound to an instance of any number of classes

Pre- and post conditions for operations in the public interface typically refer to states and/or specific attribute values of any collaborating objects


25

Collection and collaboration Collection class

maintains associations with instances of other classes but never actually interacts with those instances

that is, it never requests a service Collaborating class

Class with more extensive interactions Collection classes are less common


26

Testing Collection Classes What collection classes do

store references to these objects, representing one-to-many relationships between objects in a program

create instances of these objects delete instances of these objects

Specification refers to other objects does not refer to values computed based on

those objects


27

Identifying Collaborating Classes Non-primitive classes that are not collection

classes Use other objects in 1 or more operations as part of

their implementation When a post condition of an operation in a class’

interface refers to the state of an instance of an object and/or specifies that some attribute of that object is

used or modified,

that class is a collaborating class.


28

Choosing testing “chunk” size Number of potential collaborations can grow too

large very quickly During class test: test composing object’s

interactions with its composed attributes As successive layers of aggregation are integrated,

test the interaction between an object and its associated objects

Defect visibility issue – If too large a “chunk” is chosen, intermediate results may be incorrect but not seen at the level of test-result verification

The more complex the objects, the fewer one should integrate prior to a round of testing


29

What factors increase object complexity? Number of parameters for each method Number of methods Number of state attributes in each object

Trying to test a chunk that is too complex can result in defects that are hidden from the tests – for example, leaving an object in an inappropriate state that is not expected and therefore not looked at in the test.


30

Location of testing emphasis What are the implications of defensive and

contract design approaches when testing interactions?


31

Specifying interactions Remainder of slides assume

Operations defined by a class are specified by preconditions, post conditions, and class invariants

contract design approach


32

Contract vs. defensive approach & testing Contract More responsibility on

human designer Less class-level testing

due to fewer paths due to less error-checking code

MORE interaction testing required to ensure human designer complied with client side of contract using precondition constraints – (are preconditions in receiver met by sender?)

“illegal” for test cases to violate preconditions

Defensive More responsibility on

error-checking code More class-level testing

due to increased paths due to more code

Once the class-level error-checking code is tested, the interaction testing is simplified since there is no need to test if preconditions are met

Appropriate to violate preconditions to see if receiver catches that


33

Sampling Exhaustive testing: every possible test case

covering every combination of values Not reasonable much of the time

Want to select the ones that will find the faults in which we are the most interested Without prior information, random selection is

as good as we can do A sample is a subset of a population (e.g. all

possible test cases) that has been selected based on some probability distribution


34

Use profile (operational profile) Gave us a way to associate each use of the

system with a frequency These can be ranked by frequency Ranks can be turned into probabilities The higher the frequency of use, the larger the

probability of selection


35

Operational profile – stratified sample Stratified sample is a set of samples in which each

sample represents a specific subpopulation Example

Select test cases that exercise each component of the architecture

Divide a population of tests into subsets so that a subset contains all of the tests that exercise a specific component.

Sample on each subset independent of the others Need a basis for stratifying


36

A basis for stratification of test cases Use the actors from the use cases Select a sample of test cases from the uses of each actor

in the use cases Each actor uses some subset of the possible uses with

some frequency; rank by frequency Stratifying the test case samples by each actor provides a

way to increase the reliability of system Running selected tests

uses the system the way it will be used in typical situations;

finds defects likely to be found in typical use gives largest possible increase in reliability with least

effort


37

Test data generation for a range of values Generate test data first using the specification as

basis Select values based on boundary values

just within and just on the boundaries if contract just outside the boundaries also if defensive

Select values within intervals by sampling


38

Test data generation for a range of values, cont. Consider a random function over range 0.. N

such as int (random( ) * N) which generates a pseudo random value between 0 and 1 according to a uniform distribution. +: many different values will be tested over

many iterations Need to log actual values used in case of failure

so that the failed test case can be re-created. A randomly chosen value causing failure is

explicitly added to test suite and used to test the repaired software


39

Systematic sampling (+ boundary value testing)

Want to be able to increase level of coverage systematically. We use boundary values coupled with sampling

over ranges We never omit boundary value test data


40

Issues about sampling for interaction testing Sampling issues

Different states can cause two objects from the same class to behave differently

A class family is a subset of a larger family Possibility of combinatorial explosion in the

number of test configurations


41

Impact of inheritance hierarchy on s.t.d.’s

Each member of the family may have different states that can cause two objects from the same class to behave differently.

In families of classes, the state transition tables are related along the lines of the inheritance hierarchy. as we look down the inheritance hierarchy, there

will be the same number of states or more states in the derived class as in the base class

should cover the states defined for each class, emphasizing the new states added at that level in inheritance hierarchy.


42

Class families as subsets of others If a class family is a subset of a larger family, after

the tests are designed, they can be applied to any of the classes in the family, assuming the substitution principle has been followed during design (see next slide)


43

Substitution Principle Only the following changes are allowed in defining

the behavior associated with a new subclass: Preconditions for each operation must be the

same or weaker – less constraining – than those of the superclass

Post conditions for each operation must be the same or stronger – do at least as much as defined by the superclass

Class invariant – must be the same or stronger; add more constraints


44

Orthogonal array testing applied to OO Orthogonal array testing is a sampling

technique to limit the explosion of test cases Define pair-wise combinations of a set of

interacting objects because most faults result from interactions due to two-way interactions

An orthogonal array is an array of values in which each column represents a factor.


45

Orthogonal array testing applied to OO An orthogonal array is an array of values in

which each column represents a factor. A factor is

a variable in an experiment; in our case, a specific class family in the software system

Or states of that class family Each variable takes on a certain set of values

called levels (rows); in our case, specific classes or states of those classes

Each cell is an instance of the class, that is, a specific object or a particular state


46

OATS – orthogonal array testing system The factors (columns) are combined pair-wise

rather than representing all possible combinations of the levels for the factors.

Suppose we have three factors – A, B, C – and each has three levels

How many possible combinations of these values are there? 3 x 3 x 3

How many pair-wise combinations? That is, how many combinations where a given level appears exactly twice?

factor=class family; level=class; cell=instance


47

How many pair-wise combinations? A B C 1 1 1 3 2 1 2 2 3 1 3 1 4 2 1 2 5 2 2 1 6 2 3 3 7 3 1 1 8 3 2 3 9 3 3 2


48

Mapping the terms of OATS to OO sw

array factor:

class family

factor:

class family state

level:

specific class

an instance of the specific class

A state of that instance

level:

specific class



level:

specific class



A state of that nstance




49

OATS – uses a balanced design Every level of a factor appears exactly the same

number of times as every other level of that factor Think of the rows of a table as test cases This is a systematic way of reducing the number

of test cases. If we decide to add more later, we know which

ones have not been run. Also logical – most errors are between pairs of

objects rather than among several objects


50

Look at elevator example (M&S, p. 230) Want to test interactions between elevator

controller, elevator, and wall button-panel Suppose:

Elevator can be basic, hi-speed, or express Elevator Controller can control basic, hi-speed,

or express Wall button-panel can be one-button panel or

two-button panel Elevator states: Moving to a floor, stopped, idle Elevator Controller states: idle, scheduling,

dispatching Wall button-panel states?


51

Steps from M & S, p. 230-232

1. Identify all factors (class families in this case)

2. Determine levels for each factor (states)

3. Locate a standard orthogonal array that fits the problem

1. We have 2 class families that each have 3 levels and three states

2. We have one class family that has _a_ levels and _b_ states

That’s how many possible test cases? 3 x 3 x 3 x 3 x a x b


52

Excerpt from Fig. 6.16, M&S, 21 x 37

1 2 3 4 5 6

1 1 1 1 1 1 1

2 1 1 2 2 2 2

3 1 1 3 3 3 3

4 1 2 1 1 2 2

5 1 2 2 2 3 3

6 1 2 3 3 1 1

7 1 3 1 2 1 3

8 1 3 2 3 2 1

9 1 3 3 1 3 2

10 2 1 1 3 3 2

11 2 1 2 1 1 3


53

Need a way to map Fig 6.16 to thisElevator Elevator

StateElev Ctrl

Elev Ctrl State

Wall Button Panl

Wall Button PnlStte

1

2

3

4

5

6

7

8


54

Mapping classes and states The class family with two levels (Wall Button

Panel) maps to Column 1 Its states map to Column 2 Column 3 can have a class family with three

levels; either elevator or elevator controller will do; Map Elevator Controller family to column 3 Map its states to column 4

What’s left? Map Elevator family to column 5 Map its states to column 6


55

Mapping levels – Wall Button Panel For Wall Button Panel

1 = 1-button panel 2 = 2-button panel

One-button Panel states (column 2) 1 = off 2 = on

For two-button panel (column 2) 1 = both off 2 = 1 requesting 3 = both requesting


56

Mapping levels – Elevator Controller Column 3 – elevator controller class family

1 = basic 2 = hi-speed 3 = express

Column 4 – elevator controller class states 1 = idle 2 = scheduling 3 = dispatching


57

Mapping levels -- Elevator Column 5 = elevator class family

1 = basic 2 = hi-speed 3 = express

Column 6 = elevator class states 1 = Moving to a floor 2 = stopped 3 = idle


58

Label Fig 6.16 with headers that “fit”

Wall-Button Panel

Wall-Button Pnl stte

Elev Ctrlr

Elev Ctrlr State

Elevator ElevatorState

1 1 1 1 1 1 1

2 1 1 2 2 2 2

3 1 1 3 3 3 3

4 1 2 1 1 2 2

5 1 2 2 2 3 3

6 1 2 3 3 1 1

7 1 3 1 2 1 3

8 1 3 2 3 2 1

9 1 3 3 1 3 2


59

When mapped to all of Fig 6.16 Each row becomes a test case Decode the level numbers for a row in the array

back to the individual lists for each factor In Fig 6.16, there are actually two more columns.

We can ignore them because we don’t need them.

Or we may decide we can include one more class family and its states in the future


60

What about mismatches? When there is a difference in the number of levels,

we can use a column that matches or exceeds the maximum

What about the Wall-Button Panel at level 1 where there is only one button? That only has two states but there can be a 3 in

column 2 associated with the 1 in the column 1 We interpret the 3 as if it were either 1 or 2.

What do you want to emphasize in the testing? Arbitrary assignment High frequency High risk


61

Other what-ifs? See M & S, p. 232


62

OATS adequacy criteria Ability to vary how completely the software under

test is covered Exhaustive – all combinations of all factors Minimal – only interactions between the base

classes from each hierarchy are tested Random – tester haphazardly selects cases from

several of the classes; not statistically random Representative – uniform sample ensures every

class is tested to some level Weighted representative – adds cases to the

representative approach based on relative importance or risk associated with the class

january 27, 2002 ecen5033 university of colorado -- class testing 1 adequacy of a class test suite...

Documents