software test coverage testtcov 1 test coverage measures code coverage measurement levels of code...

26
Software Test Coverage TEST TCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch & Path Coverage White Box Test Coverage Data Coverage Measurement Levels of Data Coverage Class Coverage Interface Coverage State Coverage Data Table Coverage 2 3 6 7 5 4 8 9 10 13 Data Coverage Report Functional Coverage Measurement Levels of Functional Coverage Black Box Test Coverage Functional Test Coverage Predicted Error Coverage Inheriting Test Coverage Non-Transitivity of Test Coverage CRUD Object Access Coverage CRUD Object Access Coverage Matrix Document Coverage 14 15 18 19 17 16 20 21 22 11 23 12 24

Upload: carley-porritt

Post on 14-Dec-2015

248 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

Software Test Coverage

TEST TCOV

1 Test Coverage Measures

Code Coverage Measurement

Levels of Code Coverage

Method Coverage

Method, Statement, Branch & Path Coverage

White Box Test Coverage

Data Coverage Measurement

Levels of Data Coverage

Class Coverage

Interface Coverage

State Coverage

Data Table Coverage

2

3

6

7

5

4

8

9

10

13 Data Coverage Report

Functional Coverage Measurement

Levels of Functional Coverage

Black Box Test Coverage

Functional Test Coverage

Predicted Error Coverage

Inheriting Test Coverage

Non-Transitivity of Test Coverage

CRUD Object Access Coverage

CRUD Object Access Coverage Matrix

Document Coverage

Differences in Test Coverage

14

15

18

19

17

16

20

21

22

11 23

12 24

Page 2: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

Test Coverage Measures

TEST TCOV-1

Test Coverage

FunctionalCoverage

CodeCoverage

DataCoverage

Object Coverage

Attribute Coverage

Value Coverage

State Coverage

Function Coverage

Function Outcome Coverage

Function Chain Coverage

Function State Coverage

Method Coverage

Statement Coverage

Branch Coverage

Path Coverage

Page 3: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

Code Coverage Measurement

TEST TCOV-2

MethodCoverage

Path

Coverage

Coverage

Coverage

Branch

Statement Test Coverage = 100%could mean:

Every Method was invoked

Every Statement was executed

Every Branch was traversed

Every Control Path was traversed

Page 4: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

Levels of Code Coverage

TEST TCOV-3

Method coverage

Interface coverage

Statement coverage

Branch coverage

Path coverage

1

2

3

4

5

Increasing Reliability

Increasing Costs

Page 5: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

A_Retcode A_Method (Param_A1, Param_A2, Param_A3){

Invoke Method

B_Retcode B_Method (Param_B1, Param_B2, Param_B3){ if (Param_B1 > Attr_B9) return 1;

}Statement Coverage

C_Retcode C_Method (Param_C1, Param_C2, Param_C3){ if (Param_C1 > Attr_C9) return 1 else continue; return 0;}

Branch Coverage

1 Test

1 Test

2 Tests

Method Coverage

TEST TCOV-4

Page 6: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

F1

B1

B2

B3

A4

E2

E1

A2

A3

Method Coverage:(1 test case)

Statement Coverage:(2 test cases)

Branch Coverage:(3 test cases)

Path Coverage:(5 test cases)

F1, B1, E1 (Method Coverage)

F1, B1, E1 (Statement Coverage)F1, B1, B2, A2, B3, A3, A4, E2

F1, B1, E1 (Branch Coverage)F1, B1, B2, A2, B3, A3, A4, E2F1, B1, B2, B3, A4, E2

F1, B1, E1 (Path Coverage)F1, B1, B2, A2, B3, A3, A4, E2F1, B1, B2, A2, B3, A4, E2F1, B1, B2, B3, A3, A4, E2F1, B1, B2, B3, A4, E3

Test Cases

Method, Statement, Branch and Path Coverage

TEST TCOV-5

Page 7: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

C1 = All Methods are executed at least once.(Method Coverage)

C2 = All Statements of all Methods are executed at least once. (Statement Coverage)

C3 = All Branches of all Methods are executed at least once.(Branch Coverage)

D1 = All Attributes of an Object are altered. (Attribute Coverage)

D2 = All Attributes of an Object take on all specified values.(Value Coverage)

D3 = All Combinations of all values occur.(State Coverage)

S1 = All Messages are sent at least once.(Interface Coverage)

S2 = All Parameters of all Messages are altered. (Parameter Coverage)

S3 = All Combinations of all Parameters occur.(Interface State Coverage)

White-Box Coverage Measures (Class Test)

TEST TCOV-6

Page 8: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

Data Coverage Measurement

TEST TCOV-7

ObjectCoverage

State

Coverage

Coverage

Coverage

Value

Attribute Test Coverage = 100%could mean:

Every Object was instantiated

Every Attribute was altered

Every relevant value was set

Every possible state was generated.

Page 9: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

Levels of Data Coverage

TEST TCOV-8

1

2

3

4

5

Increasing Reliability

Increasing Costs

Window/File/DB Coverage

Object Coverage

Attribute Coverage

Value Coverage

State Coverage

Page 10: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

Object StatesMethods

AccountNumberAccountOwner

AccountBalanceAccountStatus

CreditLimit

Open_Account

Deposit_Money

Withdraw_Money

Transfer_Money

Close_Account

Account non existant

AccountBalance = 0

AccountBalance = plus

AccountBalance = minus,but < CreditLimit

AccountBalance = minusand > CreditLimit

Account closed

Attributes

How many test cases are required to test all Methods with all Object states?

Class Coverage

TEST TCOV-9

Page 11: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

Parameter Coverage

= the Interface is generated one time with a repesentative value for each individual Parameter

Value Coverage

= the Interface is generated as many times as there are representative values for each Parameter.

State Coverage

= the Interface is generated as many times as there are combinations of representative values for all Parameters.

Interface Coverage

TEST TCOV-10

Page 12: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

All Paths

All Paths

All Branches

All Branches

AllState

Transitions

AllState

Transitions

AllStates

AllStates

AllEvents

AllEvents

Subsumption Hierarchy

State Coverage

TEST TCOV-11

Page 13: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

1 OBJECT-A

2 ATTR-A1 'A' 'B' 'C'2 ATTR-A2 100 200 300 4002 ATTR-A3 'XYZ' 'ABC' 'EFG'2 ATTR-A4 -1 0 +1

1 PANEL-B

2 FIELD-B1 50 100 1502 FIELD-B2 100 -50 02 FIELD-B3 'XY' 'AB' 'EF'2 FIELD-B4 'B' 'A' 'C'

Value Coverage:

Every relevant Value of every Attribute & Field should be set at least one time.(you need as many Objects as the longest array of values)

Data Table Coverage

TEST TCOV-12

Page 14: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

Data Coverage Report

TEST TCOV-13

| DATA COVERAGE REPORT | | MODUL NAME: SVVL104 DATUM: 98.05.17 |+--------------------------------------------------------------------------------------------+| DATA NAME |RECORD NAME |BASE NAME|PREDIC.|INPUT|OUTPUT|NOT USED|+--------------------------------------------------------------------------------------------+| LISTNUMMER |FILLER |SVVL104 | | * | * | * || K-ENDE-DER-LISTE |ALLGEMEINE-VIS-KONSTANTEN|SVVL104 | | * | | * || CURRENT-DATE-MONAT |ALLGEMEINE-VIS-VARIABLEN |SVVL104 | | | * | || CURRENT-DATE-TAG |ALLGEMEINE-VIS-VARIABLEN |SVVL104 | | | * | || CURRENT-DATE-JAHR |ALLGEMEINE-VIS-VARIABLEN |SVVL104 | | | * | || SH-ENDE-DER-LISTE |ALLGEMEINE-VIS-SCHALTER |SVVL104 | * | | * | || SUMME-AWGR-VERK-MG |LISTSTEUERDATEN |SVVL104 | | | * | || FIRMA |ABSATZWEGGRUPPENSTATISTIK|SVVL104 | | * | * | || TEXT-ENDE-DER-AUSWERTUNG |ABSATZWEGGRUPPENSTATISTIK|SVVL104 | | | * | || FIRMA-GRUPPE |LISTPARAMETER |SVVL104 | | * | * | || LISTENNUMMER |LISTPARAMETER |SVVL104 | * | * | * | || FIRMA-GRUPPE |LISTPARAMETER |SVVL104 | | | * | || LISTENNUMMER |LISTPARAMETER |SVVL104 | | | * | |+--------------------------------------------------------------------------------------------+| || TOTAL NUMBER OF (NOT AUTOMATIC) DATA ITEMS REFERENCED = 13 || NUMBER OF (NOT AUTOMATIC) DATA ITMES USED = 11 || DATA COVERAGE RATION = 85% |

S O F T E S T

Page 15: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

Functional Coverage Measurement

TEST TCOV-14

FunctionCoverage

Function State

Coverage

Coverage

Coverage

Function Chain

Function Outcome

% of Application Functions

Page 16: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

Levels of Functional Coverage

TEST TCOV-15

Every application use case is executed at least once.

Every application function is executed at least once.

Every application function outcome is produced at least once.

Every application function sequence is traversed at least once.

Every possible combination of application functions is tested.

1

2

3

4

5

Increasing CostsIncreasing Reliability

Page 17: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

F1 = All specified Functions are tested at least once.

F2 = All specified Functions are tested with all valid Arguments at leastone time.

F3 = All specified Functions are tested with all valid Arguments and all expected Results at least once.

Results (n) = Function (Arguments (m))

Result (i) = Function (Argument (j))

i = 1 to n j = 1 to m

Black-Box Test Coverage (Component Test)

TEST TCOV-16

Page 18: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

Functional Test Coverage

TEST TCOV-17.1

TRAN:In|OutART_NR:999Amount:999Use Case Coverage = > 1 Test Case

TRANART_NRAmount

One update transaction

Function Coverage = > 2 Test Cases

> one Increase> one Decrease

TRANSART-NRMenge

Function Outcome Coverage

= > 6 Test Cases

> one Increase without Storno> one Increase with Storno of an existing order, generated earlier> one Increase with an error message> one Decrease without an order> one Decrease which generates an order> one Decrease with an error message

Page 19: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

TEST TCOV-17.2

TRANSART_NRAmount

Function State Coverage = > 10 Test Cases

> an Increase with invalid Art_Nr.> an increase with invalid amount> an increase to a non existing article> an increase with no storno> an increase with storno of existing order> a decrease with invalid Art_Nr.

> a decrease with invalid amount> a decrease to a non existing article> a decrease without an order> a decrease which generates an order> an invalid transaction

Page 20: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

Predicted Error Coverage

TEST TCOV-18

100

200

300

400

500

1 2 3 4 5 6

M o n t h s

N u

m b

e r

o

f

E r

r o

r s

Specification Errors

Design Errors

CodingErrors

UnitTest

Integration Test

System Test

Page 21: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

Base Class

Derived Class

Is tested first with all possible variations

Inherits test coverage ofthe base Class.

Statement,Branch,Method & State Coverage

Inheriting Test Coverage

TEST TCOV-19

Page 22: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

BaseClass

SubClassX

BaseClass

SubClassY

BaseClass

SubClassZ

Environment X Environment Y Environment Z

The Base Class must be retested in every new target Environment.

According to Perry/Kaiser Study of 1990

Non Transitivity of Test Coverage

TEST TCOV-20

Page 23: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

For each Object Type Create = 4 Test Cases e.g. Account Retrieve

UpdateDelete

For each Object State Create = 16 Test Cases e.g. plus Account Retrieve minus Account Update overdrawn Account Delete locked Account

Testing the Creation, Retrieval, Update and Deletion of persistent Objects(relative to one, several or all Object Types and all Object States)

CRUD Object Access Coverage

TEST TCOV-21

Page 24: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

Use Cases

A_UseCase

B_UseCase

C_UseCase

D_UseCase

E_UseCase

N_UseCase

C R U D C R U D C R U D

Test Object A_Object B_Object C_Object

According to R. Binder, 1999

CRUD Object Access Coverage Matrix

TEST TCOV-22

Page 25: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

Class test Component test System test

Requirements Documentation

User Documentation

Use case Documentation

Application Object Model

System Object Model

Object Interaction Model

Interface Specification

Class Specification

Method Specification

X

X

X X

X X

X

X

X X

X

X

Software Document Software Testing Level

Document Coverage

TEST TCOV-23

Page 26: Software Test Coverage TESTTCOV 1 Test Coverage Measures Code Coverage Measurement Levels of Code Coverage Method Coverage Method, Statement, Branch &

Unit = Procedural ModuleUnit-Testing = Module TestWhite-Box Test is in the foregroundData are shared (global)Functional DecompositionFunctional CouplingData and Call CouplingSub Program calls are stubbedEntity/Relationship-DiagramsData Flow DiagramsStructure DiagramsReuse is secondaryDevelopment is TOP-DOWNWaterfall Project Model

Unit = Classe or ObjektUnit-Testing = Class TestGrey-Box Test is in the foregroundData are encapsulatedClass Inheritance and AssociationObject CouplingInheritance and Message CouplingForeign Methods are invokedObject Sequence DiagramsObject State Transition Diagrams Object Collaboration DiagramsReuse is primaryDevelopment is BOTTOM-UPIterative Project Model

Procedural Software Object-orientierted Software

Differences between procedural and object-oriented Software

Differences in Test Coverage

TEST TCOV-24