module 6: software verification and validation cseb233 fundamentals of software engineering badariah...

Module 6: Software Verification and Validation

CSEB233 Fundamentals of Software Engineering

Badariah Solemon 2011

Objectives• To introduce the fundamental concepts of software

verification and validation• To explain how to conduct software testing and when

to stop• To discuss several types of testing: unit testing,

integration testing, validation testing, and system testing

• To introduce the standard for software test documentation

• To introduce a set of techniques for the creation of test cases that meet overall testing objectives and the testing strategies


Verification & Validation• Is a whole software engineering process

– V & V must be applied at each framework activity in the software process.

– Verification refers to the set of tasks that ensure that software correctly implements a specific function.

– Validation refers to a different set of tasks that ensure that the software that has been built is traceable to customer requirements. Boehm [Boe81] states this another way:

• Verification: "Are we building the product right?" • Validation: "Are we building the right product?”


Verification & Validation(cnt’d)• Has two principal objectives:

1. Discover defects in a system;2. Assess whether or not the system is useful and

useable in an operational situation.• V&V should establish confidence that the

software is fit for purpose.– This does NOT mean completely free of defects.– Rather, it must be good enough for its intended

use and the type of use will determine the degree of confidence that is needed.


Verification & Validation(cnt’d)• V & V activities include:

– Technical reviews– Quality and configuration audits– Performance monitoring– Simulation– Feasibility study– Documentation review– Database review– Algorithm analysis– Development testing– Qualification testing– Acceptance testing– Installation testing


SQA activities*

Testing

Software Testing• The process of exercising a program with the

specific intent of finding errors prior to delivery to the end user.

• Must be planned carefully to avoid wasting development time and resources, and conducted systematically .

• What testing shows?


errorsrequirements conformance

performance

an indicationof quality

Who Tests the Software?


developerdeveloper independent testerindependent tester

Understands the system Understands the system

but, will test "gently"but, will test "gently"

and, is driven by "delivery"and, is driven by "delivery"

Must learn about the system,Must learn about the system,

but, will attempt to break itbut, will attempt to break it

and, is driven by qualityand, is driven by quality

Who Tests the Software?(cnt’d)• Misconceptions :

– The developer should do no testing at all– Software should be “tossed over the wall” to stranger who

will test it mercilessly– Testers are not involved with the project until it is time for it

to be tested• The developer and Independent Test Group (ITG)

must work together throughout the software project to ensure that thorough tests will be conducted.– An ITG does not have the “conflict of interest” that the

software developer might experience.– While testing is conducted, the developer must be available

to correct errors that are uncovered.Badariah Solemon 2011

Testing Strategy

• Identifies steps to be undertaken, when these steps are undertaken, how much effort, time, and resources will be required.

• Any testing strategy must incorporate: Test planning Test case design Test execution Resultant data collection and evaluation

• Should provide guidance for the practitioners and a set of milestones for the manager.


Testing Strategy (cnt’d)• Generic characteristics of software testing

strategies that have been proposed in the literature:1. To perform effective testing, you should conduct effective

technical reviews. • By doing this, many errors will be eliminated before testing

commences.2. Testing begins at the component level and works "outward"

toward the integration of the entire computer-based system. 3. Different testing techniques are appropriate for different software

engineering approaches and at different points in time.4. Testing is conducted by the developer of the software and (for

large projects) an independent test group.5. Testing and debugging are different activities, but debugging must

be accommodated in any testing strategy.


Overall Software Testing Strategy


System engineering

Analysis modelingDesign modeling

Code generation Unit testingIntegration testing

Validation testing

System testing

• Maybe viewed in the context of the spiral.

• Begins by ‘testing-in-the-small’ and move toward ‘testing-in-the-large’

Overall Software Testing Strategy (cnt’d)• Unit Testing

– focuses on each unit of the software (e.g., component, module, class ) as implemented in source code

• Integration Testing – focuses on issues associated with verification and program

construction as components begin interacting with one another• Validation Testing

– provides assurance that the software validation criteria (established during requirements analysis) meets all functional, behavioral, and performance requirements

• System Testing – verifies that all system elements mesh properly and that overall

system function and performance has been achievedBadariah Solemon 2011

Unit Testing• Focuses on assessing:

— internal processing logic and data structures within the boundaries of a component (module).

— proper information flow of module interfaces.— local data to ensure that integrity is maintained.— boundary conditions. — basis (independent) path.— all error handling paths.

• If resources are scarce to do comprehensive unit testing, select critical or complex module and unit test only those.


Unit Testing (cnt’d)


modulemoduleto beto betestedtested

test casestest cases

resultsresults

softwaresoftwareengineerengineer

boundary conditionsboundary conditions

error handling pathserror handling pathsindependent pathsindependent paths

local data structureslocal data structuresinterfaceinterface

Integration Testing• After unit testing of individual modules the

modules are combined together into a system.

• Question commonly asked once all modules have been unit tested:– “If they work individually, why do you doubt

that they’ll work when we put them together?”• The problem is “putting them together” –

interfacing. Examples:– Data can be loss across an interface– Global data structures can present problems– Subfunctions, when combined, may not

produce the desired function


Integration Testing (cnt’d)• Incremental integration testing strategies:

1. Bottom-up integration2. Top – down integration3. Regression testing4. Smoke testing


Bottom-up Integration• An approach where the lowest level modules are tested

first, then used to facilitate the testing of higher level modules. – The process is repeated until the module at the top of the hierarchy

is tested.– Top level modules are the most important yet tested last.

• Is helpful only when all or most of the modules of the same development level are ready.


Bottom-up Integration (cnt’d)


drivers are replaced one at a drivers are replaced one at a

time, "depth first"time, "depth first"

‘‘worker modules’ are grouped into worker modules’ are grouped into

builds and integratedbuilds and integrated

AA

BB

CC

DD EE

FF GG

clustercluster

The steps:1.Test D, E individually (using a dummy program called ‘driver’)2.Low-level components are combined into clusters that perform a specific software function.

Test C such that it call D (If an error occurs we know that the problem is in C or in the interface between C and D)

Test C such that it call E (If an error occurs we know that the problem is in C or in the interface between C and E)

3.The cluster is tested.4.Drivers are removed and clusters are combined moving upward in the program structure.

Top-down Integration• The steps:

1. Main/top module used as a test driver and stubs are substitutes for modules directly subordinate to it.

2. Subordinate stubs are replaced one at a time with real modules (following the depth-first or breadth-first approach).

3. Tests are conducted as each module is integrated.4. On completion of each set of tests and other stub is replaced

with a real module.5. Regression testing may be used to ensure that new errors not

introduced. • The process continues from step 2 until the

entire program structure is built


Top-down Integration (cnt’d)


top module istop module is

tested with stubstested with stubs

stubs are replaced one at stubs are replaced one at

a time, "depth first"a time, "depth first"

as new modules are integrated, as new modules are integrated,

some subset of tests is re-runsome subset of tests is re-run

AA

BB

CC

DD EE

FF GG

Example steps:1. Test A individually (use stubs for

other modules)2. Depending on the integration

approach selected, subordinate stubs are replaced one at a time with actual components In a ‘depth-first’ structure:

a. Test A such that it calls B (use stub for other modules) If an error occurs we know

that the problem is in B or in the interface between A and B

b. Replace stubs one at a time, ‘depth-first’ and re-run tests

Regression Testing• Focuses on retesting after changes are made

– Whenever software is corrected, some aspect of the software configuration (the program, its documentation, or the data that support it) is changed.

– Regression testing helps to ensure that changes (due to testing or for other reasons) do not introduce unintended behavior or additional errors.

• In traditional regression testing, we reuse the same tests.• In risk-oriented regression testing, we test the same areas

as before, but we use different (increasingly complex) tests.

• Regression testing may be conducted manually, by re-executing a subset of all test cases or using automated capture/playback tools.


Smoke Testing• A common approach for creating “daily builds”

for product software• Smoke testing steps:

– Software components that have been translated into code are integrated into a “build.”

– A build includes all data files, libraries, reusable modules, and engineered components that are required to implement one or more product functions.

– A series of tests is designed to expose errors that will keep the build from properly performing its function.

– The intent should be to uncover “show stopper” errors that have the highest likelihood of throwing the software project behind schedule.

– The build is integrated with other builds and the entire product (in its current form) is smoke tested daily.

– The integration approach may be top down or bottom up.Badariah Solemon 2011

Validation Testing• Focuses on uncovering errors at the

software requirements level.• SRS might contain a ‘Validation Criteria’

that forms the basis for a validation-testing approach

• Validation-Test Criteria:– Ensure that:

all functional requirements are satisfied, all behavior characteristics are achieved, all content is accurate and properly presented, all performance requirements are attained,

documentation is correct, and usability and other requirements are met.


Validation Testing (cnt’d)• An important element of the validation process is a

configuration review/audit– Ensure that all elements of the software configuration have been properly

developed, are cataloged, and have the necessary detail to strengthen the support activities.

• A series of acceptance tests are conducted to enable the customer to validate all requirements– To make sure the software works correctly for intended user in

his or her normal work environment. – Alpha test – version of the complete software is tested by

customer under the supervision of the developer at the developer’s site

– Beta test – version of the complete software is tested by customer at his or her own site without the developer being present


System Testing• A series of different tests to verify that

system elements have been properly integrated and perform allocated functions.

• Types of system tests:1. Recovery Testing2. Security Testing3. Stress Testing4. Performance Testing5. Deployment Testing


System Testing (cnt’d)1. Recovery Testing

forces the software to fail in a variety of ways and verifies that recovery is properly performed

2. Security Testing verifies that protection mechanisms built into a

system will, in fact, protect it from improper penetration

3. Stress Testing executes a system in a manner that demands

resources in abnormal quantity, frequency, or volume


System Testing (cnt’d)4. Performance Testing

test the run-time performance of software within the context of an integrated system

5. Deployment Testing examines all installation procedures and specialized

installation software that will be used by customers, and

all documentation that will be used to introduce the software to end users.


Software Test Documentation• IEEE 829 2008 Standard for Software Test

Documentation– IEEE standard that specifies the form of a set

of documents for use in eight defined stages of software testing

• The documents are:– Test Plan– Test Design Specification– Test Case Specification– Test Procedure Specification– Test Item Transmittal Report– Test Log– Test Incident Report– Test Summary Report


Software Test Documentation (cnt’d)• Test Plan: a management planning document that

shows:– How the testing will be done (including SUT (system under

test) configurations).– Who will do it– What will be tested– How long it will take (although this may vary, depending upon

resource availability).– What the test coverage will be, i.e. what quality level is

required• Test Design Specification:

– detailing test conditions and the expected results as well as test pass criteria.


Software Test Documentation (cnt’d)• Test Procedure Specification:

– detailing how to run each test, including any set-up preconditions and the steps that need to be followed

• Test Item Transmittal Report:– reporting on when tested software components have

progressed from one stage of testing to the next• Test Log:

– recording which tests cases were run, who ran them, in what order, and whether each test passed or failed

• Test Incident Report: – detailing, for any test that failed, the actual versus expected

result, and other information intended to throw light on why a test has failed.


Software Test Documentation (cnt’d)• Test Summary Report:

– A management report providing any important information uncovered by the tests accomplished, and including assessments of the quality of the testing effort, the quality of the software system under test, and statistics derived from Incident Reports.

– The report also records what testing was done and how long it took, in order to improve any future test planning.

– This final document is used to indicate whether the software system under test is fit for purpose according to whether or not it has met acceptance criteria defined by project stakeholders.


Test-case Design• Focuses on a set of techniques for the

creation of test cases that meet overall testing objectives and the testing strategies.

• These techniques provide a systematic guidance for designing tests that1. Exercise the internal logic and interfaces of

every software component/module2. Exercise the input and output domains of the

program to uncover errors in program function, behaviour, and performance.


Test-case Design (cnt’d)• For conventional application, software is tested

from two perspectives:1. ‘White-box’ testing

– focus on the program control structure (internal program logic).

– Test cases are derived to ensure that all statements in the program have been executed at least once during testing and all logical conditions have been exercised.

– Performed early in the testing process2. ‘Black-box’ testing

– Examines some fundamental aspect of a system with little regard for the internal logical structure of the software

– Performed during later stages of testingBadariah Solemon 2011

White-box Testing• Using white-box testing method, you may

derive test-cases that:1. Guarantee that al independent paths within

a module have been exercised at least once2. Exercise all logical decisions on their true and

false sides3. Execute all loops at their boundaries and

within their operational bounds 4. Exercise internal data structures to ensure

their validity• Example method: basis path testing.


White-box Testing(cnt’d)


Basis path testing:

• Test cases derived to exercise the basis set are guaranteed to execute every statement in the program at least one time during testing

Deriving Test Cases• Steps to derive the test cases (by applying

the basis path testing method):1. Using the design or code, draw a

corresponding flow graph. The flow graph depicts logical control flow using the

notation illustrated in next slide. Refer Figure 18.2 in page 486 - comparison

between a flowchart and a flow graph2. Calculate the Cyclometic Complexity V(G) of

the flow graph3. Determine a basis set of independent paths4. Prepare test cases that will force execution of

each path in the basis set.


Deriving Test Cases (cnt’d)


SequenceIF

WHILE

UNTIL

CASEIF

Flow graph notation:

Drawing Flow Graph: Example 1


void foo (float y, float a *, int n)

{

float x = sin (y) ;

if (x > 0.01)

z = tan (x) ;

else

z = cos (x) ;

for (int i = 0 ; i < x ; + + i)

{

a[i] = a[i] * z ;

Cout < < a [i];

}

}

1

2 3

4

5

8

R1

R2

R3

Predicate nodes

Predicate nodes

12

3

5

6

8

7

67



void foo (float y, float a *, int n)

{

float x = sin (y) ;

if (x > 0.01)

z = tan (x) ;

else

z = cos (x) ;

for (int i = 0 ; i < x ; + + i)

{

a[i] = a[i] * z ;

Cout < < a [i];

}

}

1

2 3

4

5

8

6,7

R1

R2

R3

Predicate nodes

Predicate nodes

12

3

5

6

8

7



See Figure 18.2 in page 486

Deriving Test Cases (cnt’d)• The arrows on the flow

graph, called edges or links, represent flow of control and are analogous to flowchart arrows.

• Area bounded by edges and nodes are called regions.

• When counting regions, we include the area outside the graph as region. Badariah Solemon 2011

edges

Region 1

Region 2

Region 3

Region 4

Region 5

Region 6

Deriving Test Cases: Example 1

Step 1: Draw a flow graph


elemArray [mid] != key

elemArray [mid] > key elemArray [mid] < key

1

2

3

4

5

6

7

8

9

14 10

11

12 13

bottom > top while bottom <= topbottom > top

elemArray[mid] = key

Predicate nodes

Predicate nodes

Predicate nodes

R1 R2

R3

R4

Deriving Test Cases: Example 1 (cnt’d)

Step 2: Calculate the Cyclomatic complexity, V(G)• Cyclomatic complexity can be used to count the

minimum number of independent paths.• A number of industry studies have indicated that

the higher V(G), the higher the probability or errors.

• The SEI provides the following basic risk assessment based on the value of code:


Cyclomatic Complexity Risk Evaluation

1 to 10 a simple program, without very much risk

11 to 20 a more complex program, moderate risk

21 to 50 a complex, high risk program

> 50 an un-testable program (very high risk)


• Ways to calculate V(G):1. V(G) = the number of regions of the flow

graph.2. V(G) = E – N + 2 ( Where “E” are edges &

“N” are nodes)3. V(G) = P + 1 (Where P is the predicate nodes

in the flow graph, each node that contain a condition)

• Example:1. V(G) = Number of regions = 42. V(G) = E – N + 2 = 16 – 14 + 2 = 43. V(G) = P + 1 = 3 + 1 = 4



Step 3: Determine a basis set of independent paths• Path 1: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14• Path 2: 1, 2, 3, 4, 5, 14• Path 3: 1, 2, 3, 4, 5, 6, 7, 11, 12, 5, …• Path 4: 1, 2, 3, 4, 6, 7, 2, 11, 13, 5, …

Step 4: Prepare test cases • Test cases should be derived so that all of these paths are

executed• A dynamic program analyser may be used to check that

paths have been executed


Deriving Test Cases: Example 2Refer pages 486 – 488

* Other topics related to testing conventional application such as control structure testing, black-box testing, model-based testing , patterns for software testing are described in pages 489- 508.


When to Stop Testing?• Testing is potentially endless.

– We can not test till all the defects are unearthed and removed -- it is simply impossible.

• At some point, we have to stop testing and ship the software. The question is when.

• According to Pan (1999):– Realistically, testing is a trade-off

between budget, time and quality. – It is driven by profit models.


When to Stop Testing?(cnt’d)

• The pessimistic, and unfortunately most often used approach is to stop testing whenever some, or any of the allocated resources -- time, budget, or test cases -- are exhausted.

• The optimistic stopping rule is to stop testing when either reliability meets the requirement, or the benefit from continuing testing cannot justify the testing cost.


Summary• Software testing plays an extremely important role in V&V,

but many other SQA activities are also necessary.• Testing must be planned carefully to avoid wasting

development time and resources, and conducted systematically.

• The developer and ITG must work together throughout the software project to ensure that thorough tests will be conducted.

• The software testing strategy is to begins by ‘testing-in-the-small’ and move toward ‘testing-in-the-large’

• The IEEE 829.2009 standard specifies a set of documents for use in eight defined stages of software testing.

• The ‘white-box’ and ‘black-box’ techniques provide a systematic guidance for designing test cases.

• We need to know when is the right time to stop testing.


References and credit• Contents in the slides are adapted from

the book and the slides that accompanied the book by R.S. Pressman, Software Engineering: A Practitioner’s Approach, 7th. Edition, McGraw Hill, 2009.

• Pan, J. (1999). Software Testing. Retrieved 25th January 2011 from http://www.ece.cmu.edu/~koopman/des_s99/sw_testing/


http://www.ece.cmu.edu/~koopman/des_s99/sw_testing/

http://www.ece.cmu.edu/~koopman/des_s99/sw_testing/

module 6: software verification and validation cseb233 fundamentals of software engineering badariah...

Documents

software testing

testing slide

validation testing

software developer

testing strategy

unit testing

integration testing

types of testing