Software Testing Using Software Testing Using Model ProgramModel Program

DESIGN BYDESIGN BY

HONG NGUYEN & SHAH RAZAHONG NGUYEN & SHAH RAZA

Dec 05, 2005Dec 05, 2005

OverviewOverview Software Testing Using Model ProgramSoftware Testing Using Model Program

Briefly introduction of M-mp testingBriefly introduction of M-mp testing What is M-mp testing?What is M-mp testing? How it work?How it work? Why we need program base testingWhy we need program base testing

The problem of software testingThe problem of software testing Oracle testOracle test

N-Version programmingN-Version programming What is N-Version Programming?What is N-Version Programming? DisadvantagesDisadvantages AdvantagesAdvantages

Overview (continues)Overview (continues)

M-mp testingM-mp testing Framework for M-mp testingFramework for M-mp testing Asymptotic behavior of M-mp testingAsymptotic behavior of M-mp testing Input domain partitioningInput domain partitioning An M-mp ExperimentAn M-mp Experiment

Background and motivationBackground and motivation Selecting the applicationSelecting the application Phases of the experimentPhases of the experiment

ConclusionConclusion

IntroductionIntroduction What is M-mp testing?What is M-mp testing?

Testing using M Model programTesting using M Model program Abbreviated to ‘M-mp testing’Abbreviated to ‘M-mp testing’ It is alternative to software testing base on It is alternative to software testing base on

manual outcome predictionmanual outcome prediction M-mp testing is base on N-Version programming M-mp testing is base on N-Version programming

technique technique Which is apply specially in the well-known software Which is apply specially in the well-known software

fault tolerant technique.fault tolerant technique.

Introduction (Continues)Introduction (Continues)

How it workHow it work Model program implements selected part of functional Model program implements selected part of functional

specification of the software to be tested.specification of the software to be tested. The M-mp testing strategy requires that M (M>= 1) The M-mp testing strategy requires that M (M>= 1)

Model programs as well as the program under test, should be Model programs as well as the program under test, should be independently developedindependently developed

P and M model program are then subjected to the same test P and M model program are then subjected to the same test data. data.

Difference analysis is conducted on the outputs and Difference analysis is conducted on the outputs and appropriate corrective action takenappropriate corrective action taken

P and the M model program jointly constituted an P and the M model program jointly constituted an approximate test oracle (out come correct or not)approximate test oracle (out come correct or not)

Introduction (Continues)Introduction (Continues)

Why we need program base testingWhy we need program base testing For small number of test data, it may be For small number of test data, it may be

feasible to work out the expected output feasible to work out the expected output manually manually

For thorough testing which required large For thorough testing which required large test datasets manual testing could be time test datasets manual testing could be time consuming and error proneconsuming and error prone That is why M-mp base testing may be more That is why M-mp base testing may be more

feasible for those kinds of data setfeasible for those kinds of data set

The problem of software testingThe problem of software testing

The difficult task in software testing is to assess the The difficult task in software testing is to assess the correctness to the outcome of a program that is correctness to the outcome of a program that is subjected to particular test inputsubjected to particular test input

In testing theory, the mechanism to adjudicate on In testing theory, the mechanism to adjudicate on whether or not an output is correct, somewhat whether or not an output is correct, somewhat neglected this problem is referred to as “the oracle neglected this problem is referred to as “the oracle problem.problem.

Testing theory concern with the choice of test and Testing theory concern with the choice of test and testing methods, usually ignore the oracle problems.testing methods, usually ignore the oracle problems.

The problem of software testing The problem of software testing (continues)(continues)

Oracle TestOracle Test It means of determining whether a program passed It means of determining whether a program passed

or failed a test.or failed a test.

The problem of software testing The problem of software testing (continues)(continues)

Oracle test (continues)Oracle test (continues) P is the program to be tested P is the program to be tested Test strategy determines the set of test data to be usedTest strategy determines the set of test data to be used To get the test result we compare P’s output with the oracle To get the test result we compare P’s output with the oracle

‘s output‘s output For small out put oracle manually base oracle technique is feasibleFor small out put oracle manually base oracle technique is feasible But for large industry base application manually base oracle testing But for large industry base application manually base oracle testing

is not feasibleis not feasible One of alternative of manually base oracle is N- Version One of alternative of manually base oracle is N- Version

programmingprogramming

N-Version ProgrammingN-Version Programming

N-Version is comprises N independently written N-Version is comprises N independently written version of the softwareversion of the software

All has the same functional specificationAll has the same functional specification At runtime voting base on majority agreement is used At runtime voting base on majority agreement is used

to decided the probable outputto decided the probable output DisadvantageDisadvantage::

Concern in N-Version systems is the fact that correlated Concern in N-Version systems is the fact that correlated failures may limit the practical gain in reliabilityfailures may limit the practical gain in reliability

N-Version systems should not be casually assumed to be N-Version systems should not be casually assumed to be completed reliable.completed reliable.

N-Version (continues)N-Version (continues)

Disadvantage (continues)Disadvantage (continues) Is the increased development cost because at least three Is the increased development cost because at least three

version being required for a voting system to workversion being required for a voting system to work AdvantageAdvantage

N-Version design appear to offer more reliability that N-Version design appear to offer more reliability that we can gain from any other waywe can gain from any other way

Even though the cost of failure is high:Even though the cost of failure is high: It would be more cost effective to build N-Version systems It would be more cost effective to build N-Version systems

rather than focusing on building ‘one good version’rather than focusing on building ‘one good version’

M-mp TestingM-mp Testing

Framework for M-mp testingFramework for M-mp testing

M-mp Testing (continues)M-mp Testing (continues)

Framework (continues)Framework (continues) In M-mp testing P is primary program (the program under In M-mp testing P is primary program (the program under

test) test) mp1 –mpmp1 –mpMM are M so called ‘model program of P are M so called ‘model program of P M-mp testing treat P and mp1-mpM-mp testing treat P and mp1-mpMM to be same test input to be same test input Any disagreement on the output indicates the presence of Any disagreement on the output indicates the presence of

specification defects Or software faults in at least one of specification defects Or software faults in at least one of programprogram

Once defects are detected and removed. The program are Once defects are detected and removed. The program are re-runre-run

M-mp Testing (continues)M-mp Testing (continues) Framework (continues)Framework (continues)

The cycle is repeat until all disagreements for a particular The cycle is repeat until all disagreements for a particular

dataset are resolveddataset are resolved.. Asymptotic behavior of M-mp testingAsymptotic behavior of M-mp testing

Difference between the M-mp and manual approaches to Difference between the M-mp and manual approaches to testing is the cost of outcome verificationtesting is the cost of outcome verification

M-mp Testing (continues)M-mp Testing (continues) Asymptotic behavior of M-mp testing (continues)Asymptotic behavior of M-mp testing (continues)

On the graph: the cost of M-mp outcome verification is On the graph: the cost of M-mp outcome verification is high initially but it increase only slightly.high initially but it increase only slightly.

The cost growth is modeled as linearThe cost growth is modeled as linear In manual approach, the growth of the output verification In manual approach, the growth of the output verification

cost may be expected to be much steeper.cost may be expected to be much steeper. Input domain partitioningInput domain partitioning

Divided into a standard domain and an exception domainDivided into a standard domain and an exception domain Exception domain may be split into an incomplete data domain and Exception domain may be split into an incomplete data domain and

an invalid data domain.an invalid data domain. The standard domain consists of inputs that a program can process The standard domain consists of inputs that a program can process

‘as is’‘as is’

M-mp Testing (continues)M-mp Testing (continues)

Input domain partitioning (continues)Input domain partitioning (continues) The invalid domain contains the input that a The invalid domain contains the input that a

program can not process it should be rejected with program can not process it should be rejected with appropriate error messageappropriate error message

The incomplete data domain is ‘between’ the The incomplete data domain is ‘between’ the standard and invalid data domainsstandard and invalid data domains

It is made up of inputs that the program has to It is made up of inputs that the program has to completed with default values before processing themcompleted with default values before processing them

An M-mp ExperimentAn M-mp Experiment Background and motivationBackground and motivation

M-mp Testing (continues)M-mp Testing (continues) Background and motivation (continues)Background and motivation (continues)

The primary program is a scheduling application, a non-The primary program is a scheduling application, a non-interactive data processing subsystem of a large interactive data processing subsystem of a large logistics management systemlogistics management system

The program can be characterized as algorithmically The program can be characterized as algorithmically complexcomplex

In practice, the testing process in the company was well In practice, the testing process in the company was well defined and supported by standards and procedures but defined and supported by standards and procedures but had several drawbackhad several drawback

Designing test cases based on manual outcome Designing test cases based on manual outcome prediction was not only difficult, error prone and time prediction was not only difficult, error prone and time consumingconsuming

But consequently also unattractive and demoralizingBut consequently also unattractive and demoralizing

M-mp Testing (continues)M-mp Testing (continues) Selecting the applicationSelecting the application

The scheduling application was chosen as the primary program for The scheduling application was chosen as the primary program for following reasonsfollowing reasons

Developer was not familiar with the scheduling domain and applicationDeveloper was not familiar with the scheduling domain and application The cost of developing from scratch could be assesses more The cost of developing from scratch could be assesses more

authenticallyauthentically The risk of correlated failures caused by common design error was The risk of correlated failures caused by common design error was

reducesreduces scheduling application is representative of the family of operation scheduling application is representative of the family of operation

management programmanagement program The time the experiment started, the scheduling subsystem had been in The time the experiment started, the scheduling subsystem had been in

use for more than yearuse for more than year This seem a good opportunity to test the fault detection ability of M-mp This seem a good opportunity to test the fault detection ability of M-mp

testing, as the likelihood of finding defect was expected to be lowtesting, as the likelihood of finding defect was expected to be low

M-mp Testing (continues)M-mp Testing (continues)

Phase of the experimentPhase of the experiment

M-mp Testing (continues)M-mp Testing (continues)

Phase of the experimentPhase of the experiment The experiment entailed a test design, a test execution and The experiment entailed a test design, a test execution and

test evaluation phase as depictedtest evaluation phase as depicted Test design phase comprised test data selectionTest design phase comprised test data selection

Which includes activities related to test data generation, Model Which includes activities related to test data generation, Model program design and model program implantationprogram design and model program implantation

The test execution phase includes activities relating to The test execution phase includes activities relating to setting up the test environment and executing the programsetting up the test environment and executing the program

The test evaluation phase includes the disagreement The test evaluation phase includes the disagreement analysis procedures whereby disagreements between the analysis procedures whereby disagreements between the program need to be arbitrated.program need to be arbitrated.

ConclusionConclusion

M-mp approach could test a scheduling M-mp approach could test a scheduling program more adequately than manually program more adequately than manually designed test and at lower costdesigned test and at lower cost

M-mp testing should be judged on a case by M-mp testing should be judged on a case by case basiccase basic

M-mp approach will tend to be better option M-mp approach will tend to be better option than the manual one when the number of tests than the manual one when the number of tests required to achieve a particular adequacy level required to achieve a particular adequacy level become largebecome large

QuestionQuestion

FinishedFinished Please be easy on us.Thank youPlease be easy on us.Thank you