8. fault tolerance in software 8.5 construction of acceptance tests goal goal: describe the types...

8. Fault Tolerance in Software

8.5 Construction of Acceptance Tests

GoalGoal: describe the types and selection criteria for acceptance tests

Two levelsTwo levels where acceptance tests can be performed:

The higher-levelhigher-level: tests if the outputs of the program are consistent with the functional requirements (functional acceptance test).

The lower-levellower-level: tests sections of the code to ensure that key variable and functions have been properly executed.

High-level tests are always requiredalways required!



Acceptance tests may be devised such that they test for what a

program should do or for what a program should not do:

e.g., acceptance tests in a SW controlling an automated rail transit

system can be written to test for conformance to specified

velocities at each location on the route or for violation of safety

rules (not exceed a certain velocity around curves or in the

vicinity of a station).



Acceptance tests fall into four categoriesfour categories , which serve as a useful

starting point for the designer confronting a section of critical

code:

Satisfaction RequirementsSatisfaction Requirements

Accounting TestsAccounting Tests

Reasonableness TestsReasonableness Tests

Computer Run-Time TestsComputer Run-Time Tests




An example: one the the simplest forms is the use

in the inversion of mathematical operations.




Devising satisfaction of requirement tests may be a difficult Devising satisfaction of requirement tests may be a difficult

and subtle problemand subtle problem::

Another example: at the completion of a sort program, the acceptance test

checks if the elements are in uniformly descending order and if the # of

elements in the sorted set is equal to the # of elements in the original set.

This test is not complete. Changes in an element during execution

would not be detected. An additional test ensuring that every

element in the sorted set appeared in the unsorted set should be

rejected because of run-time requirements.




Text editing systems, compilers, and similar programs, tests Text editing systems, compilers, and similar programs, tests

for satisfaction of requirements constitute the most for satisfaction of requirements constitute the most

promising approach.promising approach.



Accounting ChecksAccounting Checks

Suitable for transaction-oriented applications with simple Suitable for transaction-oriented applications with simple

mathematical operations, such as airline reservation mathematical operations, such as airline reservation

systems, library records, and bank transactions.systems, library records, and bank transactions.

The simplest form is the Checksum, where all records of a

particular program can be compared between source and

destination.




Detect SW failures by use of precomputed ranges, expected Detect SW failures by use of precomputed ranges, expected sequences of program states, or other relationships that are sequences of program states, or other relationships that are

expected to prevail.expected to prevail.

The difference between The difference between Satisfaction of Requirements TestsSatisfaction of Requirements Tests and and Reasonableness TestsReasonableness Tests is that the latter are based on is that the latter are based on physical physical

constraintsconstraints while the former type are based on while the former type are based on logical or logical or mathematical relationshipsmathematical relationships..




E.g. (1), determination of the true airspeed (E.g. (1), determination of the true airspeed (a computed quantitya computed quantity) from the ) from the indicated airspeed (indicated airspeed (a sensed quantitya sensed quantity) in a flight control system.) in a flight control system.

The first level of acceptance test is based on a precomputed range is that the The first level of acceptance test is based on a precomputed range is that the speed must be within the structural capabilities of the airframe (140 to speed must be within the structural capabilities of the airframe (140 to

1,100 km/h) for a commercial subsonic aircraft. 1,100 km/h) for a commercial subsonic aircraft.

Thus, if the true airspeed is outside this range, there is something wrong with Thus, if the true airspeed is outside this range, there is something wrong with either the sensor or the computer (an additional possiblity is that tha either the sensor or the computer (an additional possiblity is that tha

aircraft is out of control).aircraft is out of control).




FurFurtther refinementsher refinements on this test would use a on this test would use a reasonable range of changes to reasonable range of changes to true airspeedtrue airspeed: if changes between the current airspeed and the previous : if changes between the current airspeed and the previous value value indicate accelerations beyond the aircraft limitindicate accelerations beyond the aircraft limit, then an abnormal , then an abnormal

condition exists:condition exists:

E.g., if the previous true airspeed is E.g., if the previous true airspeed is 1,000 km/h1,000 km/h and the subsequent and the subsequent calculation (next tenth of a second) results in an airspeed of calculation (next tenth of a second) results in an airspeed of 1,020 km/h1,020 km/h, ,

the accetance test will detect an the accetance test will detect an error conditionerror condition because the implied because the implied acceleration is almost acceleration is almost 6g6g!!




A second example of Reasonableness Test is based on A second example of Reasonableness Test is based on progression progression between subsequent statesbetween subsequent states: :

e.g., in a e.g., in a telephone switching systemtelephone switching system, it is not reasonable to proceed , it is not reasonable to proceed from a from a connected stateconnected state to a to a ringing stateringing state or or line-busy stateline-busy state. .

However, such a test is However, such a test is not exhaustivenot exhaustive, it would not detect the , it would not detect the premature terminationpremature termination of a connection. of a connection.



Run-Time ChecksRun-Time Checks

Consist if those provided by most current computers as continuous Consist if those provided by most current computers as continuous execution sequences which are often HW implemented and/or execution sequences which are often HW implemented and/or

embedded in special SW or OS.embedded in special SW or OS.

E.g., tests that detect anomalous states such as divide-by-zero, E.g., tests that detect anomalous states such as divide-by-zero, overflow, underflow, undefined operation code, unauthorized overflow, underflow, undefined operation code, unauthorized

entries to subroutines, and write-protection violations.entries to subroutines, and write-protection violations.

Upon detection of a run-time condition error, a status register is set Upon detection of a run-time condition error, a status register is set and transfer to an alternate routine can be effected.and transfer to an alternate routine can be effected.


8.5 Construction of Acceptance Tests 8.5.1 Placement of Acceptance Tests within the Program

The placement of Acceptance Tests is based on the The placement of Acceptance Tests is based on the development of development of “fault trees” below the functional “fault trees” below the functional

requirementsrequirements for the application program. for the application program.

The development of fault trees involves a description The development of fault trees involves a description of the of the structural aspectsstructural aspects of of subroutines subroutines, , proceduresprocedures, ,

or or data structuresdata structures contained within the program. contained within the program.

Development of fault trees Development of fault trees for the placement of for the placement of

acceptance tests.acceptance tests.



To illustrate the approach, take the To illustrate the approach, take the fault-tolerant multiprocessor fault-tolerant multiprocessor computercomputer (FTMP) developed by the C. S. Draper Labs.: consists of (FTMP) developed by the C. S. Draper Labs.: consists of 10 processors configured as 3 triads and a single spare.10 processors configured as 3 triads and a single spare.

OR gates mean that OR gates mean that all potential failures must be testedall potential failures must be tested by the by the Acceptance Test. Acceptance Test.

AND gates mean that only AND gates mean that only one event must be testedone event must be tested. E.g., “Stuck-. E.g., “Stuck-in Applications Routine” failure can occur only when two conditions in Applications Routine” failure can occur only when two conditions prevail simultaneously: prevail simultaneously: (1)(1) Interval time is not set, and Interval time is not set, and (2)(2) a design a design flaw in the applications routine causes a failure to terminate.flaw in the applications routine causes a failure to terminate.



““Clock FailureClock Failure” is considered a ” is considered a HW failureHW failure. Thus, it . Thus, it is notis not

checked by Acceptance Tests.checked by Acceptance Tests.

Failures indicated by “Failures indicated by “——” are ” are out of the scopeout of the scope of SW of SW

reliability.reliability.



Top-level fault tree for the FTMP dispacher.Top-level fault tree for the FTMP dispacher.



Timing failures.Timing failures.



As conclusionAs conclusion: :

It is evident from this example thatIt is evident from this example that considerable insight into considerable insight into the operation of the SW system the operation of the SW system is required for drawing of is required for drawing of

fault trees.fault trees.

So, special attention must be driven to the system under So, special attention must be driven to the system under development in terms ofdevelopment in terms of specification, coding, and specification, coding, and

documentation.documentation.



8. fault tolerance in software 8.5 construction of acceptance tests goal goal: describe the types...

Documents

categories acceptance

tests sections

highlevel tests

fault tolerance

acceptance test checks

runtime requirements

sort program

particular program