SIM5102 Software Evaluation

Data Collection

What is good data?

• Correctness - data collected according to the exact rules of definition of the metric

• Accuracy - difference between the data and the actual value

• Precision - number of decimal places needed to express the data

• Consistent - no much difference in value when measured using different device or by different person, or repeatedly

What is good data?

• Time-stamp - associate with a particular activity or time period to know when it was collected.

• Replication - replicate for difference circumstances.

How to define the data?

• Metric terminology must be clear and detailed, understood by all involved.

• Two kinds of data• Raw data• Refined data

• Direct and indirect measurement• Most organizations are interested in

software quality, cost and schedule.

The problem with problems

• Fig 5.2 (Software Quality Terminology)• Fault • occurs when a human error results in a

mistake in some software product• Eg: developer misunderstand the user-

interface requirement, and create a design with the wrong understanding, the design fault results in incorrect code.

The problem with problems

• Failure• Departure of a system from its required behavior• Can be discovered during testing and operation

• Can think of fault and failure as inside and outside views of the system.

• Faults - problems that developers see • Failures - problems that the users see• Not every fault corresponds to a

The problem with problems

• Reliability of software is defined in terms of failures observed during operation not faults.

• Terminology is very important here. Refer to page 157 (error, anomalies, defects, bugs, crashes)

• If investigation of failure reveal the fault, change is made to the product to remove it.

The problem with problems

• Key elements of a problem:• Location - where did it occur• Timing - when did it occur• Symptom - what was observed• End result - which consequences resulted• Mechanism - how did it occur• Cause - why did it occur• Severity - how much was the user affected• Cost - how much did it cost

The problem with problems

• These 8 attributes are mutually independent (applies only to initial measurement).

• Is known as orthogonality• Orthogonality also refers to a classification

scheme within a particular category.• Non-orthogonality can lead to data loss or

corruption, Eg 5.1

The problem with problems

• The 8 attributes should suffice for all types of problems, but are answered differently based on whether it is faults, failures or changes.

Failures

• Focus on external problems of the system: installation, chain of events, effect on user or other system, cost.

• Fig 5.3 (failure report), page 160• Location - code that uniquely identifies the

installation and platform on which the failure was observed.

• Timing - real time of occurrence, execution time up to the occurrence of failure.

Failures

• Cause • type of trigger, type of source• Often cross-referenced to fault and change reports.

• Severity • how serious the failure’s end result• Classification for safety-critical system:

• Catastrophic• Critical• Significant• Minor

• Can also be measured in terms of cost

Failures

• Cost - how much effort and other resources to diagnose and response to the failure

• Ninth category; Count - number of failures in a stated time interval.

• Mechanism, cause and cost can only be completed after diagnosis.

• Data collection form for failure should include at least five categories.

Failure Report• Location: such as installation where failure was observed• Timing: CPU time, clock time or some temporal measure• Symptom: type of error message or indication of failure• End result: description of failure, such as “operating system crash”,

“ services degraded”, “loss of data”, “wrong output”, “no output”• Mechanism: chain of events, including keyboard commands and

state data, leading to failure• Cause: reference to possible fault(s) leading to failure• Severity: reference to a well-defined scale, such as “critical”,

“major”, “minor”• Cost: cost to fix plus cost of lost potential business

Faults

• Focus on the internals of system• Only the developer can see it• Location • which product(identifier and version) or part of the

product contains the fault.• Spec, code, database, manuals, plan and procedures,

report, standard/policy• Requirements, functional, preliminary design, detailed

design, product design, interface, database, implementation.

Faults

• Timing - when the fault is created, detected, corrected.

• Symptom - what is observed during diagnosis (Table 5.2, page 166)

• End result - actual failure caused by fault, should be cross-referenced to failure report.

• Cause• human error that led to the fault• Communication, conceptual, clerical

Faults

• Severity - impact on the user• Cost - total cost to system provider• Count - number of faults found in a

product or subsystem or during a given period of operation.

Fault Report

• Location: module or document name• Timing: phases of development during which fault was

created, detected, and corrected• Symptom: type of error message reported or activity

which revealed fault• End result: failure cause by the fault• Mechanism: how cause was created, detected,

corrected• Cause: type of human error that led to fault• Severity: refer to severity of resulting or potential failure • Cost: time or effort to locate and correct

Changes

• Once a failure is experienced and its cause determined, the problem is fixed through one or more changes

• Problem is fixed through 1 or more change to any or all of the development product

• Changes report (Fig 5.5) - report the changes and track the most affected products

Changes

• Cause of change may be• Corrective - correcting a fault• Adaptive - system changes in some way so the

product have to be upgraded• Preventive - find faults before they become failure• Perfective - redo something to clarify the system

structure

• Count - number of changes made in a given time or given system component.

Change Report

• Location: identifier of document or module changed• Timing: when change was made• Symptom: type of change• End result: success of change, as evidence by

regression or other testing• Mechanism: how and whom change was performed• Cause: corrective, adaptive, preventive or perfective• Severity: impact on rest of the system• Cost: time and effort for change implementation and test

How to collect data

• Requires human observation and reporting

• Manual data collection – bias, error, omission and delay -> uniform data collection form

• Automatic data capture – desirable, essential – recording the execution time

• To ensure data are accurate and complete, planning is essential.

How to collect data

• Planning involves:• Decide what to measure based on GQM

analysis.• Determine the level of granularity (individual

modules, subsystem, function etc)• Ensure that product is under configuration

control (which version)• Form design

How to collect data

• Planning involves (continue)• Establish procedures for handling the forms,

analyzing the data and reporting the results, setting a central collection point

How to collect data

• Keep procedure simple

• Avoid unnecessary recording

• Train staff in recording data and procedure

• Provide results to original providers promptly

• Validate all data collected at a central collection point

Data collection forms

• Encourages collecting good, useful data• Self-explanatory• Should allow fixed-format data and free-

format comments• Table 5.3 (Data collection forms for

software reliability evaluation)

When to collect data

• Data collection planning when project planning begins

• Actual data collection takes place during many phases of development.

• Can be collected at the beginning of the project to establish initial values and collected again later to reflect activities and resources being studied.

When to collect data

• Data-collection activities should become part of the regular development process.

• Should be mapped to the process model. (eg: Fig 5.9)

When to collect data

• Eg:• Data relating to project personnel (qualifications or

experience) can be collected at the start of the project. • While other data collection, such as effort, begins at

project start and continues through operation and maintenance

• Count of the number of specification and design faults can be collected as inspections are performed

• Data about changes made to enhance the product can be collected as enhancements are performed

How to store and extract data

• Raw data should be stored in database set up using DBMS.

• Can define data structure, insert, modify, delete and extract refined data.

• Formats, ranges, valid values etc can be check automatically as they are input.

How to store and extract data

• Raw database structure• Eg: Figure 5.10 (data structure that supports

reliability measurement)• box - table, arrow denotes a many-to-one

mapping, double arrow