12 reliability qa standards

Software Reliability

Organization of this Lecture:

• Introduction. • Reliability metrics• Reliability growth modelling• Statistical testing• Summary

Software Reliability – Alternate Definitions

• Informally denotes a product’s trustworthiness or dependability.

• Probability of the product working “correctly” over a given period of time.


• a software product having a large number of defects is unreliable.

• reliability of a system improves if the number of defects is reduced.

Reliability

• User’s concern– An important attribute determining

the quality of the product.

• Demand for quantitative estimation of reliability before making buying decision.

Software Reliability Estimation

• Tricky problem– Several factors contribute to making

measurement of software reliability difficult.

Problems in Reliability Measurements

• Errors do not cause failures at the same frequency and severity.– measuring latent errors alone not

enough

• The failure rate is observer-dependent

Difficulties in Software Reliability Measurement (1)

• No simple relationship observed between system reliability and the number of latent software defects.

• Removing errors from parts of software which are rarely used makes little difference to the perceived reliability.

The 90-10 Rule

• Experiments from analysis of behavior of a large number of programs:– 90% of the total execution time is

spent in executing only 10% of the instructions in the program.

– This 10% part is the core of the program.

Effect of 90-10 Rule on Reliability

• removing 60% defects from least used parts would lead to only about 3% improvement to product reliability.

• Reliability improvements from correction of a single error depends on whether the error belongs to the core or the non-core part of the program.

Difficulty in Software Reliability Measurement (2)

• The perceived reliability depends to a large extent upon how the product is used

• In technical terms on its operation profile.

Effect of Operational Profile on Software Reliability Measurement• If we select input data so that only

“correctly” implemented functions are executed none of the errors will be exposed perceived reliability of the product will be high.

• On the other hand, if we select the input data such that only functions containing errors are invoked perceived reliability of the system will be low.


• Different users use a software product in different ways. – defects which show up for one user

may not show up for another.

• Reliability of a software product– clearly observer-dependent – cannot be determined absolutely

Difficulty in Software Reliability Measurement (3)

• Software reliability keeps changing through out the life of the product each time an error is detected and corrected

Hardware vs. Software Reliability

• Hardware failures inherently different from software failures.

• Most hardware failures are due to component wear and tear:– some component no longer functions

as specified.


• A logic gate can be stuck at 1 or 0,– or a resistor might short circuit.

• To fix hardware faults:– replace or repair the failed part.


• Software faults are latent:– system will continue to fail unless

changes are made to the software design and code.


• Because of the difference in effect of faults, metrics appropriate for hardware reliability measurements are not good software reliability metrics


• When a hardware is repaired:– its reliability is maintained

• When software is repaired:– its reliability may increase or

decrease.


• Goal of hardware reliability study :– stability (i.e. interfailure times

remains constant)

• Goal of software reliability study – reliability growth (i.e. interfailure

times increases)

Reliability Metrics

• Different categories of software products have different reliability requirements: – level of reliability required for a

software product should be specified in the SRS document.

Reliability Metrics

• A good reliability measure should be observer independent, – so that different people can agree on

the reliability.

Rate Of Occurrence of Failure

• ROCOF measures: – frequency of occurrence failures.– observe the behavior of a software

product in operation over a specified time interval and calculate the total number of failures during the interval.

Mean Time To Failure

• MTTF is average time between two successive failures: – Determine by observing over a large

number of failures.


• MTTF is not as appropriate for software as for hardware:– Hardware fails due to a component’s

wear and tear thus indicates how frequently the component fails

– When a software error is detected and repaired the same error never appears.


• We can record failure data for n failures:– let these be t1, t2, …, tn– calculate (ti+1-ti)– the average value is MTTF

(ti+1-ti)/(n-1)

Mean Time to Repair

• Once failure occurs additional time is lost to fix faults

• MTTR measures average time it takes to fix faults.

Mean Time Between Failures

• We can combine MTTF and MTTR:– to get an availability metric:

MTBF = MTTF + MTTR

• MTBF of 100 hours would indicate– Once a failure occurs, the next failure

is expected after 100 hours of clock time (not running time).

Availability

• Measures how likely the system shall be available for use over a period of time: – considers the number of failures

occurring during a time interval, – also takes into account the repair

time (down time) of a system.

Availability

• Important for Systems having real time requirement:– telecommunication systems, – operating systems, etc. which are

supposed to be never down– where repair and restart time are

significant and loss of service during that time is important.

Failure Classes/ Category

• Transient: – Transient failures occur only for certain

inputs.

• Permanent: – Permanent failures occur for all input

values.

• Recoverable: – When recoverable failures occur the system

recovers with or without operator intervention.

Failure Classes

• Unrecoverable: – the system may have to be restarted.

• Cosmetic: – May cause minor irritations. Do not

lead to incorrect results. • Eg. mouse button has to be clicked twice

instead of once to invoke a GUI function.

Reliability Growth Modelling

• A reliability growth model:– a model of how software reliability

grows as errors are detected and repaired.

• A reliability growth model can be used to predict when (or if at all) a particular level of reliability is likely to be attained.– i.e. how long to test the system?


• There are two main types of uncertainty which render any reliability measurement inaccurate:

• Type 1 uncertainty:– our lack of knowledge about how the

system will be used, i.e. • its operational profile


• Type 2 uncertainty:– reflects our lack of knowledge about

the effect of fault removal.• When we fix a fault we are not sure if the

corrections are complete and successful and no other faults are introduced

• Even if the faults are fixed properly we do not know how much will be the improvement to interfailure time.

Step Function Model

• The simplest reliability growth model:– a step function model

• The basic assumption:– reliability increases by a constant

amount each time an error is detected and repaired.

Step Function Model

ROCOF

Time

Step Function Model

• Assumes:– all errors contribute equally to

reliability growth– highly unrealistic:

• we already know that different errors contribute differently to reliability growth.

Jelinski and Moranda Model

• Realizes each time an error is repaired reliability does not increase by a constant amount.

• Reliability improvement due to fixing of an error is assumed to be proportional to the number of errors present in the system at that time.

Jelinski and Moranda Model• Realistic for many applications,• Shortcomings

– Most probable failures (failure types which occur frequently) discovered early during the testing process.

– Repairing these faults contribute maximum to the reliability growth initially.

– This implies rate of reliability growth should be large initially slow down later on

Littlewood and Verall’s Model

• Assumes different fault have different sizes, thereby contributing unequally to failures.

• Allows for negative reliability growth

Littlewood and Verall’s Model

• Large sized faults tends to be detected and fixed earlier

• As number of errors is driven down with the progress in test, so is the average error size, causing a law of diminishing return in debugging

Other Reliability growth models

• Variations exists– LNHPP (Littlewood non homogeneous

Poisson process) model

• Goel – Okumoto (G-O) Imperfect debugging model– GONHPP

• Musa – Okumoto (M-O) Logarithmic Poisson Execution Time model

Applicability of Reliability Growth Models

• There is no universally applicable reliability growth model.

• Reliability growth is not independent of application.

Applicability of Reliability Growth Models

• Fit observed data to several growth models.– Take the one that best fits the data.

Statistical Testing

• A testing process:– the objective is to determine

reliability rather than discover errors.– uses data different from defect

testing.

Statistical Testing

• Different users have different operational profile:– i.e. they use the system in different

ways– formally, operational profile:

• probability distribution of input

Operational profile: Example

• An expert user might give advanced commands:– use command language interface,

compose commands

• A novice user might issue simple commands:– using iconic or menu-based interface.

How to define operational profile?

• Divide the input data into a number of input classes:– e.g. create, edit, print, file operations,

etc.

• Assign a probability value to each input class:– a probability for an input value from

that class to be selected.

Steps involved in Statistical testing (Step-I)

• Determine the operational profile of the software:– This can be determined by analyzing

the usage pattern.

Step 2 in Statistical testing

• Manually select or automatically generate a set of test data: – corresponding to the operational

profile.


• Apply test cases to the program:– record execution time between each

failure– it may not be appropriate to use raw

execution time


• After a statistically significant number of failures have been observed:– reliability can be computed.

Statistical Testing

• Relies on using large test data set. • Assumes that only a small

percentage of test inputs:– likely to cause system failure.

Statistical Testing

• It is straight forward to generate tests corresponding to the most common inputs:– but a statistically significant

percentage of unlikely inputs should also be included.

• Creating these may be difficult:– especially if test generators are used.

Advantages of Statistical Testing

• Concentrate on testing parts of the system most likely to be used:– results in a system that the users find

more reliable (than actually it is!).

Advantages of Statistical Testing

• Reliability predictions based on test results:– gives an accurate estimation of

reliability (as perceived by the average user) compared to other types of measurements.

Disadvantages of Statistical Testing

• It is not easy to do statistical testing properly:– there is no simple or repeatable way

to accurately define operational profiles.

• Statistical uncertainty.

Software Quality

a measure of usefulness and

practicality

Quality

a degree or level of innate excellence

extent of departure from the standard,

degree of conformity

a measure of the skill of the creator and value to the person for whom it was created

comparative-dictionary

fitness-for-purpose-Juran

quantitative -manufacturing

subjective -artistic...

Introduction

• Consider a software product:– functionally correct– but unusable user interface.

• Functional correctness alone does not determine quality

Modern view of quality• Associates several quality factors with a

software product :– Correctness– Reliability– Efficiency (includes efficiency of resource

utilization)– Portability– Usability– Reusability– Maintainability

Correctness

• A software product is correct, – if different requirements as specified

in the SRS document have been correctly implemented.

– Accuracy of results.

Portability

• A software product is said to be portable, – Work on different operating systems – Or on different machines– Or with other software products

Reusability

• Different modules of the product can easily be reused to develop new products.

Usability

• A software product has good usability, if different categories of users (i.e. both expert and novice users) can easily invoke the functions of the product.

• HCI aspect

Maintainability

• A software product is maintainable, – If faults can be corrected– Functionalities can be added or

modified (customized)

Software Quality

• the totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs .

– ISO 8402

Difficulties in achieving software quality

• its ethereal nature• its discreteness (no tolerances)• informal/semiformal discipline• volatility of requirements• inherent complexity of large

software systems

Cont…

• intellectual (read, semi-disciplined) input

• complex intra-team communication• deadline pressures• rapid advances in technology• high (often unreasonable)

expectations of clients

McCall’s Model of Software Quality

• Operation

• Revision• Transition

Operation Attributes

usability

correctnessintegrityreliability

efficiency

Revision Attributes

maintainability

testability

flexibility

Transition Attributes

portability reusability

interoperability

Crosby’s Maturity Level

Level 1: Uncertainty

Level 2: Awakening

Level 5: Certainty

Level 3: Enlightenment

Level 4: Wisdom

based on management

attitudes

SQA Environment

• Quality Policy• Quality Management• Quality Assurance• Quality Control• Quality Management System• Total Quality Management

Evolution of Quality Systems

• Initial product inspection method :– gave way to quality control (QC).

• Quality control:– not only detect the defective products

and eliminate them – but also determine the causes behind

the defects.

Quality control (QC)

• Quality control aims at correcting the causes of errors: – not just rejecting defective products.

• Statistical quality control– quality of the output of the process is

inferred using statistical methods – in stead of inspection or testing of all

products

Quality control (QC)

• The next breakthrough, – development of quality assurance

principles

Quality assurance

• Basic premise of modern quality assurance: – if an organization's processes are

good and are followed rigorously, • the products are bound to be of good

quality.

Quality assurance

• All modern quality paradigms include:– guidance for recognizing, defining,

analyzing, and improving the production process.

Total quality management (TQM)

• Advocates: – continuous process improvements

through process measurements.

Quality Policy

the overall quality intentions and direction of an organization as regards quality, as formally expressed by top management.

- ISO 8402

Quality Management

that aspect of the overall management function that determines and implements the quality policy.

- ISO 8402

Quality Assurance

a planned and systematic pattern of all actions necessary to provide adequate confidence that the item or product conforms to established technical requirement.

- IEEE 729

Quality Control

the operational techniques and activities that are used to satisfy the quality requirements.

- ISO 8402

Quality Management System

the organizational structure, responsibilities, procedures, processes and resources for implementing quality management.

- ISO 8402

Total Quality Management

Quality is to satisfy customer's requirements continuallyTotal quality is to achieve quality at low costTotal quality management is to obtain total quality by involving everyone's daily commitment

- Kanji

SQA Techniques - Error detection & removal

• inspections• walkthroughs• reviews• testing• demonstrations

SQA Techniques - Error avoidance• systematic and error-proof

determination of user needs• systematic and error-proof translation of

user needs into software products• methods and tools• rigorous specifications• configuration management• adherence to standards• root cause analysis

Role of SQA

• evolve and approve a Software Quality Assurance Plan (SQAP) for every project

• Audit the development process for proper execution in conformance with SQAP and Software Development Plan (SDP)

• Participate in the assessment of internal deliverables and of overall software quality

Role of SQA

• Perform, witness or audit tests; and attend selected life cycle reviews to assure appropriate Verification & Validation

• Participate in configuration management and QIP

• Provide appropriate visibility in situation of non-conformance

Quality Guidelines

• start with management commitment to ensure proper attention

• specify quality goals to promote understanding

• establish strategy to achieve quality to ensure proper resources for quality

• apply predictive measures to evaluate quality and increase insight to identify deficiencies and problem areas

Quality Guidelines

• assess quality achieved in final product to determine how well project satisfied goals

• analyze results to identify areas to improve

• provide feedback to exploit lessons learnt

Quality System Activities:• Auditing of projects• Development of:

– standards, procedures, and guidelines, etc.

• Production of reports for the top management – summarizing the effectiveness of the

quality system in the organization.• Review of the quality system itself.

Business Process reengineering

• A term related to TQM. • Process reengineering goes a step

further than quality assurance:– aims at continuous process

improvement.

Business Process reengineering

• BPR aims at reengineering the way business is carried out in any organization – not just software development

organizations.

Total quality management (TQM)

• TQM goes beyond documenting processes – optimizes them through redesign.

• Over the years the quality paradigm has shifted:– from product assurance to process

assurance.

ISO 9000

• ISO (international Standards Organization): – Consortium established to formulate

and foster standardization.

• ISO published its 9000 series of standards in 1987.

What is ISO 9000 Certification?

• ISO 9000 certification: – serves as a reference for contract

between independent parties.

• The ISO 9000 standard: – specifies guidelines for maintaining a

quality system.

What is ISO 9000 Certification?

• ISO 9000 specifies: – guidelines for repeatable and high

quality product development.– Also addresses organizational aspects

• responsibilities, reporting, procedures, processes, and resources for implementing quality management.

ISO 9000

• A set of guidelines for the production process.– not directly concerned about the

product it self.– a series of three standards:

• ISO 9001, ISO 9002, and ISO 9003.

ISO 9000

• Based on the premise:– if a proper process is followed for

production: • good quality products are bound to

follow.

ISO 9001:

• Applies to:– organizations engaged in design,

development, production, and servicing of goods.

– applicable to most software development organizations.

ISO 9002:• ISO 9002 applies to:

– organizations who do not design products: • but are only involved in production.

• Examples of this category of industries:– steel or car manufacturing industries– buy the product and plant designs from

external sources:• only manufacture products.

– not applicable to software development organizations.

ISO 9003

• Applies to organizations involved only in installation and testing of the products.

ISO 9000 for Software Industry

• ISO 9000 is a generic standard:– applicable to any industry

• Many clauses of ISO 9000 documents:– use generic terminologies– very difficult to interpret them in the

context of software organizations.

Software vs. other industries

• No direct interpretation exist in context of software industry


• Software is intangible. Therefore difficult to control. – In contrast, in a car manufacturing unit:

• we can see a product being developed through stages such as fitting engine, fitting doors, etc.

• one can accurately tell about the status of the product at any time.

– Software project management is an altogether different ball game.

Software vs. other industries• During software development:

– the only raw material consumed is data.• For any other product development:

– Lot of raw materials consumed• e.g. Steel industry consumes large volumes

of iron ore, coal, limestone, etc.

• ISO 9000 standards have many clauses corresponding to raw material control .

• not relevant to software organizations.


• Radical differences exist between software and other product development, – difficult to interpret various clauses of

the original ISO standard in the context of software industry.

ISO 9000 Part-3

• ISO released a separate document called ISO 9000 part-3 in 1991 – to help interpret the ISO standard for

software industry.

• At present official guidance is inadequate

Benefits of ISO 9000 Certification?

• Gain confidence of customers on organization

• Government contracts. This is especially true in the international market.

How to Get ISO 9000 Certification?

• An organization intending to obtain ISO 9000 certification: – applies to a ISO 9000 registrar for

registration.

• ISO 9000 registration process consists of several stages.


• Application stage:– Applies to a registrar for registration.

• Pre-assessment:– the registrar makes a rough

assessment of the organization.


• Document review and adequacy audit:– process and quality-related

documents.– the registrar reviews the documents – makes suggestions for improvements.


• Compliance audit: the registrar checks – whether the suggestions made by it

during review have been complied.


• Registration:– The registrar awards ISO 9000

certificate after successful completions of all previous phases.

• Continued surveillance:– The registrar continues monitoring

the organization periodically.

ISO 9000 Certification

• An ISO certified organization – can use the certificate for corporate

advertizements– cannot use the certificate to advertize

products.• ISO 9000 certifies organization's process • not any product of the organization.

– An organization using ISO certificate for product advertizements:

• risks withdrawal of the certificate.

Summary of ISO 9001 Requirements

• Management responsibility(4.1):– Management must have an effective

quality policy. – The responsibility and authority of all

those whose work affects quality:• must be defined and documented.

Management responsibility(4.1)

• Responsibility of the quality system. – independent of the development

process, – can work in an unbiased manner.

• The effectiveness of the quality system: – must be periodically by audited.

Quality system (4.2) and contract reviews (4.3):

• A quality system must be maintained and documented.

• Contract reviews (4.3):– Before entering into a contract, an

organization must review the contract • ensure that it is understood,• organization has the capability for

carrying out its obligations.

Design control (4.4):

• The design process must be properly controlled, – this includes controlling coding also.

• A good configuration control system must be in place.

Design control (4.4):

• Design inputs must be verified as adequate.

• Design must be verified.• Design output must be of required

quality.• Design changes must be

controlled.

Document control (4.5):

• Proper procedures for – document approval, issue and

removal.

• Document changes must be controlled. – use of some configuration

management tools is necessary.

Purchasing (4.6):

• Purchased material, including bought-in software:– must be checked for conforming to

requirements.

Purchaser Supplied Products (4.7):

• Material supplied by a purchaser,– for example,

• client-provided software must be properly managed and checked.

Product Identification (4.8):

• The product must be identifiable at all stages of the process. – In software development context this

means configuration management.

Process Control (4.9) :

• The development must be properly managed.

• Quality requirements must be identified in a quality plan.

Inspection and Testing (4.10) :

• In software terms this requires effective testing i.e., – unit testing, integration testing and

system testing.

• Test records must be maintained.

Inspection, measuring and test equipment(4.11):

• If integration, measuring, and test equipments are used, – must be properly maintained and

calibrated.

Control of nonconforming product (4.13) :

• In software terms, – keeping untested or faulty software

out of released product, • or other places whether it might cause

damage.

Corrective Action (4.14) :

• This is both about correcting errors when found, – investigating why they occurred– improving the process to prevent

further occurrences.

• If an error reoccurs despite the quality system, – the system needs improvement.

Handling (4.15) and Quality audits (4.17):

• Handling (4.15) Deals with: – storage, packing, and delivery of the

software product.

• Quality Audits (4.17) :– quality system audit must be carried

out to ensure its effectiveness.

Training (4.18) :

• Training needs must be identified and met.

• Most items of ISO standard– are largely common sense.

Salient features of ISO 9001 requirements:

• All documents concerned with the development of a software product – should be properly managed,

authorized, and controlled.

• Proper plans should be prepared– progress against these plans should

be monitored.

Salient features of ISO 9001 requirements:

• Important documents independently checked and reviewed: – for effectiveness and correctness.

• The product should be tested :– against specification.

• Several organizational aspects: – e.g., management reporting of the

quality team.

Shortcomings of ISO 9001 Certification (1)

• ISO 9000 requires a production process to be adhered to:– but does not guarantee the process to

be of high quality.– Does not give any guideline for

defining an appropriate process.


• ISO 9000 certification process– not fool-proof– no international accredition agency

exists. – likely variations in the norms of

awarding certificates: • among different accredition agencies and

among the registrars.


• Organizations qualifying for ISO 9001 certification:– tend to downplay domain expertise. – tend to believe that since a good

process is in place, • any engineer is as effective as any other

engineer in doing any particular activity relating to software development.


• In manufacturing industry– clear link between process quality and

product quality– once a process is calibrated:

• can be run again and again producing quality goods

• Software development is a creative process:– individual skills and experience is significant


• Many areas of software development are very specialized: – special expertize and experience

(domain expertize) required.

• ISO 9001– does not automatically lead to

continuous process improvement, – does not automatically lead to TQM.


• ISO 9001 addresses mostly management aspects.

• Techniques specific to software development have been ignored– Configuration management– Reviews– Release builds– Problem Notification system– Intranets

SEI Capability Maturity Model

• Developed by Software Engineering Institute (SEI) of the Carnegie Mellon University, USA:– to assist the U.S. Department of

Defense (DoD) in software acquisition.

– The rationale was to include:• likely contractor performance as a factor

in contract awards.


• Major DoD contractors began CMM-based process improvement initiatives: – as they vied for DoD contracts.

• SEI CMM helped organizations: – Improve quality of software they developed– Realize adoption of SEI CMM model had

significant business benefits.

• Other organizations adopted CMM.


• In simple words,– CMM is a model for apprising the

software process maturity of a contractor into different levels.

– Can be used to predict the most likely outcome to be expected from the next project that the organization undertakes.


• Can be used in two ways:– Capability evaluation– Software process assessment.

Capability Evaluation

• Provides a way to assess the software process capability of an organization– Helps in selecting a contractor– Indicates the likely contractor

performance

Software Process Assessment

• Used by an organization to assess its current process:– Suggests ways to improve the

process capability.– This type of assessment is for purely

internal use.


• The SEI CMM classifies software development industries into: – Five maturity levels.– Stages are ordered so that

improvements at one stage provide foundations for the next

– Based on the pioneering work of Philip Crosby


Initial (1)

Repeatable (2)

Defined (3)

Managed (4)

Optimizing (5)

Level 1: (Initial)

• Organization operates– without any formalized process or

project plans

• An organization at this level is characterized by – ad hoc and often chaotic activities.

Level 1: (Initial)

• Software production processes are not defined, – different engineers follow their own

process – development efforts become chaotic. – The success of projects depend on

individual efforts and heroics.

Level 2: (Repeatable)

• Basic project management practices – tracking cost, schedule, and functionality

are followed.

• Size and cost estimation techniques – function point analysis, COCOMO, etc. used.

• Production process is ad hoc – not formally defined– also not documented.

Level 2: (Repeatable)

• Process used for different projects might vary between projects: – earlier success on projects with

similar applications can be repeated.– Opportunity to repeat process exist

when a company produces a family of products.

Level 3: (Defined)

• Management and development activities: – defined and documented.– Common organization-wide

understanding of activities, roles, and responsibilities.

Level 3: (Defined)

• The process though defined, – process and product qualities are not

measured.

• ISO 9001 aims at achieving this level.

Level 4: (Managed)

• Quantitative quality goals for products are set.

• Software process and product quality are measured: – The measured values are used to

control the product quality.– Results of measurement used to

evaluate project performance • rather than improve process.

Level 4: (Managed)

• Organization sets quantitative quality goals

• World-wide about 100 organizations assessed at this level.

Level 5: (Optimizing)

• Statistics collected from process and product measurements are analyzed:– continuous process improvement

based on the measurements.• Known types of defects are prevented

from recurring by tuning the process• lessons learned from specific projects

incorporated into the process

Level 5: (Optimizing)

• Identify best software engineering practices and innovations:– tools, methods, or process are

identified– transferred throughout the

organization

• World-wide about 50 organizations have been assessed at this level.

Key Process Areas

• Each level is associated with a key process area (KPA) identifies– where an organization at the previous

level must focus to reach this level

Level 2 KPAs

• Software project planning– Size, cost, schedule.– project monitoring

• Configuration management• Subcontract management

Level 3 KPAs

• Process definition and documentation

• Reviews• Training program

Level 4 KPAs

• Quantitative measurements• Process management

Level 5 KPAs

• Defect prevention• Technology change management• Process change management

Comparison between ISO 9001 and SEI CMM

• ISO 9001 awarded by an international standards body– can be quoted in official documents

and communications

• SEI CMM assessment is purely for internal use.


• SEI CMM was developed specifically for software industry:– addresses many issues specific to

software industry.

• SEI goes beyond quality assurance– aims for TQM– ISO 9001 correspond to SEI level 3.


• SEI CMM provides a list of key areas– on which to focus to take an organization

from one level to the other

• Provides a way for gradual quality improvements over several stages.– e.g trying to implement a defined process

before a repeatable process:• counterproductive as managers are

overwhelmed by schedule and budget pressure.

Remarks on Quality Model Usage• Highly systematic and measured approach

to software development process suits certain circumstances– negotiated software, safety-critical software, etc

• What about small organizations?• Typically handle applications such as internet, e-comm. • without an established product range,• without revenue base, experience on past projects, etc.• CMM may be incompatible

Small Organizations

• Small organizations tend to believe:– We are all competent people hired to

do a job, we can’t afford training– We all communicate with one another

• Osmosis works because we are so close

– We are all heroes• We do what needs to be done• Therefore rules do not apply to us

Small Organizations

• Often have problems:– Undocumented requirements– Inexperienced managers– Documenting the product– Resource allocation– Training– Peer reviews

Small Organizations

• A two week CMM-based appraisal is probably excessive:

• Small organizations need to operate more efficiently at lower levels of maturity– Must first fluorish if eventually they

are to mature

Personal Software Process (PSP)

• Based on the work of Humphrey• PSP is a scaled down version of

industrial software process– suitable for individual use

• Even CMM assumes that engineers use effective personal practices


• A process is the set of steps for doing a job

• The quality and productivity of an engineer – largely determined by his process

• PSP is framework that– helps software engineers to measure

and improve the way they work.


• Helps developing personal skills and methods– Estimating and planning method– Shows how to track performance against

plans– Provides a defined process

• can be fine tuned by individuals• Recognizes that a process for individual use is

different from that necessary for a team project.

Time Management

• Track the way you spend time– Boring activities seem longer then

actual– Interesting activities seem short

• Record time for– Designing – Writing code– Compiling– Testing


Planning

Design

Code

Compile

TestPostmortem

Logs

Project plan summary

PSP-Planning

• Problem definition• Estimate max, min, and total LOC• Determine minutes/LOC• Calculate max,min, and total

development times• Enter the plan data in project plan

summary form• record the planned time in Log

PSP-Design

• Design the program• Record the design in specified

format• Record the Design time in time

recording log

PSP-Code

• Implement the design• Use a standard format for code

text• Record the coding time in time

recording log

PSP-Compile

• Compile the program• Fix all the defects• Record compile time in time

recording log

PSP-Test/Postmortem

• Test– Test the program– Fix all the defects found– Record testing time in time recording log

• Postmortem– Complete project plan summary form

with actual time and size data– Record postmortem time in time record


PSP 0

PSP 1

PSP 2

PSP 3

Personal measurement

Basic size measures

Personal planning

Time and schedule

Personal quality management

Design and code reviews

Personal process

evolution

Fitting pieces

Building the CMM Structure

Capability Maturity Models• CMMI SM – CMM IntegrationSM • SW-CMM – Capability Maturity Model® for

Software • P-CMM – People Capability Maturity Model • SA-CMM – Software Acquisition Capability

Maturity Model • SE-CMM – Systems Engineering Capability

Maturity Model • IPD-CMM – Integrated Product

Development Capability Maturity Model

SW - CMM

• Describes principles and practices underlying software process maturity

• Intended to help software organizations improve the maturity of their software processes

SE - CMM

• Describes essential elements of an organization's systems engineering process

• Provides a reference for comparing actual systems engineering practices against these essential elements.

IPD - CMM• Guide organizations in IPD design,

development, appraisal, and improvement.

• Involves a teaming of the functional disciplines to integrate and concurrently apply all necessary processes to produce an effective and efficient product that satisfies the customer's needs

CMMI

• The content and characteristics of SW-CMM [draft version 2(c)], SE-CMM [EIA/IS 731], and IPD-CMM [Version 0.98] models provide the basis for the CMMI product suite

• CMMI models eventually replace SW-CMM Version 1.1

Motivation for Integrations

• Models created within an environment of evolving national and international standards and frameworks

• Maintaining harmonization between them and improvement models become a continuing challenge, particularly across discipline

Frameworks Quagmire

CMM Integration

• Objective is to develop a product with a set of integrated products to support process and product improvement.

• Preserve investment in process improvement and enhance the use of multiple models.

• Output will be integrated model(s), assessment method(s), and training material.

CMMI Objectives• Eliminating inconsistencies• Reducing duplications• Increasing clarity and understanding• Assuring consistency with ISO 15504

Longer-term objective is to lay foundation for later addition (acquisition, security)

CMM Based Appraisals• CMM-Based Appraisal for Internal Process

Improvement (CBA IPI): To determine the state of one's own processes

• Software Capability Evaluation (SCE):To determine the state of someone else's processes.

• To ensure that these methods are both rigorous and consistent, the Process Program developed and maintains the CMM Appraisal Framework (CAF), which defines the standards that an appraisal method must meet.

When to apply CBA IPI

• For investigating your organization, using mostly your own team members, and making principal use of the results yourself, then a CBA IPI assessment is the recommended tool. This method assumes a collaborative environment and a collegial approach to gathering data.

When to apply SCE• If, on the other hand, you are investigating

somebody else's organization, using your own team members (not theirs), and you will use the results for your purposes (even if you plan to share the results with the organization), then an SCE is the recommended tool. This method assumes a potentially audit-like environment and a more intrusive approach to gathering data

SEI’s Role In CMM-Based Appraisals• Authorized Lead Assessors for CBA

IPI are required to return a report to the SEI for each completed assessment. The data is entered into the Process Appraisal Information System. All assessment data that is returned to the SEI is kept confidential.

• The SEI trains and authorizes qualified persons to lead appraisals

Cont…

• The SEI does not validate or certify appraisal results.

• An assessment's findings and maturity level are owned by the assessment sponsor. The sponsor may publicize this information at their discretion.

• The Maturity Profile Report is published by the SEI twice a year

That’s all?

• Not exactly• SQA Activities

– Cleanroom methodology– Six Sigma

• Assessment models– SPR Assessment– Malcolm Baldrige National Quality

Award

Cleanroom Software Engineering

• Metaphor coming from integrated circuit manufacturing– Free from dust, flecks of skin etc

• Whenever defect occurs, they are considered as defect in process than product– Invoke process inspection, review and

improvement cycle

Six Sigma

• Motorola’s baby that won them MBNQA award for quality in 1988

• Six sigma is a quantitative approach to eliminate defects

• Applicable to any industry - manufacturing, product development, service

Six Sigma• To achieve six sigma

– a process must not produce more than 3.4 defects per million opportunities.

– 5 Sigma -> 230 defects per million– 4 Sigma -> 6210 defects per million

• Six sigma methodologies– DMAIC (Define, Measure, Analyze, Improve,

Control)– DMADV: (Define, Measure, Analyze, Design,

Verify)

Six Sigma Methodologies

• The methodologies are implemented by Green belt and Black belt workers– Supervised by Master black belt worker

• Pareto Chart:– Simple bar chart to represent defect

data– Identify the problems that occurs with

greatest frequency• or incur the highest cost

Software Productivity Research Inc.• Methodology developed parallel to SEI

Process maturity model• SPR question covers both Strategic

corporate issues and tactical project issues that affect quality, productivity and user satisfaction

• 400 questionnaire as against SEI’s 85 questions– Lickert Scale for response (Excellent, Good,

Average, Marginal, Poor)

Malcolm Baldrige National Quality Award• Established in 1988 by US Department

of Commerce• Examination criteria against seven

categories– Leadership– Information and analysis– Strategic Quality Planning– Human Resource Utilization– Quality assurance of products and services– Quality Results– Customer satisfaction

References

• CMMI Distilled– Ahern, Clouse, Turner. Addison Wesley

• CMM in Practice– Pankaj jalote. Pearson Education

• Introduction to Personal SW Process, Introduction to Team SW Process– Watts Humphrey. Addison Wesley

Web References• General Info

– http://www.sei.cmu.edu– http://www.sei.cmu.edu/sema/profile.html

• PSP, TSP– http://interactive.sei.cmu.edu

/Features/1999/June/Background/Background.jun99.pdf [Both]

– http://www.sei.cmu.edu/pub/documents/articles/pdf/psp.pdf [PSP]

– http://wuarchive.wustl.edu/languages/ada/sei/documents/00.reports/pdf/00tr023.pdf [TSP]

Alternatives/Additions• ISO Standards

– Generic Specifications in areas of Quality Requirements (9000), Environmental Management (14000)

– Latest on Information Technology (ISO/IEC TR 15504-#:1998)

, where 0<#<10

– Some client demand for ISO Certified institutes

http://www.iso.ch/iso/en/iso9000-14000/tour/magical.html

Other References

• http://www.software.org/quagmire/• Presentation by Rajib Mall

12 reliability qa standards

Documents