Software Quality &

Testing

US Govt. Study on Software Development Projects reveals

60 % of Projects are behind Schedule

50% were over cost

45 % of delivered projects were unusable

IT SCENARIO

Success/Failure Rate of Projects

PROJECT SUCCESS HISTORY

0

10

20

30

40

50

60

1994 1996 1998

SuccessFailedChallenge

16

53

3127

4033

2628

46

%

SDLC

Coding and

Testing Implementation

Architectural DESIGN (HLD)

Detailed Design (LLD)

Maintenance

Requirement Analysis

Design27%

Coding7%

Others10%

Requirement56%

Distribution of Bugs

Source: Writing Testable Requirements by Dick Bender

Requirement82%

Coding1% Others

4%

Design13%

S ou rce: W r it in g T esta b le R eq u ire m en ts b y D ick B en d e r

DISTRIBUTION OF EFFORTS

Quality ?

Problem with Quality?

The problem with quality is not what people don’t know about it?

The problem is what they think they do know.

Quality has much in common with Sex Bill Crossby Everybody is for it.

(Under certain conditions, of course)

Everyone feels they understand it

(Even though they would not want to explain it)

Everyone thinks execution is only a matter of following natural inclinations.

(After all we do get along somehow)

Most people feel that problems in these areas are caused by others.

(if only they would take the time to do things right)

What Is Quality?

As defined by a producer of a product

Meeting specified requirements of an acceptable

PRODUCT

As defined by a customer

FITNESS FOR USE Should be available at the RIGHT

TIME, PLACE, FORM, PRICE

What Is Quality?

As defined by ISO 9000 : The totality of characteristics of an entity (product or service) that bear

on its ability to satisfy

STATED

and/or

IMPLIED needs.

Software Quality :McCall’s Quality Factors Factors that can be directly measured.

Factors that can be measured only indirectly.

Compare the software to some datum and arrive at an indication of quality.

Software Quality :McCall’s Quality Factors

PRODUCT TRANSITIONPRODUCT REVISION

PRODUCT OPERATION

Software Quality : Product Operation Correctness The degree to which the

software satisfies stated needs as indicated by: Suitability Accuracy Interoperability Compliance Security

Software Quality : Product Operation Reliability: The amount of time that the

software is available for use as indicated by: Maturity

Fault Tolerance

Recoverability

Software Quality : Product Operation Usability: The degree to which the software is

easy to use as indicated by: Understandability

Learnability

Operability

Software Quality : Product Operation Integrity:

The extend to which access to Software or Data by unauthorized persons can be controlled

Software Quality : Product Operation Efficiency: The degree to which the software

makes optimal use of system resources as indicated by: Time Behavior

Resource Behavior

Software Quality : Product Revision Maintainability: The ease with which repair

may be made to the software is easy to use as indicated by: Analyzability Changeability Stability Testability

Software Quality : Product Revision Flexibility

The effort required to modify the programs

Software Quality : Product Revision Testability

The effort required to test the programs to ensure that it performs its intended functions

Software Quality : Product Transition Portability: The ease with which software can

be transposed from one environment to another as indicated by: Adaptability Installability Conformance Replaceability

Software Quality : Product Transition Reusability The extend to which a program

[or parts of programs] can be reused in other applications - related to the packaging and scope of the functions that the program perform.

Software Quality : Product Transition Interoperability

The effort required to couple the system to another

Software Cost of Errors at Each Stage of the SDLC

Software Economics

Relative Cost of Software Defects

0

20

40

60

80

100

120

140

Req Design Code Test Production

Line 1

Testing Economics

Stage Err. Int. Traditional Continuous

& Cost Left Cost Left Cost

------------------------------------------------------------------

Req. - 1 20 20 000 10 010

Des. - 3 20 40 000 15 055

Code - 5 20 60 000 18 140

Test - 10 12 480 04 280

Prdn -100 00 1680 00 680

Definitions

Errors

Error is an oversight or a wrong decision

Defects

Non fulfillment of an intended usage, requirement or reasonable expectation. A deviation from specification or standard Anything that causes customer dissatisfaction on

conformance to standards. Anything less/extra requirement from specification

Definitions

Software Verification & Validation

Software V & V processes determine whether development products of a given activity conform to the requirements of that activity & whether software satisfies its intended use and user needs

Definitions

Verification

IEEE defines Verification as the process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase

e.g. Do we have clear and complete requirements

Is this a good design

Definitions

Validation

IEEE defines Validation as the process of evaluating a system or component during or at the end of the development whether it satisfies specified requirements.

e.g. It involves executing the actual software

or a simulated mock type.

V & V Objectives

Facilitate early detection and correction of errors

Enhance management insight into process and product risk

Support software life cycle processes to ensure compliance with program performance, schedule and budget requirements.

What is Quality Assurance (QA)?

All those planned and systematic activities implemented within the quality system and demonstrated as needed to provide adequate confidence that an entity will fulfil requirements for quality

What is Quality Control (QC)?

The operational techniques and activities that are used to fulfil requirements for quality

Quality Assurance Vs Quality control

Product based

Reactive

S/W Test cases

Require testers

QC can be automated.

Inspection/Audit

Process Based

Proactive

Has to be designed, evolved as a culture

Requires Quality Specialists

Selection of QA tools.

Processes

Quality Assurance Vs Quality control (contd..)

Satisfies the Process

Time/Project Bound

Dynamic Software Testing, Test Reviews etc.

Satisfies the Customer

Continuous Improvement

Training on quality (Awareness, Processes, Costing).

Quality Models

Spice

SEI CMM

SEI PCMM

SEI CMMI

Kaizen

Total Quality Management (TQM)

Tick-IT

ISO–9000

CMM - Capability Maturity Model

Managed(4)

|Optimizing (5)

Defined(3)

Repeatable(2)

Initial(1)

Improvement institutionalized

Quantitative control

Technical practices are integrated with management practices and institutionalized

Project Management practices are institutionalized

Process is informal & adhoc

Disciplined

process

Standard consistent process

Predictable process

Continuously improving process

5 levels of software process maturity

Optimizing (5)

Process change management Technology change mgmt. Defect prevention

Managed (4)Quality management Quantitative Process management

Defined (3)Peer reviews Inter group coordination Software product engineering Integrated software management Training program Organization process definition Organization process focus

Repeatable (2)Software configuration management Software quality assurance Software subcontract management Software project tracking and oversight Software project planning Requirements management

Intial(1)

The Maturity Levels

1

2

3

4

5

Process unpredictable, poorly controlled, and reactive

Process characterized for projects and is often reactive

Process characterized for the organization and is proactive

Process measuredand controlled

Focus on continuous process improvement

Optimizing

QuantitativelyManaged

Defined

Initial

Managed

Optimizing

Defined

CMMI®-SE/SW (Staged Representation)

Organizational Innovation and DeploymentCausal Analysis and Resolution5 Optimizing

4 Quantitatively Managed

3 Defined

2 Managed

ContinuousProcess Improvement

QuantitativeManagement

ProcessStandardization

BasicProjectManagement

Organizational Process PerformanceQuantitative Project Management

Requirements DevelopmentTechnical SolutionProduct IntegrationVerificationValidationOrganizational Process FocusOrganizational Process DefinitionOrganizational Training Integrated Project ManagementRisk ManagementDecision Analysis and Resolution

Requirements Management Project PlanningProject Monitoring and ControlSupplier Agreement ManagementMeasurement and AnalysisProcess and Product Quality AssuranceConfiguration Management

QualityProductivity

RiskRework1 Initial

Process AreasMaturity Level Focus

Software Quality Assurance

Software Quality Assurance includes:

Scope Management

Time Management

Cost Management

Quality Management

Change Management

Software Quality Assurance includes:

Human resources Management

Communication Management

Risk Management

Procurement Management

Integration Management

Software Engineering Solutions

Joint Application Development (JAD) Structured Methodology Visual Diagramming/Prototyping Model Iterative Model Spiral Model

Software Engineering Solutions Joint Application Development (JAD)

Implemented by IBM User and Systems Team Work together thru

the entire Cycle Creates an Ownership Reduces Communication Gap Detection of Errors on Time User gets trained during the development

Software Engineering Solutions Structured Methodology

Based on 4 Principles

1. Abstraction

2. Formality

3. Divide and Rule

4. Hierarchical Ordering

Software Engineering Solutions Prototyping/Visual Modeling

1. Make Prototype for User Communication

2. Use Diagrammatic Techniques

3. Visual Modeling helps better understanding

Software Engineering Solutions Spiral Model

Consists of a Cyclic approach

1. Planning

2. Risk Assessment

3. Engineering

4. Review/Evaluation

SoftwareRequirements

Change ControlProcess

System TestingProcess

User Documentation

Process

Design and Construction

Process

Project PlanningProcess

Relationship of Requirements to Project Development Processes

Changes

Are the

baseline for

Are input to

Requirements Engineering

SoftwareRequirementsSpecifications

Requirements Management

Requirements ChangeManagement

Requirements Traceability

Requirements Definition

Requirements Elicitation Requirements Analysis Requirements Documentation Requirements Review