white paper on software quality

While metrics are important means to quantify

performance of a process, we often define metrics to

comply with process standards, overlooking actual

alignment with business factors. We may put

substantial effort and discipline in getting metrics

deployed, but we might not be using the ones that

really matter. How can we define metrics that remain

anchored to business needs? In other words, how can

we translate business requirements to the right set of

metrics? Once the right metrics are defined, how can

we monitor and benchmark it to control performance

and make breakthrough improvements? This white

paper discusses best practices around metrics, from its

very definition to using it for continuous improvement.

Software Quality - Getting Right Metrics, Getting Metrics Right

How to set the right performance metrics and then benchmark it for

continuous improvement?

1


About the Authors

Dr. Kanhaiya Jethani

Kanhaiya is a senior Process and Quality consultant at Tata Consultancy

Services. He has extensive experience in the fields of software project

management, software process and quality, CMM and CMMI. He has led

and executed process and quality consulting engagements for various IT

organizations in USA, Spain, Poland, Taiwan, Singapore, Japan and India.

He is a Certified Software Quality Analyst (CSQA), Project Management

Professional (PMP) and Candidate SCAMPI® Lead Appraiser. A Doctorate in

Chemical Engineering from Oklahoma State University, USA, he was also a

Post-Doctoral Research Fellow at Oklahoma State University, USA and

University Institute of Chemical Technology (UICT), Mumbai. He is a

member of PMI.

Vanishri Veloo

Vanishri is a senior Process and Quality consultant at Tata Consultancy

Services. She is a Candidate Lead Assessor for People CMM®, a qualified

auditor for ISO 9001:2000, a trained assessor for CMM® and SCAMPISM and

an External Assessor for Tata Business Excellence Model (TBEM). She has

led and executed process and quality consulting engagements for various

IT organizations in USA, UK, Germany, The Netherlands, Spain, Taiwan and

India.

Vanishri has worked in the financing of renewable energy, e-business and

financial services industry before moving to the Information Technology

Industry. Vanishri is a graduate in Biomedical Engineering from Mumbai

University, India and has done her Masters in Management Studies

(M.M.S.) in Finance from the Sydenham Institute of Management Studies,

Research & Entrepreneurship Education, Mumbai University, India.

Pragnya Misra

Pragnya Misra is a senior Process and Quality consultant at Tata

Consultancy Services. She has extensive experience in the fields of

application development and maintenance, software project

management, software process and quality, CMM and CMMI®. She has led

and executed process definition and deployment engagements for

various customers across the world. She is a Certified Software Quality

Analyst (CSQA).She is an SEI authorized assessor for CMM® and CMMI® and

also ISO-9000 Internal Auditor. She is a Masters in Business Administration

(Finance) and a Bachelor of Engineering (Instrumentation and Control)

from Gujarat University, India.

2

Table of Content1. Introduction 3

2. Ask the Right Question 3

3. GQM Keeps Us Anchored to Changing 5

Business Imperatives

4. Using SPC to Control Process Performance 5

5. Continuous Improvement Needs 7

Breakthroughs where it Matters

6. Conclusion 9


3

Introduction

Ask the Right Question

Pursuit of process frameworks and quality standards like CMMI® or ITIL has brought more discipline and maturity to

software development and IT services. The flip side is that we go overboard with a plethora of metrics, which often

are dictated by our preoccupation for compliance to such frameworks. With little alignment to the business-needs,

many of these metrics can turn ritualistic and time consuming, let alone the questionable sanity of the data. For

example, while we may remain complacent with low defect levels; we may ignore a high defect concentration in a

critical module or the severity of defects.

Many IT organizations have a metrics handbook that suggests the set of metrics to be followed. If such a handbook

prescribes a set of mandatory metrics, we would like such metrics to be rather small in number. A larger set of metrics

can remain optional, leaving it to the discretion of the project. Now the question is - how can one decide what would

be the right metric. It is the question of finding the right metric first before getting the metric right; the irony that

comes out of Joseph Juran’s adage do right things before doing things right.

Consider this: a module in software was found to have abnormally high defects. The metric that was closely

monitored was the defect density - defects per unit functionality (for example defects per function point). This

module was escalated for re-engineering. On inspection of the design, it was found to be good in terms of

maintainability and defect prevention. Questioning the rationale for re-engineering, the defects were revisited. It

was found that surprisingly, the defects were more in number but low in severity, being concentrated mostly on

output formats. Unfortunately, the severity of the defects was not factored in the defect density metric.

Metrics will remain meaningless ceremonies unless they are aligned to business goals. In this example, the business

goal was to reduce injection of defects critical to the quality of the product.

While we inherit or improvise the metrics that would support the monitoring of the project, it is essential to be

anchored to the business needs. In fact, the business goal should translate to the right set of metrics required to

monitor and control quality. A three step process, Goal-Question-Metric (GQM), would be very handy here: It is a way

to identify the right metrics by asking questions around the business goal. GQM helps translate the business needs

from conceptual level to the operational level by asking appropriate questions, thereby identifying the measures

that can provide objective insight into the achievement of the business goals. For example, if our goal is to improve

estimation, what questions would we have in mind? We would most probably ask how good the current estimation

method is. Effort overrun would then be a good metric to indicate the accuracy of our estimation method.


4

Figure 1: Translating business needs at conceptual to operational and quantitative parameters


ReduceCosts

MinimizeDefects

Improve Estimation

What is the rework

Effort?

What arecurrent defectlevels?

How goodis effort

estimation?

ReworkEffort

Defect Density

Effort Slippage

Goal Question Metric

- Identifying value added activities- Detecting defects early- Reducing rework

non- -

- Detecting defects as early as possible- Minimize acceptance testing defects

- Estimating effort based on requirements- Planning based on estimation

Approach

Establish measurementgoals aligned with

business needs i.e. whatwe want to achieve

Establish the set of data,objective or subjective,

that will answer eachquestion in a

quantitative way

Generate a set ofquestions that can be

used to determineachievement of goals

Conceptual Level

Operational Level

Quantitative Level

5

GQM Keeps Us Anchored to Changing Business Imperatives

Using SPC to Control Process Performance

A large project calculated billable value in terms of the project effort expended until the time of costing. However, the

value in terms of project deliverables and milestones met were found to be lesser than the effort spent. This was

primarily due to schedule slippages at certain phases in the project. Due to the enormity of the project, the actual

value delivered during the project remained confusing. In this case, the business need was to deliver the milestones

as per schedule. The question that comes to mind is what milestones have been delivered and what was the

corresponding estimated effort? This suggested tracking the delivered value in terms of Earned Value. Earned value

is the value of the deliverables made in terms of estimated effort, irrespective of the slippages. This gave the project a

better understanding of how much of true value is being delivered by the project at a given point in time, vis- a- vis

the total effort spent.

Now, when we have selected the right metric for monitoring project or process performance, how do we ensure that

the performance suggested by the metric is within acceptable limits? Or, how do we know that the project will be

able to deliver the deliverable as per plan? Statistical Process Control (SPC) is a technique that is useful in this. It

defines the operational limits for acceptable variation in the process data.

Statistical Process Control primarily follows four steps. These are:! Measuring the process! Reducing variation to acceptable limits! Monitoring and controlling! Improving the process to achieve target value

While many tools are used in process improvement, with each being applicable to specific contexts, we will discuss

the use of Control Charts to control process variation. Control chart is plotting of the process parameter over time to

identify causes of any abnormal variation. Any parameter would have some variation that would be confined to an

acceptable or “normal” range. An instance of the high variation would suggest an underlying assignable or “special”

cause, which would then have to be eliminated. Normally, an instance of variation lying outside the range of u + 3s (u

is the mean and s the standard deviation) is considered to be abnormal or “outlier”. The limits u + 3s and u – 3s are

referred to as Upper Control Limit (UCL) and Lower Control Limit (LCL) respectively. Processes having variation within

the range LCL-UCL are considered stable.


Figure 2

The reason we chose to discuss control chart in this paper is that we find the technique useful in gauging the

capability of the process, than just the stability. Besides UCL and LCL, the process can set its own limits (target for

process performance). This is referred to as Lower Specification Limit (LSL) and Upper Specification Limit (USL). A

capable process would have a variation well within the LSL-USL range. The LCL-UCL of a capable process is called

Process Capability Baseline (PCB). As the process matures to be more capable, the USL-LSL range is shortened with a

more tapered peak (“high Kurtosis”) at the mean.

6


1020

1010

1000

990

980

970

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Sample number

LCL

UCL

Mean

Out ofcontrol

Abnormal variationdue to assignable sources

Abnormal variationdue to assignable sources

Normal variationdue to chance

3s3s

LCL UCL

USLLSL

ProcessCapability

Band

The bell-curve becomes:! “Taller”! “Thinner”with more process improvements

Time X Time X Time X

Improvements

Capability Over Time

Further Improvements

Continuous Improvement Needs Breakthroughs where it Matters

Continuous improvement essentially involves improving the performance of the process so that the specification

limits are satisfied and the process becomes more capable. Once the specification limits are satisfied, tighter

specification limits or “stretch targets” are defined and another cycle of continuous improvement is initiated. We call it

a continuous improvement journey because the improvement actions are not warranted by exceptions or “special

causes”, but are more proactive in nature. It addresses the “common causes”, which are inherent in the nature of the

process. In the context of control charts, capability improvement in processes requires identifying the right

breakthroughs, which would shift the mean and reduce variation to make it centered on the mean. In simpler words,

this means reducing variation to the extent that it improves end quality and shifts the mean to the desired

specifications. Often, we reduce variation on a parameter even when the process is robust with the existing level. For

example, we may go overboard on improving the performance variation of a transaction when its highest variation is

well accommodated in the actual transaction sequence. Rather, the business objective would be to reduce the mean

time taken to perform the series of related transactions in the practical business scenario. Reduction of the mean

would require a “breakthrough” action or process change. This is because the old process may have already hit its

maximum performance state given the existing process definition. To improve from here, one would need to

introduce new interventions or activities within the process. For example, we may often find that, despite sufficient

testing, there are a high number of defects during production. A process alteration such as peer review of source

code can help in detection of potential defects.

7


As depicted in the following figure, breakthrough improvements will lead to change in the mean performance of the

process.

To summarize, quality improvement starts with identifying the right metrics. We apply techniques such as control

charts to improve the stability of the process. Thereafter, we embark on a continuous improvement journey by

increasing the capability of the process. All through, a fact-based approach is needed. The typical approach is

illustrated as follows:

8


Original zone ofQuality control

Chronic waste

BreakthroughImprovement

New zone ofQuality control

Less chronic waste

Business Objectives

Sub-Selection

process

Measure

StatisticalAnalysis

(OPP)

ProcessStable?

ProcessCapable?

Remove SpecialCauses

Remove Common

Causes

Process/ Technology

Change

Yes

No

Yes

No

Process Change

Project-Change

level

Org.Change

Piloting


9

Conclusion

A quantitatively managed process would have metrics aligned to business needs. Such needs are better

understood when we define the business goals. One can translate the goals to a set of representative metrics by

asking the right questions, which is an important part of the exercise. The GQM approach helps in this translation.

Once the right metrics are defined, SPC helps in monitoring the performance of the metrics and bringing the

process under control. We have discussed control chart as one of the tools that is useful here. As we move into the

continuous improvement journey, we try to reduce variance where it matters and find breakthroughs/ process

changes to shift the mean. This makes the process more capable.

At the end of the day, all a good process needs is the right metrics used to monitor it and good benchmarks set to

control its performance and continuously improve its performance.

All content / information present here is the exclusive property of Tata Consultancy Services Limited (TCS). The content / information contained here is correct at the time of publishing. No material from here may be copied, modified, reproduced, republished, uploaded, transmitted, posted or distributed in any form without prior written permission from TCS. Unauthorized use of the content / information appearing here may violate copyright, trademark and other applicable laws, and could result in criminal or civil penalties.

Copyright © 2008 Tata Consultancy Services Limited

About Tata Consultancy Services (TCS)Tata Consultancy Services is an IT services, business solutions and

outsourcing organization that delivers real results to global

businesses, ensuring a level of certainty no other firm can match.

TCS offers a consulting-led, integrated portfolio of IT and IT-

enabled services delivered through its unique Global Network

Delivery ModelTM recognized as the benchmark of excellence in

software development.

A part of the Tata Group, India's largest industrial conglomerate,

TCS has over 100,000 of the world's best trained IT consultants in 50

countries. The company generated consolidated revenues of US

$5.7 billion for fiscal year ended 31 March 2008 and is listed on the

National Stock Exchange and Bombay Stock Exchange in India. For

more information, visit us at www.tcs.com.

www.tcs.com

[email protected]

About TCS IT Process and Service Management ConsultingIT Process and Service Management Consulting partners with

organizations to enable IT process improvement aligned to

business needs. Leveraging TCS' experience in successfully

adopting best-inclass frameworks like CMMI®, ISO-9001, ITIL®,

P-CMM®, BS15000, BS7799, and Six Sigma, the consulting group

provides services with a practitioner’s approach to assessing,

defining, improving and deploying IT processes.

CMMI® appraisal services and training

ITIL deployment and assessment

Agile development for distributed environment

Six Sigma-based improvements

Quality improvement initiatives

Digitized enablers which reduce 'Time to Deploy' and sustain

productivity for process

l

l

l

l

l

l

white paper on software quality

Documents