1

Douglas GabelDirector, Quality Management Office

"Implementing a Successful Goal-Driven KPI Approach in Quality"

2

Agenda

• cover the formula for SUCCESSFUL

• personal choice KPI’s / Metrics

3

“Count what is countable, measure what is measurableand what is not measurable, make measurable.”

- Galileo, (1564 – 1642)

4

Why do we measure quality?

The first answers that pop into your head might be:• you can’t manage what you don’t measure

• what you measure gets done • we have to be accountable • they have to be held accountable

• they told us to

5

Today

• many in management still feel quality testing processes are fluff, overhead, or added process

• quality teams need to give management more visibility into the value of quality testing

• the challenge has always been determining what to measure and what KPI’s to use

• we need to align quality testing with business outcomes to determine if the testing processes really contribute to the improvement of business objectives

6

Going Forward

• we need to determine:– what to measure– what metrics to use– what is the best process to follow– what KPI’s are important to the business– what metrics are really needed to learn or

understand the business– what can drive improvement

7

Basis for presentation

• For those who have been delivering quality testing for some time, this basic approach is a derivation from the Mercury Quality Model. It has been proven successful and is a logical and simple series of steps that can benefit any Quality organization as a measurement strategy.

To read more, Reference “Optimize Quality For Business Outcomes”, A Practical Approach to Software Testing by

Andreas Golze, Charlie Li, Shel Prince

8

Formula for SUCCESSFUL

1. Set business goals2. Understand the impact of departments on those

business goals3. Choose supporting business processes4. Create business-process goals5. Examine what to measure6. Standardize measurements across departments7. Scope data source and integration needs8. Formalize indicators and thresholds9. Utilize and execute appropriate action plans10. Lay the groundwork and baseline

9

Set business goals

• establish high-level (specific) goals to measure against – not simple objectives

• document them – write them down – communicate them

• for each initiative you should have at least 3 or so goals

• be sure to identify milestones / checkpoints

10

Understand the impact of each department/group on business goals

• groups having direct impact on the goal like Development, QA, Operations, Support, etc…should be evaluated

• leveling resources, adjusting budgets, narrowing scope may be necessary activities to ensure success

• evaluate non-IT demand departments for consequential impact

11

Choose supporting business processes

• Evaluate the various attributes of each business process that affects IT costs (staff, software, hardware, training, etc…)

• challenge where streamlining can occur

• value in stepping back and strategize at a high level where tactical changes can be made to the overall business model

12

Create business process goals

• these goals are sub-goals that assist with accomplishing a quality outcome

• keep these goals to a limited set for tracking and reporting purposes

13

Examine what to measure

• establish a quantifiable set of questions to ask for each sub-goal (it’s easy to identify what metrics to use or measure when we ask the right questions – of the right people)

• continue to ask why we measure, and take the accuracy of what you are measuring seriously

14

“Everything that can be counted does not necessarily count; everything that counts cannot

necessarily be counted.” Albert Einstein

15

Standardize measurements across departments

• avoid using metrics that only one department will benefit from

• keep metrics simple to avoid mis-interpretation

• build consistency into any measures you utilize for greatest acceptance

• find ways to interpret results with accuracy and be able to properly compare results across departments

16

Scope data source and integration needs

• always weigh the feasibility and ease of getting the data when creating metrics

• allocate enough resources and time to set up the technology required to capture the data you want

17

• data / result thresholds must be agreed upon so anyone looking at the data / results will interpret it the same way

• thresholds help with the structure of reports and dashboards that management can understand and use

• avoid representing every bit of data / results in a report / dashboard – this only clutters and confuses the message / findings

Formalize indicators and thresholds

18

Utilize and execute appropriate action plans

• all KPI’s / thresholds aligned with business goals build the foundation for action plans

• be prepared to act on pre-defined thresholds and ready to respond to dynamic business changes when thresholds are met

19

Lay the groundwork and baseline

• understand your environment, the state of your business

• take a snapshot of the initial state of the business and use this as a baseline for comparison in the future

• communicate, communicate, communicate your adherence and actions taken

20

In Summary

• SUCCESSFUL is a goal-driven approach with a logical and simple series of steps to follow to create a measurement strategy

• this is an iterative process – revisit when goals are met or need to be realigned

21

Useful Measures / Metrics / KPI’s / Baselines

22

What does a good set of metrics look like?

A good system of metrics provides a basis for the following:

• Summary charts, showing the application related defect counts over time, their open and closure rates, and the progress towards delivery, policy compliance and risk assessment goals

• the numbers necessary to answer management’s questions about “Is the application contributing to our business goals?”, “How robust and/or secure is the application?” or “Does this application put us at risk in meeting our business objectives?”

• application/security defect density (that is, the average number of application/security defects per unit of code is being monitored and the rate is going in the right direction: Down!)

23

Useful References

The following are a number of useful KPI’s / Metrics for most organizations:

• Defect Removal Efficiency (DRE) – # defects in test phase / (# defects in test phase + # defects in prod 1st 30 days)

• Test Case Effectiveness (TCE) – # defects not mapped to test cases / # defects mapped to test cases

• Traceability / Code Coverage– % of test cases that map directly back to requirements and % of requirements

that map directly to test cases

• Planned vs. Actual Execution– line chart test cases planned vs. executed daily

24

Useful References

The following are a number of useful KPI’s / Metrics for most organizations:

• Defects Distribution by Severity/Status– bar chart defects by Severity (by project/cycle?)– bar chart defects by Status (by project/cycle?)

• Defects Distribution by Root Cause– bar chart defects by Root Cause (by project/cycle?)

• Testing Effort– actual testing effort (person hours) / total overall project effort (person hours)

25

Useful References

Suggested Baselines to start:

• Defect Removal Efficiency (DRE) – 90% defects removed

• Test Case Effectiveness (TCE) – 10% not mapped

• Traceability / Code Coverage– 98%

• Planned vs. Actual Execution– 8% deviation +/- from plan

• Defects Distribution by Severity/Status– bar chart actuals

• Defects Distribution by Root Cause– bar chart actuals

• Testing Effort– 25 to 30%