infostretch - business driven quality metrics: beyond counting defects

phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]

©2004-2011 InfoStretch Corporation. All rights reserved.All trademarks are the property of their respective owners.©2004-2010 InfoStretch Corporation. All rights reserved.

Business Driven Quality MetricsBeyond Counting Defects

March, 2011


©2004-2011 InfoStretch Corporation. All rights reserved.

1. Introduction

2. Types QA Metrics and Examples

3. Business Driven Metrics – Going Beyond SDLC

Industry Focused, Competitive - Success Metrics

Cost, Timelines, Effectiveness – Optimization Metrics

End-User / Customer focused Metrics

4. Metrics Driven Actions

5. Q & A

Agenda

INFOSTRETCH CORPORATE(p) (408) 727-1100 | (w) ww.infostretch.com | (e) [email protected]©2004-2010 InfoStretch Corporation. All rights reserved.



(p) (408) 727-1100 | (w) ww.infostretch.com | (e) [email protected] | Presenter: Rutesh Shah


InfoStretch OverviewElevating Confidence

mailto:[email protected]



Organization HQ in Silicon Valley Offices in New York, Seattle 105+ Clients and 160+ Projects Strong contributor to QA Community

People Average 6 years experience 300+ Engineers/Consultants Industry Expertise in Finance,

Retail, Mobile and Healthcare Strong Project Mgmt. team

Infrastructure Extensive Platform

Compatibility and Performance Test Labs

2000+ Mobile Handset Test Lab ISO 9000-2001 Certified High Availability and 24 hours

secure offshore centers with ISO 27001 Audited

Overview

INFOSTRETCH CORPORATE(p) (408) 727-1100 | (w) ww.infostretch.com | (e) [email protected] | Presenter: Rutesh Shah©2004-2011 InfoStretch Corporation. All rights reserved.

Partners




RecognitionUnique Perspective – Leadership Team with Entrepreneurial, Startup, Fortune 500, Top Technology & Big 4 Consulting Firms Experience.

Long Term Relationships – Focus on building long term relationships and competencies around client 100% reference-able. Clients including T-Mobile, AT&T, Amazon, Intel, TNS, Harmony & Wells Fargo

Strong Growth – Strong Year over Year Growth and Profitable Operation Since 2004. Significant Investments in Platform Compatibility and Device Labs

(p) (408) 727-1100 | (w) ww.infostretch.com | (e) [email protected]

19th among Fast growing

privately owned

companies in Bay Area

Awarded Inc 500/5000

company of the year 2010

Part of 10 Most Promising

Testing Services Company

Gartner Top 20 QA Service Provider

Exceptional QA Services and

Solutions

GESIA Rising Star Award 2010 – Best

Software Product

http://www.inc.com/inc5000/2010/index.html

http://www.thesmarttechie.com/index.php

http://www.gesia.org/index.php



(p) (408) 727-1100 | (w) ww.infostretch.com | (e) [email protected] | Presenter: Rutesh Shah


QA MetricsWe best Manage, What we can Measure






Metrics – The Map

“Standards of measurement by which efficiency, performance, progress, or quality of a plan,

process, or product can be assessed."





If only Metrics ruled the World

Note: Cartoon by Andy Glover





Quality Metrics – The Guide

“Measuring whether or not a given process will produce

products and services that meet quality standards.”





Business Metrics – The Compass

“Measuring the outcome of business objectives with-in the defined constraints –

Predictability of process

Resource utilization

product/service quality

Timeliness

Market Impact

Customer Satisfaction

Financial Sanity





Lost without a Guide or Quality Metrics





What is a successful outcome?

Measurement

Impact Criteria

Happy Customer &

Business Success

Outcome

Quality

Financial &BudgetTimeline

CustomerSatisfaction

Emp Happine

ss

Competition Impact

Sales

< Cust.Support





Influence Metrics

Influence Metrics from Quantity to Quality



Why Quality Metrics?Companies that consistently produce high quality products and services will have high customer loyalty and a strong corporate brand.

Is this a good business practice?

Is this guaranteed?

What about Cost, Timelines, Competitive Threat?



Business MetricsLet us to do what is important to Business



Types of Business Metrics – QA Centric

Defect Metrics

Resource Utilization

Metrics

Process Effectiveness

Metrics

Requirements Coverage Metrics

Test Coverage Metrics

Automation ROI Metrics

Quality Trend (Optimization)

Metrics

Compliance Metrics

Performance Metrics



Quality MetricsDefect Reports

Lookup33%

Updates3%

Cache8%

Playlist30%

Lists3%

Core12%

Submit6%

Discover5%

Defect Density by Com-ponent

Rel1

Rel1.1

Rel2

Rel2.1

Rel3 Rel4 Rel5 Rel6

Rel6.1

Rel6.2

Rel6.3

Rel7

0

20

40

60

80

100

120

0

5

10

15

20

25

30

Defect trends

Total Defects

Total Fixed Defects

Total Open defects




1 (Catastrophic)Severity

9%

2 (Critical)Severity

50%

3 (Moderate)Severity

31%

4 (Minor)Severity

10%

Defect Severity Report

<4 weeks 4-10 weeks >10 weeks

0

10

20

30

40

50

60

70

80

90

100

Open Defects (Aging Report)

Open Defects




Rel1

Rel1.1

Rel2

Rel2.1

Rel3 Rel4 Rel5 Rel6

Rel6.1

Rel6.2

Rel6.3

Rel7

02468

101214161820

Defects By Release

Total Defects

Defects Fixed

Defects Deferred

Release

De

fec

ts

Rel1

Rel1.

1Rel

2

Rel2.

1Rel

3Rel

4Rel

5Rel

6

Rel6.

1

Rel6.

2

Rel6.

3Rel

7

4

8

4

1113

1

6

11

18

4

12

8

1 1 1 0 0 0 0 0 1 0 0 0

Pre and Post Product De-fects

Pre Production Defects Post Production Defects




GUI2%

DataBase1%

Cosmetic12%

Business Logic46%

Error Handling2%

Server Response21%

Performance/ Optimization1%

Logical Errors6% Security Errors

8%GUIDataBaseCosmeticBusiness LogicError HandlingCompatibilityServer ResponsePerformance/ Op-timizationLogical ErrorsSecurity Errors

Rel1

Rel1.

1Rel

2

Rel2.

1Rel

3Rel

4Rel

5Rel

6

Rel6.

1

Rel6.

2

Rel6.

3Rel

70.00

20.00

40.00

60.00

80.00

100.00

120.00

Defect Removal Efficiency

Series1



QA Metrics – Examples & Importance

•These metrics give the status of quality of products based on defects and its impact on quality

Defect Based Metrics

• These metrics show the test as well as business requirement coverage as part of the testing effort

Coverage Based Metrics

•These metrics give the details of resource utilization and efficiency details which helps to optimize the QA/Testing operation. These metrics give the details surrounding effectiveness of testing process to meet quality objectives

Process Effectiveness/

Resource Efficiency / Optimization

Metrics



QA Metrics – Examples & Importance

• These metrics give the overall quality picture for product over the life span which enables to understand what is working and what is not working

Quality Trend Metrics

•These are special purpose metrics created specifically for business outcomes

Solution/Client Specific

Metrics

•It also includes the impact of Automation related ROI in testing organization

Automation ROI Metrics

• These metrics are usually produced as part of performance testing engagements.

Performance Metrics



Business Quality MetricsEfficiency and Optimization

4.3.

84.

3.9

4.3.

104.

4.0

4.4.

24.

3.0

4.3.

24.

3.9

050

100150200250300350400

Testcase Maintenance For Release

Deleted TestcasesModified TestcasesNew Testcases

John Klien

Jeff Warner

Rakesh Kamani

Leslie Debra Nick Mencha

0

50

100

150

200

250

Resource Utilization and De-liverables

Testcases DevelopedDefects LoggedTestcases Automated



Business Quality MetricsEfficiency and Optimization

4.3.

6.1

4.3.

6.2

4.3.

6.3

4.3.

7.1

4.3.

7.2

4.3.

7.3

4.3.

7.4

4.3.

7.5

4.3.

7.6

4.3.

8.1

4.3.

8.2

4.3.

8.3

4.3.

8.4

4.3.

8.5

4.3.

9.1

4.3.

9.2

4.3.

9.3

4.3.

9.4

4.3.

10.1

4.3.

10.3

4.3.

10.4

4.3.

11.1

4.3.

11.2

0

200

400

600

800

1000

1200

1400

0

20

40

60

80

100

120

140

160

180

Product Test Optimization per Release

Time for Testing Total Testcases

Tota

l Tes

tcas

es

Time For Testing



Business Quality MetricsValue Driven - Automation and ROI

1 2 3 4 5 6 7 80

200

400

600

800

1000

1200

1400

Automation ROI

Manual Testcases Automated TestcasesManual Execution Time Automated Execution Time



Business Quality MetricsEnd-Result Driven – User Satisfaction and Trend

Rel1 Rel2 Rel3 Rel402468

1012141618

Usability Measure

New Features Features Used in 1st monthDefects logged

Rel1 Rel2 Rel3 Rel40

200

400

600

800

1000

1200

1400

1600

Regression Efficiency

Regresion TestcasesRegression Time

Axis Title



Business Driven Quality MetricsOverall Indicator

4.10

4.20

4.30

4.40

4.50

4.60

4.70

4.80

4.90

5.00

5.10

Stability IndexStability IndexStability Index

Stability IndexStability Index


Stability Index





Stability Index

Stability Index

Stability Index

Stability Index

Stability Index

Product - Stability Index

Stability Index

Index Description

1Poor Quality2Becoming Stable3Needs Improvement4Desired Stability5Highly Stable

Quality indicators of the testing effort

using complex formula

reasonably indicates product quality

overall quality variation from

release to release

Stability Index



Metrics Driven ActionMake the change and measure to analyze the impact



QA Metrics – Actions and Impacts

Metrics Description Frequency Actions

Effort & Schedule Variance

Effort spent versus Effort planned & Actual End Date Versus Planned End Date

Monthly • Adjust Resources and/or Project Timeline

• Incorporate learning for next phase planning

Time Variance

Number of releases that were successfully delivered to QA within the timelines

Monthly • Calculate the cost/quality impact due to delay

• Development schedule check




Metrics Description Frequency Actions

Test Containment

Efficiency

# Defects detected in Testing / (# Defects detected in Testing + Escaped defects detected by Client or subsequent testing)

Monthly • Modify testcases and coverage as necessary

• Retire the testcases that are not aiding value or repetative

Defect Sanity

Number of Valid Defects / Total Number of Invalid + Valid defects)

Monthly • Review testcases and refersh tester knowledge about the product

• Ensure completeness of defect description




Metric Description Frequency Actions

Test Case Efficiency

Number of Test Cases which are finding defects Vs Total Number

of Test Cases

End of testing round

• Review and Enhance/Retire testcases

• Link testcases to requirement, Usecases and/or defects

Cost of Testing

Ratio of Testing effort versus Total Project effort. Cost

effectiveness for Onsite vs offshore testing cost.

Monthly • Adjust tester focus based on the module/build/release importance

• Cross-train testers for different types of testing




Metric Description Frequency Actions

ROI on Automation

Calculate the Return on Investment on Automation (ROI

Calculator)

Quarterly • Measure reduction in regression time and compare with faster time-to-market

• Framework/Keyword driven automation for reduced maintenance

Usability and Performance

Measure the Usability of application with customer’s usage of features intuitively

(How many tech support calls?)

Release • Try new features with different UI

• Measure the usability and performance with customer data volume



Best Theory

Mobile App DesignIt’s an art, not a science

Presenter: Morgan Russell, InfoStretch Corporation

All trademarks are the property of their respective owners.©2004-2011 InfoStretch Corporation. All rights reserved.

InfoStretch Next Webinar

Mark your CalendarApril 21, 2011

Mark your CalendarApril 21, 2011

Santa ClaraHeadquarters2880 Lakeside Drive, #200Santa Clara, CA 95054(408) 727-1100

Partners

New York17 Tanglewood Rd,Scarsdale, NY 10583(703) 395-9791

Ahmedabad5th Floor, Regency Plaza, Anand Nagar, Ahmedabad-15, Gujarat, India91-79-40266266

MumbaiVashi InfoTech Park, B103, Tower # 1, Vashi Rail Station, New Mumbai, Maharashtra, India91-22-6947900

Website: www.infostretch.com



infostretch - business driven quality metrics: beyond counting defects

Technology

infostretch corporation

rights rights

infostretch w

e email

business metrics

business driven metrics

types qa metrics

metrics driven actions