Metrics Training Program Conducted by QAI

Measurement Framework

ProcessPerformance Measurement

Performance Goals

Improvement Plan

Improvement Comparison

Measurement

Analysis

What Should Change?

Did it Work?

Projects… Project Managers

Processes… PEG, QAD

Business… Business Manager

Multiple Level of Responsibilities

Improvements and Measurement

Identify Scope

Collect Data

Analyze Data

Define Procedures

Evolve Process

GQM

CMM

Continuous Improvement

Model

Measurement Need

Your Measurement Process Need to be a Dynamic One!!

Without the right information, you are just another person with an

opinion- Tracy O’Rourke

Why We Measure What We Measure• To Characterize Project• To Evaluate Product• To Predict Quality• To Improve Process

Metrics- KISS Approach

Easy to Understand, Able to Improve, Simple to

Define, Easy to Measure

Measurement Scales• Nominal- Type of Project• Ordinal- Customer Satisfaction

Level• Interval- Temperature• Ratio- Mean Time Between Failure• Absolute- Difference of One, Origin

Present, e.g.-Number of Defects

Philip B Crosby-Quality Without Tears

Material

Information

INPUTS OUTPUTS

Performance Standards

Procedures

Facilities and Equipment

Training and Knowledge

What to Measure?• Basic Measures-SizeEffortScheduleDefectCost

Common Assumptions

1. Cost is Proportional to Effort

2. Schedule= Cycle Time

Cycle Time is the Minimum Completion Time for a Particular Type of Project

What to Measure?• Derived MeasuresProductivityQualityChangeDefect LeakageEffort DistributionRate of Defect InjectionReview EffectivenessRisk

Derived Measures• Productivity- WC per PH, Hours per WC• Defect Injection Rate- Denominator

may not be in PH alone (Defect/Page for Tech. Doc. Activity)

• Change- Change in Size is a Surrogate measure for Change in Requirement

Derived Measures• Quality-Cost of

Quality={(Review+Rework+Testing+Change+Audit+Training Effort+Defect Prevention)/Total Effort}*100

Cost of Poor Quality={(Rework+Testing+Change+Audit Effort)/Total Effort}*100

Try to Identify the Value Added Items

Value Added Items• Product Quality- if Delivered Defect/Size Delivered Defect= Defect Injection

Rate*Size*(1-Defect Containment Efficiency)

Defect Prevention, Training reduces DIRReview Increases Defect Containment

Efficiency

What is Risk?• Risk is an event• Risk is unpredictable• It always has negative impact

Risk is a Derived Measure

Consider External Dependencies before assessing Risks

Review and Test Effectiveness

• Review Effectiveness= Weighted Defects found in Review/Review Effort

• Testing Effectiveness= Weighted Defects found in Testing/Testing Effort

The Management Dashboard

WorkloadDemand

Quality

Tools Readiness

Capability

Productivity

DeliveryDevelopment/Maintenance

QualityFunctional Technical Satisfaction DefectsQuality of Design Service Level MTBFO/P Quality

Delivered Defects

Reliability Interop- Security erability Usability Maintain- ability

Quality may be measured w.r.t. the implicit and specified requirements

Productivity

Size

WC/PH

Size

WC/Day

Delivery Rate

Deviation from estimate

%Commitment

Development:Maintenance Ratio

• D:M Ratio=Effort in Creation/(Total Effort-Effort in Creation)

Tool Penetration• Tool Penetration

Phase

Tool A

Tool B

% Using the tool

% Not Using the tool, who are supposed to use it

Others• Capability- In terms of Maturity

Level• Readiness- Response Time, Server

Downtime• Productivity- Use Run Charts to

track over a period of time, Team Size, Domains, Technology

Project Management• Metrics Required for- Team Members/Module Leaders Project Managers Senior Management Quality Management

Metrics- TM/ML• Individual Planning (PSP)- Tasks

and Durations• Individual Tracking- Allocated vs

Actual Schedule, No. of Hours for Completion of Tasks

• Quality Tracking- Reviews/ Testing Vs Faults

Metrics- Project Managers

• Required Data for Project Management- Size (Actual and Estimated), Task Allocation and Completion Schedule, Actual and Estimated Effort, Impact of Size Variation on Schedule

• Project Review Meetings

Project Review- Progress Tracking

• Schedule- Original, Latest,Actual• Risk- Planned, Forecast, Actual, Index• Requirement Volatility-Impact on Effort• Staffing• Cumulative Milestones• Variation Chart

Project Review- Quality Tracking

• In Phase Defects-Errors• Planned vs Actual- Product Quality• Cost of Quality, Cost of Poor

Quality• Phase Containment Efficiency• Delivered Defect Density

Metrics –Senior Management

• Planned Completion Date vs Status• Planned Efforts vs Status• Planned Size vs Status• Injected and Detected Defects-Planned vs Actual• Testing-Test Progress vs Status of Completion• Filed Test Bugs-Status• Delivered Defects-Status• Technology Changes- Progress (Schedule)• Process Changes (Schedules)• Customer Satisfaction Measurement Status

Management of Quality• Review Data• Testing Data• Delivered Defect Data

Process Control• Process Metrics• PDB• PCB

Project’s Process Control

• Process Capability Levels• Actual Measurement• Special Cause of Variation• Projects Process Control - Control Schedules - Control Efforts - Control Quality

Organization’s Process Improvements

• Decrease of Variation - Estimation Variance - Defects Variance - Rate of Completion• Increase of Performance - Improve in Productivity - Decrease of Cost (CoQ) - Decrease in Cycle Time - Decrease of Delivered Defects

Metrics and CMM KPAs• Metrics at Level 2- Estimates and

Actuals• Metrics at Level 3- Threshhold for

Management of Size, Effort, Schedule, Cost, Quality, Critical Resources, Risks

• Metrics Database• Defect Data analysis

Metrics and CMM KPAs• Metrics at Level 4-• Process Management - Baselines - Process Stabilization - SPC - Process Goals and Projects’ Process Control

• Quality Management- - Quality Goals - Quality Progress - Defect Goals

Metrics and CMM KPAs• Metrics at Level 5-• Defect Prevention - Reduce CoQ, CoPQ• Technology and Process Change

Management- - Reduce Cycle Time - Improve Productivity - Reduce Defect Density

Seven QC Tools• Flow Diagrams• Cause-Effect Diagrams• Check Sheets• Histograms• Pareto Charts• Scatter Diagrams• Control Charts

Seven QC Tools• Cause and Effect Diagram: Three

Type• Dispersion Type• Production Process Classification

Type• Cause Enumeration Type

(Methods, People, Input, Tools)

Seven QC Tools• Histogram- Be aware of the Time Interval• SPC- Peer Review (Separate for Doc.

Inspection and Code Inspection) and Test Effectiveness can be under SPC

• Achieve Process Stability before using control charts

• Update Control Limits at Regular Intervals• Pareto Charts- Always Mention the Time

Period• Structured Brainstorming

Causal Analysis Method• Analyze Defect Data• Pareto Analysis• Brainstorming• Cause-Effect Analysis• Action Planning• Implementation of Action Plan• Check for DP Effectiveness- Decrease in CoQ, CoPQ- Absent of Similar Defects in future reviews

Recommended Readings

• Measures for Excellence- Putnam• Measuring the Software Process-

Florac, Carlton• Practical Software Metrics- Brady• Memory Jogger II• Humphrey’s book on Software

Process