measuring roi in low maturity - wibas...basics of improvement: process focus the engineering process...

SPIN - Darmstadt - 18 Jan 2008

©Q:PIT Ltd 2008 1

ROI in Low Maturity ©Q:PIT Ltd 2005-2008 1

Copyright StatementSM Architecture Tradeoff Analysis Method, ATAM, CMM Integration, COTS Usage

Risk Evaluation, CURE, EPIC, Evolutionary Process for Integrating COTS-based systems, Framework for Software Product Line Practice, IDEAL, Interim Profile, OAR, OCTAVE, Operationally Critical Threat, Asset, and Vulnerability Evaluation, Options Analysis for Reengineering, Personal Software Process, PLTP, Product Line Technical Probe, PSP, SCAMPI, SCAMPI Lead Appraiser, SCE, SEPG, SoS Navigator, T-Checks, Team Software Process, TSP, are service marks of Carnegie Mellon University

TM Carnegie Mellon Software Engineering Institute (stylized), Carnegie Mellon

ROI in Low Maturity

g g g ( y ), gSoftware Engineering Institute (and design), Simplex, Stylized hexagon are trademarks of Carnegie Mellon UniversityQ:PIT and the Q:PIT logo are trademarks of Q:PIT Ltd, Milton Keynes, UK

® Capability Maturity Model, Capability Maturity Modeling, Carnegie Mellon, CERT, CERT Coordination Center, CMM, CMMI, FloCon, OCTAVE are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University

©Q:PIT Ltd 2005-2008 2

Contents5 Reasons for FailureBasics for ImprovementPrinciples of MeasurementRequirements for Measurement


qProcess for MeasurementPutting into Practice


©Q:PIT Ltd 2008 2


Five Reasons to FailObjectives

Focus on the satisfying the model, not on the business

BudgetDemonstrate return on investment within the year

Technical change


Technical changeLet the techies do this: it is their problem

We know what is neededDon’t get the customer involved today, they will be happy that we improve

Do it quicklyWe can’t afford to wait



©Q:PIT Ltd 2008 3

Basics of Improvement:Business Focus

Focus on what you need to deliver to your market:Less defects, increased qualityReduced costs, better investmentQuicker time to marketBeing Maturity Level X is rarely a business issue


Basics of Improvement: Culture

Process Improvement is not a technical activity:Focus on the cultural change that is neededEnsure that the cultural aspects are understood by managementGeneric goals are more important in the long-term than specific goals


p gWe need people to focus on quality and on improvement, not on complianceWe need management to demonstrate what they want

Basics of Improvement: Leadership

Keep your eye on the horizonLead by exampleLearn from mistakesEncourage the right attitude rather than the short-term result


Lead so that people can follow


©Q:PIT Ltd 2008 4

Basics of Improvement: Quality Assurance

Assurance that people know what they are doingAssurance that you know how they are doingAssurance that they know why they are doing itIt is only by knowing what is really happening that we


y y g y pp gcan learn and progressQuality assurance needs to be seen as a proactive support, not just compliance audits and punishment

Basics of Improvement: Configuration Management

Configuration management gives us the foundations on which the rest can be built

Every area, every process needs to ensure the integrity of the systemThis is more than just version controlThis does require investment in a tool


This does require investment in a tool

Basics of Improvement: Process Focus

The Engineering Process Group needs to be implemented at Level 1

Even though OPF is a maturity level 3 process area, the ownership and co-ordination of processes and improvement paths need to be implemented from the beginningH i th OPF i l d t th t d t


Having the OPF in place does not mean that you need to jump over ML2 and have everyone operating at ML3


©Q:PIT Ltd 2008 5

Basics of Improvement: Process Framework

Set up the framework for the futureConsider what you are planning on doingConsider how you want to structure the processesSet up a process architectureSet up a process repositoryS t f ti d t ti l


Set up a means for reporting processes and potential improvementsIdentify what and how you want to measure

Basics of Improvement: Measurement

If you don't know what you want to achieve, you will not achieve itIf you cannot measure your achievement, you will not achieve itIf you don't know how far you have come you will not


If you don t know how far you have come, you will not know if you have arrivedIf you cannot demonstrate that you have improved, you will lose your support

Setting Up the BasicsSetting up all these things requires a significant investment

How much should you invest?What will you get out of it?How to persuade management that this is an investment and not a cost?


and not a cost?How to persuade the board / the shareholders that they are getting benefits out of this?How to ensure that this is improving quality of products and the life of the participants, not just creating more bureaucracy?


©Q:PIT Ltd 2008 6


Purpose of MeasurementEnsure that you are improving what needs to be improvedDemonstrate that the improvements are useful to the businessDemonstrate that progress is supporting the business


Demonstrate that progress is supporting the businessDemonstrate that this is an investment, not a cost

Principles Of Measurement 1/2

Specifying the objectives of measurement and analysis such that they are aligned with identified information needs and objectivesSpecifying the measures, data collection and storage mechanisms, analysis techniques, and reporting and feedback mechanisms


feedback mechanismsImplementing the collection, storage, analysis, and reporting of the dataProviding objective results that can be used in making informed decisions, and taking appropriate corrective actions

Adapted from CMMI® - SEI 2003


©Q:PIT Ltd 2008 7

Principles Of Measurement 2/2

Objective planning and estimatingTracking actual performance against established plans and objectivesIdentifying and resolving process-related issues


Providing a basis for incorporating measurement into additional processes in the future


What to MeasureThere is not that much we can measure:

Timeo Elapsedo Effort

CostQuality (defects)


Quality (defects)Size

Progress and BaselinesAbsence of measures is the key reason that organizations are ML1: if you knew the cost of quality, you would already have improved itAbsence of baselines is the key reason that ML1 organizations do not measure


We don't measure, because we have no measurements!The time to establish the baselines is start measuring is now

But where to start?


©Q:PIT Ltd 2008 8

Establishing MeasuresThink outside the box:

What do you need?What do you have?How long do you need to establish a baseline?


Quick BaselinesThink outside the box

Establish a 1-week baseline

Could the productivity of the coming week be representative of your general productivity?


p y

Measure the TrendStart measuring the trend without a baseline:

What is the change between two projects?What is the different with yesterday or last week?

Y d t d t h


You do not need to have a baseline in order to determine the difference between two or three points

02468

101214

M.01 M.02 M.03 M.04 M.05 M.06 M.07 M.08 M.09 M.10 M.11


©Q:PIT Ltd 2008 9

Reverse BaselineMeasure your previous project to establish a baseline

What happened previously?What were the results?What went wrong?What needs to be changed?


Estimate the OppositeEstimate the cost of note having done it:

What if you had not identified this risk?How long would it take if you had only identified this requirements defect in acceptance testing, not in the requirements review?What would have been the cost if QA had not identified


Qthis?

What Do You Have?Consider what you already have

Milestone slippagesDefects reportedTime in testingBudgetO ti


Overtime


©Q:PIT Ltd 2008 10


CMMI®Align Measurement and Analysis Activities

Establish Measurement ObjectivesSpecify MeasuresSpecify Data Collection and Storage ProceduresSpecify Analysis Procedures


Provide Measurement ResultsCollect Measurement DataAnalyse Measurement DataStore Data and ResultsCommunicate Results


FAA-ICMMGoals

Measures related to goals, objectives and major issues are establishedMeasurement data are collected, analysed, and results are reportedMeasurement data and results are stored for use


Some general categories of measures:Schedule and progressResources and costProduct size and stabilityProcess performanceProduct/service qualityContractor performanceTechnology effectiveness

Adapted from FAA-ICMM - Published by the Federal Aviation Administration, 2001.


©Q:PIT Ltd 2008 11

Trillium Measurement MapLevel 2

CostsProduct MeasurementProject MeasurementProcess MeasurementProcess and Product FailuresCustomer Satisfaction

Level 3Measurement PlanningProcess MeasurementProduct MeasurementData AnalysisProcess and Product FailuresCustomer Satisfaction


Level 5Statistical Process Control

Level 4Process MeasurementQuality Management

Adapted from Trillium (Roadmap 3.4 – Measurements) - © Copyright Bell Canada 1994

Trillium Level 2 Measurements 1/2

CostsProject cost factors

Product MeasurementSize, system resources, performanceAdditions, changes and deletions to requirements,


, g q ,design, code and test specifications Activities for managing requirements (e.g. status, change activity, cumulative number of changes)

Project MeasurementMilestone completion, work completed, effort extendedEffort and resources, change activity for the development plan


Trillium Level 2 Measurements 2/2

Process MeasurementSupplier costs and delivery date compared to planQuality system costs and number of audits compared to planConfiguration management effort and funds, completion of milestones compared to plan


p p

Process and Product FailuresFailures and statistics throughout the life-cycle of the product



©Q:PIT Ltd 2008 12


The 6-Sigma Approach: DMAICDefine

Understand what are the key processes

MeasureIdentify and collect measurements of success

Analyse


AnalyseSee what are the issues

ImproveMake useful improvements

ControlManage the process

DMAIC : DefineCharterCustomer needs and requirementsHigh-level process mapAgree on the high-level steps


g g p


©Q:PIT Ltd 2008 13

DMAIC : DefineCharter

Business case Why do we want to change anything? What is the business reason for spending time on this?

Problem statementA detailed description of what is the problemPossibly include a quantitative description of the expected benefits and targetsWhat is the scope of this project hat is incl ded


Project scope What is the scope of this project, what is included, where are the limits?

Goals and objectives Description of what we hope to achieve through this approach

Milestones Dates and deliverables for each of the main phases

Roles and responsibilities

Who is taking responsibility for this problem? Who is the sponsor? Who are the process improvement team members?

DMAIC : DefineCustomer needs and requirements


DMAIC : DefineHigh-level process map 1/2

Name the processEstablish the start and stop points of the processDetermine the output(s) of the processDetermine the customer(s) of the processDetermine the supplier(s) of the processD t i th i t( ) f th


Determine the input(s) of the process


©Q:PIT Ltd 2008 14

DMAIC : DefineHigh-level process map 2/2


DMAIC : MeasureDefine how you are going to measure:

Create a data collection mapCreate data collection forms

Implement your data collection planStart making some statistics:


Start making some statistics:Calculate baseline sigmaEstablish a basic control chart

DMAIC : MeasureData collection map creation

What to measureType of measureType of dataOperational definitions


pTargets / specificationsData collection formsSampling


©Q:PIT Ltd 2008 15

DMAIC : Measure Data collection plan

Data collection formsDetermine what a defect isDetermine reason codes for defectsDetermine the time frame for data collectionDetermine a grid for data collection


DMAIC : Measure Calculating Baseline

Standard Sigma Baseline:1000000 * (number of defects / (number of units * number of opportunities))

The Sigma baseline is not a requirement but a possibility

O h d f t t bli h


Once you have made a few measurements, establish a baselineThis does not have to be based on a statistically representative sample, but it does need to be understood for what it is

DMAIC : AnalyseData Analysis

DiscreteContinuous

Process analysisSub-process mapping


p pp gNature of work


©Q:PIT Ltd 2008 16

DMAIC : AnalyseData Analysis 1/2

DiscreteParetoPie chart

ContinuousFrequency distribution check sheet


q yo Machines in the processo Materials used in the processo Methods in the processo Nature or environmento The measurement itselfo People

Run chart

DMAIC : AnalyseData Analysis 2/2

8%

16%

13%

11%23%

11%

8%

6% 4%

0

5

10

15

20

25

0 1 2 3 4 5 6 7 8 9 10

Proj

ect S

taff

Initial PlanEstimatedSpent


0 1 2 3 4 5 6 7 8 9 10

Months

-50

0

50

100

150

200

0 3 6 9 12 15 18 21 24

Months

Cum

ilativ

e K

LOC

Unit Tested (New and Modified) Integrated

DMAIC : AnalyseProcess Analysis

Sub-process mappingNature of work

Sub-process selectiono Customer considers the sub-process importanto There is a physical changeo It is done right the first time


gNon-value-added activities

o Internal failureo External failureo Delayso Controls and inspectionso Preparation and set upo Moveso Value enabling

Micro-problem statement


©Q:PIT Ltd 2008 17


Define ObjectiveBasic objectives for process improvement:

Cycle time (efficiency)Cycle time slippage (repeatability)Effort to find and fix defectsEffort slip rate (predictability)C t t d d f t ( lit )


Customer reported defects (quality)Feature slip rate

CharterBusiness case Improving the satisfaction of existing customers will directly

impact repeat business

Problem statement Customers are currently not satisfied because we regularly deliver late

Project scope All projects within the local development centre

Goals and objectives Identify how we can reduce the time to deliver


Goals and objectives Identify how we can reduce the time to deliver

Milestones

Define:Measure:Analyse:Improve:Control:

1 week from now3 weeks from now4 weeks from now2 months from now6 months from now

Roles and responsibilities

Manager to provide dataMeasurement specialist to provide analysis


©Q:PIT Ltd 2008 18

Define Customer NeedsCustomer

Satisfaction

Meet Promises Faster thanCompetition


Deliver on Time Few Defects

Able to promisewhat we can deliver

Plan reasonabledeadlines Meet deadlines

All Functionality

Define High-Level Process Map


Define Data Collection MapWhat to measure

Cycle time

Type of measureTime

Type of dataMinutes

Operational definitions


Operational definitionsTime between moment customer enters the restaurant and first bite into productTime taken to complain and respond is added to value

Targets / specificationsBring total time below 4 minutes

Data collection formsSee attached

SamplingMeasure the time for every 20th customer


©Q:PIT Ltd 2008 19

Measure ResultsNr Entered Ordered Served Complaints Total

42 12:15 12:17 12:18 - 3

43 12:25 12:27 12:32 - 7

44 12:35 12:40 12:43 - 8


45 12:50 12:58 13:05 00:02 17

Average:

Variation

15

20

25

Cus

tom

ers Time to Serve


0

5

10

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Minutes

Num

ber o

f C

TrendMean = Σx / NStd Deviation = (Σx2 / N)-M2Control Limit = ±(3 SD)

14

16


0

2

4

6

8

10

12

1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106 111 116 121 126 131 136 141


©Q:PIT Ltd 2008 20

Identify Sub-ProcessesSub-Processes:

QueuingChoosingWaiting for selection

o Sandwicho Fries


o Frieso Drinko Deserts

Paying

Identify the detailed measurements

Now Do ItDefine your improvementsKeep on measuringDemonstrate that you are doing the right thingBecause, if you cannot measure it, you cannot improve


, y , y pit



©Q:PIT Ltd 2008 21

Measurement ChartDefine what you need to measureDefine what are the added-value elements

Step Value Added Non VA Time

Request bid X 24

Bids are reviewed

Xwait 48

Finalists contacted X 1


Re-negotiate bid Xerror 24

Try alternative vendor

Xprep. 48

Wait for final submission

Xwait 48

Award contract X 1

Contact vendor X 1

Total 196

Cause and Effect Diagram

Why are there somany waits

Machine Methods Measurement

Drive throughmicrophone

problems

Fryer Failure

Fryer warm-up

Basket holes

Order delays

Pushproduction

ProductionLayout

CommunicatingOrders

Cooking timeaccuracy

Altitude adjustmentin cooking time

Turnover

Lack ofmotivation

SubstandardOil

Food toofrozen

Need tore cook


Environment People Materials

re-cook

Identify the root causes of problems, classify them into general categoriesSuggested general categories are Machine, Methods, Measurement, Materials, People and Environment, but others are possible

HistogramClassify sequential events, see the “bell-curve”, in which the most frequent case is bound by examples of lesser frequency tapering off in both directions


0

5

10

15

20

25

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Minutes

Num

ber o

f Cus

tom

ers Time to Serve


©Q:PIT Ltd 2008 22

Pareto Chart

3.6

3

2.2

no money

forgotten item

misc

Group elements based on frequency of a problem occurring, allows to rapidly identify the biggest issue.


36.3

22

13.3

11.2

8.4

0 5 10 15 20 25 30 35 40

price check

override

wrong item

register tape

bag

Some Additional ReadingSix-Sigma for Everyone

George Eckes, 2003John Wiley and Sons, EditorISBN 0-471-28156-5

Application of the Indicator Template for Measurement and Analysis


yWoflhart Goethert and Jeannine Siviy, 2004Software Engineering Measurement and Analysis InitiativeSEI Technical Note CMU/SEI-2004-TN-024

Metrics and Models in Software Quality EngineeringStephen H. Kan 2003Addison Wesley / Pearson EducationISBN 0-201-72915-6

Contact InformationPeter Leeson

Q:PIT Ltd1 Harby CloseEmerson ValleyMilton KeynesMK4 2BBUnited Kingdom

ROI in Low Maturity

United KingdomDirect Line: +44 (0)20 8433 4120Tel / Fax: +44 (0)1 908 506 908Mobile/Cell: +44 (0)773 998 98 67E Mail: [email protected]: http://www.qpit.ltd.uk

©Q:PIT Ltd 2005-2008 66

measuring roi in low maturity - wibas...basics of improvement: process focus the engineering process...

Documents