measuring dollar savings from software process improvement with cocomo ii betsy clark software...
Post on 22-Dec-2015
218 views
TRANSCRIPT
![Page 1: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/1.jpg)
Measuring Dollar Savings from Software Process Improvement
with COCOMO II
Betsy Clark
Software Metrics Inc.October 25, 2001
Acknowledgment: This presentation describes work being done by TeraQuest Metrics
![Page 2: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/2.jpg)
Outline
• Background• Measuring the Impact of Software Process
Improvement (SPI)• Some Initial Results
![Page 3: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/3.jpg)
Customer Background
• Large financial institution • Actively involved in software process improvement
(SPI)– Software-CMM– System Test
• Began summer of 2000 at CMM Level 1• Incrementally adding Key Process Areas• Two pilot organizations
– Planning Level 2 assessment end of this year
![Page 4: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/4.jpg)
Background (continued)
• Strong emphasis on measuring impact of SPI, especially hard dollar savings
• CIO: “If process improvement saves us money, I should be able to go down the street to my competitor’s bank and get a loan to fund our process improvement initiative.”
![Page 5: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/5.jpg)
Outline
• Background• Measuring the Impact of Software Process
Improvement (SPI)• Some Initial Results• Conclusions
![Page 6: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/6.jpg)
“Maturity levels are meaningless if they cannot be explained in terms of business objectives”
John D. VuBoeing
Level 5 Organization
![Page 7: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/7.jpg)
Business Objectives
• Reduce the cost of software activities• Reduce delivery time• Improve product quality• Increase customer satisfaction
– customers are internal to the bank (e.g., wholesale and retail mortgage, investment division)
![Page 8: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/8.jpg)
Measurement Objectives
• Measure impact of SPI in terms of these business objectives
• Impacts of SPI will be measured by comparing a set of baseline projects to pilot projects
![Page 9: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/9.jpg)
Measuring Hard Savings
• CFO’s initial understanding - – “If we have savings from SPI, we can reduce IT budget in
the future.”– First point of discussion - need to measure work load– Led to concept of unit savings, holding IT organization
accountable for those savings
• Brought IT manager into the discussion -– “But events occur outside of my control that can affect unit
costs. For example, I can lose my top staff.”
![Page 10: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/10.jpg)
Measuring Hard Savings
• The IT manager was talking about variability due to factors outside of SPI.
• That variability is addressed by parametric cost models.
• Approach - measure COCOMO II cost drivers for baseline projects and for SPI projects. Use them to adjust unit costs.– Backout all influences on unit costs except SPI
![Page 11: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/11.jpg)
Measuring Hard Savings (cont)
• Savings due to SPI– Difference in adjusted unit costs between baseline and SPI
projects
![Page 12: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/12.jpg)
Setting Expectations
• SPI is a staged, long term initiative– implemented on pilot projects first, then on a wider scale
• Initially, we will estimate savings based on pilot results– few data points, wide variation
• As SPI is implemented on a wider scale, we will have more data points, clearer trends
• Moving from CMM Level 1 to Level 2 lays the foundation for unit cost savings– a few studies do show cost savings from Level 1 to 2
• major effect is in better estimation and planning• reduction in rework due to stable requirements
![Page 13: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/13.jpg)
Measures
1) estimation accuracy: effort
2) estimation accuracy: schedule
3) productivity
4) unit costs
5) project delivery rate (cycle time)
6) system test effectiveness
7) delivered defect density
8) customer satisfaction
9) requirements volatility
![Page 14: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/14.jpg)
Approach
• Attempted to “mine” existing data sources (e.g., time tracking, financial, problem reporting systems)– not successful, sporadic and inconsistently used
• Selected a representative set of completed projects from the two pilot organizations
• Goal was 10-15 projects per pilot organization– 13 projects from one– 11 from the other
• Constructed a survey, met with project managers to collect data
• Followed-up with each manager to verify data
![Page 15: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/15.jpg)
Measures
1) estimation accuracy: effort
2) estimation accuracy: schedule
3) productivity
4) unit costs
5) project delivery rate (cycle time)
6) system test effectiveness
7) delivered defect density
8) customer satisfaction
9) requirements volatility
![Page 16: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/16.jpg)
Planned Labor Hours
Per
cen
t d
iffe
ren
ce b
etw
een
act
ual
an
d e
sti
mat
ed
0
Overruns
Underruns
Calculation:(Actual labor hours - estimated) / estimated
Estimation Accuracy - Effort
![Page 17: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/17.jpg)
Planned duration
Per
cen
t D
iffe
ren
ce b
etw
een
act
ual
an
d e
sti
mat
ed
0
Overruns
Underruns
Calculation:(Actual calendar months - estimated) / estimated
Estimation Accuracy - Schedule
![Page 18: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/18.jpg)
Measures of Interest
• Median - very stable across organizations• standard deviation• Goals with SPI:
– median should approach zero– standard deviation should be smaller
![Page 19: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/19.jpg)
Measures
1) estimation accuracy: effort
2) estimation accuracy: schedule
3) productivity
4) unit costs
5) project delivery rate (cycle time)
6) system test effectiveness
7) delivered defect density
8) customer satisfaction
9) requirements volatility
![Page 20: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/20.jpg)
Productivity and Unit Costs
• High variability• Median is stable across divisions
![Page 21: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/21.jpg)
Initial Results
• Used COCOMO II parameters to adjust size• Led to a reduction in the standard deviation• Helped explain:
– why lower productivity projects had difficulty– why higher productivity projects had an easier time
• Projects with very high productivity seemed to do everything right– capable staff, low turnover, managing requirements…– these are good things that should improve with SPI– don’t want to penalize organization for improvement in these other
(non-SPI) areas– management controllables vs noncontrollables
![Page 22: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/22.jpg)
Measures
• 1) estimation accuracy: effort• 2) estimation accuracy: schedule• 3) productivity• 4) unit costs• 5) project delivery rate (cycle time)• 6) system test effectiveness• 7) delivered defect density• 8) customer satisfaction• 9) requirements volatility
![Page 23: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/23.jpg)
Project Delivery Rate
• Calculation:– Function points / calendar months
• Goal: Increasing
![Page 24: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/24.jpg)
Function Points
Fu
nct
ion
po
ints
pe
r ca
len
da
r m
on
ths
Project Delivery Rate
![Page 25: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/25.jpg)
Measures
1) estimation accuracy: effort
2) estimation accuracy: schedule
3) productivity
4) unit costs
5) project delivery rate (cycle time)
6) system test effectiveness
7) delivered defect density
8) customer satisfaction
9) requirements volatility
![Page 26: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/26.jpg)
System Test Effectiveness
• Calculation:– (Defects Found in System Test / Total Defects)– where – Total Defects = (Defects Found in System Test +
Delivered Defects found in first 30 days)– Example:– Defects found in System Test = 45– Defects found in first 30 days of operations = 5– Test Effectiveness = 90%
• Goal: 100%• Result: Wide variation in effectiveness
![Page 27: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/27.jpg)
Measures
1) estimation accuracy: effort
2) estimation accuracy: schedule
3) productivity
4) unit costs
5) project delivery rate (cycle time)
6) system test effectiveness
7) delivered defect density
8) customer satisfaction
9) requirements volatility
![Page 28: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/28.jpg)
Delivered Defect Density
• Calculation:– Defects found in first 30 days of operations / function points
• Goal: 0
![Page 29: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/29.jpg)
Function Points
Def
ects
per
fu
nct
ion
po
ints
0
COTS
Custom
Delivered Defect Density
![Page 30: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/30.jpg)
(Very Preliminary) Finding of Interest• In contrast to custom development, defect density for
COTS projects appears unrelated to size
![Page 31: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/31.jpg)
Measures
1) estimation accuracy: effort
2) estimation accuracy: schedule
3) productivity
4) unit costs
5) project delivery rate (cycle time)
6) system test effectiveness
7) delivered defect density
8) customer satisfaction
9) requirements volatility
![Page 32: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/32.jpg)
Customer Satisfaction, Rqts Volatility
• Data do not exist• Strategy was altered to request the manager’s
estimate
![Page 33: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/33.jpg)
Message to Executive Level
• Measurement – can be a powerful foundation for understanding and
managing IT– is a cultural change and not a scoreboard– will improve as process maturity improves
![Page 34: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/34.jpg)
Response from Executive Level (CIO and direct reports)• Intense interest in the measures and in
benchmarking• Basis for excellent discussions about need for
visibility into – requirements management– quality– customer satisfaction
• Collection of the nine measures has been made part of executive compensation– Moving forward to put supporting processes, tools and
training in place
![Page 35: Measuring Dollar Savings from Software Process Improvement with COCOMO II Betsy Clark Software Metrics Inc. October 25, 2001 Acknowledgment: This presentation](https://reader030.vdocuments.site/reader030/viewer/2022032704/56649d785503460f94a5a201/html5/thumbnails/35.jpg)
To be continued...