The Future of Software Analysis & Measurement

The Future of

Software Analysis & Measurement

Daniel D. Galorath

40+ years Software & Software

Management Experience

Founder & CEO Galorath Incorporated,

SEER By Galorath

1980 MBA Management,

California State Universities

1984 Began working software

estimation, planning & control

2009 Society of Cost Estimation and

Analysis (SCEA) Lifetime Achievement

award

2001 ISPA Freiman Award,

lifetime achievement award

2006 Book: Software Sizing, Estimation,

and Risk Management

SEER parametrically estimates project

cost, effort, duration, cost & risk

Answer Fundamental Management

Questions

How much effort is required

to complete an activity?

How much calendar time is

needed to complete an activity?

What is the total cost of an activity?

How reliable will it be?

What can we do better?

What If?

www.galorath.com

SEER® Empowering Project Estimation, Planning & Control Since 1988

David Herron

Business Development Manager and VP of

Knowledge Solution Services for David

Consulting Group

Consulting and coaching services for a variety

of IT organizations throughout the US and

Canada

Acknowledged authority in the areas of

performance measurement, process

improvement and organizational change

management

Noted author and lecturer; co-authored several

books on topics relating to IT performance

measurement and Function Point Analysis

David Consulting Group

DCG is an international IT process improvement and measurement

company managing value-driven engagements with companies and

government agencies around the world

Software Process Improvement

Utilizing CMMI, Six Sigma, Lean and Agile methods

Software Sizing

Using IFPUG Function point Counting and alternative sizing techniques

Software Measurement

Providing roadmap planning, estimation models, performance benchmarks

and outsourcing SLA support

IT Performance Improvement

Improving IT operations through ITIL and IT Governance

Dr. Bill Curtis

Senior Vice President and Chief Scientist

Industry luminary responsible for influencing

CAST’s scientific and strategic direction

Best known for leading development of the

Capability Maturity Model (CMM) which has become

the global standard for evaluating the capability of

software development organizations.

Prior to joining CAST,

Co-Founder of TeraQuest, the global leader in CMM-

based services,

Directed the Software Process Program at the Software

Engineering Institute (SEI) at Carnegie Mellon University.

CAST Application Intelligence Platform

Most enterprises measure everything but the product delivered to the business

CAST Application Intelligence Platform (AIP) measures the product itself

Robustness Performance Security Changeability Transferability Size

Planning Estimation Scheduling Time Tracking Cost Tracking

Product Process

Time & Duration

Effort & Budget Function &

Scope

Quality

&Size

Requirements Earned Value User Acceptance Usability

CAST AIP

QUESTIONS

What are the single most important business drivers that lead an organization

to embark on a SAM initiative?

How does software analysis and measurement relate to business outcomes?

What organizational behaviors are most favorable for a successful SAM

program?

What is the relationship between SAM and IT governance?

How can measurement and analysis interact with estimating to provide

improved visibility and information to make the best management decisions

regarding software?

How can anyone provide a viable estimate software development, software

maintenance, or continuing innovation work and what use it is?

How can we determine which software projects with have the highest ROI?

What are the single most important

business drivers that lead an organization

to embark on a SAM initiative?

HBR Article explains this Phenomenon:

Humans seem hardwired to be optimists

We routinely exaggerate benefits and discount costs

Delusions of Success: How Optimism Undermines Executives' Decisions

(Source: HBR Articles | Dan Lovallo, Daniel Kahneman | Jul 01, 2003)

11

Solution: Temper with “outside view”

Software Analysis & Measurement can temper

Viable estimation can temper

Don’t remove optimism, but balance optimism & realism

Human Nature Optimism in Software

Cutter Consortium Software Project Survey:

62% overran original schedule by more than 50%

64% more than 50% over budget

70% had critical product quality defects after release

Standish Group CHAOS Report

46% challenged

19% failed

35% successful

~$875 billion spent on IT

~$300 billion spent on IT projects

~$57 billion wasted annually

Ever increasing “Technical Debt”

ROI of Applied Software Analysis & Measurement is HUGE

IT Failures are Pervasive: And Even Successful Projects may not be

Manual Estimates

Human reasons for error (Software Analysis & Measurement can

help)

Desire for “credibility” motivates overestimate behavior (80%

probability?)

So must spend all the time to be “reliable”

Better approach force 50% probability & have “buffer” for overruns

Technical pride causes underestimates

Buy-in causes underestimates

13

Important Business Drivers for Structural Measurement

1 Poor software

from suppliers

2 Embarrassing

disasters

3 High cost of

maintenance

How does software analysis and

measurement relate to business outcomes?

Goal Question Metric Approach

Combine goal-orientation bottoms up, decision-support & other

operational management techniques

Going to weather.com and deciding to bring an umbrella is decision support

Goal

Question

Development

Contractors

Metric

Organizational Goal

Question

Development

Organizations

Metric

How Does Measurement Relate to Business Outcomes

TRANSFERABILITY

allows new teams to quickly

begin working with an application

CHANGEABILITY

makes an application easier and quicker to modify

ROBUSTNESS

improves application stability & reduces injecting new defects

PERFORMANCE

Reduces degraded response

times and increases scalability

SECURITY

affects an application’s ability to

prevent unauthorized intrusions

Optimize work

productivity

Maximize

customer loyalty

Maximize business

agility

Minimize business

risks

Minimize IT costs

Business

Value

Application Health

Factors

Tactical Objectives

Reduce learning curves

Improve software readability

Ease team handoffs

Reduce vendor lock-in

Reduce modification effort

Reduce cost of ownership

Accelerate new function delivery

Reduce application rework

Maximize application availability

Minimize liquidated damages

Minimize degraded service

Maximize application scalability

Reduce injected defects

Reduce application mistakes

Maximize speed of response

Maximize information retrieval

Maximize information protection

Maximize customer confidence

Maximize standards compliance

Minimize unwanted breaches

Cost

Risk

What organizational behaviors are most

favorable for a successful SAM program?

Key Organizational Behaviors

Trust but verify

Viable measurement and analysis

Answering the right questions

Having actions based on measurement & analysis

Measure what can make a difference

Measure to the audience needs (Goal / Question / Metrics)

Generating the Business Value Side of the Equation (Benefits)

The business owns benefit calculations

IT should participate

Exception: projects solely improving internal IT

Beware of subjectivity translating soft benefits

Use probability and risk

Increased Revenue

Increased Profit

Reduction in cost (people, processes,

cash out) Internal benefits

Intangible Benefits

Conditions that Favor Structural Measurement

Low

maturity

Fed up

managers

Mature Disciplined

processes

High

maturity

Quantitative

management

What is the relationship

between SAM and IT governance?

(Source: Fraunhofer)

Financial

Customer Business Process

Innovation / Growth

Vision & Strategy

Goals

Data Collection

Measurement Metrics

Answers Questions

Goal Attainment

Software Development & Maintenance

6σ

GQM

PSM

CoBIT…

Business/ Organization

Level

IT Governance

IT Governance

Customer Focus

The Gap Between IT & Management Needs

How Structural Measurement is Tied to IT Governance

The Evolution of Governance

Govern the

Department

Budget headcount

Govern the

Project

Schedule defects

Govern the

Product

Structural quality

How can measurement and analysis interact

with estimating to provide improved visibility

and information to make the best management

decisions regarding software?

What a Parametric Model can Tell You

26

What is likely to happen

Feel lucky?

Firm Fixed Price?

Understand the risk before you commit!

SAM Feeds Estimation and Provides Insight & Choice

Cost, schedule, risk

Should we update or

redevelop?

What is the risk of continuing

with software with increasing

“technical debt”?

SEER-SEM Software

Estimation Model

Size

Complexity

Maintainability/

Defect Potential

People, Process, Technology

SAM Outputs

Effort

Schedule

Risk

Reliability

/Defects

Historical Results

(SAM Outputs)

Constraints

Maintenance Effort

Required

Looking at Maintenance and Technical Debt

Software Estimation Basic Model & Associated Metrics

Start Finish

Effort K

Size St ReuseDIT

Size Se (work units)

Size (Effective Se & Total St)

Defects Count (Qi Qr)

Calendar Time

On-going Iterations of Effort (ACWP or Spent)

Progress (BCWP or Earned Value) Defects (Qi Qr )

Growth (Sg)

Software Development

Process

Staffa &

Constraints

Effective Technology Cte

People Process

Technology

Stakeholder Requirements

Delivered Software

Effective complexity D

Maintenance/ Block Change Development

Process

Development Legacy,

Maintenance Specifics & Constraints

and/or

Block Changes As Redevelopment

Fundamental Metrics

Size

AKA Volume, Mass

Units: Source Lines of Code

(SLOC); Function Points (FP)

Use Cases

New versus rework

COTS & Packages

Effective Technology

AKA Productivity Potential,

Efficiency

Units: none

Time

AKA Duration, Schedule

Units: Calendar Months,

Calendar Weeks

Effort

AKA Work, Labor

Units: Staff Months, Staff Hours

Cost

AKA Budget, Money

Units: $, other currencies

Staffing

AKA Manpower Loading

Units: FTE People

Defects

AKA Reliability, Quality

Units: Defect Count

Other Key Metrics Help Track Project Performance

Track defect discovery and removal rates

against expected rates

Increased defect reporting rate

shows a worsening trend

Heath and Status Indicator shows status and trends from

the previous snapshot

Thresholds are user definable

When & Why to Collect Data

When to Collect

1. Up-front: when scoping new

project data from completed

projects

2. In-Process: During

development for management,

to identify issues and progress

3. Post Mortem: Upon

development completion to

improve corporate history

repository

4. In Service: During

maintenance to continue

learning & improving

Why People Don’t Want to

Provide Data

They could be proven wrong

It could be used against them

Data often doesn’t exist

Even if processes dictate data

requirements

If it exists it may not be clean

It may give away corporate

productivity & bid strategy

How can anyone provide a viable estimate

software development, software maintenance, or

continuing innovation work and what use it is?

10 Step System Estimation Process 2011

1. Establish Estimate Scope

2. Establish Technical Baseline, Ground

Rules, Assumptions

4. Refine Technical Baseline Into

Estimable Components

4. Collect data / estimation inputs

5. Estimate Baseline Cost, Schedule, Affordability Value

6. Validate Business Case Costs &

Benefits (go / no go)

6. Quantify Risks and Risk Analysis

8. Generate a Project Plan

9. Document Estimates and Lessons

Learned

10. Track Project Throughout

Development

Level 0 Informal or no estimating

Manual effort estimating without a process

Level 1 Direct Task Estimation

Spreadsheets Ad Hoc Process

Level 2 Formal

Sizing (e.g. function points)

Direct Task Estimation

Simple model (Size *

Productivity) or informal SEER

Use

Some measurement

& analysis

Informal Process

Level 3 Formal Sizing

Robust Parametric estimation

(SEER)

Estimate vs. actual capture

Formalized Multiple Estimate Process

Rigorous measurement

& analysis

Parametric planning &

Control

Risk Management

Repeatable process

Level 4 Formal sizing Repeatable

process

Robust parametric estimating

(SEER)

Rigorous measurement

& analysis

Parametric estimation

with tracking & control

Risk Management

Process improvement via lessons

learned

Level 5 Formal sizing Repeatable

process

Robust parametric estimating

(SEER)

Rigorous measurement

& analysis

Parametric estimation

with tracking & control

Risk Management

Continuous process

improvement

Why should we care? Maturity is related to estimate viability… With better estimation process, projects more likely to be successful in execution

Estimation Organizational Maturity V1.7

ESTIMATION & PLANNING: An Estimate Defined

An estimate is the most knowledgeable statement you can make at a

particular point in time regarding:

Effort / Cost

Schedule

Staffing

Risk

Reliability

Estimates more precise with progress

A WELL FORMED ESTIMATE IS A DISTRIBUTION

36

Avoid Surprises with Estimation Process & Tools

Challenged projects

Would you still go forward if you knew

Schedule would be significantly longer?

Cost would be dramatically higher?

Probably: but perhaps more insight could identify mitigation

Plan functionality differently

Certainly you could notify stakeholders of real costs

Ensure staffing is appropriate for the constraints

Failed Projects

Would you start a project you knew was unaffordable? Or if schedule was

completely unrealistic?

If knowing up-front could you do something about it?

often better to kill project before it begins than waste resources & let the

organization down

How can we determine which software

projects with have the highest ROI?

Software Projects Must Return Business Value

“Economics is primarily a science of choice… software

economics should provide methods for analyzing the

choices software projects must make.” Leon Levy

“Base choices on those providing the maximum business

value to the organization” Eli Goldratt

Measurement and its uses such as estimating and defect

analysis help this science of choice

Some say business value is not our problem

While others generally need to perform benefit analysis

We need to build systems that optimize the business

Make IT part of the solution

If IT & measurement don’t generate sufficient profit, money will go elsewhere

Software & IT Systems are about Business Value

Cost

Value

An ROI Analysis of an Upgrade: Software Analysis & Measurement Provides Valve Cost

Can we do better? Will stakeholders tolerate a loss for 3 years?

What is the risk?

Measurement Perspective

We measure to ultimately

produce business value to the

organization

MEASURED Measurement

should not ultimately be a cost

The analysis of measurements

produces decisions that

produce business value

Management & Stakeholder Perspective

Speak so I can understand

Give me actionable items

Don’t just give me problems…

Give me solutions

Help me make the best decisions

so we can produce business

value

Measurement itself is not the answer….

Management decision making, better performance, quality improvements and better serving stakeholders is

Measurement Manifesto for Software & IT

Learn more about CAST

www.castsoftware.comblog.castsoftware.com

www.facebook.com/castonquality www.slideshare.net/castsoftware www.twitter.com/OnQuality