Software Metrics

ByTouseef Tahir

Agenda

- Introduction

- Metrics in the Process Domain

- Metrics in the Project Domain

- Software Measurement

- Integrating Metrics within the Software Process

Definitions• Measure - quantitative indication of extent, amount, dimension,

capacity, or size of some attribute of a product or process.– E.g., Number of errors

• Metric - quantitative measure of degree to which a system, component or process possesses a given attribute. “A handle or guess about a given attribute.”– E.g., Number of errors found per person hours expended

• Indicator- a combination of metrics that provide insight into the software process or project or product itself.

What are Software Metrics?

AppliedTo

MeasurementBased

Techniques

Engineering &ManagementInformation

ToImprove

SoftwareProcesses,Products &

Services

ToSupply

Measurement Defined

Entity:Attribute:

Features & Properties

Mapping:Numbers

&Symbols

6

Metrics of Process Improvement

• Focus on Manageable Repeatable Process

• Use of Statistical SQA on Process

• Defect Removal Efficiency

7

Statistical Software Process ImprovementStatistical Software Process Improvement

All errors and defects are categorized by origin

The cost to correct each error and defect is

recorded

No. of errors and defects in each category is counted and ranked in descending order

The overall cost in each category is computed

Resultant data are analyzed and the

“culprit” category is uncovered

Plans are developed to eliminate the errors

8

Metrics of Project Management

• Budget• Schedule/Resource

Management• Risk Management• Project goals met or

exceeded• Customer satisfaction

9

Metrics of the Software Product

• Focus on Deliverable Quality

• Analysis Products• Design Product

Complexity – algorithmic, architectural, data flow

• Code Products• Production System

Two Historic Schools of Thought1. Collect data on everything -- then find

meaning

2. Implement a random selection of metrics

Jeopardy approach to metrics -- start with the answer and try to guess the

question

Motivation for Metrics

• Estimate the cost & schedule of future projects

• Evaluate the productivity impacts of new tools and techniques

• Establish productivity trends over time

• Improve software quality

• Forecast future staffing needs

• Anticipate and reduce future maintenance needs

12 Steps to Useful Software MetricsStep 1 - Identify Metrics Customers

Step 2 - Target Goals

Step 3 - Ask Questions

Step 4 - Select Metrics

Step 5 - Standardize Definitions

Step 6 - Choose a Model

Step 7 - Establish Counting Criteria

Step 8 - Decide On Decision Criteria

Step 9 - Define Reporting Mechanisms

Step 10 - Determine Additional Qualifiers

Step 11 - Collect Data

Step 12 - Consider Human Factors

Metric Classification

• Products– Explicit results of software development activities– Deliverables, documentation, by products

• Processes– Activities related to production of software

• Resources– Inputs into the software development activities– hardware, knowledge, people

Product vs. Process

• Process Metrics – Insights of process paradigm, software engineering

tasks, work product, or milestones – Lead to long term process improvement

• Product Metrics – Assesses the state of the project– Track potential risks– Uncover problem areas– Adjust workflow or tasks– Evaluate teams ability to control quality

15

What are Metrics?• Software process and project metrics are quantitative

measures• They are a management tool• They offer insight into the effectiveness of the software

process and the projects that are conducted using the process as a framework

• Basic quality and productivity data are collected• These data are analyzed, compared against past averages,

and assessed• The goal is to determine whether quality and productivity

improvements have occurred• The data can also be used to pinpoint problem areas• Remedies can then be developed and the software process

can be improved

16

A Quote on Measurement

“When you can measure what you are speaking about and express it innumbers, you know something about it; but when you cannot measure,when you cannot express it in numbers, your knowledge is of a meagerand unsatisfactory kind; it may be the beginning of knowledge, but youhave scarcely, in your thoughts, advanced to the stage of science.”

LORD WILLIAM KELVIN (1824 – 1907)

Role of Measurement

Processes,Products &Resources

Characterize Predict

Evaluate Improve

18

Reasons to Measure

• To characterize in order to– Gain an understanding of processes, products, resources, and

environments– Establish baselines for comparisons with future assessments

• To evaluate in order to– Determine status with respect to plans

• To predict in order to– Gain understanding of relationships among processes and products– Build models of these relationships

• To improve in order to– Identify roadblocks, root causes, inefficiencies, and other opportunities for

improving product quality and process performance

Metrics in the Process Domain

20

Metrics in the Process Domain

• Process metrics are collected across all projects and over long periods of time

• They are used for making strategic decisions• The intent is to provide a set of process indicators that lead to long-

term software process improvement• The only way to know how/where to improve any process is to

– Measure specific attributes of the process– Develop a set of meaningful metrics based on these attributes– Use the metrics to provide indicators that will lead to a strategy for

improvement

(More on next slide)

21

Metrics in the Process Domain(continued)

• We measure the effectiveness of a process by deriving a set of metrics based on outcomes of the process such as– Errors uncovered before release of the software– Defects delivered to and reported by the end users– Work products delivered– Human effort expended– Calendar time expended– Conformance to the schedule– Time and effort to complete each generic activity

22

Etiquette of Process Metrics

• Use common sense and organizational sensitivity when interpreting metrics data

• Provide regular feedback to the individuals and teams who collect measures and metrics

• Don’t use metrics to evaluate individuals• Work with practitioners and teams to set clear goals and metrics that will

be used to achieve them• Never use metrics to threaten individuals or teams• Metrics data that indicate a problem should not be considered “negative”

– Such data are merely an indicator for process improvement• Don’t obsess on a single metric to the exclusion of other important

metrics

Metrics in the Project Domain

24

Metrics in the Project Domain

• Project metrics enable a software project manager to– Assess the status of an ongoing project– Track potential risks– Uncover problem areas before their status becomes critical– Adjust work flow or tasks– Evaluate the project team’s ability to control quality of software work

products

• Many of the same metrics are used in both the process and project domain

• Project metrics are used for making tactical decisions– They are used to adapt project workflow and technical activities

25

Use of Project Metrics

• The first application of project metrics occurs during estimation– Metrics from past projects are used as a basis for estimating time and effort

• As a project proceeds, the amount of time and effort expended are compared to original estimates

• As technical work commences, other project metrics become important– Production rates are measured (represented in terms of models created, review

hours, function points, and delivered source lines of code)

– Error uncovered during each generic framework activity (i.e, communication, planning, modeling, construction, deployment) are measured

(More on next slide)

26

Use of Project Metrics (continued)

• Project metrics are used to– Minimize the development schedule by making the adjustments necessary to

avoid delays and mitigate potential problems and risks

– Assess product quality on an ongoing basis and, when necessary, to modify the technical approach to improve quality

• In summary– As quality improves, defects are minimized

– As defects go down, the amount of rework required during the project is also reduced

– As rework goes down, the overall project cost is reduced

Software Measurement

28

Categories of Software Measurement

• Two categories of software measurement– Direct measures of the

• Software process (cost, effort, etc.)

• Software product (lines of code produced, execution speed, defects reported over time, etc.)

– Indirect measures of the• Software product (functionality, quality, complexity, efficiency, reliability,

maintainability, etc.)

29

Size-oriented Metrics

• Derived by normalizing quality and/or productivity measures by considering the size of the software produced

• Thousand lines of code (KLOC) are often chosen as the normalization value

• Metrics include – Errors per KLOC - Errors per person-month– Defects per KLOC - KLOC per person-month– Dollars per KLOC - Dollars per page of documentation– Pages of documentation per KLOC

(More on next slide)

30

Size-oriented Metrics (continued)

• Size-oriented metrics are not universally accepted as the best way to measure the software process

• Opponents argue that KLOC measurements– Are dependent on the programming language– Penalize well-designed but short programs– Cannot easily accommodate nonprocedural languages– Require a level of detail that may be difficult to achieve

31

Function-oriented Metrics

• Function-oriented metrics use a measure of the functionality delivered by the application as a normalization value

• Most widely used metric of this type is the function point:

FP = count total * [0.65 + 0.01 * sum (value adj. factors)]

32

Function Point Controversy

• Like the KLOC measure, function point use also has proponents and opponents

• Proponents claim that– FP is programming language independent– FP is based on data that are more likely to be known in the early stages of

a project, making it more attractive as an estimation approach• Opponents claim that

– FP requires some “sleight of hand” because the computation is based on subjective data

– FP has no direct physical meaning…it’s just a number

33

Reconciling LOC and FP Metrics

• Relationship between LOC and FP depends upon– The programming language that is used to implement the software

– The quality of the design

• FP and LOC have been found to be relatively accurate predictors of software development effort and cost– However, a historical baseline of information must first be established

• LOC and FP can be used to estimate object-oriented software projects– However, they do not provide enough granularity for the schedule and

effort adjustments required in the iterations of an evolutionary or incremental process

• The table on the next slide provides a rough estimate of the average LOC to one FP in various programming languages

34

LOC Per Function Point

Language Average Median Low High

Ada 154 -- 104 205

Assembler 337 315 91 694

C 162 109 33 704

C++ 66 53 29 178

COBOL 77 77 14 400

Java 55 53 9 214

PL/1 78 67 22 263

Visual Basic 47 42 16 158

www.qsm.com/?q=resources/function-point-languages-table/index.html

Metrics and Software Quality

FURPS• Functionality - features of system• Usability – Training time, skill level necessary

to use, Increase in productivity, subjective questionnaire or controlled experiment

• Reliability – frequency of failure, security• Performance – speed, throughput• Supportability – maintainability

Measures of Software Quality• Correctness – degree to which a program operates according to

specification– Defects/KLOC– Defect is a verified lack of conformance to requirements– Failures/hours of operation

• Maintainability – degree to which a program is open to change– Mean time to change– Change request to new version (Analyze, design etc)– Cost to correct

• Integrity - degree to which a program is resistant to outside attack– Fault tolerance, security & threats

McCall’s Triangle of QualityMaintainabilityMaintainability

FlexibilityFlexibility

TestabilityTestability

PortabilityPortability

ReusabilityReusability

InteroperabilityInteroperability

CorrectnessCorrectness

ReliabilityReliability

EfficiencyEfficiency

IntegrityIntegrity

UsabilityUsability

PRODUCT TRANSITIONPRODUCT TRANSITIONPRODUCT REVISIONPRODUCT REVISION

PRODUCT OPERATIONPRODUCT OPERATION

A Comment

McCall’s quality factors were proposed in theearly 1970s. They are as valid today as they were

in that time. It’s likely that software built to conform to these factors will exhibit high quality well into

the 21st century, even if there are dramatic changesin technology.

Quality Model

product

operation revision transition

reliability efficiency usability maintainability testability portability reusability

Metrics

High level Design Metrics

Structural ComplexityData ComplexitySystem ComplexityCard & Glass ’80• Structural Complexity S(i) of a module i.

– S(i) = fout2(i)

– Fan out is “A count of the number of modules that call

– module i”

Design Metrics

• Data Complexity D(i)– D(i) = v(i)/[fout(i)+1]– v(i) is the number of inputs and outputs passed

to and from i

• System Complexity C(i)– C(i) = S(i) + D(i)– As each increases the overall complexity of the

architecture increases

System Complexity Metric

• Another metric:– length(i) * [fin(i) + fout(i)]2

– Length is LOC– Fan in is the “number of modules that invoke i”

• Cyclomatic complexity

43

Metrics for Software Quality

• Correctness– This is the number of defects per KLOC, where a defect is a verified lack of

conformance to requirements– Defects are those problems reported by a program user after the program is

released for general use

• Maintainability– This describes the ease with which a program can be corrected if an error is

found, adapted if the environment changes, or enhanced if the customer has changed requirements

– Mean time to change (MTTC) : the time to analyze, design, implement, test, and distribute a change to all users

• Maintainable programs on average have a lower MTTC

44

Defect Removal Efficiency

• Defect removal efficiency provides benefits at both the project and process level

• It is a measure of the filtering ability of QA activities as they are applied throughout all process framework activities– It indicates the percentage of software errors found before software

release

• It is defined as DRE = E / (E + D)– E is the number of errors found before delivery of the software to the end

user– D is the number of defects found after delivery

• As D increases, DRE decreases (i.e., becomes a smaller and smaller fraction)

• The ideal value of DRE is 1, which means no defects are found after delivery

• DRE encourages a software team to institute techniques for finding as many errors as possible before delivery

45

Object-oriented Metrics

• Average number of support classes per key class– Key classes are identified early in a project (e.g., at requirements analysis)– Estimation of the number of support classes can be made from the number

of key classes– GUI applications have between two and three times more support classes

as key classes– Non-GUI applications have between one and two times more support

classes as key classes• Number of subsystems

– A subsystem is an aggregation of classes that support a function that is visible to the end user of a system

Coupling

• Data and control flow– di – input data parameters– ci input control parameters– do output data parameters– co output control parameters

• Global– gd global variables for data– gc global variables for control

• Environmental– w fan in – r fan out

Metrics for Coupling

• Mc = k/m, k=1

– m = di + aci + do + bco + gd + cgc + w + r

– a, b, c, k can be adjusted based on actual data

Component Level Metrics

• Cohesion (internal interaction) - a function of data objects

• Coupling (external interaction) - a function of input and output parameters, global variables, and modules called

• Complexity of program flow - hundreds have been proposed (e.g., cyclomatic complexity)

• Cohesion – difficult to measure– Bieman ’94, TSE 20(8)

Using Metrics

• The Process– Select appropriate metrics for problem– Utilized metrics on problem– Assessment and feedback

• Formulate• Collect• Analysis• Interpretation• Feedback

Metrics for the Object Oriented

• Chidamber & Kemerer ’94 TSE 20(6)

• Metrics specifically designed to address object oriented software

• Class oriented metrics

• Direct measures

Weighted Methods per Class

WMC =

• ci is the complexity (e.g., volume, cyclomatic complexity, etc.) of each method

• Viewpoints: (of Chidamber and Kemerer)

-The number of methods and complexity of methods is an indicator of how much time and effort is required to develop and maintain the object

-The larger the number of methods in an object, the greater the potential impact on the children

-Objects with large number of methods are likely to be more application specific, limiting the possible reuse

n

iic

1

Depth of Inheritance Tree

• DIT is the maximum length from a node to the root (base class)

• Viewpoints:• Lower level subclasses inherit a number of methods

making behavior harder to predict

• Deeper trees indicate greater design complexity

Number of Children• NOC is the number of subclasses immediately

subordinate to a class

• Viewpoints:• As NOC grows, reuse increases - but the abstraction may be diluted

• Depth is generally better than breadth in class hierarchy, since it promotes reuse of methods through inheritance

• Classes higher up in the hierarchy should have more sub-classes then those lower down

• NOC gives an idea of the potential influence a class has on the design: classes with large number of children may require more testing

Coupling between Classes• CBO is the number of collaborations between

two classes (fan-out of a class C)– the number of other classes that are referenced in the class

C (a reference to another class, A, is an reference to a method or a data member of class A)

• Viewpoints:• As collaboration increases reuse decreases

• High fan-outs represent class coupling to other classes/objects and thus are undesirable

• High fan-ins represent good object designs and high level of reuse

• Not possible to maintain high fan-in and low fan outs across the entire system

Response for a Class

• RFC is the number of methods that could be called in response to a message to a class (local + remote)

• Viewpoints: As RFC increases• testing effort increases

• greater the complexity of the object

• harder it is to understand

Lack of Cohesion in Methods

• LCOM – poorly described in Pressman

• Class Ck with n methods M1,…Mn

• Ij is the set of instance variables used by Mj

LCOM

• There are n such sets I1 ,…, In

– P = {(Ii, Ij) | (Ii Ij ) = }

– Q = {(Ii, Ij) | (Ii Ij ) }

• If all n sets Ii are then P =

• LCOM = |P| - |Q|, if |P| > |Q|

• LCOM = 0 otherwise

Example LCOM

• Take class C with M1, M2, M3

• I1 = {a, b, c, d, e}• I2 = {a, b, e}• I3 = {x, y, z}• P = {(I1, I3), (I2, I3)}• Q = {(I1, I2)}

• Thus LCOM = 1

Explanation

• LCOM is the number of empty intersections minus the number of non-empty intersections

• This is a notion of degree of similarity of methods

• If two methods use common instance variables then they are similar

• LCOM of zero is not maximally cohesive• |P| = |Q| or |P| < |Q|

Some other cohesion metrics

Class Size

• CS – Total number of operations (inherited, private,

public)– Number of attributes (inherited, private, public)

• May be an indication of too much responsibility for a class

Number of Operations Overridden

• NOO

• A large number for NOO indicates possible problems with the design

• Poor abstraction in inheritance hierarchy

Number of Operations Added

• NOA

• The number of operations added by a subclass

• As operations are added it is farther away from super class

• As depth increases NOA should decrease

Method Inheritance Factor

MIF = .

• Mi(Ci) is the number of methods inherited and not overridden in Ci

• Ma(Ci) is the number of methods that can be invoked with Ci

• Md(Ci) is the number of methods declared in Ci

n

iia

n

iii

CM

CM

1

1

)(

)(

MIF

• Ma(Ci) = Md(Ci) + Mi(Ci)

• All that can be invoked = new or overloaded + things inherited

• MIF is [0,1]

• MIF near 1 means little specialization

• MIF near 0 means large change

Coupling Factor

CF= .

• is_client(x,y) = 1 iff a relationship exists between the client class and the server class. 0 otherwise

• (TC2-TC) is the total number of relationships possible

• CF is [0,1] with 1 meaning high coupling

)(

),(_2 TCTC

CCclientisi j ji

Polymorphism Factor

PF = .

• Mn() is the number of new methods

• Mo() is the number of overriding methods

• DC() number of descendent classes of a base class

• The number of methods that redefines inherited methods, divided by maximum number of possible distinct polymorphic situations

i iin

i io

CDCCM

CM

)()(

)(

Operational Oriented Metrics

• Average operation size (LOC, volume)

• Number of messages sent by an operator

• Operation complexity – cyclomatic

• Average number of parameters/operation– Larger the number the more complex the collaboration

Encapsulation

• Lack of cohesion

• Percent public and protected

• Public access to data members

Inheritance

• Number of root classes

• Fan in – multiple inheritance

• NOC, DIT, etc.

Metric tools

• McCabe & Associates ( founded by Tom McCabe, Sr.)

– The Visual Quality ToolSet – The Visual Testing ToolSet – The Visual Reengineering ToolSet

• Metrics calculated

– McCabe Cyclomatic Complexity– McCabe Essential Complexity – Module Design Complexity – Integration Complexity – Lines of Code – Halstead

CCCC• A metric analyser C, C++, Java, Ada-83, and Ada-95 (by

Tim Littlefair of Edith Cowan University, Australia)

• Metrics calculated– Lines Of Code (LOC)– McCabe’s cyclomatic complexity– C&K suite (WMC, NOC, DIT, CBO)

• Generates HTML and XML reports

• freely available

• http://cccc.sourceforge.net/

Jmetric• OO metric calculation tool for Java code (by Cain and

Vasa for a project at COTAR, Australia) • Requires Java 1.2 (or JDK 1.1.6 with special extensions)

• Metrics – Lines Of Code per class (LOC)– Cyclomatic complexity– LCOM (by Henderson-Seller)

• Availability– is distributed under GPL

• http://www.it.swin.edu.au/projects/jmetric/products/jmetric/

JMetric tool result

GEN++ (University of California,  Davis and  Bell Laboratories)

• GEN++ is an application-generator for creating code analyzers for C++ programs

– simplifies the task of creating analysis tools for the C++

– several tools have been created with GEN++, and come with the package

– these can both be used directly, and as a springboard for other applications 

• Freely available

• http://www.cs.ucdavis.edu/~devanbu/genp/down-red.html

Integrating Metrics within the Software Process

77

Arguments for Software Metrics

• Most software developers do not measure, and most have little desire to begin

• Establishing a successful company-wide software metrics program can be a multi-year effort

• But if we do not measure, there is no real way of determining whether we are improving

• Measurement is used to establish a process baseline from which improvements can be assessed

• Software metrics help people to develop better project estimates, produce higher-quality systems, and get products out the door on time

78

Establishing a Metrics Baseline

• By establishing a metrics baseline, benefits can be obtained at the software process, product, and project levels

• The same metrics can serve many masters• The baseline consists of data collected from past projects• Baseline data must have the following attributes

– Data must be reasonably accurate (guesses should be avoided)– Data should be collected for as many projects as possible– Measures must be consistent (e.g., a line of code must be interpreted

consistently across all projects)– Past applications should be similar to the work that is to be estimated

• After data is collected and metrics are computed, the metrics should be evaluated and applied during estimation, technical work, project control, and process improvement

79

Software Metrics Baseline ProcessSoftware

EngineeringProcess

SoftwareProject

SoftwareProduct

Data Collection

MetricsComputation

MetricsEvaluation

Measures

Metrics

Indicators

80

Getting Started with Metrics

1) Understand your existing process2) Define the goals to be achieved by establishing a metrics program3) Identify metrics to achieve those goals

– Keep the metrics simple– Be sure the metrics add value to your process and product

4) Identify the measures to be collected to support those metrics

(More on next slide)

81

Getting Started with Metrics(continued)

5) Establish a measurement collection processa) What is the source of the data?b) Can tools be used to collect the data?c) Who is responsible for collecting the data?d) When are the data collected and recorded?e) How are the data stored?f) What validation mechanisms are used to ensure the data are correct?

6) Acquire appropriate tools to assist in collection and assessment7) Establish a metrics database8) Define appropriate feedback mechanisms on what the metrics indicate

about your process so that the process and the metrics program can be improved

IEEE Software Metrics Methodology

1. Establish software quality requirements

2. Identify software quality metrics

3. Implement the software quality metrics

4. Analyze the software metrics results

5. Validate the software quality metrics

Establish Software Quality Requirements

• What group is empowered to define software quality requirements?

• How should customers provide input?

• How are requirements conflicts resolved?

Identify Software Quality Metrics

• Specify important quality factors and subfactors• Identify direct metrics

– Name– Costs– Target value– Tools– Application– Data items– Computation

Example of Documenting a Metric

Item Description

Name Number of defects detected in selected modules

Costs Minimal: data can be obtained from bug-tracking tool

Target Value 5

Tools Spreadsheet

Application Metric is used for relative comparison to values obtained for other modules

Data Items Count of defects detected at code inspections

Computation Sum number of defects reported against specific modules

Implement the Collection of Data

Item Description

Name Name given to a data item

Metrics Metrics associated with the data item

Definition Straightforward description of the data item

Source Location of where the data originates

Procedures Procedures (manual or automated) for collecting the data

Representation Manner in which data is represented, for example, precision, format, units, etc.

Storage Location of where the data is stored

Analyze Software Quality Metric Results

• Results need to be analyzed within the context of the project’s overall software quality requirements

• Any metrics that fall outside of their respective targets should be identified for further analysis

Validate the Software Quality Metrics

• Assess the statistical significance of the metrics to the quality factors they represent

• See the IEEE Standard 1061-1998 for a thorough description of this process

Metrics that Support Software Verification Activities

• Complexity Metrics– The McCabe Cyclomatic Complexity Metric– Halstead’s Software Science Complexity Metric

• Defect Metrics

• Product Metrics

• Process Metrics

Complexity Metrics Can Be Used to Identify

• Candidate modules for code inspections• Areas where redesign may be appropriate• Areas where additional documentation is

required• Areas where additional testing may be

required

Product Metrics

• Number and type of defects found during requirements, design, code, and test inspections

• Number of pages of documentation delivered

• Number of new source lines of code created

• Number of source lines of code delivered

• Total number or source lines of code delivered

• Average complexity of all modules delivered

Product Metrics (cont’d)

• Average size of modules

• Total number of modules

• Total number of bugs found as a result of unit testing

• Total number of bugs found as a result of integration testing

• Total number of bugs found as a result of validation testing

• Productivity, as measured by KLOC per person-hour

Process Metrics

• Average find-fix cycle time

• Number of person-hours per inspection

• Number of person-hours per KLOC

• Average number of defects found per inspection

• Number of defects found during inspections in each defect category

• Average amount of rework time

• Percentage of modules that were inspected

Attributes of a Measurement Program – according to Humphrey

• The measures should be robust

• The measures should suggest a norm

• The measures should relate to specific product and process properties

• The measures should suggest an improvement strategy

Attributes of a Measurement Program – according to Humphrey (cont’d)

• The measures should be a natural result of the software development process

• The measures should be simple• The measures should be predictable and

trackable• The measures should not be used as part of a

person’s performance evaluation

Template for Software Quality Goal Definition

• Purpose: To (characterize, evaluate, predict, monitor, etc.) the (process, product, model, metric, etc.) in order to (understand, plan, assess, manage, control, engineer, learn, improve, etc.) it.– Example: To evaluate the maintenance process in

order to improve it.

Template for Software Quality Goal Definition (cont’d)

• Perspective: Examine the (cost, effectiveness, correctness, defects, changes, product measures, etc.) from the viewpoint of the (developer, manager, customer, etc.)– Example: Examine the effectiveness from the

viewpoint of the customer.

Template for Software Quality Goal Definition (cont’d)

• Environment: The environment consists of the following: process factors, people factors, methods, tools, constraints, etc.– Example: The maintenance staff are poorly

motivated programmers who have limited access to tools.

Determining Metrics

Goal Questions Metrics

Evaluate How fast are fixes to customer reported problems made?

What is the quality of fixes delivered?

Average effort to fix a problem

Percentage of incorrect fixes

What We Are Talking About Today

• What Is Software Cost Estimation

• How It’s Done (Models, Methods, Tools)

• Issues and Problems

04/20/23 100

Introduction To Software Cost Estimation

• A Few Definitions (Vidger/Kark 1994)

– Software Cost• Manpower Loading

• Effort

• Duration

– Software Cost Estimation Process• Set of techniques and procedures that an

organization uses to arrive at a software cost estimate

04/20/23 101

Classical View - Inputs and Outputs

Classical View Of Software Estimation Process [Vidger/Kark]

04/20/23 102

Why and How Cost Estimates Are Used

• Understanding why and how cost estimates are used within an organization will likely determine how cost estimations will be done

• Common Examples (Vidger/Kark 1994)

– Planning and Budgeting– Project Management– Communication Among Team Members

04/20/23 103

Cost Estimation Process

• Determine Objectives– Who needs what data for what purpose(s)

• Gather Data– Focus Should Be Given To ‘Hard’ Data

• Well-Defined Requirements

• Available Resources

• Analyze Data Using A Variety Of Methods

04/20/23 104

Cost Estimation Process (Cont’d)

• Re-estimate Costs Throughout The Project– Effective Monitoring– Refine and Make Changes As Necessary

• Compare End Costs With Estimated Costs– That is, if project actually completes!

04/20/23 105

Issues and Problems

Inaccurate Estimation Models, Methods, and Tools Ineffective Management of Requirements Duration, Size, and Number of People Involved Ineffective or Non-existent Monitoring of Project New Technology Mixed Technology Inexperienced Project Managers and Estimators Lack of Application Domain Expertise Software Processes and Process Maturity Lack of Historical Data Business / Legal / Monetary Issues

04/20/23 106

What is Needed To Make Reliable Cost Estimates

• A Combination Of Models, Methods, and Tools

• Gathering/Improving of Historical Data• Well-defined and Well-controlled Software

Development Processes• Better Managing of Requirements• Experienced Project Managers, Estimators,

and Team Members

04/20/23 107

Expert Judgment Method

Expert judgment techniques involve consulting with software cost estimation experts to use their experience and understanding of the proposed project to arrive at an estimate of its cost.

Technique: Delphi technique, a group consensus technique.

• Advantages: Empirical• Disadvantages: Subjective

04/20/23 108

Estimating by Analogy

Estimating by analogy means comparing the proposed project to previously completed similar project where the project development information is known.

• Advantages: Based on actual experience of projects• Disadvantages: Difficult to ensure the degree of

representative between previous projects and new one

04/20/23 109

Top-Down Estimating Method

Top-down estimating method is also called Macro Model. Using it, a cost estimation is derived from the global properties of the software project, and then the project is partitioned into various low-level components.

• Advantages: Efficient and system level view• Disadvantages: Too rough

04/20/23 110

Bottom-up Estimating Method

Bottom-up estimating method, the cost of each software components is estimated and then combines the results to arrive at an estimated cost of overall project.

Advantages: Detailed and stable Disadvantages: overlook many of the system-

level costs, inaccurate, more time-consuming

04/20/23 111

Algorithmic Method

The algorithmic method is designed to provide some mathematical equations to perform software estimation.

Models: COCOMO & COCOMO II, Putnam, ESTIMACS and SPQR/20.

04/20/23 112

Algorithmic Method (Cont’d)

Advantages: 1. Objective, repeatable

2. Has modifiable and analyzable formulas

3. Efficient and able to support sensitivity analysis

4. Objectively calibrated to previous experience.

04/20/23 113

Algorithmic Method (Cont’d)

Disadvantages:

1. Unable to deal with exceptional conditions

2. Poor sizing inputs and inaccurate cost driver rating will result in inaccurate estimation

3. Some experience and factors can not be easily quantified

04/20/23 114

The Selection and Use of Estimation Methods

No one method is necessarily better or worse than the other, in fact, their strengths and weaknesses are often complementary to each other.

1. Do not depend on a single cost or schedule estimate, use several techniques or cost models, compare the results

2. Document the assumptions made when making the estimates3. Monitor the project to detect when assumptions that turn out to

be wrong jeopardize the accuracy of the estimate. 4. Improve software process5. Maintaining a historical database

04/20/23 115

Commercial Tools of Cost Estimation (Cont’d)

Some Promising Tools:

• ACEIT • COCOMO II *• Construx Estimate *• COSMOS *• COSTAR *

04/20/23 116

• Cost Xpert • ESTIMATE Pro *

• PRICE-S • SEER-SEM

• SLIM-Estimate

Most parametric models are likely to have the COCOMO II equations at the core...

The COCOMO Model

Model to estimate the development cost and schedule of a software project.

Introduced by Barry Boehm of USC-CSE in 1981.First two letters of the words Constructive Cost

ModelPrimarily based on the software development

practices prior to 1980s, i.e. based on the Waterfall model.

04/20/23 117

The COCOMO Model

• Effort equation is the basis of the COCOMO II model.

• The nominal effort equation of a project of a given size is given by the equation – PM(nominal) = A * (Size)B where PM(nominal) is the nominal effort in person months A is the multiplicative effect of cost drivers B is the constant representing the affect of scale factors

04/20/23 118

COCOMO Model

• MM = a * KLOC b

• Time for development = c * MM d

a b c d

organic 2.4 1.05 2.5 0.38

Semi-Detached 3.0 1.12 2.5 0.35

Embedded 3.6 1.20 2.5 0.3204/20/23 119

The COCOMO II Model

• Has three series of models:– The Application Composition Model– The Early Design Model– The Post-Architecture Model

04/20/23 120

The COCOMO II Model

Cost Drivers are used in the model to adjust the nominal effort in the software project

Cost Drivers are multiplicative factors required to determine the effort required to complete the software project. Ratings range from VL, L, N, H, VH, EH

Model has 17 cost drivers divided into 4 categories:ProductComputerPersonnelProject

04/20/23 121

The COCOMO II Model

• Cost drivers in the Product category:– Required Software Reliability– Database Size– Software Product Complexity– Required Reusability (new)– Documentation match to life-cycle needs

04/20/23 122

The COCOMO II Model

• Cost drivers in the Computer Category:– Execution Time Constraint– Main Storage Constraint– Platform Volatility

04/20/23 123

The COCOMO II Model

• Cost drivers in Personnel Category:– Analyst Capability– Programmer Capability– Applications Experience– Platform Experience– Language and Tool Experience– Personnel Continuity

04/20/23 124

The COCOMO II Model

• Cost drivers in Project category:– Use of Software Tools– Multisite Development– Required Development Schedule

04/20/23 125

The COCOMO II Model

Scale Drivers are important factors determining the cost and duration of the software development effort.

The five scale drivers in the COCOMO II model are:Precedentedness Development Flexibility Architecture / Risk Resolution Team Cohesion Process Maturity

04/20/23 126

Scaling Factors• cause an exponential cost increase

04/20/23 127

Cost Drivers Very Low

Low Nom High Very High

Extra High

Precedentedness (PREC)

5 4 3 2 1 0

Development flexibility (FLEX)

5 4 3 2 1 0

Archictecture/risk resolution (RESL)

5 4 3 2 1 0

Team cohesion (TEAM)

5 4 3 2 1 0

Process Maturity (PMAT)

5 4 3 2 1 0

Scaling Factors

04/20/23 128

Effort Multipliers (Post-Architecture)

• Product factors

04/20/23 129

Cost Drivers Very Low

Low Nom High Very High

Extra High

Reliability (RELY) 0.75 0.88 1.00 1.15 1.39

Database size (DATA) 0.93 1.00 1.09 1.19

Complexity (CPLX) 0.75 0.88 1.00 1.15 1.30 1.66

Reusability (RUSE) 0.91 1.00 1.14 1.29 1.49

Documentation (DOCU) 0.89 0.95 1.00 1.06 1.13

Effort Multipliers (Post-Architecture)

• Platform factors

04/20/23 130

Cost Drivers Very Low

Low Nom High Very High

Extra High

Execution time constraints (TIME)

1.00 1.11 1.31 1.67

Main storage constraints (STOR)

1.00 1.06 1.21 1.57

Platform volatility (PVOL)

0.87 1.00 1.15 1.30

Effort Multipliers (Post-Architecture)

• Personnel factors

04/20/23 131

Cost Drivers Very Low

Low Nom High Very High

Extra High

Analyst capability (ACAP)

1.50 1.22 1.00 0.83 0.67

Programmer .. (PCAP) 1.37 1.16 1.00 0.87 0.74

Application experience (APEX)

1.22 1.10 1.00 0.89 0.81

Platform .. (PLEX) 1.24 1.10 1.00 0.92 0.84

Language/tool ..(LTEX) 1.25 1.12 1.00 0.88 0.81

Personnel continuity (PCON)

1.24 1.10 1.00 0.92 0.84

Effort Multipliers (Post-Architecture)

• Project factors

04/20/23 132

Cost Drivers Very Low

Low Nom High Very High

Extra High

Use of software tools (TOOL)

1.24 1.12 1.00 0.86 0.72

Multi-site development (SITE)

1.25 1.10 1.00 0.92 0.84 0.78

Required development schedule (SCED)

1.29 1.10 1.00 1.00 1.00

Effort Multipliers (Post-Architecture)

04/20/23 133

every 1 mo.

Effort Multipliers (Post-Architecture)

04/20/23 134

Application Composition Model intended mainly for prototyping activitiesuses Object Point (OP) counts (not FPs)

a count of the number and complexity of large granularity entities such as screens, reports and components

and factors in code reuse and productivity new object points, NOP = OP x (100 - % reuse) / 100

productivity, PROD = NOP / person-months

effort, PM = NOP / PROD

04/20/23 135

Functional Size Measurement

136

Functional Size measurement

137

Applicability of FSM

• FUR can be extracted from software engineering artifacts BEFORE the software exists… (using UML for instance).

• Inputs:– Requirements definition artifacts

– Data analysis / modeling artifacts

– Artifacts from functional decomposition of requirements

– FUR can also be extracted from software engineering artifacts AFTER the software has been constructed.

– Physical programs and screens

– Software operations manuals and procedures

– Physical data storage artifacts

138

ISO 14143-1 Terminology for FSM (I)

• Functional User Requirements (FUR): A sub-

set of the user requirements. The FURs

represent the user practices and procedures

that the software must perform to fulfill the

users’ needs.

• Functional Size: A size of the software derived

by quantifying the FUR.

139

ISO 14143-1 Terminology for FSM (II)

• Base Functional Component (BFC): An

elementary unit of FUR defined by and used

by an FSM Method for measurement

purposes.

• BFC Type: A defined category of BFCs. A

BFC is classified as one and only one BFC

Type.140

Functional Size Measurement

Function points (FP)p( )

International Function Point Users Group

(IFPUG) FP

Common Software Measurement International

Consortium (COSMIC) FP

141

Functional Size Measurement

• Function points (FP)– International Function Point Users Group

(IFPUG) FP– Common Software Measurement International

Consortium (COSMIC) FP

142

ISO Approved FSM Methods

• IFPUG Function Point Analysis (ISO/IEC 20926)

• Mark II Function Point Analysis (ISO/IEC 20968)

• NESMA Functional Size Measurement Method

• (ISO/IEC 24570)

• COSMIC Function Points (ISO/IEC 19761)143

IFPUG Function Point AnalysisMethod

144

IFPUG FPA Measurement Procedure

• Identify Elementary processes from the• Functional User RequirementsRequirements.• Identify the BFCs and their types.• Rate the complexity of each BFC Type.• Assign Function Points to each BFC Type• according to the complexity rates.• Calculate the functional size by summing the• FPs assigned to each distinct BFC TypeType.

145

IFPUG FPA - BFC

• Elementary Process: the smallest unit of activity that is meaningful to the user(s)user(s).

– E.g., FUR: “The user will be able to add a new employee to the application.” is an Elementary Process.

• The elementary process must be self-contained and leave the business of the application being counted in a consistent state.

• E.g., If all the employee information is not added, an employee has not yet been created. Adding some of the information alone leaves the business of adding employee in an inconsistent state.

146