metrics for mofel-based systems development

®

IBM Software Group

© 2013 IBM CorporationInnovation for a smarter planet

MetricsYou can’t control what you don’t measure

Dr. Bruce Powel Douglass, Ph.D.Chief EvangelistGlobal Technology [email protected]: @BruceDouglassYahoo: http://tech.groups.yahoo.com/group/RT-UML

mailto:[email protected]

http://tech.groups.yahoo.com/group/RT-UML

http://www.ibm.com/software/rational

http://www.ibm.com/software/rational

IBM Software Group | Rational software

Innovation for a smarter planet

Agenda

Introductions

On the importance of being metric

Types of metrics Design Metrics Quality Metrics Project Metrics

Q&A



Dr. Bruce Powel Douglass

Chief EvangelistGlobal Technology Ambassador

IBM Rational


Innovation for a smarter planet 4

On the Importance of Being Metric

Companies that measure

Companies that don’t measure

On-time projects 75% 45%

Late projects 20% 40%

Cancelled projects 5% 15%

Defect removal >95% Unknown

Cost estimates Accurate Optimistic

User satisfaction High Low

Software status High Low

Staff morale High Low

Source: Applied Software Measurement 3rd Edition by Capers Jones 2008

Results improve when measurements are used



On the Importance of Being Metric A metric is a measurement of something important to you

A metric includes A scalar value which is measured Range Units Consistent technique for measurement acquisition Standard analysis of outcome

Metrics should be Easy to measure Correlate with the actual information desired Have easy to understand appropriate interpretation for decision makers

Example: BMI BMI = Weight in Kilograms / ( Height in Meters)2 What is the BMI for a body builder?

Example: LOC Lines of code as a measure of work performed If I optimized from 700K lines to 500K lines have I done negative work?



On the Importance of Metrics

Metrics are used To understand an aspect of a project or system

Static metric measures a static property of system (e.g. size, complexity, defect density)

Dynamic metric measures something that changes (e.g. velocity, iteration burndown, defect rate)

To reduce risk To answer a question To enable informed decision making

Metrics are best used to change from open-loop (ballistic) decision making to closed loop (dynamic evidence-based) decision making

Plan

Act

Measure

Analyze



Goal-Based versus Plan-Based Metrics

A goal is a designed outcome

Plans try to achieve goals but may themselves but suboptimal or in error

Ergo, Goal-based metrics are generally preferred to plan-based metrics

Plan-based metrics On planned schedule

Metric: Number of hours worked on project Metric: Number of lines of code generated Metric: Weight lost

Goal based metrics Progressing towards system delivery

Metric: Requirements delivered/designed/implemented/verified/validated Metric: Amount of verified functionality delivered Metric: body fat percentage



When are metrics useful?

When there is a consensus on what to measure

When all relevant

measurements are made

When the measurement

is timely

When the measurement

correlates to the desired

information

When the measurement is

precise and accurate enough

When it allows you to make an informed correct

decisionWhen all

performers know how to

make the measurement

When the measurement is

properly analyzed

When the analysis leads to

proper action



Key Metric Success Factors

Clear and easy performance of measurement

Training for relevant

personnel for measurement performance

Support for metrics at all project levels

Governance and enforcement of metric policies

Clear assignment of roles and

responsibilities

Outcomes displayed in ways

meaningful to consumers

Retention of metrics for historical reference

Consistent use of outcomes for

decision making

Early establishment of

metrics



Breaking Bad (metrics)

Lack of consensus and

buy in

Measurements inconsistently

gathered

Measurements inaccurate or lack consistent

accuracyMeasurements

gathered too late

Analysis inappropriate for

metric

Analysis not applied or is

ignored

Metric not retained or referenced

Metric not highly correlated with

desired information



Use dashboards to provide summary views

11



Best Metric Practices

Explicitly link metrics to goals

Define what will be done with the results

Train staff in metrics gathering, analysis, and interpretation (as appropriate)

Prefer trends over statics

Use short plan act measure analyze plan… cycles

Audit metric acquisition to ensure consistency of data source, methods, frequency, and analysis

Change metrics when they stop driving improvements

Automate metric acquisition and analysis where possible

Render the analytic results in a form useful to the consumers of the metric

Apply intelligence to interpretation, don’t just blindly accept the obvious conclusion

Use previously captured metrics as guidelines for future planning but don’t slavishly follow them



Type of metrics

Project

Quality

Design



Project Metrics

Project metrics measure project concerns Earned Value Schedule variance Velocity Iteration burndown Release burndown Requirements churn / enhancement request trend Age of enhancement (responsiveness metric) Cost per unit work

Project

Quality

Design



Earned Value

Planned value (AKA Budgeted Cost of Work Scheduled BCWS)

Earned Value is how much value has been created so far:

Application Define the work as a set of mutually exclusive tasks Assign a Planned Value to each task (when complete) Define earning rules (0/100 simplest rule)

Schedule Variance

Schedule Performance Index (>1 is ahead of plan)

Project

Quality

Design



Velocity

A team’s velocity is the amount of functionality it completed in the previous iteration If something isn’t done then its points isn’t counted towards a teams velocity that iteration.

Therefore a 5 point story that is 80% done is counted as 0 and not as 4 points that iteration.

Velocity is typically measured in a point system unique to the individual team You cannot compare teams using points because Team A measures using a different point

system than Team B. Points are typically called story points or user story points by teams that have adopted a

user-story driven approach to development

Project

Quality

Design



Iteration burndown

Shows amount of work taken on by the team for a single iteration and how much work is left to do

Usage: Enables the team to identify where to adjust scope or resources

to finish the iteration successfully Provides delivery progress for the iteration

Project

Quality

Design



Release burndown

Shows the estimated functionality remaining to complete the current release Usage:

Track actual progress Estimate release date based on remaining work/velocity

Project

Quality

Design



Enhancement Request Trend Many agile teams work on a new release of a

solution which is already deployed into production

Shows the trend of enhancement requests received, approved, and closed during the project lifecycle

Usage: Few enhancement requests may indicate lack of

interest in the current production release OR may indicate satisfaction in the current release

A high number of enhancement requests can indicate that the current production release is not functioning as stakeholders expect

A growing backlog of enhancement requests may indicate an inability of the team to respond to changing requirements

Project

Quality

Design



Age of enhancement requests

Tracks the length of time stakeholder enhancement requests remain open

Usage: Indicates responsiveness of delivery team Unaddressed requests can impact the stakeholders' perception of value

Project

Quality

Design



Cost per unit of work

Tracks the cost of delivering a single unit of work (such as a user story point or use case point) across iterations.

Usage: Used to monitor costs throughout the project lifecycle based on the cost of the

team and their velocity. Monitoring this metric in each iteration helps the team understand if their

spending is sustainable.

Project

Quality

Design

Scott Ambler

Doesn't seem all that useful for an agile team???



Design Metrics

Organizational Complexity

Requirements Complexity

Model Architectural Complexity

Model Semantic Complexity

Model Design Complexity

Project

Quality

Design

These metrics are instrumented in a Rhapsody wizard that is available at Merlin’s Cave:http://merlinscave.info/Merlins_Cave/Wizards/Entries/2010/1/8_Model_Metrics.html

http://merlinscave.info/Merlins_Cave/Wizards/Entries/2010/1/8_Model_Metrics.html





Design Metrics

Organizational

Project

Quality

Design

Name Purpose DescriptionNP Number of Packages Identifies the number of “work” or “configuration”

units in the modelDPC Depth of Package

ContainmentA measure of the depth of the model organization unit

EP Number of Elements In Package

The number of classes and other elements (such as use cases and types) in a specified package, a measure of the size and granularity of the package

AEP Average Number of Elements Per Package

A measure of the overall granularity of the model organization

MEPP Maximum Number of Elements Per Package

A measure of the maximum complexity of model organizational elements

PU Package Utility Number of developers the number of people who have read or usage access to a package / the number of developers who write element of a package



Project

Quality

Design

Requirements Complexity

Name Purpose DescriptionNUC Number of Use Cases A measure of the number of independent capabilities of

the systemFPUCP

Function PointsUse Case Points

An estimate of the complexity of the problem to be solved, maps well to the NUC metric

NA Number of Actors The number of actors associated with a given use caseNUCA Number of Use Cases

per ActorGiven an actor, the number of use cases associated with the it

NUCSD Number of Use Case Sequence Diagrams

The total number of black-box sequence diagrams used as exemplars for use cases

AUCSD Average number of Use Case Sequence Diagrams

NUCSD / NUC. This is a measure of the average scope of a use case

NUCS Number of Use Case States

Total number of states + activities used to specify the use cases

UCDC Use Case Decomposition Complexity

The number of use cases derived from a single use case – this includes generalization and dependencies (both «includes», and «extends» relations)

IIC Information Item Count Total number of Information Items in the use case modelIICUC Information Item Count

per Use CaseIIC / NUC



Project

Quality

Design

Model Architecture Complexity

Name Purpose DescriptionNS Number of

Subsystems The number of large-scale architectural units of a system.

NT Number of Tasks Number of «active» objects in a systemNAS Number of Address

SpacesA measure of the scope of the distribution of a model across address spaces or computers.

CASC Cross-Address Space Coupling

A measure of the cohesion within address spaces versus cohesion across address spaces.

RAS Redundant Architecture Scope

Number of redundant architectural units for use in the Safety and Reliability Architecture (either homogeneous or heterogeneous)

NP Number of Processors Number of processor nodes in the system. This may (or may not) be identical with the NAS metric

NCP Number of Components per Processor

Measures the cohesion of functionality within a processor node, assuming that a component provides a coherent set of functionality.

NUCS Number of Use Cases per Subsystem

For systems that decompose system use cases into subsystem level use cases



Project

Quality

Design

Model Semantic Complexity

Name Purpose DescriptionNoC Number of Classes A measure of the number of model sizeCC Class Coupling Measures the cohesion of the classes by computing the

number of associations a class has with its peersTSC Total number of

subclassesMeasures the global use of generalization within a model

CID Class Inheritance Depth The maximum length of a given class generalization taxonomy

NC Number of Children Number of direct descendent (subclasses), a measure of the class reuse

NM Number of Methods Number of methods within a classNP Number of ports Number of unique identifiable connection points of a

classEF Encapsulation Factor Number of class features (attributes, methods, and event

receptions) publicly visible divided by the total number of such features – a measure of the degree to which the internal structure of a class is encapsulated

SF Specialization Factor The number of operations and statechart action sets which are specialized in subclasses



Project

Quality

Design

Behavioral Semantics Complexity

Name Purpose DescriptionSDS Sequence Diagram size Number of messages x number of lifelinesDCC Douglass Cyclomatic

ComplexityModified McCabe cyclomatic complexity to account for nesting and and-states

WMC Weighted Methods per Class

A measure of (non-reactive) class complexity = sum of methods x complexity for all methods. For classes without activity diagrams, method complexity can be estimated by Lines of Code in the method.

ND Nesting Depth State and activity nesting depth – number of levels of nesting

NE Nesting Encapsulation Number of transitions (other than default) that cross levels of nesting

NAS Number of And-States Total number of and-states within a statechartSCN Statechart

completenessNumber of events received by a statemachine / number of transitions

NPS Number of pseudostates Number of non-default pseudostates such as history, conditional, fork, join, junction, and stubs



Quality Metrics

Quality metrics focus on Compliance to plans Deviation of expected functionality and correctness

Compliance to standards AKA “Syntactic correctness” Audit / Review Performance Percentage Audit / Review Pass Percentage

Correctness AKA “Semantic correctness” Defect Density Defect Trend Requirements Coverage Trace Completeness

Project

Quality

Design



Defect Density

Tracks the number of defects found, fixed, and still remaining during a given period of time per thousand source lines of code (KSLOC), per model, or design unit

Usage: Indication of the quality of the product Indication of the effectiveness of testing efforts

Project

Quality

Design



Defect Trend Chart

Shows defect arrival and closure rates, indicates the remaining defect backlog, projects the future defect arrival/close rate up to and post-ship

Usage:– Indication of the quality of the product– Indication of the effectiveness of testing efforts

Project

Quality

Design



Test coverage of requirements

Indicates the percentage of requirements linked to validating tests For agile teams, this is often the percentage of user stories which have one or

more acceptance tests associated with them Usage:

When the coverage isn’t 100% it indicates that the solution isn’t fully tested When the coverage is 100% we need other metrics to determine sufficiency of testing

Project

Quality

Design



Requirements-based Code Coverage

Indicates the completeness of the coverage of the code versus the requirements This metric is required for evidence by some safety standards (e.g. DO-178) Usage:

When the coverage isn’t 100% it indicates that the solution isn’t fully tested or that there is code for which there are no requirements

Project

Quality

Design



Trace Completeness

Indicates the degree of consistency among engineering work products

Typically traces among elements from requirements, architecture, design, code, test, safety analysis (required by some safety standards e.g. DO-178, ISO26262)

Usage When it is important to ensure consistency of the design due to high consequences of

failure

Project

Quality

Design

Design / Implementation Elements

D1 D2 D3 D4 D5

Requiremen

ts

R1 x x

R2

R3 x

R4 x

Gold plating?

Unimplementedrequirement



DimensionsTeam

(In Process)Middle Management

(Development Mgmt.)Development Executive

(VP Development)Time-to-Delivery / Schedule

User Story Points / Use Case PointsIteration Burndown, Blocking Work Item

Release Burndown

Product Value: Iteration VelocityStakeholder Feedback, # of Enhancement Request, Age of Enhancement Request

Tested and Delivered Requirements, Business Value Velocity, Customer Satisfaction

Product Cost / Expense

Effort (Man-hours)Cost / Unit of work

Development / Maintenance Costs

Product Quality Technical Debt (Defect trend, defect density)Test Status, Test Coverage of Requirement, Test Execution Status

Quality at ShipPredictability User Story Points / Use Case Points

Planned/Actual Cost and VelocityTrend Variance. Likelihood of on-time delivery

Note: Bold indicates that there is Out-Of-The-Box report supported by Rational tools

From In Process (Team) To Executive Value: Appropriate Metrics for Each Management level



Adding Metrics into your process: Define your metrics

Concern

s

• Identify your concerns

Goals

• Specify your goals

Select

• Define your metrics

Implement

• Define when and how metrics will be gathered

Specify analytics

• Define how metrics outcomes will be used



Adding Metrics into your process: Update your process

Gathering

• Identify how the metric data will be captured

Instrumenti

ng

• Tool up for metrics

Process

update

• Add into your process definition

Train

• Train workers to properly capture and/or use metric data

Deploy

• Instrument your project



Summary: Metric Guidance

Clearly define goals and explicitly link metrics to them

Prefer goal-based over plan-based metrics

Obtain buy in and consensus

Train and allocate resources

Use short enough cycles that metrics can positively affect outcomes

Collect only a few key measurements

Automate where possible

Metrics that do not result in changed actions are worse than useless

Understand Metrics are not the goal Most issues require more than a single metric Metrics augment but do not replace intelligent judgment



Final Thoughts

metrics for mofel-based systems development

Technology

importance of metrics

smarter planet metrics

metric analysis

applied software measurement

metrics useful

metric companies

metric practices

referenced metric