482612main 2010 tuesday 4 cmmi kulpa margaret executive overview of cmmi v1.4b

Empowering People through Process

© AgileDigm, Inc.

Executive Overview of CMMI®

18 Aug 2010NASA IT SummitMargaret Kulpa

Chief Operating Officer of AgileDigm, Inc.

www.agiledigm.com

® CMMI is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.

© 2010 Executive Overview of CMMI

2

Copyright NoticesThe presentation material in this seminar is copyrighted by AgileDigm, Incorporated, 2010. For further information, please contact:

AgileDigm, Inc.11 Twelve Oaks TrailOrmond Beach, Florida 32174 U.S.A.Phone: +1.386.673.1384Email: [email protected]

Terms like these are often used in the following material:CMMI

SCAMPIsm and SCAMPI Lead Appraisersm

CMMI is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.SM SCAMPI and SCAMPI Lead Appraiser are service marks of Carnegie Mellon University.


3

Who is AgileDigm?AgileDigm, Incorporated is a US based corporation working with organizations around the world

– providing consulting, training, and appraisals for organizations that build software-intensive systems

– Software Engineering Institute (SEI) Partner– Scrum Training Institute (STI) Partner

Authors of Interpreting the CMMI, best selling book on using the CMMI. Unique experience working with one of the only CMMI Maturity Level 5 Agile companies.

CMMI is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.SM SCAMPI and SCAMPI Lead Appraiser are service marks of Carnegie Mellon University.


4

What is the CMMI?• CMMI is an acronym for Capability Maturity Model

Integration. • Approach used by organizations to improve.• CMMI is a collection of best practices

– Over 100 organizations formally assisted in its development, including NASA

• There is a wide range of freedom in how an organization decides to embrace the CMMI. You can –– Go slow or fast– Choose the areas you need to focus on– Choose the appraisal mode you are comfortable with (SCAMPI -

A,B,C, High Maturity)


5

Three Constellations• CMMI for Development

– Development and maintenance of products– Integrates software engineering, systems engineering,

collaborative teams, acquisition from the supplier’s side – Not just for software development! – Can be used for developing complex systems (healthcare,

bridges, valves, cleanroom technology)• CMMI for Services

– Organizations that provide services (from taxis to call centers)• CMMI for Acquisition

– Acquiring products and services • Common Process Areas are embedded within all three• This presentation focuses on CMMI for Development.


6

CMMI Concepts

What the CMMI is not:–CMMI is not a process, standard, directive,

regulation, or description.–Not a list of “Shalls” that might require big

changes in how you do your work.You can make the CMMI in your organization as intrusive and painful as you want --- OR NOT! It’s up to you.


7

Does the CMMI tell me how to do stuff?

• NO!• The CMMI documents best practices from many

organizations.• The CMMI describes WHAT to do, not HOW to

do it.• Organizations need to determine how to apply

the best practices of the CMMI to how the organization works in order to improve how stuff gets done.


8

Why Should I Use the CMMI?• Structures the effort (Where do I start? How

do I know when I am done?)• Helps identify an organization’s improvement

objectives and priorities• Provides benefit from a larger community’s

experience• Has an an appraisal methodology to diagnose

the state of improvement efforts.


9

Why Should I Use the CMMI?• CMMI focuses on “Process”• A process is a set of practices performed

to achieve a given purpose (from CMMI Glossary)

• Four major elements of process are tools, methods, materials, and people– Major determinants of product cost, schedule,

and quality are tools, methods, materials, and people.


10

CMMI Performance Results Summary

Source: Performance Results of CMMI-Based Process Improvement, Aug 2006, Technical Report CMU/SEI-2006-TR-004

This is the summary of improvements from all sources collected regardless of maturity levels.

Improvement Nbr. of Data PointsCategory Median Lowest Highest

Cost 34% 3% 87% 29

Schedule 50% 2% 95% 22

Productivity 61% 11% 329% 20

Quality 48% 2% 132% 34

Customer Satisfaction

14% -4% 55% 7

Return on Investment

4.0 : 1 1.7 : 1 27.7 :1 22


11

Cost – Benefits and Impact• Cost metrics (and reductions in cost) defined as:

– Cost of delivery – Cost of quality/Cost of poor quality

– Overhead rate – Costs of rework/Defect find & fix cost

– Software unit costs – Number/Cost of process staff– Variation in cost performance index (CPI)

• Organizations have reported improved– Budget estimation accuracy – Average in cost performance index (CPI)


Cost example on next slide – cost per function reduced by 52%


12

Cost Savings at DB Systems GmbH



13

Schedule – Benefits and Impact• Schedule metrics (and reductions) defined as:

– Variation in schedule – Schedule performance index (SPI)– Number of days late – Days variance from plan– Slippage of project delivery

• Organizations have reported improved or increased– Cycle time – Proportion of milestones met– Average SPI – Estimation accuracy

• Example from NCR Corporation (on next slide)– Average days schedule variance reduced from approximately

130 days to less than 20 days one year after reaching CMMI maturity level 2



14

Schedule Improvement at NCR



15

Productivity – Benefits and Impact• Productivity measures (and improvements)

defined as: – Time comparisons by build – Source statements per

month – Lines of code per labor hour – Number of releases per

year– Testing rates – Software production– Function points (FP) per full time equivalent (FTE) staff

• Example from IBM Australia (on next slide)– Over 20 percent improvement in account productivity as the

organization moved from SW-CMM maturity level 3 toward CMMI maturity level 5.



16

Productivity Improvement at IBM Australia


Account Productivity (FP/FTE)


17

Quality – Benefits and Impact


• “Quality” most frequently defined as reductions in numbers of defects.

• Measurements include:– Defect counts by phase of injection– Defect counts by phase of discovery– Total defect density

• Example from IBM Australia (on next slide)– 40 percent reduction in overall production and over 80 percent

reduction in Severity 1 problems, as the organization moved from SW-CMM maturity level 3 toward CMMI maturity level 5


18

Quality Improvement at IBM Australia



19

Customer Satisfaction – Benefits and Impact

• Customer satisfaction most frequently defined by the results of customer surveys.– Award fees are sometimes used as surrogate

measures.• An example from Lockheed Martin Management

and Data Systems– Increased award fees by 55 percent



20

ROI – Benefits and Impact


• ROI defined by cost avoidance measures and improvements including:– Rework avoided due to fewer defects– Improved productivity– Increased revenue due to shorter cycle times

• Example from Raytheon Corporation, anonymous site– 6:1 ROI in a CMMI maturity level 3

organization


21

From personal experience …• Ensuring I am working on the latest (correct) version of

the code• Running tests that prove requirements are satisfied

(not that the system is just not blowing up)• Predicting the number of projects to start up and the

number of people I need ahead of time• Increasing communication throughout the organization

(and hearing things I sometimes did not want to hear)• Not making phone calls to my QA person while she

was in labor.


22

CMMI StructureCMMI is a collection of best practices

– These “best practices” reside in 22 focus areas called “Process Areas.”

– Each process area has specifically stated goals that have to be met in order to have “maturity” or “capability” in the process area.

– Each process area has specific practices, subpractices, typical work products (outputs), examples, and other informative material

– Most of the goals are specific to the process area, but there are some generic or “global” goals that are common to all of the Process Areas like:

• Provide Resources for this area• Train People for this area• Identify Stakeholders for this area

Let’s look at the book …


23

Structure of the CMMI• Maturity Level or Capability Level

– Specific and Generic Goals• Specific and Generic Practices

– Subpractices, Typical Work Products, Examples• Two types of Goals and Practices

– We will see these examples in the Example Process Area ☺

– Specific Goals pertain to one Process Area (SG1 in Project Planning is “Establish Estimates”)

– Specific Goals have associated Specific Practices (For SG1 above, SP 1.4 is Determine Estimates of Effort and Cost

– There are also Generic Goals and Generic Practices that pertain to all of the Process areas throughout the model. They are “Global.”


24

Example Process Area (Maturity Level 3)

Decision Analysis and ResolutionThe purpose of Decision Analysis and Resolution (DAR) is to analyze possible decisions using a formal evaluation process that evaluates identified alternatives against established criteria.

The CMMI then continues with half a page of Introductory Notes that further describe what this Process Area is about.Note: Underlines and emphasis added by the presenter here and in the following slides. I have omitted some of the Informative Material.

Informative material is very useful in interpreting the intent of the process area/practice/subpractice and should not be ignored.


25

Decision Analysis and Resolution• SG1: Evaluate Alternatives

– SP1.1 Establish Guidelines for Decision Analysis– SP1.2 Establish Evaluation Criteria– SP1.3 Identify Alternative Solutions– SP1.4 Select Evaluation Methods ***– SP1.5 Evaluate Alternatives– SP1.6 Select Solutions

Note: Other Process Areas may have more Specific Goals and more Specific Practices. There is more informative material presented in the book.

***This is the practice used in the following example.

SG – Specific GoalSP – Specific Practice


26

Decision Analysis and ResolutionSP 1.4 Select the Evaluation MethodsTypical Work Products:1. Selected evaluation methods

Subpractices

1. Select the methods based on the purpose for analyzing a decision and on the availability of the information used to support the method.

Typical evaluation methods include the following: – Modeling and simulation– Engineering studies– Manufacturing studies– Cost studies– Business opportunity studies


27

Decision Analysis and Resolution– Surveys– Extrapolations based on field experience and prototypes– User review and comment– Testing– Judgment provided by an expert or group of experts

2. Select evaluation methods based on their ability to focus on the issues at hand without being overly influenced by side issues.

3. Determine the measures needed to support the evaluation method. – Consider the impact on cost, schedule, performance, and risks.


28

Generic “Common” Goals and Practices for All Process Area

Generic Goal 2 Institutionalize a Managed ProcessGP 2.1 Establish an Organizational PolicyGP 2.2 Plan the ProcessGP 2.3 Provide ResourcesGP 2.4 Assign ResponsibilityGP 2.5 Train PeopleGP 2.6 Manage ConfigurationsGP 2.7 Identify and Involve Relevant StakeholdersGP 2.8 Monitor and Control the ProcessGP 2.9 Objectively Evaluate AdherenceGP 2.10 Review Status with Higher-Level Management

Generic Goal 3 Institutionalize a Defined Process GP 3.1 Establish a Defined ProcessGP 3.2 Collect Improvement Information

GG – Generic Goal GP -Generic Practice


29

Two representations of the model• Staged Representation (or “View”):

– Most like other currently accepted models– Focuses on all process areas contained within a level– Used for Maturity Level ratings for contract awards– Start with any Process Area within a level.

• Continuous Representation (or “View”):– Was designed to focus on specific, individual process areas– Process areas are arranged into functional categories– Start with any Process Area.

• Staged is the most “popular” CMMI representation.– When your customers ask for a level or when an organization

says “We are Level N”, they are referring to a Maturity Level of the staged representation.


30

Not to Worry• The information contained is the same in

both!• The only difference is in your approach for

implementation and appraisal.


31

Process Areas by Maturity Level (Staged)

1 Initial

Requirements Management Project PlanningProject Monitoring and ControlSupplier Agreement ManagementMeasurement and AnalysisProcess and Product Quality AssuranceConfiguration Management

2 Managed

Requirements Development Technical SolutionProduct Integration VerificationDecision Analysis and Resolution Validation Organizational Process FocusOrganizational Process Definition +IPPDOrganizational TrainingIntegrated Project Management +IPPDRisk Management

3 Defined

Organizational Process PerformanceQuantitative Project Management

4 Quantitatively Managed

Organizational Innovation and DeploymentCausal Analysis and Resolution

5 OptimizingProcess AreasMaturity Level

1 Initial

Requirements Management Project PlanningProject Monitoring and ControlSupplier Agreement ManagementMeasurement and AnalysisProcess and Product Quality AssuranceConfiguration Management

2 Managed

Requirements Development Technical SolutionProduct Integration VerificationDecision Analysis and Resolution Validation Organizational Process FocusOrganizational Process Definition +IPPDOrganizational TrainingIntegrated Project Management +IPPDRisk Management

3 Defined

Organizational Process PerformanceQuantitative Project Management

4 Quantitatively Managed

Organizational Innovation and DeploymentCausal Analysis and Resolution

5 OptimizingProcess AreasMaturity Level


32

Process Areas by Category (Continuous)Process Management Project ManagementOrganizational Process FocusOrganizational Process Definition (with IPPD)

Organizational TrainingOrganizational Process PerformanceOrganizational Innovation & Deployment

Project PlanningProject Monitoring and ControlSupplier Agreement ManagementIntegrated Project Management (with IPPD)

Risk ManagementQuantitative Project Management

Engineering SupportRequirements ManagementRequirements DevelopmentTechnical SolutionProduct IntegrationVerificationValidation

Measurement & AnalysisProcess and Product Quality AssuranceConfiguration ManagementDecision Analysis & ResolutionCausal Analysis and Resolution


33

CMMIMaturity Levels

Level 1 – InitialLevel 2 – ManagedLevel 3 – DefinedLevel 4 – Quantitatively ManagedLevel 5 - Optimizing

Capability LevelsLevel 0 - IncompleteLevel 1 – PerformedLevel 2 – ManagedLevel 3 – DefinedLevel 4 – Quantitatively ManagedLevel 5 - Optimizing

“Continuous”“Staged”

Process ManagementProject ManagementEngineeringSupport

Process Areas (22)

Process Areas (22) Active PA Subset

SpecificGoals

SpecificPractices

GenericGoals

GenericPractices

It’s a Model,not a specification

It can be hierarchical.

It’s highly detailed..

It’s complex, butnothing is

complicated.

There are no“shalls”.

Appraisals yield

Ratings Realmof

Appraisals


34

What is a SCAMPIsm Appraisal?• SCAMPIsm – Standard CMMI Appraisal Method for

Process Improvement– Used for baselining current behaviors to begin

improvement activities or see into ongoing improvement activities

– To assign a formal rating for contract award or fees (if requested by appraisal sponsor)

– Several appraisal methods (class A, B, C, plus High Maturity)

– Uses the CMMI as reference model.• SCAMPI goal is to provide an accurate picture of your

organization relative to the reference model.


35

Reported Appraisals

CMMI® For Development SCAMPIsm Class A Appraisal Results 2009 End-Year Update

Red indicates countries with reported SCAMPI Class A Appraisals


36

Maturity Profile – End of Year 2009• The percentage of organizations at each maturity level

based on most recent appraisals of 2753 organizations.

Maturity Profile - CMMI/SCAMPI Results published March 2010

0.5%5.2%1.3%

25.2%

63.0%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

Level 1 Level 2 Level 3 Level 4 Level 5


37

Appraisal Classes A, B, and C


38

Appraisal Team (SCAMPI “A”)• Minimum team size is four - A Lead Appraiser and

three additional team members. • At least one team member must be from the

appraised organization.• All team members must be trained in the official

CMMI and SCAMPI courses• Conflicts of interest must be avoided

– Exclude process authors and individuals directly impacted by the outcome of the appraisal

• Team members cannot be managers of any of the selected projects or groups

• Appraisal sponsors cannot be team members.


39

What does the SCAMPI Team do?

• Reviews an organization’s documentation– Processes, procedures, policies, status reports,

meeting minutes• Conducts interviews

– To ensure the documentation is actually used• Maps results to the CMMI

– Practices, Goals, Process Areas• Assigns ratings (if requested and allowed by

the type of SCAMPI).


40

What does the Team NOT do?• Does not go looking for the documentation

– Easter egg hunts are not allowed!• Does not judge the “goodness” of the

process• Does not judge how well you are doing

your job.


41

PPQA Example EvidenceID Goal/Practice Potential ArtifactsSG1 Objectively Evaluate Processes and Work ProductsSP1.1 Objectively Evaluate Processes QA audit report

SP1.2 Objectively Evaluate Work Products and Services

QA audit report

SG2 Provide Objective InsightSP2.1 Communicate and Ensure

Resolution of Noncompliance Issues

QA audit report

SP2.2 Establish Records QA Reports, Status of Corrective Actions, Report of Quality Trends


42

Simplified Point of the SCAMPI:• Do you have a (documented) process?• Do you follow it?• How closely does it map to the CMMI?


43

Conclusion (1of 2)

• Organizations can and have benefited from using the CMMI.

• Implement the CMMI to support your organization’s business objectives, goals, and culture.

• “Engage your brain.” Interpretation of the information must be used.

• Do not ignore the informative information and the subpractices. They help to understand the intent of the practice or Process Area.

• Use a SCAMPI to guide your implementation, measure your progress, and verify your achievements.


44

Conclusion (2 of 2)

• My favorite websites for more information:– www.sei.cmu.edu (main SEI website)– seir.sei.cmu.edu (contains helpful

presentations and articles from the community)

http://www.sei.cmu.edu/


45

Questions?• Time for some questions?


46

Contact Information

Margaret KulpaChief Operating OfficerAgileDigm, Inc.11 Twelve Oaks TrailOrmond Beach, FL 32174 USA+1 386 673 [email protected]

The presentation material in this seminar is copyrighted by AgileDigm, Incorporated, 2010. For further information, please contact:

mailto:[email protected]

http://www.agiledigm.com/


47

Backup Slides


48

Combined View of Staged and Continuous


49

Maturity Levels (MLs)• A maturity level signifies the level of

performance that can be expected from an organization. There are 5 maturity levels. Following are the key features of each maturity level (ML):– ML 1 (Initial) – ad hoc processes– ML 2 (Managed) – basic project management system – ML 3 (Defined) – process standardization – ML 4 (Quantitatively Managed) – quantitative

management “by the numbers”– ML 5 (Optimizing) – continuous process improvement

ML 2 provides a foundation for ML 3, ML 3 for ML4, and ML 4 for ML 5.


50

Appraisal Sponsor• SCAMPI appraisals require an appraisal sponsor.

– Usually this is a senior manager.– The appraisal sponsor is responsible for

• Verifying that the appraisal team leader has the appropriate experience, knowledge, and skills to lead the appraisal.

• Ensuring that the appropriate organizational units (projects and support) participate in the appraisal.

• Supporting the required confidentiality and non-attribution rules.

– Some appraisals have multiple sponsors.


51

“Typical” Appraisal Roadmap

Train Team• Train Appraisal Team• Begin PIID creation• Finalize Plan

Readiness Review

Plan• Plan SCAMPI activities• Monitor and adjust plan

Appraisal Onsite• Conduct Appraisal

• Organization Develops PIID documents

May 2010

August 2010 (1 – Week)

October 2010 (1 – Week)

December 2010 (1 – 3 Weeks)Reviewing PIIDs is a key focus of the required readiness review


52

Presenter – Margaret Kulpa• Chief Operating Officer of AgileDigm,

Incorporated• She is the primary author of the book

Interpreting the CMMI: A Process Improvement Approach, Second Edition, Auerbach (2008) which details how to identify, define, and improve business, systems and software processes on both enterprise-wide and project levels.

• Ms. Kulpa has developed and taught courses in software quality assurance, project management, peer reviews, requirements development and management, process baselines and modeling, safety and security, and other software engineering-related subjects. She has developed courses in CMMI and has served as an external member of numerous SCAMPIsm Class A Appraisals.

482612main 2010 tuesday 4 cmmi kulpa margaret executive overview of cmmi v1.4b

Documents