joan weszka lockheed martin mission systems systems & software resource center © copyright...

Joan WeszkaLockheed Martin Mission Systems

Systems & Software Resource Center

© Copyright Lockheed Martin Corporation 2003

CMMI® Technology Conference 2003November 19, 2003

.

Transition from SW-CMM to CMMI: The Benefits Continue!

® CMM and CMMI are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.

11/19/03 © Copyright Lockheed Martin Corporation 2003

Joan Weszka - 2

Agenda

Context Background on Lockheed Martin’s (LM) CMMI®

transition approach Benefits in the Software CMM® and CMMI® eras

LM Systems Integration (Owego, NY)LM Maritime Systems & Sensors – Tactical Systems (Eagan, MN)LM Maritime Systems & Sensors – Radar Systems (Syracuse, NY)LM Maritime Systems & Sensors – Undersea Systems (Manassas, VA)


Joan Weszka - 3

Context

A number of Lockheed Martin organizations that tracked quantitative process improvement benefits during their SW-CMM high maturity journey have now transitioned to CMMI

Experience to date indicates that these benefits have continued with CMMI implementation

Benefits derived are not attributable only to CMMI

Many initiatives are underway concurrently with CMMI (and SW-CMM) deployment


Joan Weszka - 4

Lockheed Martin’s CMMI® Deployment Approach

Many Lockheed Martin (LM) companies have institutionalized best-of-breed integrated processes (e.g., IPPD)

Multiple process models and standards are in use LM identified industry and internal best practices

as sources for corporate-wide process requirements The LM Integrated Engineering Process (LM-IEP)

standard synthesizes these requirements

Lockheed Martin’s Integrated Engineering Process Standard is being deployed under corporate policy.


Joan Weszka - 5

Key Tenets of Lockheed Martin’s CMMI® Transition

Address CMMI® in the context of your organization’s business requirementsLockheed Martin’s Integrated Engineering Process

(LM-IEP) standard includes CMMI®, in addition to other standards and requirements (e.g., ISO/IEC 15288, ISO 9001:2000)

Adopt an incremental appraisal approach Lockheed Martin Continuous Appraisal Method (CAM)

has been successfully deployed with CMMI® and is being extended for use with LM-IEP


Joan Weszka - 6

LM-IEP, CAM and CMMI® Relationship

CMMI® provides a set of integrated process and appraisal method requirements

LM-IEPLM-IEPLM-IEPLM-IEP

CMMICMMI®®

ProcessProcessRequirementsRequirements

CMMICMMI®®

ProcessProcessRequirementsRequirements

ARC* Class AARC* Class AAppraisalAppraisal

RequirementsRequirements**

ARC* Class AARC* Class AAppraisalAppraisal

RequirementsRequirements** Lockheed Martin Integrated Engineering Process

* Appraisal Requirements for CMMI®® (ARC)

CAM

CAM meets ARC Class A requirements.


Joan Weszka - 7

LM CPSs

ANSI/EIA-632

ISO 9001:2000

IEEE 1220

LM HardwareProcess Standard

ISO/IEC-12207

CMMI®® V1.1

ISO/IEC-15288

LM-IEP Context Diagram

Common Source Standards

OrganizationalStandard

Process

LM-IEPStandard

Industry Standards

GovernmentStandards

Domain Specific Standards

Industry Stds

Government Stds

Project Specific Standards

ProjectDefinedProcess

In LM-IEP V2 issued 1/03

Reference only

Integrated Processes

Integrated Work Products

Integrated Methods

Tailored OSP

EPI Process & Methods

Informative only

CPS – Corporate Policy StatementEPI – Engineering Process Improvement

Company or Business Unit


Joan Weszka - 8

Continuous Appraisal Method (CAM) Design Goals

Minimize appraisal preparation and reduce cost Integrate process improvement with process

appraisal activities Facilitate appraisal scheduling and minimize

disruption for participants Provide an appraisal environment conducive to

process improvement Promote institutionalization

CAM is being used with CMMI® and being extended for appraisal of LM-IEP requirements.


Joan Weszka - 9

Project Appraisals

Overview of Incremental Appraisal Using CAM

Maintenance Review

Institutionalization focus with minimal project disruption

OSP = Organizational Standard ProcessPA = Process Area

OSPAppraisal

Appraise OSP Fix Weaknesses Re-appraise

Fix Weaknesses Re-appraise Appraise PA 1

...

Planning




Joan Weszka - 10

Feedback on CAM usage CAM has been or is being deployed at 12 Lockheed Martin

operating units using CMMI®

6 prior CAMs have been completed using EIA/IS 731

Experience with CAM has been positive:More focus on process improvementLess invasive to programsLess stressful to the organizationMore value-add, in-depth findingsMore cost effective


Joan Weszka - 11

Lockheed Martin Systems Integration - Owego Process CredentialsLockheed Martin Systems Integration - Owego Process Credentials

Q-100 Award

19901990

19931993 IBM Baldrige Gold Award

New York State Excelsior Award

19941994

19951995 RIT / USA TODAYQuality Cup Finalist

19961996 ISO 9001:1994 /SW CMM Level 3

19971997 SW CMM Level 5

19981998 LM21 Best Practices

20012001 ISO 9001:2000 /SEI CMMI Pilot Site

20032003 CMMI Level 5 /AS 9100A

20002000 Lean / Six Sigma

TQM

20022002

20032003

20022002

20012001

20002000

19981998

19971997

19961996

19951995

19941994

19931993

19901990

Joan Weszka - 12 © Copyright Lockheed Martin Corporation 2003

11/19/03

Lockheed Martin Systems Integration Owego - QualityLockheed Martin Systems Integration Owego - Quality

Software Defects per Million Delivered Source Lines of Code

-80%

-60%

-40%

-20%

0%

1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002

% R

edu

ctio

n S

TR

s /

MS

LO

C Improvements Since ‘92

80.5% Overall

Inte

grat

ed P

roce

sses

SW

CM

M L

evel

5

CM

MI

Leve

l 3

Cap

able

CM

MI

Leve

l 5

* STR / MSLOC = Software Trouble Report/Million Source Lines of Code

SW

CM

M L

evel

3

2003

® ® CMMI and CMM are registered in the U.S. Patent and Trademark Office by Carnegie Mellon UniversityCMMI and CMM are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University..


Joan Weszka - 1311/19/03

© Copyright Lockheed Martin Corporation 2003Joan Weszka - 13

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

120.0%

140.0%

160.0%

1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002

CM

MI

Leve

l 5in

200

3

SW

CM

M L

evel

3

SW

CM

M L

evel

5

Improvements Since 1992:

9.8% Average per Year

ISD

Pro

cess

es

SW

CM

M L

evel

3 C

apab

le

Lockheed Martin Systems Integration Owego Lockheed Martin Systems Integration Owego -- Software ProductivitySoftware Productivity

Software Productivity (All Software including Reuse)

Contributors:•Increased Reuse (Domain Specific)•Process Maturity and Compliance•Process Consistency•Increased use of High Order Language / 4th Generation / Object Oriented•Use of development and test tools

CM

MI

Leve

l 3

Cap

able

2003


Joan Weszka - 14

LM Maritime Systems & Sensors Tactical SystemsProcess Improvement Credentials

Oct. 1999 – Attained SW-CMM® level 4 Dec. 2000 – Attained Systems Engineering Capability

Model (EIA 731) level 3 Jan. 2001

Began focus on integrated process improvementBegan transition to CMMI®

June 2002 – May 2003 CAM AppraisalOSP: Target profile 5 for CMMI®-SE/SW/IPPD/SS Projects: Target profile 3 for CMMI®-SE/SW

August 2003 – SCAMPISM AppraisalAchieved target profile 3 for CMMI®-SE/SW

® SW-CMM is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.


Joan Weszka - 15

LM Maritime Systems & Sensors Tactical SystemsInitiatives

Streamlining of references and guidance documents which supplement the OSP

Value stream mapping of engineering process/business model

LM-IEP gap analysisSelf audit process complianceAirworthiness manual Risk Management methodology best practiceUpgrade of Process Asset Library (PAL)Mechanical engineering guidebook Change management pilotDefect prevention pilot


Joan Weszka - 1611 /19 /03

© C op yrigh t L ockh eed M artin C orp ora tion 2003Joa n W es zk a - 16

L M M aritim e System s & Sensors T actical System s Softw are P roductiv ity

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

1979 1984 1989 1994 1999 2004

Year

So

urc

e S

tate

men

ts p

er H

ou

r

Level 3C M M ®

Level 4 C M M ®

Level 3 C M M I®


Joan Weszka - 17

LM Maritime Systems & Sensors Radar SystemsProcess Improvement Credentials

SW-CMM® level 5 (CBA IPISM) in Dec. 1999

Systems Engineering Capability Model (EIA/IS 731) level 3+ (CAM) in Dec. 1999

Focus on integrated process improvement including hardware began in 2000

Transition to CMMI® began in 2000 CMMI®-SE/SW/IPPD target profile 4

(CAM) in Nov. 2002

® SW-CMM and CMMI are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.


Joan Weszka - 18

LM Maritime Systems & Sensors Radar Systems Functional Excellence Objectives - 2003

LM 21 business excellence Enhance program performance Improve productivity Continue to manage the business with quantitative data Improve quality of the product and process Provide a supportive infrastructure for process

improvement Demonstrate continued process maturity leadership Manage technology innovations to support program

performance Document processes and procedures core to our success Develop and train employees for current and future

assignments


Joan Weszka - 19

LM Maritime Systems & Sensors Radar SystemsSoftware Productivity and Quality

Lockheed Martin: NE&SS-Syracuse - Software Productivity & Quality Performance History

0%

100%

200%

300%

400%

500%

600%

700%

800%

1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003

SoftwareProductivityGrowth (%)

-100%

-90%

-80%

-70%

-60%

-50%

-40%

-30%

-20%

-10%

0%

Software Error RateReduction (%)

Software Productivity and Quality Performance Application of Best Practices and Investment Has Resulted in Significant Improvements in Quality and Cost. As error rates declined,

productivity increased by 80+%.

CM

M

L4

CM

M

L5

CM

M

L2

CM

M

L3

Error Rate

Productivity Growth

CM

Mi

L4

L


Joan Weszka - 20

LM Maritime Systems & Sensors – Undersea Systems Process Credentials

Systems & Software Engineering* Software CMM® Level 4 (CBA IPISM) - June 1995 Software CMM® Level 5 (CBA IPISM ) - February 1999 CMMI® & EIA-731 Level 3 (CAM) - October 2001 CMMI® Level 5 (CAM) - October 2002

Quality Management ISO 9001:1994 - September 1995 AS9000 - November 1997 Defense Contract Management Agency (DCMA) ISO 9001

Qualified - December 1997 ISO 9001:2000 - December 2000 AS9100A – December 2002

Certificate No.

10061Certificate No.

7092Certificate No.

9227

Certificate No.

E00015Certificate No.

EN10002

*Assessed programs comprise over 80% of the Undersea Systems development programs, and all parts of the development cycle.

*Assessed programs comprise over 80% of the Undersea Systems development programs, and all parts of the development cycle.

SM CBA IPI is a service mark of Carnegie Mellon University


Joan Weszka - 21

LM Maritime Systems & Sensors – Undersea SystemsProcess Chronology

1970s Top-down Structured Programming Design & Code Inspections 1980-2 Functional Decomposition SW Engineering Workshop Advanced Design Workshop 1983 SW Management Workshop 1984 Ada Workshop 1985 Requirements Inspections 1986 FSC Practices & Measurements 1988 SW Technology Steering Group Organizational Operating Procedures

1990 SW Engineering Process Group Formed First SW-CMM® Assessment (Level 3) Formal Estimation Procedures

1991Market Driven QualityReuse Focus 1992 Defect Prevention Process

1993Integrated TeamsStandard Development EnvironmentIntegrated Process Group 1994 Automated Metrics (MAX) Process Coordination Group (PCG) 1995-6 Integrated Process Library ISO 9001 Registration Software CMM® Level 4) 1997-9 ISO 14001 Registration, AS9000 and DCMA ISO 9001 qualification Software CMM® Level 5

2000-2 ISO 9001: 2000

Certification EIA-731 Level 3 CMMI® SE/SW/IPPD/SS

Level 5


Joan Weszka - 22

1 1 / 1 9 / 0 3© C o p y r i g h t L o c k h e e d M a r t i n C o r p o r a t i o n 2 0 0 3

J o a n W e s z k a - 2 2

L M M a r i t i m e S y s t e m s & S e n s o r s – U n d e r s e a S y s t e m sS o f t w a r e P r o d u c t Q u a l i t y ( D e f e c t R a t e )

0

5 0 0

1 0 0 0

1 5 0 0

2 0 0 0

2 5 0 0

3 0 0 0

1 9 8 0 1 9 8 4 1 9 8 8 1 9 9 2 1 9 9 6 2 0 0 0

Maj

or D

efec

ts P

er M

illio

n D

eliv

ered

Sou

rce

Sta

tem

ents

C M M ®

L e v e l 3

C M M ®

L e v e l 4

C M M ®

L e v e l 5

C M M I ®

L e v e l 5

O u r q u a l i t y im p r o ve d a s o u r m a t u r i t y i n c r e a s e d .


Joan Weszka - 23

1 1 / 1 9 / 0 3© C o p y r i g h t L o c k h e e d M a r t i n C o r p o r a t i o n 2 0 0 3

J o a n W e s z k a - 2 3

L M M a r i t i m e S y s t e m s & S e n s o r s – U n d e r s e a S y s t e m sP r o c e s s I m p r o v e m e n t R e t u r n - o n - I n v e s t m e n t S u m m a r y

N o t e : O t h e r i n i t i a t i v e s u n d e r w a y d u r i n g t h i s p e r i o d i n c l u d e

• I S O 9 0 0 1 r e g i s t r a t i o n , f o l l o w e d b y A S 9 0 0 0• I n t e g r a t e d T e a m i n g , a n d c r e a t i o n o f a n I n t e g r a t e d P r o c e s s L i b r a r y• I n t e g r a t i o n o f S y s t e m s E n g i n e e r i n g a n d S W E n g i n e e r i n g

S W - C M M ® L e v e l 3



C M M I ® L e v e l 5

I m p r o v e m e n t M e t r i c 1 9 9 0 1 9 9 5 1 9 9 9 2 0 0 2Q u a l i t y D e f e c t s / M D S S 6 0 0 3 0 0 1 5 0 5 1P r o d u c t i v i t y E S S / L a b o r M o n t h 2 2 0 2 8 0 3 4 0 3 7 9C o s t & S c h e d u l e + - V a r i a n c e 1 5 % 1 0 % 8 % 8 %R e w o r k e x p r e s s e d a s a %

o f i n d u s t r y a v g 6 % 3 % 2 % 2 %R e u s e P e r c e n t 6 8 % 7 5 % 8 2 % 8 2 %


Joan Weszka - 24

0.5373

1.7010

0.4617

2.7931

4.4119

2.5100

0.00

1.00

2.00

3.00

4.00

5.00

6.00

HDHD DDDD CDCD UTUT CTCT SWITSWIT T&IT&I ATAT Life-cycle phasesMaj

or

Def

ects

/ K

-Eq

uiv

alen

t S

ou

rce

Sta

tem

ents

In this case, representing an actual program, Latent Defect Rate = 0.51 Defects / Thousand Equivalent Source Statements

We use defect data from formal inspections to project product quality.

We take corrective actionsif projections show a deviation from the goals.

Latent Defect Projection Model(best fit to Exponential, Inflected-S, Rayleigh)

We produce high quality products by monitoring the quality level throughout the program’s performance.

LM Maritime Systems & Sensors – Undersea SystemsProduct Quality


Joan Weszka - 25

11/19/03© Copyright Lockheed Martin Corporation 2003

Joan Weszka -25

SigmaDefects/MSS1 690,000.02 308,537.03 66,807.04 6,210.05 233.06 3.4

0

100,000

200,000

300,000

400,000

500,000

600,000

700,000

800,000

1.00 2.00 3.00 4.00 5.00 6.00

Sigma Number

Def/M

illion P

arts

5 Sigma

Our quality rate is 20 times better than the average industry rates.

LM Maritime Systems & Sensors –Undersea Systems Software Quality

MS = Million Source Statements

Product Quality Level is in Five Sigma Range


Joan Weszka - 26

1 1 /1 9 /0 3© C op yr ig h t L o ck h e ed M a rtin C orp o ra tion 2 0 0 3

J oa n W e s zk a - 2 6

Product Productivity Percent ImprovementDelivered Source Lines of Code per Labor Month

All Languages

0%

50%

100%

150%

200%

250%

300%

350%

400%

1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999

L e v e l 1

L e v e l 2

L e v e l 3

L e v e l 4 ,5

L M M a ritim e S y ste m s & S e n so rs – U n d e rse a S y ste m s S o ftw a re P ro d u c tiv ity v s . S W -C M M ® M a tu rity L e v e l


Joan Weszka - 27

Reference: “A Correlational Study of the CMM® and Software Development Performance”Lawlis, Flowe & Thordahl, CROSSTALK, September 1995

SCATTER PLOTS BETWEEN “1” AND “3” RATINGS ARE INDUSTRY SAMPLES.

LOCKHEED MARTIN DATA HAS BEEN ADDED IN RED.

LM Maritime Systems & Sensors – Undersea SystemsCost & Schedule Estimates vs. Actuals at Maturity Level 5

DATA FROM 8 PROGRAMS IN 2Q 2003

Cost Performance Index (CPI) & Schedule Performance Index (SPI)

0

1

2

3

1 2 3 4,50

1

2

3

1 2 3 4,5

+

++++

+

+

++

++

+++

+++=+++= =

==++

+++

++++++

++++

+ ==

====

CPI

SPI

CMM® Rating

+ +++

CMM® Rating


Joan Weszka - 28

Summary

At Lockheed Martin, benefits derived during SW-CMM® implementation continue to be realized as CMMI® maturity evolves

Allocating benefits to their sources is difficult when implementing multiple models/standards

SW-CMM® and CMMI® are viewed as significant (but not sole) contributors to process improvement benefits to date


Joan Weszka - 29

Contact Information

Joan WeszkaLockheed Martin Mission SystemsSystems & Software Resource Center(301) 240-7013 / Fax [email protected]


Joan Weszka - 30

Acronyms ARC – Appraisal Requirements for CMMI®

CAM - Continuous Appraisal Method CMMI® - Capability Maturity Model Integration ESLOC or ESS – Equivalent SLOC/SS; a normalized value derived from

new development, plus SLOC/SS that are modified, retained, ported, etc. IPPD - Integrated Product and Process Development LM - Lockheed Martin LM-IEP - Lockheed Martin Integrated Engineering Process OSP – Organizational Standard Process PA - Process Area SLOC – Source Line of Code SS – Source Statement (sometimes called a “Logical SLOC”)

joan weszka lockheed martin mission systems systems & software resource center © copyright...

Documents

appraisal of lm

lmiep lmiep

cmmi experience

appraisal requirements

cmmi v1

cmmi arccamcam

cmmi relationshipcmmi

engineering process