Toward Ontology for SE-ROI 1

Honourcode, Inc.

Toward an Ontology for Measuring Systems Engineering

Return on Investment

Advancing the knowledge of systems engineering

Eric Honour+1 (850) 479-1985ehonour@hcode.com Dr. Ricardo Valerdi+1 (617) 253-8583rvalerdi@mit.edu

Version

Toward Ontology for SE-ROI 2Honourcode, Inc.

Topics

SE-ROI Project “Ontology” concept COSYSMO work toward ontology Categorization from current standards Future directions

Toward Ontology for SE-ROI 3

Honourcode, Inc.

Systems Engineering Return on Investment

Summary of the SE-ROI Project

Toward Ontology for SE-ROI 4Honourcode, Inc.

Heuristic Claim of SE

Better systems engineering leads to Better system quality/value Lower cost Shorter schedule

SYSTEMDESIGN

DETAILDESIGN

PRODUCTIONINTEGRATION TEST

Traditional Design

Time

Risk

SavedTime/Cost

“System Thinking” Design Time

Risk

Toward Ontology for SE-ROI 5Honourcode, Inc.

0.4

0.6

0.8

1.0

1.2

0% 4% 8% 12% 16% 20% 24%

SE Effort

Dev

elo

pm

ent

Qu

alit

y(C

ost

/Sch

edu

le B

ased

)

Value of SE – 2004 Results

Problems/challenges: Quantitative data on SE not

available in program databases All data points were subjective

Detailed structure not availableSource: SECOE 01-03INCOSE 2003

Value = 1.0 if program met cost/schedule goals

Greater SE Effort led to better cost/schedule compliance and better predictability

Each dot is one program, with sizes between $1M and $6.5B

Toward Ontology for SE-ROI 6Honourcode, Inc.

SE-ROI Project

Interviews• Just-completed programs•Key PM/SE/Admin•Translate program data

into project structure

•Program characterization•Program success data•SE data (hours, quality,

methods)

Statistical correlation

Desired Results1. Statistical correlation

of SE methods with program success.

2. Leading SE indicators that can be used during a program.

3. Identification of good SE practices under different conditions.

0.6

1.0

1.4

1.8

2.2

2.6

3.0

0% 4% 8% 12% 16% 20% 24%

SE Effort = SE Quality * SE Cost/Actual Cost

Act

ual

/Pla

nn

ed S

ched

ule

Toward Ontology for SE-ROI 7

Honourcode, Inc.

“Ontology” Concept

What is this word and how does it relate to systems engineering?

Toward Ontology for SE-ROI 8Honourcode, Inc.

Current State of SE Definition

Fragmented by domain opinions

•Military – DOD/MOD•Space - NASA/ESA•Commercial products•Aircraft•Automobiles•Nuclear waste•Process engineering•Tool vendors•Etc. Etc. Etc.

Fragmented by discipline opinions

•Technical leaders•System architects•System analysts•Requirements

engineers•Operations analysts•Design engineers

Fragmented by standards

•ANSI/EIA-632• IEEE-1220• ISO-15288•CMMI•MIL-STD-499C

Toward Ontology for SE-ROI 9Honourcode, Inc.

Ontology

“…“…a branch of metaphysics concerned with a branch of metaphysics concerned with the nature and relations of being”the nature and relations of being”

functions

inputs

outputs

structure

understanding

aesthetics

components

interfaces

• POSIWID – The Purpose of a Systems Is What It Does - Jack Ring

• The purpose of systems engineering is different in the eyes of different people, because they perceive different actions/results from SE

methods

categories

Toward Ontology for SE-ROI 10Honourcode, Inc.

Purpose of this Paper

Explore the variety of what people see in SE

Formulate some general categories Interpret historical SE effort data Provide a structure for the data-gathering

in the SE-ROI project.

Toward Ontology for SE-ROI 11

Honourcode, Inc.

COSYSMO work toward ontology

An exploration of ontology as performed in the COSYSMO project

Toward Ontology for SE-ROI 12Honourcode, Inc.

COSYSMO Effort Profile

How is Systems Engineering effort distributed over time?

Phase Conceptualize DevelopOperational

Test & Eval

Transition to

Operation

%Effort(STDEV)

23 (12) 36 (16) 27 (13) 14 (9)

Toward Ontology for SE-ROI 13Honourcode, Inc.

Effort Distribution Across ANSI/EIA 632 Fundamental Processes

ANSI/EIA 632 Fundamental Process

AverageStandard Deviation

Acquisition & Supply 7% 3.5

Technical Management 17% 4.5

System Design 30% 6.1

Product Realization 15% 8.7

Technical Evaluation 31% 8.7

Toward Ontology for SE-ROI 14Honourcode, Inc.

Systems Engineering Effort Profile

Toward Ontology for SE-ROI 15

Honourcode, Inc.

Categorization from Current Standards

The start of an ontology, by identifying the widely-accepted categories.

Toward Ontology for SE-ROI 16Honourcode, Inc.

Categories in the Standards

Mission/Purpose Definition Requirements Management System Architecting System Implementation Technical Analysis Technical Management/Leadership Verification & Validation

ANSI/EIA-632

IEEE-1220

ISO-15288

CMMI

MIL-STD-499C

Colored boxes on following slides show the terminology used by each standard

Toward Ontology for SE-ROI 17Honourcode, Inc.

Mission/Purpose Definition

Define the mission or purpose of the new/changed system. Typically described in the language of the system

users rather than in technical language

ANSI/EIA-632•Not included

IEEE-1220•Define customer

expectations ISO-15288•Stakeholder

needs definition

CMMI•Develop customer

requirements

MIL-STD-499C•Not included

Toward Ontology for SE-ROI 18Honourcode, Inc.

Requirements Management

Creation and management of requirements Efforts to define, analyze, validate, and manage

the requirements

ANSI/EIA-632•System design•Requirements

definition

IEEE-1220•Requirements

analysis ISO-15288•Requirements

analysis

CMMI•Requirements

development•Requirements mgmt

MIL-STD-499C•System requirements

analysis and validation

Toward Ontology for SE-ROI 19Honourcode, Inc.

System Architecting

Define the system in terms of its component elements and their relationships Diagrams that depict the system, its

environment, components, and relationships

IEEE-1220•Synthesis

ISO-15288•Architectural design•System life cycle mgmt

CMMI•Technical solution ANSI/EIA-632

•System design•Solution definition MIL-STD-499C

•System product technical requirements analysis and validation

•Design or physical solution representation

Toward Ontology for SE-ROI 20Honourcode, Inc.

System Implementation

Development/completion of the system Specific system-level efforts in the standards are

system integration and transition to use

IEEE-1220•Not included

ISO-15288• Implementation• Integration •Transition

CMMI•Product integration

ANSI/EIA-632•Product realization• Implementation•Transition to use

MIL-STD-499C•Not included

Toward Ontology for SE-ROI 21Honourcode, Inc.

Technical Analysis

System-level technical analysis Assessment of system performance Trade-off analysis of alternatives

IEEE-1220•Functional analysis•Requirements trade

studies and assessments•Functional trade studies

and assessments •Design trade studies and

assessmentsISO-15288•Requirements analysis

CMMI•Measurement and analysis

ANSI/EIA-632•Technical evaluation•System analysis

MIL-STD-499C•Functional analysis,

allocations and validation•Assessments of system

effectiveness, cost, schedule, and risk

•Tradeoff analyses

Toward Ontology for SE-ROI 22Honourcode, Inc.

Technical Management/Leadership

Guiding the engineering teams involved in system design programs Size/complexity of teams demands leadership

IEEE-1220• Technical mgmt• Track analysis data• Track requirements

and design changes• Track performance• Track product metrics• Update specifications• Update architectures• Update plans• Maintain database

ISO-15288• Planning, Assessment,

Control• Decision mgmt• Config mgmt• Acquisition, Supply• Resource mgmt • Risk mgmt

CMMI• Project planning• Project monitoring & control• Supplier agreement mgmt• Process/product quality assur.• Configuration mgmt• Integrated project mgmt• Decision analysis/resolution• Quantitative project mgmt • Risk mgmt

ANSI/EIA-632• Technical Mgmt• Planning• Assessment• Control MIL-STD-499C

• Planning• Monitoring• Decision making, control,

and baseline maintenance• Risk mgmt• Baseline change control• Interface mgmt• Data mgmt• Subcontract mgmt• Technical reviews/audits

Toward Ontology for SE-ROI 23Honourcode, Inc.

Verification & Validation

Verification: comparison of the system with its requirements through objective evidence.

Validation: comparison of the system or requirements with the intended mission

IEEE-1220•Requirement verification•Functional verification•Design verification

ISO-15288•Verification•Validation•Quality mgmt

CMMI•Verification•Validation

ANSI/EIA-632•Technical Evaluation•Requirements validation•System verification•End products validation MIL-STD-499C

•Design or physical solution verification and validation

Toward Ontology for SE-ROI 24

Honourcode, Inc.

Future Directions

Where is SE-ROI going?

Toward Ontology for SE-ROI 25Honourcode, Inc.

Project Advisory Group

Group of interested people/organizations Communicate via web, telecon, meetings Help define the data organization Build public interest in the project Provide access to real programs View interim (protected) data as it develops

AF Institute of TechnologyNorthrop GrummanDOD Office of Secy of DefenseDRSJohns Hopkins UnivMITThe Mitre Corp

NAVAIRRaytheonRand CorporationRafaelSystems & Software ConsortiumUniv of South AustraliaUSN Chief Engineer

Current members come from:

For information, see http://www.hcode.com/seroi/

Toward Ontology for SE-ROI 26Honourcode, Inc.

Summary

Systems engineering current state of knowledge is fragmented

Broadly-accepted ontology is needed SE-ROI project needs categorization now

Structure the data to be correlated Discover leading SE indicators Identify SE best practices and methods

Categorization across standards helps develop the needed ontology

Toward Ontology for SE-ROI 27

Honourcode, Inc.

Questions?

Eric Honour+1 (850) 479-1985ehonour@hcode.com Dr. Ricardo Valerdi+1 (617) 253-8583

rvalerdi@mit.edu

For information, see http://www.hcode.com/seroi/For information, see http://www.hcode.com/seroi/

Toward Ontology for SE-ROI 28Honourcode, Inc.

Survey of SE-ROI Knowledge

1992 Gruhl (NASA)

Project Definition – NASA

Program definition 10-15% reduces cost overruns

1990 Ancona/ Caldwell

Boundary Management Study

Boundary management averages 14%; more is better

2000 Miller (MIT)

Large Engineering Projects Study

Programs value cost over schedule over tech Leadership important

1995 Franz (Boeing)

Impact of Systems Engineering on Quality and Schedule

Better SE led to significant cost reduction

2003 Barker (IBM)

Systems Engineering Effectiveness

Better SE reduced parametric costs by 30%

2004 Kludze (NASA)

Impact of Systems Engineering on Complex Systems

General belief that SE improves program cost

2004 Honour (SECOE)

Value of Systems Engineering

Greater SE 10-15% reduces cost/schedule overruns

Toward Ontology for SE-ROI 29Honourcode, Inc.

Next Steps

Define interview data sheets Use this categorization

Identify and interview trial projects Obtain initial data Evaluate the interview data sheets

Identify and interview projects Ongoing effort for 2-3 years

Perform statistical correlation work Ongoing effort for duration of project Interim reports to participating organizations

Final report expected 2009