systems engineering research - seari at mitseari.mit.edu/documents/presentations/sdm-thesis... ·...

Systems Engineering Research

Overview and Opportunities

February 26, 2008

Dr. Donna H. Rhodes Dr. Adam M. Ross

Massachusetts Institute of Technology

Engineering Systems Division

[email protected] [email protected]

seari.mit.edu © 2008 Massachusetts Institute of Technology 2

What is Systems Engineering?

SYSTEMS ENGINEERING (Traditional)

Systems engineering is the process of selecting and synthesizing the application of the appropriate scientific and technical knowledge in order to translate system

requirements into system design. (Chase)



SYSTEMS ENGINEERING (Advanced)

Systems engineering is a branch of

engineering that concentrates on design

and application of the whole as distinct from the parts… looking at the problem in its entirety, taking into account all the facets and variables and relating the social to

the technical aspects. (Ramo)


Changing Face

of Systems Engineering

TRADITIONAL SE

Transformation of customer requirements to design

Requirements clearly specified, frozen early

Emphasis on minimizing changes

Design to meet well specified set of requirements

Performance objectives specified at project start

Focus on reliability, maintainability, and availability

ADVANCED SE

Effective transformation of stakeholder needs to fielded (and sustainable) solution

Focus on product families and systems-of-systems

Complex interdependencies of system and enterprise

Growing importance of systems architecting

Designing to accommodate change

Emphasis on expanded set of “ilities” and designing in robustness, flexibility, adaptability in concept phase



Systems Engineering is an interdisciplinary approach and means to enable the realization of successful systems.

Systems Engineering integrates all the disciplines and specialty groups into a team effort forming a structured development process that proceeds from concept to production to operation.

Systems Engineering considers both the business and the technical needs of all customers with the goal of providing a quality product that meets the user needs.

International Council on Systems Engineering


Motivations for Research in Systems Engineering


DSB/AFSAB Report on Acquisition of National Security Space Programs

May 2003

Findings:

Cost has replaced mission success as the primary driver in managing space development programs

Unrealistic estimates lead to unrealistic budgets and unexecutable programs

Undisciplined definition and uncontrolled growth in system requirements increase cost and schedule delays

Government capabilities to lead and manage the acquisition process have seriously eroded

Industry has failed to implement proven practices on some programs


Systems Engineering Continues to Be Cited as a Source of Problems

DOD IG: Lack of systems engineering imperils missile system

Published on Mar. 20, 2006

A lack of systems engineering plans could derail a $30 billion effort to field an integrated Ballistic Missile Defense System (BMDS), the Defense Department’s inspector general said in a report released earlier this month.

The Missile Defense Agency (MDA) has not completed a systems engineering plan or developed a sustainment plan for BMDS, jeopardizing the development of an integrated BMDS, the DOD IG said.

The report emphasizes that DOD must practice strong systems engineering to effectively sustain weapons systems. That begins with design and development.


Critical Need for Systems Engineering for “Robustness”

Systems Engineering for robustness means developing systems/system-of-systems that are:

� Capable of adapting to changes in mission and requirements � Expandable/scalable� Designed to accommodate growth in capability� Able to reliably function given changes in threats and environment � Effectively/affordably sustainable over their lifecycle � Easily modified to leverage new technologies

Reference: Rhodes, D., Workshop Report – Air Force/LAI Workshop on Systems Engineering for Robustness, July 2004, http://lean.mit.edu

In a 2004 workshop, Dr. Marvin Sambur, (then) Assistant Secretary of the

AF for Acquisition, noted that average program is 36% overrun according

to recent studies -- which disrupts the overall portfolio of programs.

The primary reason cited in studies of problem programs state the

number one reason for programs going off track is systems engineering.


Today’s Failures Exhibit Global Engineering Complexities

October 8 2005 CryoSat Mission lost due to launch failure

Mr Yuri Bakhvalov, First Deputy Director General of the Khrunichev Space Centre on behalf of the Russian State Commission officially confirmed that the launch of CryoSat ended in a failure due to an anomaly in the launch sequence …. missing command from the onboard flight control system…

.

This loss means that

Europe and the worldwide

scientific community will

not be able to rely on such

data from the CryoSat

mission and will not be

able to improve their

knowledge of ice,

especially sea ice and its

impact on climate

change.

Will this event have an impact on ESA’s relationship with Russia?

Space has always been a risky business. Failures can happen on each side.

From this end I do not expect any impact on relations with Russia. I wish to

underline that in this particular case we, ESA, were customers to Eurockot, the

launch service provider, which is a joint venture between EADS Space

Transportation (Germany) and Krunichev (Russia).


Contemporary Systems

Engineering

Systems of systems

Extended enterprises

Network-centric paradigm

Delivering value to society

Sustainability of systems

Design for flexibility

Managing uncertainty

Predictability of systems

Spiral capable processes

Model-based engineering

… and more

This requires a

broader field of study

for future systems

leaders and enabling

changes in education

and research …


The Research Landscape


MIT is tackling the large-scale engineering challenges of the 21st century through a new organization….

The Engineering Systems Division (ESD) creates and shares interdisciplinary knowledge about complex engineering systems through initiatives in education, research, and industry partnerships.

– Cross-cutting academic unit including engineering, management, social sciences

– Broadens engineering practice to include context of challenges as well as consequences of technological advancement

– Dual mission: (1) evolve engineering systems as new field of study and (2) transform engineering education and practice

MIT Engineering Systems Division as Intellectual Home for Systems Research

Council of 40+ universities is collaborating on this goal (http://www.cesun.org)


ES versus SEWhat Is the Difference?

SYSTEMS ENGINEERING (Traditional)

Systems engineering is the process of selecting and synthesizing the application of the appropriate scientific and technical knowledge in order to translate system requirements into system design. (Chase)

SYSTEMS ENGINEERING (Advanced)

Systems engineering is a branch of engineering that concentrates on design and application of the whole as distinct from the parts…looking at the problem in its entirety, taking into account all the facets and variables and relating the social to the technical aspects. (Ramo)

ENGINEERING SYSTEMS

A field of study taking an integrative holistic view of large-scale, complex, technologically-enabled systems with significant enterprise level interactions and socio-technical interfaces.


ENGINEERING

SYSTEMS

PoliticalEconomy

Economics,Statistics

Systems Theory

OrganizationalTheory

Operations Research/Systems Analysis

System Architecture& Eng /ProductDevelopment

EngineeringManagement

Technology & Policy

Engineering Systems as a Field of Study


Engineering Systems RequiresFour Perspectives

1. A very broad interdisciplinary perspective, embracing technology, policy, management science, and social science.

2. An intensified incorporation of system properties (such as sustainability, safety and flexibility) in the design process. – Note that these are lifecycle properties rather than first use properties. – These properties, often called “ilities” emphasize important intellectual considerations associated with long term use of engineering systems.

3. Enterprise perspective, acknowledging interconnectedness of product system with enterprise system that develops and sustains it. – This involves understanding, architecting and developing organizational structures, policy system, processes, knowledgebase, and enabling technologies as part of the overall engineering system.

4. A complex synthesis of stakeholder perspectives, of which there may be conflicting and competing needs which must be resolved toserve the highest order system (system-of-system) need.


SEAri Research Program


Systems Engineering Advancement Research Initiative (SEAri)

Current Sponsors: US Air Force Office of Scientific Research, Singapore Defense Sciences Office, US Air Force, Aerospace Corporation, MIT Portugal Program, Draper Laboratory, Lean Advancement Initiative, US Government Agency

3 Cambridge Center

NE20 – 388/343

Mission

Advance the theories, methods, and effective practice

of systems engineering applied to complex socio-

technical systems through collaborative research


Multiple heterogeneous stakeholder

groups with divergent cost goals and

measures of success

Single or homogenous stakeholder

group with stable cost/funding profile

and similar measures of success

Cost

Intense concept phase analysis followed

by continuous anticipation, aided by

ongoing experimentation

Concept phase activity to determine

system needs

Anticipation of

Needs

SoS component systems separately

acquired, and continue to be managed

and operated as independent systems

Centralized acquisition and

management of the system

Acquisition and

Management

Enhanced emphasis on “ilities” such as

Flexibility, Adaptability, Composeability

Reliability, Maintainability, Availability

are typical ilities

System

“ilities”

Component systems can operate

independently of SoS in a useful manner

Protocols and standards essential to

enable interoperable systems

Defines and implements specific

interface requirements to integrate

components in system

System

Interoperability

Dynamic adaptation of architecture as

needs change

System architecture established early

in lifecycle; remains relatively stable

System

Architecture

Evolving new system of systems

capability by leveraging synergies of

legacy systems

Development of single system to meet

stakeholder requirements and defined

performance

Purpose

Advanced

Systems Engineering

Traditional

Systems Engineering


Knowledge Deployment

SE-Field Research

Products and

Services

Military and

SecurityEnterprises

Systems of Systems

Workshops, Tutorials, White papers, Conferences, Journals

e.g.,“Leading indicators”

Collaborative distributed SE“Enablers to sys thinking”

““PracticePractice--basedbased””Descriptive research

Prescriptive

research

Normative

research

““TheoryTheory--basedbased””

“ Empirically relevant”

““ApplicationsApplications”

System Design

Commercial Systems

“Value-driven”V-STARS

e.g.,Dynamic MATEDesign for Value RobustnessTradespace visualizations

“Resource-effective”R-STARS

e.g.,Uncertainty assessment & managementResource usageCost estimating

SE-Synthesis e.g.,SE EducationSE Policy

Knowledge

Deployment

SE-Field

Products and

Services

Military and

SecurityEnterprises

Systems of Systems

Workshops, Tutorials,

White papers,

Conferences, Journals

““PracticePractice--basedbased””Descriptive

research

Prescriptive

research

Normative

research

““TheoryTheory--basedbased””

System Design

Commercial Systems

Value-driven

V- STARS

Resource-effective

R-STARS

SE-Synthesis

Empirically-relevant

Strategy-generating

Structured with four interacting “clusters” that undertake research in a portfolio of five topics:

1. Socio-Technical Decision Making

2. Designing for Value Robustness

3. Systems Engineering Economics

4. Systems Engineering in the Enterprise

5. Systems Engineering Strategic Guidance


Research Portfolio

SOCIO-TECHNICAL DECISION MAKING

This research area seeks to develop multi-disciplinary representations, analysis methods, and techniques for improving decision making for socio-technical systems.

DESIGNING for VALUE ROBUSTNESS

This research area seeks to develop methods for concept exploration, architecting and design using a dynamic perspective for the purpose of realizing systems, products, and services that deliver sustained value to stakeholders in a changing world.

SYSTEMS ENGINEERING ECONOMICS

This research area aims at developing a new paradigm that encompasses an economics view of systems engineering to achieve measurable and predictable outcomes while delivering value to stakeholders.

SYSTEMS ENGINEERING in the ENTERPRISE

This research area involves empirical studies and case based research for the purpose of understanding how to achieve more effective systems engineering practice in context of the nature of the system being developed, external context, and the characteristics of the associated enterprise.

SYSTEMS ENGINEERING STRATEGIC GUIDANCE

This research area involves synthesis of theory with empirical and case based researchfor the purpose of developing prescriptive strategic guidance to inform the development of policies and procedures for systems engineering in practice.


Research Portfolio (1)

SOCIO-TECHNICAL DECISION MAKING

This area of research is concerned with the context of socio-technical systems. Based on a multi-disciplinary approach, decision making techniques are developed through the exploration of:

– Studies of decision processes and effectiveness of techniques – Constructs for representing socio-technical systems to perform

impact analysis, including costs, benefits, and uncertainties– Visualization of complex trade spaces and saliency of

information – Understanding and mitigating cognitive biases in decision

processes

Representations, analysis methods, and techniques for improving socio-technical decision making


The Scope of Upfront Decisions

Key Phase Activities

Concept(s) Selected

Needs Captured

Resources Scoped

DesignDesign

In SituIn Situ

Top-side sounderTop-side soundervs.

In SituIn Situ

Top-side sounderTop-side soundervs.

After Fabrycky and Blanchard 1991

~66%

Conceptual Design is a high leverage phase in system development

Reliance upon BOGGSAT could have large consequences

How can we make better decisions?


Three keys to good upfront decisions

• Structured program selection process– Choosing the programs that are right for the organization’s stakeholders

• Systems engineering– Determining stakeholder needs, generating concept of operations, and deriving requirements

• Conceptual design practices– Finding the right form to maximize stakeholder value over the product (or product family) lifetime

“Good” system decisions must include both “socio”as well as “technical” components


Uncertainty Representations

Prescriptive Analysis(Hot/Cold Spot Analysis)

Sensitivity Analysis

Benefit Calculation:

Network Analysis

Cost Calculation:

Cost (effort,$)

Benefit (utils,$)

Measure of Uncertainty/Volatility

Uncertainty/VolatilityMeasure:

Forecast

Low p

Low Cost

Low Benefit

Low p

High Cost

Low BenefitLow p

High Cost

High Benefit

High p

Low Benefit

Low Cost

Tail Connector

Wing Connector

Interchangeable

Battery Module

Low pHigh Benefit

Low Cost

High p

High Cost

Low BenefitHigh p

High Benefit

Low Cost

High p

High Benefit

High Cost

Real Options in Enterprise ArchitectureTsoline Mikaelian, Aero/Astro PhD 2009

What enterprise representation/models can be used to identify potential real option investment opportunities?

How can you quantify the value of real options in enterprises to enable the selection of an options portfolio in enterprise decision making?

Methodology for Identifying

Opportunities for Flexible DesignJennifer Wilds, Aero/Astro and TPP SM 2008

How can the Engineering Systems Matrix (ESM) be used for identifying “hot” places for applying real options in complex systems?


Visualization Constructs for Tradespace Exploration

Brzezinski, A, and Cummings, M.L., “FanVis: A Decision Support Tool for Cost-Benefit Analysis,” Human and Automation Laboratory, May 2007

McManus, H.M., Richards, M.G., Ross, A.M., and Hastings, D.E., “A Framework for Incorporating “ilities” in Tradespace Studies,” AIAA Space 2007, Long Beach, CA, September 2007.

Thrusters

Flywheel

Gas & BottlesElectronics

Metal

Electro-Magnet

Construction Labor

Electronics

Construction Labor

AttitudeControl

ConstructionLabor

ConstructionLabor

Computer

Bluetooth

RPGA

Computer

Component

Subsystem

System

(Brzezinski 2007)(McManus 2007)


Research Summary

Project Title: Application of Multi-attribute Tradespace Exploration to a Transportation Engineering System

Sponsor: MIT- Portugal

Goal: Gain knowledge about new issues in the application of MATE to civil and commercial engineering systems

Approach: Applying MATE analyses to case studies in Portuguese transportation systems (airline, rail)

Nickel, J., Ross, A.M., and Rhodes, D.H., “Cross-domain Comparison of Design Factors in System Design and Analysis of Space and Transportation Systems,” 6th Conference on Systems Engineering Research, Los Angeles, CA, April 2008.



DESIGNING for VALUE ROBUSTNESS

This area of research seeks to develop methods for concept exploration, architecting and design using a dynamic perspectivefor the purpose of realizing systems, products, and services that deliver sustained value to stakeholders in a changing world.

– Methods for dynamic multi-attribute trade space exploration

– Architecting principles and strategies for designing survivable systems

– Architecting strategies for coupling in system of systems

– Quantification of the changeability of a system design

– Techniques for the consideration of unarticulated and latent stakeholder value

– Taxonomy for enabling stakeholder dialogue on ‘ilities’

Representations, analysis methods, and techniques for designing systems for “success” in dynamic contexts


Strategies and Methods for Value Robust Systems Toward Improved Practice

Strategies for Active and Passive Value Robustness

time

value

Epoch 1a Epoch 2

original state

recovered state

disturbance

recovery

Epoch :

Time period during w ith a fixed context; characterized by static constraints, design concepts, available technologies, and articulated attributes (Ross 2006)

emergency value threshold

expected value threshold

perm itted recovery time

V x

Ve

T r

Epoch 1b

V(t)

Active Passive

Utility

Cost

State 1 State 2

U

Cost

DV2≠DV1

DV2=DV1

Utility

0

Epoch 1 Epoch 2

S1,b S1,e S2,b S2,e

T1 T2

Time

Architecting for Survivability

Filtered Outdegree# outgoing arcs from design at acceptable “cost”

(measure of changeability)

OD( )C

<C

> C

>C

OD( )C

<C

> C

>C

OD( )C

<C<C

> C> C

>C>C

OD(<C)

C

OD(< )C

C

OD(<C)

C

OD(< )COD(< )C

C

Subjective FilterOutdegree

Cost



OD( )C

<C

> C

>C

OD( )C

<C

> C

>C

OD( )C

<C<C

> C> C

>C>C

OD(<C)

C

OD(< )C

C

OD(<C)

C

OD(< )COD(< )C

C

OD(<C)

C

OD(< )C

C

OD(<C)

C

OD(< )COD(< )C

C


Cost

Quantifying Flexibility


Meeting Customer Needs

• Goal of design is to create value (profits, usefulness, voice of the customer, etc…)

• Requirements capture a mapping of needs to specifications to guide design


Deploying a “Valuable”System…

Contexts change…


Meeting Customer Needs (cont.)

• Goal of design is to create value (profits, usefulness, voice of the customer, etc…)

• Requirements capture a mapping of needs to specifications to guide design

• People change their minds…

• To continue to deliver value, systems may need to change as well…


Aspects of Dynamic MATE*

How can System Designers cope with these types of changes during design?

• System Success criteria– Expanding scope of system “value”

• Tradespace exploration– Understanding success possibilities across a large number of designs

• Change taxonomy– Specifying and identifying change types

• Tradespace networks– Analyzing changeability of designs

• System Epoch/Era analysis– Quantifying effects of changing contexts on system success

[ ]∑=

+≥+=ΨN

i

Y

CDMi

X

CDMi ttYttXt DMiDMi

1

)()()()()( εε

Change Agent

Internal(Adaptable)

External(Flexible)

None(Rigid)

Change Agent

Internal(Adaptable)

External(Flexible)

None(Rigid)

Change Effect

Parameter level(Scalable)

Parameter set(Modifiable)

None(Robust)

Change Effect



None(Robust)

Change Effect



None(Robust)

Uti

lity

Cost

Cost

Utility

Cost

Utility

U

0

Epoch 1 Epoch 2 Epoch 3 Epoch n

…S1,b S1,e S2,b S2,e S3,b S3,e Sn,b Sn,e

T1 T2 T3 Tn

Time

U

0



T1 T2 T3 Tn

Time

Experience(t)Exceeding

Expectation(t)

*MATE = Multi-Attribute Tradespace Exploration


Tradespace Exploration

Total Lifecycle Cost

($M2002)

Assessment of cost and utility of large space of possible system designs

ATTRIBUTES: Design decision metrics– Data Lifespan (yrs)– Equatorial Time (hrs/day)– Latency (hrs)– Latitude Diversity (deg)– Sample Altitude (km)

Orbital Parameters– Apogee Altitude (km)– Perigee Altitude (km)– Orbit Inclination (deg)

Spacecraft Parameters– Antenna Gain – Communication Architecture– Propulsion Type– Power Type– Total Delta V

DESIGN VARIABLES: Design trade parameters

Each point is a specific design

Attributes Utility

Design Variables

“Cost”

Analysis

Tradespace: {Design,Attributes} �� {Cost,Utility}

Cost, Utility

Value

Concept

FirmDesignerCustomerUser

Value-driven design…


Total Lifecycle Cost

($M2002)

Example “Real Systems”Spacetug vs CX-OLEV

130*148Cost

0.690.69Utility

15900**12000 – 16500***DV m/s

213*300Equipment kg

730*600Propellant kg

670*805Dry Mass kg

14001405Wet Mass kg

CX-OLEV

(2009 launch)

Electric Cruiser

(2002 study)

XTOS vs Streak

Ion gauge and atomic

oxygen sensor

Three (?)Instruments

75***75 - 72Cost $M

0.590.56 - 0.50 Modified Utility**

0.57 - 0.54*0.61 - 0.55 Utility

MinotaurMinotaurLV

321a-296p -> 200 @ 96°300 -185 km @ 20°Orbit

12.3 - 0.5Lifetime (yrs)

420325 - 450Wet Mass kg

Streak (Oct 2005 launch)XTOS (2002 study)


Tradespace Analysis: Selecting “best” designs

Cost

Utility 1

A

B

C

D

E

CostUtility 2

A

B

C

D

E

If the “best” design changes over time, how does one select the “best” design?

Time

New “best” designNew “best” designClassic “best” designClassic “best” design


Tradespace Networks

Cost

Utility

Cost

Utility

Transition rules

Transition rules are mechanisms to change one design into anotherThe more outgoing arcs, the more potential change mechanisms

Tradespace designs = nodes

Applied transition rules = arcs

1

2

3

4

Cost

1 2

1

2

3

4

Cost

1 2

Example: X-TOS Transition Rules

External (Flexible)Change all orbit, ∆∆∆∆VR8: Add Sat

External (Flexible)Increase ∆∆∆∆V, requires “refuelable”R7: Space Refuel

External (Flexible)Increase/decrease perigee, requires “tugable”R6: Perigee Tug

External (Flexible)Increase/decrease apogee, requires “tugable”R5: Apogee Tug

External (Flexible)Increase/decrease inclination, requires “tugable”R4: Plane Tug

Internal (Adaptable)Increase/decrease perigee, decrease ∆∆∆∆VR3: Perigee Burn

Internal (Adaptable)Increase/decrease apogee, decrease ∆∆∆∆VR2: Apogee Burn

Internal (Adaptable)Increase/decrease inclination, decrease ∆∆∆∆VR1: Plane Change

Change agent originDescriptionRule


Tradespace Networks: Changing designs over time

Cost

Utility 1

A

B

C

D

E

CostUtility 2

A

B

C

D

E

Select changeable designs that can approximate “best” designs in new contexts

Time

Classic “best” designClassic “best” design New “best” designNew “best” design


Determining Changeability

The Question: Is the system _____________?(Flexible, Adaptable, Robust,

Scalable, Modifiable, Changeable,

Rigid, etc…)

The Answer: It depends!

The question of changeability is partly subjective: Is the “cost” for change acceptable?

100 101 102 103 104 105 106 …

Yes No

Cost or Time


objective subjective

Changeability Metric: Filtered Outdegree



OD( )C

<C

>C

>C

OD( )C

<C

>C

>C

OD( )C

<C<C

>C>C

>C>C

OD(<C)

C

OD(< )C

C

OD(<C)

C

OD(< )COD(< )C

C


Cost



OD( )C

<C

>C

>C

OD( )C

<C

>C

>C

OD( )C

<C<C

>C>C

>C>C

OD(<C)

C

OD(< )C

C

OD(<C)

C

OD(< )COD(< )C

C

OD(<C)

C

OD(< )C

C

OD(<C)

C

OD(< )COD(< )C

C


Cost

Outdegree# outgoing arcs from a given node

ODK

RKRK+1

RK+1

ODK

RKRK+1

RK+1

ODK

RKRK+1

RK+1

Filtered outdegree is a measure of the apparent changeability of a design


Tradespace Networks in the System Era

0 0.5 1 1.5 2 2.50

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Change Tradespace (N=81), Path: 81-->10, Goal Util: 0.97

U

Total Delta C

U

Total Transition Time0 1 2 3 4

1

0

Change Tradespace (notional), Goal Util: 0.97

Temporal strategy can be developed across networked tradespace

U

0



T1 T2 T3 Tn

Time

U

0



T1 T2 T3 Tn

Time

System EraPareto Tracing across

Epochs

Rk

ODk

≈Nk

Rk

ODk

≈Nk

Changeability Quantified

as Filtered Outdegree


Example System Timeline

U Epoch 1 Epoch 2 Epoch n

…S1,b S1,e S2,b S2,e Sn,b Sn,e

0

T1 T2 Tn

Value degradation

Major failure

Service to “restore”

New Context: new value function (objective fcn)

Same system, but perceived value decrease

Service to “upgrade”

Major failure

Service to “restore”

Value outage: Servicing time

System BOL System EOL

System timeline with “serviceability”-enabled paths allow value delivery

Example system: Serviceable satellite


Ex: Desiring “No” Change: Value Robustness

Preferences t=1

Attribute kiSize 0.5Loudness 0.2

Choose 3

1 2 3 4

big>smallloud>quietred>gray>black

Change?

New

If switching costs are high, option 2 may be better choice (i.e. robust in value)2

Preferences t=2

Attribute kiSize 0.5Loudness 0.2Color 0.6

Choose 443

Attribute “priority”

Clever designs or changeable designs can achieve value robustness

U(3)>U(2)>U(1)>U(4)

U(4)>U(2)>U(3)>U(1)


Achieving Value Robustness

Utility

Cost

State 1 State 2

U

Cost

DV2≠DV1

DV2=DV1

Utility

0

Epoch 1 Epoch 2

S1,b S1,e S2,b S2,e

T1 T2

Active Passive

Research suggests two strategies for “Value Robustness”

1. Passive• Choose “clever” designs that remain high value

• Quantifiable: Pareto Trace number

2. Active• Choose changeable designs that can deliver high value when needed

• Quantifiable: Filtered Outdegree

Value robust designs can deliver value in spite of inevitable context change

Time

New Context Drivers• External Constraints• Design Technologies• Value Expectations


Designing for Value Robustness

Mindshift: recognize dynamic contexts and fallacy of static preferences; the inevitability of “change”

• Matching changing systems to changing needs leads to sustained system success

• Methods for increasing Changeability– Increase number of paths (change mechanisms)– Lower “cost” or increase acceptability threshold (apparent changeability)

• Concept-independent measure of changeability: filtered outdegree– Change mechanism drives filtered outdegree, as does consideration of alternative systems (future states)

– Quantifiable filtered outdegree couples both objective and subjective measures (irreducible subjectivity � acceptability threshold)

• Changeability can be used as an explicit and consistent metric for designing systems

Designed for changeability, systems will be empowered to become value robust, delivering value in spite of context and

preference changes


Other Research in “Designing for Value Robustness”


Designing for Unarticulated Values

Large � infiniteCannot be added through changing system (system too rigid)

Inaccessible Value4 (state 5)

Small � largeCan be added through changing system (scale or modify)

Accessible Value3 (state 4)

SmallCan exist by recombining class 0 and/or 1Combinatorial Latent Value2 (state 3)

0Exist, not assessedFree Latent Value1 (state 2)

0Exist and assessedArticulated Value0 (state 1)

Cost to DisplayProperty of ClassNameClass

Attribute Classification Framework

CBA

D

UnarticulatedArticulatedE

FC

BAD

UnarticulatedArticulatedE

F

A

A BC

BA

A BC

BA

A BC

BCA

A BC

BCA

A BC

BDC

D

A

A BC

BDC

D

Increasing “cost” to display

A

A BC

BDC

E

E D

A

A BC

BDC

E

E D

A

A BC

BDC

E

E D

FFFF

A

A BC

BDC

E

E D

FFFF

Displayed Displayed Displayed Displayed Displayed

state 1 state 2 state 3 state 4 state 5


What are Principles for Architecting for Survivability?

Matt Richards, PhD Candidate, 2009

The interdependence of

large-scale, distributed

systems has grown since

the advent of modern

telecommunications

Engineering systems are

increasingly at risk from

disturbances that rapidly

propagate through

networks, damage critical

infrastructure, and

undermine system-of-

systems time

value

Epoch 1a Epoch 2

original state

recovered state

disturbance

recovery

Epoch:

Time period during with a fixed context; characterized by static constraints, design concepts, available technologies, and articulated attributes (Ross 2006)

emergency value threshold

expected value threshold

permitted recovery time

VxVe

Tr

Epoch 1b

V(t)Time period with a fixed context; characterized by static constraints, design concepts, available technologies, and articulated attributes (Ross 2006)


Research Summary

Project Title: Quantifying Flexibility for the Operationally Responsive Space Paradigm

Sponsor: Air Force Office

of Scientific Research

Goal: Provide tools and metrics for high level decision makers to evaluate flexibility during conceptual design.

Approach: Collaborative engagement using a selected ORS model as the basis for exploratory research in applying methodology to cases with operational and environmental fluctuations.

Richards, M., Viscito, L.., Ross, A., and Hastings, D., “Distinguishing Attributes for the Operationally Responsive Space Paradigm”, Responsive Space 6, Los Angeles, CA, April 2008


Research Summary

Project Title: Dynamic Tradespace Exploration Method Applied to System Of Systems

Sponsor: Draper Laboratories

Goal: Enhance the methods for concept tradespace exploration, particularly as extended to systems of systems problems and considerations

Approach: Collaborative engagement using a selected Draper project as the basis for exploratory research in applying MATE methodology to a system of systems case Chattopadhyay, D., Ross, A.M., and Rhodes, D.H., “A Framework for Tradespace Exploration of Systems

of Systems,” 6th Conference on Systems Engineering Research, Los Angeles, CA, April 2008.


Ongoing SDM Thesis Luke Cropsey

Integrating Unmanned Aircraft Systems in the National Air Space

• Applied look at Value-Focused Thinking, Enterprise & System Architecting, and Coping with Uncertainty

– Characterization of UAS and NAS stakeholders using a value-focused approach

– Use of enterprise architecture framework to generate alternative value propositions

– Assessment of uncertainty creation/resolution to determine optimal transition path

• Targeted Contribution: A framework for putting structure on dynamic context so solutions can be designed


Future Research Directions

Socio-Technical Decision Making

• Decision criteria for “valuing ilities”: when to seek active vs. passive Value Robustness– Refined quantitative metrics

• Advanced dynamic tradespace visualization experiments and methods development

• Field study assessing current senior decision making practices

• Application of descriptive temporal utility theories to elucidate changing preferences on system designs– Can engineers predict or anticipate

certain classes of preference change?

• Assessment of current acquisition policies for research deployment opportunities

Designing for Value Robustness

• Application of Dynamic MATE to transportation system

• Application of Dynamic MATE to Systems of Systems

• Categorization of architectural approaches for increasing changeability – Towards a changeable design “toolkit”

• Application of attribute class framework to case studies

• Refinement of fundamental attribute set, validated through case studies

• In depth application of Epoch/Era Analysis to case study(s)

• Application of industrial economics to market classes and the value of changeability



SYSTEMS ENGINEERING ECONOMICS

This research area aims at developing a new paradigm that encompasses an economics view of systems engineering to achieve measurable and predictable outcomes while delivering

value to stakeholders. Examples include:

• Measurement of productivity and quantifying the ROI of systems engineering

• Advanced methods for reuse, cost modeling, and risk modeling

• Application of real options in systems and enterprises

• Leading indicators for systems engineering effectiveness


Models, Measures, and Leading Indicators for Project SuccessThrough Better Execution of Systems Engineering

Cost and schedule modelingProject Risk Assessment

Leading Indicators for Performance Systems Engineering ROI

Person Months Risk

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

55%

60%

65%

70%

75%

80%

85%

90%

95%

100%

0 25000 50000 75000 100000

Person Months

Ris

k (

= P

rob

. T

ha

t A

ctu

al

Pe

rso

n M

on

ths

Wil

l E

xc

ee

d I

nd

ica

ted

, X

-Ax

is,

Fig

ure

)

Person Months Confidence (Cumulative Probability)

0%5%10%15%20%25%30%35%40%45%50%55%60%65%70%75%80%85%90%95%100%

0 25000 50000 75000 100000

Person Months

Cu

mu

lati

ve P

rob

ab

ilit

y o

f P

ers

on

Mo

nth

s


COSYSMO

SizeDrivers

EffortMultipliers

Effort, $

Calibration

# Requirements# Interfaces# Scenarios# Algorithms

+3 Volatility Factors

- Application factors-8 factors

- Team factors-6 factors

Valerdi, R., Systems Engineering Cost Estimation with COSYSMO, Wiley, 2008.


Systems Engineering Risk Assessment


Systems Engineering Leading Indicators

• Requirements Trends

• System Definition Change Backlog Trend

• Interface Trends

• Requirements Validation Trends

• Requirements Verification Trends

• Work Product Approval Trends

• Review Action Closure Trends

• Risk Exposure Trends

• Risk Handling Trends

• Technology Maturity Trends

• Technical Measurement Trends

• Systems Engineering Staffing & Skills Trends

• Process Compliance Trends

Requirements Trends

TIME

Requirements Growth Trends

TIME

NUMBER OF REQUIREMENTS

JulyMar Apr May JuneFebJan

LEGEND

Planned Number Requirements

Actual Number Requirements

Aug Sep Oct Nov Dec

Projected Number Requirements

SRR PDR CDR ….

Corrective Action Taken

Rhodes, D., Roedler, G., Systems Engineering Leading Indicators Guide, V. 1.0, June 15, 2007


Past SDM ThesisTim Flynn

Evaluation and Synthesis of

Methods for Measuring

Systems Engineering

Efficacy within Products

and Processes

• Case study on systems engineering leading indicators in an aerospace company


SE Leading Indicators Project

Creating New Knowledge

Guide to SE Leading Indicators(December 2005)

BETA

AF/DODSE Revitalization

Policies

AF/LAI Workshop on Systems Engineering

June 2004

SE LI Working Group

With SSCI and PSM

+

Pilot Programs(several companies)

Masters Thesis(1 case study)

Validation Survey(1 company)

SE LI Working Group

With SSCI and PSM

+

+

Guide to SE Leading Indicators

June 2007

V. 1.0

Knowledge Exchange

Event

Tutorial on SE

Leading Indicators

(many companies)

(1) January 2007

(2) November 2007

Project is an example

of the power of

collaborative research



• Additional case studies on use of leading indictors on SE programs

• Case studies in application of enterprise real options framework

• Extension of leading indicators to Human Systems Integration

• Extension of COSYSMO to Human Systems Integration



SYSTEMS ENGINEERING in the ENTERPRISE

This research area involves empirical studies and case based research for the purpose of understanding how to achieve more effective systems engineering practice in context of the nature of the system being developed, external context, and the characteristics of the associated enterprise.

– Engineering systems thinking in individuals and teams

– Collaborative, distributed systems engineering practices

– Social contexts of enterprise systems engineering

– Alignment of enterprise culture and processes

– Socio-technical systems studies and models

Empirical Studies and case based research for understanding of how to achieve more effective practice


Motivation

• High variation in the scale of and nature of enterprises

• Increased diversity of stakeholders involved in and impacted by a program

• Systems engineering often involves collaboration of teams across geographies and organizations

• New methods and technologies are available for model-based SE and other advanced practices, but we lack empirical data to show under what conditions and within what enterprise contexts these can be successfully used

• Successful engineering depends upon systems thinking but ambiguity in what this means and how to develop it


Evolution of Practice of Systems Engineering

Over the past five or six decades, the discipline known as “Systems Engineering” has evolved. At one time, many years ago, development of a capability was relatively simple to orchestrate.

The design and development of parts, engineering calculations, assembly, and testing was conducted by a small number of people. Those days are long gone.

Teams of people, sometimes numbering in the thousandsare involved in the development of systems; and, what was previously only a development practice has evolved to become a science and engineering discipline.

Saunders, T., et al, System-of-Systems Engineering for Air Force Capability Development: Executive Summary and Annotated Brief, AF SAB TR -05-04, 2005


The understanding of the organizational and

technical interactions in our systems,

emphatically including the human beings who

are a part of them, is the present-day frontier of

both engineering education and practice.

Dr. Michael D. Griffin, Administrator, NASA

Boeing Lecture, Purdue University

28 March 2007


Descriptive Studies Better Understanding of Systems Engineering Practice

MIT/MITRE Social Contexts of ESE (2006 – 2008)

Collaborative Distributed SE (Utter 2007)

Empirical Studies of Systems Thinking (Davidz 2006)

Collaboration Situation and Management

AA BBKnowledge, Data and Decision Management

SE Processes and Practices

Collaboration Tool Use

CDSE Social and Cultural Environment

CDSE Benefits and Motivation

Description RecommendationLesson Learned

Issue or Barrier Success Factor Irrelevant

OR

Tool Training Network Reliability Tool VersionsTool AccessLearning Curves

Classified Data

Interview Heading Topics

Company

Transcripts

Subtopic

Interviewee Experience

Data Analysis

Example


MITRE/MIT Research on Social Contexts of Systems Engineering

Develop social science capabilities and products complementing MITRE’s technical capabilities in order to meet the challenges of Systems Engineering at the Enterprise level• Transform practical field experience of MITRE site staff

into social-scientific understanding that is usefully transferable

• Leverage experience and approaches from MIT partners

Technical Approach� Case Studies� Workshops� 2nd Round of Case Studies� Communicate Lessons Learned


RESEARCH PROJECT

Enabling Systems Thinking to Accelerate the Development of Senior Systems Engineers

Consensus on primary mechanisms that enable or obstruct systems thinking development in engineers

1. Experiential learning2. Individual characteristics3. Supportive environment

Even though systems thinking definitions diverge, there is

consensus on primary mechanisms that enable or obstruct systems thinking development in engineers

Dr. Heidi Davidz, PhD 2006

LAI Funded Research


RESEARCH PROJECT

Collaborative Distributed Systems Engineering in the Aerospace Industry

Darlene Utter, S.M. 2007

Social Factors:

Local/Company Culture DifferencesCareer Development Advancement

Distributed TeamworkCommunications

Working w/ IT and toolsProject Management

Technical Factors:

SE Process/ArchitectureDistributed Decision MakingTools/Information TechnologyKnowledge Management

Cost & ScheduleProduct Impact

CDSE

Develop heuristics for successful CDSE resulting from case studies

Recommendations to overcome barriers to successful CDSE

Recommendations for future work in this area


Research Summary

Project Title: Collaborative Systems Thinking:The role of culture and process in supporting higher level systems thinking

Sponsor: National Defense Science and Engineering Graduate Fellowship/Lean Aerospace Initiative

Goal: Identify organizational culture and process-based enablers and barriers to collaborative systems thinking within the aerospace industry

Approach: Exploratory research anchored by a series of case studies looking at the enablers and barriers to engineering teams in theaerospace industry developing collaborative systems thinking

Lamb, C.T. and Rhodes, D.H., "Collaborative Systems Thinking Research: Exploring Systems Thinking within

Teams," INCOSE International Symposium 2008, Utrecht, Netherlands, June 2008


Culture and Standard Process

• The impact of culture and process on team performance a repetitive theme in literature

• Defining leverage points will help in observing tensions and synergies

• Process is must easier to change than culture

• Alignment creates reinforcing loop

• Misalignment produces conflicting stimuli and dysfunctional behavior

Undocumented

team norms

Documented tasks

and methods

Culture Standardized

Process

Espoused

beliefs

Vision

statements

Underlying

assumptions

Strategy for

standardization

Social

networks

Process flow

maps, org-charts

Undocumented

team norms

Documented tasks

and methods

Culture Standardized

Process

Espoused

beliefs

Vision

statements

Underlying

assumptions

Strategy for

standardization

Social

networks

Process flow

maps, org-charts


Experimental Method

Literature

SearchReview

Past

Relevant

Cases

Pilot

Interviews

Theory

Synthesis

Data

Analysis

Theory

Generation

and Idea

Sharing

Coding,

statistical

analysis,

MS Excel

Case StudiesData Sources

Primary

Documentation

Interviews

Observations Surveys

Experimental Design

Identify potential case opportunities

Identify potential case opportunities

Work with point

of contact

Work with point

of contact

Collect survey dataCollect survey data

Interview engineering team members

Interview engineering team members

Observe team membersObserve team members

Case ?

Case Case Case β

Case ?

Case ?

Case α

Case Case γCase

Case ε

Case ζ

δ


Past SDM Theses

Re-Architecting the Battalion Tactical Operations Center: Transitioning to Network Centric Operations – Nathan Minami (2007)

Construction Design as a Process for Flow: Applying Lean Principles to Construction Design -- Leticia Soto (2007)

Re-Architecting the DoD Acquisition Process: A Transition to the Information Age – Kevin Brown


Past SDM Theses

An Exploration of Architectural Innovation in Professional Service Firms - Fernando Espinosa Vasconcelos (2007)

• Architectural innovation is achieved using architectural knowledge to

reconfigure an established system to link together components in a new

way that provides a competitive advantage

Concurrent Engineering for Mission Design in Different Cultures -- Akira Ogawa (2008)

• Analysis of Integrated Concurrent Engineering approach to identify the key

factors for successful implementation and operation from both systems

engineering and cultural perspectives through the case studies of a

implementation failure in a Japanese organization and some successes in

Euro-American organizations



• More extensive and rigorous studies to understand collaborative distributed systems engineering – Is there a preferred system architecture to support CDSE?

– What incentives and performance measures should be used?

– Development of an “SE Collaboration Maturity Factor”

• Additional research related to development of systems competencies in the workforce

• Field research to motivate theory and principles for developing and managing enterprises for context-harmonized interactions

• Understand the factors for effective systems engineering in commercial product and service enterprises



SYSTEMS ENGINEERING STRATEGIC GUIDANCE

This research area involves synthesis of theory with empirical and case based research for the purpose of developing prescriptive strategic guidance to inform the development of policies and procedures for systems engineering in practice.

– Systems Engineering research guidelines

– Participation in focus groups and pilot-phase reviews

– Position papers on proposed policies

– Recommendations for integrating SE research into curriculum

– Identification of SE research gaps and opportunities


Influencing Proposed Policy

• As a first step toward a prescriptive model, the Department of Defense (DoD) is developing a Guide to System of Systems Engineering (SoSE) based on best present state knowledge – The guide provides 16 DoD technical and management processes to help sponsors, program managers, and chief engineers address theunique considerations for DoD SoS

• SEAri participated as an academic review body in the pilot phase of the development of the guide

• A recent position paper by SEAri researchers included recommendations for how a normative, descriptive and prescriptive framework can be used to contribute to evolving SoSE guidance

Valerdi, R., Ross, A., and Rhodes, D., A Framework for Evolving System of Systems Engineering”, Crosstalk: The Journal of Defense Software Engineering, 2007


Access to Research

seari.mit.edu

PurposeWeb portal for sharing

research within SEAri, MIT, and systems community

Navigation

Home

About

People

Research

Related Courses

Documents

Events

Sponsors

Community

Contact

Internal