© 2005 Massachusetts Institute of Technology Adam Ross/03.04.05- 1web.mit.edu/leanweb.mit.edu/lean

Capturing Unarticulated Value: System Properties and Multi-Attribute Tradespace Exploration

Presented ByAdam Ross, ESD PhD Candidate

MIT/MITRE WorkshopMarch 4, 2005

Committee: Daniel Hastings (Chair), Deborah Nightingale, Olivier de Weck, Thomas Allen

© 2005 Massachusetts Institute of Technology Adam Ross/03.04.05- 2web.mit.edu/leanweb.mit.edu/lean

Goal of designCreate a system that fulfills some need while

efficiently utilizing resources

QuestionHow can a system designer cope with poorly

defined needs?

© 2005 Massachusetts Institute of Technology Adam Ross/03.04.05- 3web.mit.edu/leanweb.mit.edu/lean

0.98

0.985

0.99

0.995

1

100 1000

Lifecycle Cost ($M)

Util

ity

A

ED

C

B

0.98

0.985

0.99

0.995

1

100 1000

Lifecycle Cost ($M)

Util

ity

A

ED

C

Bvalue

External Stakeholders

Firm Customer

Designer User

Level 0

Level 1

Level 2

Contracts

Requir

emen

ts

Polic

y, etc

. Policy, etc.

Organizational Goals

Operational Strategy

designs uses

External Stakeholders

Firm Customer

Designer User

Level 0

Level 1

Level 2

Contracts

Requir

emen

ts

Polic

y, etc

. Policy, etc.

Organizational Goals

Operational Strategy

designs uses

External Stakeholders

Firm Customer

Designer User

Level 0

Level 1

Level 2

Contracts

Requir

emen

ts

Polic

y, etc

. Policy, etc.

Organizational Goals

Operational Strategy

designs uses

Architecture Exploration

Architecture Evaluation

Need Identification

Design Exploration

Design Evaluation

System Design(s)

Architecture Exploration

Architecture Evaluation

Need Identification

Design Exploration

Design Evaluation

System Design(s)

Architecture Exploration

Architecture Evaluation

Need Identification

Design Exploration

Design Evaluation

System Design(s)

1 2 3 4 5 6 7 10 11 14 15 18 8 9 12 13 16 17 20 21 22 25 19 23 24 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42Identify Need 1 1 1Define Mission 2 1 2 1 1Define Scope 3 1 1 3Identify all relevant decision makers 4 1 1 1 4Identify Constraints 5 1 1 1 1 5 1Propose Attribute Definitions (USER) 6 1 1 1 1 6Nail down attribute definitions (USER) 7 1 1 1 1 7Propose Attribute Definitions (CUSTOMER10 1 1 1 1 10Nail down attribute definitions (CUSTOME 11 1 1 1 1 11Propose Attribute Definitions (FIRM) 14 1 1 1 1 14Nail down attribute definitions (FIRM) 15 1 1 1 1 15Concept generation 18 1 1 1 1 1 1 1 1 1 1 1 18Utility interview (USER) 8 1 1 1 8 1Utility verification and validation (USER) 9 1 1 1 1 9Utility interview (CUSTOMER) 12 1 1 1 12 1Utility verification and validation (CUSTOM13 1 1 1 1 13Utility interview (FIRM) 16 1 1 1 16 1Utility verification and validation (FIRM) 17 1 1 1 1 17Propose Design Variables 20 1 1 1 1 1 1 1 1 20Nail down Design Variables 21 1 1 1 1 1 1 1 1 1 21Map Design variable to attributes 22 1 1 1 1 1 1 1 1 1 1 1 1 22Decompose code (develop software archit 25 1 1 1 1 1 1 1 1 1 1 1 1 1 25Organization formation (software teams) 19 1 1 19Identify I/O for entire simulation 23 1 1 1 1 1 1 1 1 1 1 1 1 1 23Write Model translation from DV to Att 24 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 24 1Integrate model 26 1 1 1 1 1 1 1 1 1 1 26Enumerate tradespace 27 1 1 1 1 1 1 1 1 1 27Navigate enumerated tradespace (intellige28 1 1 1 1 1 1 1 1 1 28Run simulation (calculate attributes) 29 1 1 1 1 1 1 29Run Utility function 30 1 1 1 1 1 1 1 1 1 1 1 1 30Verify Output 31 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 31 1 1Analyze output 32 1 1 1 1 1 1 1 1 1 1 1 1 1 1 32Perform sensitivity analysis (constants/con33 1 1 1 1 1 1 1 1 1 1 1 33Perform sensitivity analysis (utility function 34 1 1 1 1 1 1 1 1 1 1 1 1 1 1 34Refine tradespace 35 1 1 1 1 1 1 1 1 1 1 35Rerun simulation/utility function 36 1 1 1 1 1 1 1 1 1 1 1 1 1 36Analyze output 37 1 1 1 1 1 1 1 1 1 37Locate frontier 38 1 1 1 1 1 38 1Select reduced solution set 39 1 1 1 1 1 1 1 1 39 1Show to DM(s) 40 1 1 1 1 1 1 1 40Define stakeholder tradeoff function 41 1 1 1 1 1 41Select design(s) for concurrent design 42 1 1 1 1 1 1 1 42Set selected design as baseline for CE 43 1 1 1 1 1 1Develop higher fidelity CE models 44 1 1 1 1 1Perform concurrent design trades 45Converge on final design(s) 46Show to DM(s) 47Select final design(s) 48

43 44 45 46 47 48

431 441 1 45

1 1 461 1 47

1 48

MATE-CON is…

• A Decision Maker preference-directed tradespace exploration process

• Robust to changes and concepts

• A formalized framework

• A flexible philosophy Tradespace Solution Spaces

Decision Makers Formal Framework

MATE-CON OverviewDual-SMAero/Astro, TPP 2003

© 2005 Massachusetts Institute of Technology Adam Ross/03.04.05- 4web.mit.edu/leanweb.mit.edu/lean

Capturing System Properties: Tradespace “ilities”

0.98

0.985

0.99

0.995

1

100 1000

Lifecycle Cost ($M)

Util

ity

A

ED

C

B

0.98

0.985

0.99

0.995

1

100 1000

Lifecycle Cost ($M)

Util

ity

A

ED

C

B

value?

Pareto surface

Q: What if Decision Maker(s) cannot/do not articulate everything that will result in VALUE?

A: Attempt to assess system properties

0.11MaintainabilityP14

0.08ReparabilityP13

0.96ReliabilityP12

0.87QualityP11

0.24SustainabilityP10

0.34DurabilityP9

0.99SafetyP8

0.45ModularityP7

0.70ScalabilityP6

0.65AdaptabilityP5

0.15FailsafeP4

0.88RobustnessP3

0.54AgilityP2

0.66FlexibilityP1

Do these ilities apply to entire tradespace, or to each solution?

Design

From Ross, TBD

© 2005 Massachusetts Institute of Technology Adam Ross/03.04.05- 5web.mit.edu/leanweb.mit.edu/lean

“Articulated” versus “Unarticulated” Value

• Articulated Value• Traditional requirements• Objectives, attributes• “Desirements”• “Must haves”

• Unarticulated Value• Fuzzy desires,

objectives• Emergent applications• “Good to haves”• System properties?

??

“A flexible system is a good thing, but what is it and how much?”““A flexible system is a good thing, but what is it and how much?A flexible system is a good thing, but what is it and how much?””

© 2005 Massachusetts Institute of Technology Adam Ross/03.04.05- 6web.mit.edu/leanweb.mit.edu/lean

System Context: Value Lenses

SystemInputs Outputs

System Boundary

Value

“Experienced”

Value

“Decisional”

ConstraintsStatic ViewStructural Metrics

Delay

Dynamic ViewOperational Metrics

© 2005 Massachusetts Institute of Technology Adam Ross/03.04.05- 7web.mit.edu/leanweb.mit.edu/lean

Tradespace Exploration: Avoiding Point Designs

Cost

Utility

Tradespace exploration enables big picture understandingTradespace exploration enables big picture understandingTradespace exploration enables big picture understanding

Differing types of trades

1. Local point solution trades

2. Frontier subset solutions

3. Frontier solution set

Designi = {X1, X2, X3,…,Xj}

4. Full tradespace exploration

4b. Tradespace ilities (NEW)

(t)

(t)

© 2005 Massachusetts Institute of Technology Adam Ross/03.04.05- 8web.mit.edu/leanweb.mit.edu/leanFrom Shah, 2003

Technology and Programmatic Flexibility Over Time for Space-Based Radar

Flexibility Over Time: SBR

2 3 4 5 6 7 8 9 10 110.8

0.82

0.84

0.86

0.88

0.9

0.92

0.94

0.96

0.98Number of Transition Possibilities: 2002 to 2005 (decreasing altitude

Life cycle Cost ($B)

Uti

lity

123456789

e)

Cost of flexibility?

SBR

• “Transitionability”relates to “distance”between architectures

• Transition costs vary widely based on transition path

• Pareto Front may not provide best answers

• “Optimality” not defined when fitness function changes over time

Additional tech transitions possible based on investment

© 2005 Massachusetts Institute of Technology Adam Ross/03.04.05- 9web.mit.edu/leanweb.mit.edu/lean

Example Tradespace Change: Pareto Set Shift

What happens when preferences change?

0 2 4 6 8 10 120.8

0.82

0.84

0.86

0.88

0.9

0.92

0.94

0.96

0.98

1

10 year life cycle cost ($b)

Mul

tiatt

ribut

e U

tility

(us

ing

MIS

T2

data

)

0 2 4 6 8 10 120.95

0.955

0.96

0.965

0.97

0.975

0.98

0.985

0.99

0.995

1

10 year life cycle cost ($b)

Mul

tiatt

ribut

e U

tility

(us

ing

HA

ND

2 da

ta)

From Spaulding, 2003Dynamic tradespaces may reveal dynamic propertiesDynamic tradespaces may reveal dynamic propertiesDynamic tradespaces may reveal dynamic propertiesFrom Spaulding, 2003

© 2005 Massachusetts Institute of Technology Adam Ross/03.04.05- 10web.mit.edu/leanweb.mit.edu/lean

Research Design: Experiments

• Computer experiments• Isolate ilities• Utilize past data (X-TOS, B-TOS, SpaceTug)

• Experiment set #1: Dynamic context• Question: How is a tradespace affected when articulated value

changes?• Two classes of “value” changes:

• Single DM changing over time• Addition of new/different DM

• Post-process X-TOS dataset• Vary input preferences, collect statistics on output tradespace,

including Pareto Set

“Optimized” solutions may be poor choices under changing value

““OptimizedOptimized”” solutions may be poor solutions may be poor choices under changing value choices under changing value

40 42 44 46 48 50 52 54 56 58 600.2

0.3

0.4

0.5

0.6

0.7

0.8

40 42 44 46 48 50 52 54 56 58 600.2

0.3

0.4

0.5

0.6

0.7

0.8

Original Revised

© 2005 Massachusetts Institute of Technology Adam Ross/03.04.05- 11web.mit.edu/leanweb.mit.edu/lean

Research Questions

• “New” research questions:

1. What are the relationships between flexibility, adaptability, robustness, and scalability for spacesystems and how do they relate to unarticulated value?

2. How can these ilities be quantified and/or used as decision metrics when exploring tradespaces during Conceptual Design?

• Questions guide exploratory research design• Inheritance, Theory, Experiments, Case studies

© 2005 Massachusetts Institute of Technology Adam Ross/03.04.05- 12web.mit.edu/leanweb.mit.edu/lean

Expected Findings

Contributions of research• Concrete quantification of flexibility,

robustness, adaptability, and scalability in a value-centric tradespace framework

• Insight into how unarticulated value may be assessed

• Useful and expanded process/framework for Conceptual Design trade studies

• Suggestions for future work

© 2005 Massachusetts Institute of Technology Adam Ross/03.04.05- 13web.mit.edu/leanweb.mit.edu/lean

Questions?