benchmarking the benefits and current maturity of digital ... · systems engineering research...

1

Benchmarking the Benefits and Current Maturity of Digital Engineering/Model-Based SE

Tom McDermott, Stevens Inst. of Technology

Nicole Hutchison, Megan Clifford (Stevens)Eileen Van Aken, Alejandro Salado, Kaitlin Henderson (Va Tech)

www.sercuarc.org

This material is based upon work supported, in whole or in part, by the U.S. Department of Defense through the Systems Engineering Research Center (SERC) under Contract H98230-08-D-0171. The SERC is a federally funded University Affiliated Research Center (UARC) managed by

Stevens Institute of Technology consisting of a collaborative network of over 20 universities. More information is available at www.SERCuarc.org

http://www.sercuarc.org/

Systems Engineering Research Center 2

Frameworks for Evaluating DE Value

• DoD Digital Engineering Strategy

―Vision, mission, and goals can be generalized to any organization

• INCOSE Model-Based Enterprise Capability Matrix

―Benchmark for Organizational and Industry-wide DE Maturity, published January 2020

• DE/MBSE Value, Benefits, and Metrics Framework

―Developed by the SERC, four categories for DE value/benefits: Quality, Velocity/Agility, User Experience, Knowledge Transfer

• DE/MBSE Enterprise Adoption Framework

―Developed by the SERC from the Baldrige CPE Framework:Adoption


DoD Digital Engineering Strategy

STRATEGIC MANAGEMENT SYSTEMDigital Engineering (DE) Vision: Modernize how the Department designs, develops, delivers,

operates, and sustains systems.

DE Mission: Securely and safely connect people, processes, data, and capabilities across an end-

to-end digital enterprise.

WA

YS

/I N

ITIA

TIV

ES

EN

DS

EN

DS

WA

YS

/ INIT

IAT

IVE

SM

EA

NS

ME

AN

S

An enduring,

authoritative

source of truth is

used over the

lifecycle

Plan and develop the

authoritative source

of truth

Govern the

authoritative source

of truth

Use technological

innovation to

improve engineering

practices

Infuse technological

innovations to enable

the end-to-end digital

enterprise

Advance human-

machine interactions

Make use of data to

improve awareness,

insights, and decision

making

Models are used to

inform enterprise

and program

decision making

Formalize the planning

for models to support

engineering activities

and decision making

across the lifecycle

Use the authoritative

source of truth across

the lifecycle

Use models to support

engineering activities

and decision making

across the lifecycle

Improve the digital

engineering

knowledge base

Transform culture

and workforce

engineering across

the lifecycle

Build and prepare the

workforce

Lead and support

digital engineering

transformation efforts

Process, Methods, Tools, Technology, People

Infrastructure

and environments

support improved

communication and

collaboration

Develop, mature, and

use digital

engineering IT

infrastructures

Secure IT

infrastructure and

protect intellectual

property

Develop, mature, and

use digital

engineering

methodologies

Formally develop,

integrate,

and curate models


Focus: Enterprise Transformation Measures

• Gartner: “Select just 5 to 9 metrics to track, report and act on. The value of a metric lies in its ability to influence business decision making.”

• The best metrics:

―Have a defined and defensible causal relationship to a business outcome

―Work as a leading, not lagging, indicator

―Address a specific defined audience – in a way they can understand

―Drive action when they change from green to yellow to red

• There are no universal metrics – must be enterprise specific

• Good Digital Transformation metrics have some traits

―Measure people adoption, and enterprise process adoption

―Analyze breadth of usability, and issues with usability

―Measure productivity indicators

―Generate new value to the enterprise (revenue, operational efficiency, etc.)

https://www.gartner.com/smarterwithgartner/how-to-measure-digital-transformation-progress/


Summary DE Success Measures Framework

Quality:

• Reduce Errors/Defects

• Improve System Quality

• Improve Traceability

• Reduce Cost

Knowledge Transfer:

• Better access to

information

• Better communication/

info sharing

• Collaboration

An enduring,

authoritative

source of truth is

used over the

lifecycle

Use technological

innovation to

improve

engineering

practices

Models are used to

inform enterprise

and program

decision making

Transform culture

and workforce

engineering across

the lifecycle

Infrastructure

and environments

support improved

communication and

collaboration

Adoption:

• Methods/Processes

• Roles/Skills

• Training/Tools

• Leadership support

• Change Mgmt Process

• Resources

Velocity/Agility:

• More Reuse

• Improve Consistency

• Increase Efficiency

• Support Integration

• Reduce Time

User Experience:

• Manage Complexity

• Improved System

Understanding

• Automation


MBSE Benefits: Literature Review Results

• Searched papers that mention a benefit of MBSE and what the source of that benefit was: measured gains, observed gains, perceived gains (no source for benefit), reference.

―Total Papers that mention MBSE: 847

oPapers that mention benefits: 360

―Measured gains: 2

―Observed gains: 27

―Perceived gains: 236

• Identified 48 categories of benefits linked to the metrics framework across the areas of Quality, Velocity/Agility, User Experience, and Knowledge Transfer.

*Kaitlin Henderson (VT) PhD studies


Literature Review Benefit Categories (48)

Category List of Benefits

Quality

Reduce errors/ defects Improved risk analysis Improved capability

Improved traceability Improved system design More stakeholder involvement

Improved system quality Better requirements generation Strengthened testing

Reduce risk Increased accuracy of estimates Reduce cost

Increased rigor Improved predictive ability Better analysis capability

Increased effectiveness Improved deliverable quality

Velocity/ Agility

Improved consistency Increased productivityHigher level support for integration

Increased capacity for reuse Increased transparency Increased uniformity

Increased efficiency Increased confidence Increased precision

Reduce rework Increased flexibility Early V&V

Reduce time Better requirements management Reduce ambiguity

Reduce waste Ease of design customization Easy to make changes

User Experience

Higher level support for automation Improved system understanding Reduce effort

Reduce burden of SE tasks Better data management/capture

Better manage complexity Better decision making

Knowledge Transfer

Better accessibility of info Improved architectureBetter communication/ info sharing

Better knowledge management/ capture

Multiple viewpoints of model Improved collaboration


Literature Review Quantitative Results (sorted by measured, 21 most cited)

Category Perceived Observed Measured

Quality

Reduce Errors/Defects (16) Reduce Errors/Defects (26) Reduce Errors/Defects (2)

Improve Traceability (61) Improve Traceability (9) Improve Traceability (1)

Improve System Quality (21) Improve System Quality (0) Improve System Quality (1)

Reduce Risk (22) Reduce Risk (2) Reduce Risk (1)

Increased Rigor (0) Increased Rigor (0) Increased Rigor (1)

Reduce Cost (33) Reduce Cost (4) Reduce Cost (0)

Velocity/ Agility

Consistency (44) Consistency (6) Consistency (1)

Reuse (37) Reuse (5) Reuse (1)

Efficiency (13) Efficiency (2) Efficiency (0)

Improve Standardization Improve Standardization Improve Standardization (0)

Collaboration/Info Sharing (68) Collaboration/Info Sharing (11) Collaboration/Info Sharing (0)

Integration/V&V (11) Integration/V&V (3) Integration/V&V (1)

Reduce Time (24) Reduce Time (8) Reduce Time (1)

User Experience

Automation (0) Automation (0) Automation (2)

Reduce SE Task Burden (0) Reduce SE Task Burden (0) Reduce SE Task Burden (1)

Manage Complexity (48) Manage Complexity (2) Manage Complexity (0)

Productivity (14) Productivity (0) Productivity (0)

System Understanding (24) System Understanding (2) System Understanding (0)

Knowledge Transfer

Information Access (27) Information Access (5) Information Access (2)

Knowledge Capture/Sharing (13) Knowledge Capture/Sharing (0) Knowledge Capture/Sharing (2)

Architecture/Sys Understanding (23) Architecture/Sys Understanding (2) Architecture/Sys Understanding (1)


mbsematuritysurvey.sercuarc.org

Final Survey data as of February 1, 2020.Quantitative metrics reflect “effective” participants. Information derived from text

responses reflects “all” participants


MBSE Survey OverviewTopics Summary of Survey Questions

7. Model Sharing and Reuse

19. Support model libraries for reuse20. Have interfaces around models for stakeholder use21. Shared models are used to consistently manage

programs across lifecycle22. Identify practices for data/model discovery, reuse?

8. Modeling Environments

23. Our modeling environment is secure24. Our modeling environment protects IP25. Have defined processes for tools, data

interoperability26. Identify benefits from collaborating on models

across disciplines

9. Organizational Implementation

27. Identify most challenging org obstacles for MBSE28. Identify best organizational enablers for MBSE29. Identify biggest changes our org needs for MBSE

10. Workforce 30. Have defined critical roles to support MBSE31. Identify top MBSE roles in your organization32. Have sufficient staffing to fill MBSE-related roles

11. MBSE Skills 33. Have defined critical skills for MBSE34. Identify the most critical skills for MBSE

12. Demographics

Organizational size, domain, MBSE experience, role

Topics Summary of Survey Questions

1. MBSE Usage

1. MBSE strategy is integrated with product strategy at the enterprise level

2. MBSE processes & tools are integrated with product-level processes and tools

3. Most important reasons for integrating MBSE

2. Model Manage-ment

4. Have a taxonomy for modeling across organization5. Have defined processes/tools for model management6. Have standard guidance for model management/tools7. Identify business value from consistent model management

3. Technical Manage-ment

8. Use modeling as the basis for technical processes9. MBSE process supports our technical review process10. Identify benefits or challenges of MBSE in technical

reviews

4. Metrics 11. Modeling provides measurable improvement across projects

12. Have consistent metrics across programs/enterprise13. Identify any metrics that have proven useful

5. Model Quality

14. Have defined processes/tools for V&V of models15. Have defined processes/tools for data/model quality

assurance

6. Data Manage-ment

16. Have effective approaches for managing the data interface between tools17. Data is managed independent of tools for portability18. Identify new data management roles/processes

Survey content is derived from the draft INCOSE Digital Engineering Capabilities Definition

*Questions in italics elicit free text responses


Demographics - Organization type:

Aerospace and defense, 56

Automotive, 7

Consultancy and Services, 6

Tools & MBSE services, 4

Agricultural, 4

Manufacturing , 2

Meteorology, 1

Ship building, 1 Rail, 1

R&D, 1

Medical device, 1 Health IT, 1Genomics, 1

Food & beverage packaging, 1

Electronics Manufacturing, 1

Construction and infrastructure, 1

Industry-Reported Market Areas


Demographic – type, size, experience

0

5

10

15

20

25

<50

0

50

1-2

00

0

20

01

-10

000

<50

0

50

1-2

00

0

20

01

-10

000

>10

000

<50

0

50

1-2

00

0

20

01

-10

000

>10

000

Academia Government Industry

Less Than 1 Year

1-3 Years

4-6 Years

More than 6 years


-29

-63

-86

-98

-92

-118

-77

30

-153

-61

-73

-91

-111

18

-85

-107 243

195

-10

-70

-165

-53

-24

-250 -150 -50 50 150 250

1. Mature use strategy

2. Mature process/tool strategies

4. Consistent lexicon & taxonomy across enterprise

5. Mature model management processes

6. Standard program & business guidance for models

8. Models are the basis for technical processes

9. MBSE is the basis for Technical Reviews

11. Modeling provides measurable improvements

12. Have consistent metrics across enterprise

14. Consistent data/model V&V processes

15. Consistent data/model quality assurance processes

16. Processes to manage data interface between tools

17. Data is portable across organizations & tools

19. Support model libraries for model reuse

20. Libraries support discoverable knowledge

21. Consistent use of shared models

23. Trust that environment is secure

24. Trust that environment protects IP

25. Have processes and tool selection & interoperability

30. Have clearly defined roles supporting MBSE

32. Have sufficient staffing for all roles

33. Have defined critical skills supporting MBSE

35. Training is linked to critical skills

Overall Survey Scores

MBSE Usage

Model Management

TechnicalManagement

Metrics

Model Quality

Data Management

Model Sharing& Reuse

ModelingEnvironment

Workforce

MBSE Skills

AGREEDISAGREE


Demographics – Participant roles

Systems Engineer, 92

Project, Product or other

Management, 29

Executive Management, 17

Modeling and Simulation

Professional, 13

Other, 12

Acquisition Professional, 4

Other Engineering or

Software Disciplines, 3


Overall Survey Scores by Role

MBSE Usage

Model Management

TechnicalManagement

Metrics

Model Quality

Data Management

Model Sharing& Reuse

ModelingEnvironment

Workforce

MBSE Skills

-40

-30

-20

-10

0

10

20

30

40

1. M

atu

re u

se s

trat

egy

2. M

atu

re p

roce

ss/t

oo

l sta

tegi

es

4. C

on

sist

en

t le

xico

n &

tax

on

om

y ac

ross

en

terp

rise

5. M

atu

re M

od

el M

anag

emen

t P

roce

sses

6. S

tan

dar

d P

rogr

am &

Bu

sin

ess

Gu

idan

ce f

or

Mo

del

s

8. M

od

els

are

the

bas

is f

or

Tech

nic

al P

roce

sses

9. M

BSE

is t

he

bas

is f

or

Tech

nic

al R

evie

ws

11. M

od

elin

g p

rovi

des

mea

sura

ble

imp

rove

men

ts

12

. Hav

e co

nsi

sten

t m

etri

cs a

cro

ss e

nte

rpri

se

14

. Co

nsi

sten

t d

ata/

mo

del

V&

V p

roce

sses

15

. Co

nsi

sten

t d

ata/

mo

del

QA

pro

cess

es

16. P

roce

sses

to

man

age

dat

a in

terf

ace

bet

we

en t

oo

ls

17. D

ata

is p

ort

able

acr

oss

org

aniz

atio

ns

& t

oo

ls

19. S

up

po

rt m

od

el li

bra

ries

fo

r m

od

el r

euse

20. L

ibra

ries

su

pp

ort

dis

cove

rab

le k

no

wle

dge

21

. Co

nsi

sten

t u

se o

f sh

ared

mo

del

s

23. T

rust

th

at e

nvi

ron

men

t is

sec

ure

24. T

rust

th

at e

nvi

ron

men

t p

rote

cts

IP

25

. Hav

e p

roce

sses

fo

r to

ol s

elec

tio

n &

inte

rop

erab

ility

30

. Hav

e cl

earl

y d

efin

ed r

ole

s su

pp

ort

ing

MB

SE

32

. Hav

e su

ffic

ien

t st

affi

ng

for

all r

ole

s

33. H

ave

def

ined

cri

tica

l ski

lls s

up

po

rtin

g M

BSE

35. T

rain

ing

is li

nke

d t

o c

riti

cal s

kills

Executive Management

Program/Project/Other Management

System Engineers and Other roles

AG

REE

DIS

AG

REE


MBSE Usage and Model Management Survey Results

Our MBSE processes and tools are integrated with our overall product-level processes and tools.

Our organization has standard business and program guidance that defines our model management processes and tools.


Technical Management Survey Results

Our organization uses modeling as the basis for our technical processes consistently across the enterprise. Our MBSE process fully supports our technical review process.

Free-text question: Please identify any benefits or challenges your organization has found in the use of MBSE (or 'digital engineering') in the technical review process.


0

2

4

6

8

10

12

14

16

MB

SE m

eth

od

s/p

roce

sses

Org

aniz

atio

nal

cu

ltu

re

Co

mm

un

icat

ing

succ

ess

sto

ries

/pra

ctic

es

Cu

sto

mer

/sta

keh

old

er b

uy-

in/e

nga

gem

ent

MB

SE t

erm

ino

logy

/on

tolo

gy/l

ibra

ries

Lega

cy/c

urr

ent

pro

cess

es

Peo

ple

will

ing

to u

se M

BSE

to

ols

Trai

nin

g

MB

SE t

oo

ls

Co

st t

o u

se M

BSE

to

ols

Dem

on

stra

tin

g b

enef

its/

resu

lts

mo

re s

take

ho

lder

invo

lvem

ent

Lead

ersh

ip s

up

po

rt/c

om

mit

men

t

Incr

ease

d t

race

abili

ty

mu

ltip

le v

iew

po

ints

of

mo

del

Co

mm

un

icat

ion

/in

fo s

har

ing

Incr

ease

d c

apac

ity

for

reu

se

imp

rove

d s

yste

m u

nd

erst

and

ing

Red

uce

ris

k

Red

uce

tim

e

Red

uce

err

ors

Red

uce

d S

E ta

sk b

urd

en

Incr

ease

d c

on

sist

ency

Dem

on

stra

tin

g b

enef

its/

resu

lts

Obstacles vs. Enablers

Comparing Obstacles vs. Enablers in Tech Review Processes

Benefits of DE/MBSEAdoption of DE/MBSE

Participants were asked to respond to the prompt, “Please identify any benefits or challenges your organization has found in the use of MBSE (or 'digital engineering') in the technical review process.”


DE/MBSE Metrics Survey Result

Modeling activities in our organization provide measurable improvements within and across projects.

We have consistent metrics across our program(s)/enterprise that include our modeling activities.

Free-text question: Please identify any metrics that have proven to be useful for measuring the performance of your MBSE activities.


Model Quality Survey Results

Our organization has defined processes and tools for V&V of models at appropriate levels and program phases.

Our organization has defined processes and tools for data and model quality assurance.


Data Management Survey Results

Our organization has effective approaches for managing the data interface between tools.

Data is managed independent of tools and allows for portability across different organizational structures and related disciplines.


Model Sharing & Reuse Survey Result

Our organization supports model libraries for the purpose of model reuse.

Shared models are being used to consistently manage systems across the lifecycle.

Free-text question: Please identify any practices your organization has implemented to improve data and model discovery and reuse, either within or

between teams. Include examples of appropriate model reuse if possible.


Data/Model Discovery & Reuse

Participants were asked to respond to the prompt, “Please identify any practices your organization has implemented to improve data and model discovery and reuse, either within or between teams. Include examples of appropriate model reuse if possible.”

0

5

10

15

20

25

30

Lib

rari

es

Too

l Sp

ecif

ic

Ch

alle

nge

s

Un

iqu

e ve

rsu

s In

tegr

ated

Solu

tio

ns

EA F

ram

ewo

rk

Mo

del

Cu

rati

on

Fra

me

wo

rk

Mo

del

or

Too

l In

tegr

atio

n

Cre

ated

MB

SE R

ole

s

Dat

a M

anag

emen

t P

roce

sses

Org

aniz

atio

nal

Tax

on

om

y an

dO

nto

logy

Oth

er

Total Responses

None or NATBD or Too Immature

Analyzable responses

Q10 Data and Model Discovery and Reuse

97 13 26 58

Foundational

Advanced


Modeling Environment Survey Results

Our organization takes steps to make sure that our modeling environment protects our intellectual property.

Our organization has defined processes and work instructions that cover tool selection, use, and related data interoperability concerns.


Workforce and Skills Survey Results

Our organization has clearly defined the critical roles to support MBSE. Our organization has clearly defined critical skills for MBSE.


Survey Results of Critical Skills

0

5

10

15

20

25

System

sArchitecture

System

sThinking

Requ

irem

entsEnginering

Dom

ainKn

owledge

SEProcesses

Abstractio

n

System

sAnalysis

V&V

Mod

elArchitecture

AbilitytoM

akeDecisions

Allocatio

n

AnalyticalThinking

DesignThinking

Efficiency

Implem

entatio

n

Prod

uctD

evelop

men

t

Prod

uctLife

cycle

Managem

ent

RiskBalance

System

ofSystems

System

sScience

Traceability

MostCriticalSystemsEngineeringSkills

Question34ThemostcriticalskillsforMBSEare:

020406080

100120

Syst

ems

Engi

nee

rin

g…

Too

l Exp

erti

se

Dig

ital

Engi

nee

rin

g

No

n-T

ech

nic

alSk

ills

Att

rib

ute

s

Ap

plie

dEx

per

ien

ce

Soft

war

eD

evel

op

men

t

Ch

alle

nge

s

Mu

lti-

Dis

cip

linar

y

Org

aniz

atio

nal

Co

nte

xt

Pro

ject

Man

agem

ent

Question 34The most critical skills for MBSE are:


Survey Results of Critical Skills

05

1015202530

Mo

del

ing

Dat

a Sc

ien

ce

Sim

ula

tio

n

MB

SEEn

viro

nm

ent

Mo

del

Go

vern

ance

Most Critical Digital Skills


05

1015202530

Gen

eral

To

ol

Exp

erti

se

Mo

del

ing

Lan

guag

es

Too

lIn

tegr

atio

n

Ab

ility

to

Lear

n N

ewTo

ols

Most Critical Tool Expertise


020406080

100120

Syst

ems

Engi

nee

rin

g…

Too

l Exp

erti

se

Dig

ital

Engi

nee

rin

g

No

n-T

ech

nic

alSk

ills

Att

rib

ute

s

Ap

plie

dEx

per

ien

ce

Soft

war

eD

evel

op

men

t

Ch

alle

nge

s

Mu

lti-

Dis

cip

linar

y

Org

aniz

atio

nal

Co

nte

xt

Pro

ject

Man

agem

ent



• Survey questions related to benefits are: ―Q3. What do you see as the most important reasons for integrating MBSE processes

with program and business management processes?

―Q7. Please provide one or more descriptions of the business value you are realizing from consistent model management processes and tools.

―Q26. Please identify any additional benefits you find from collaborating on models across disciplines.

Survey Free-Response Analysis - Benefits

Non-

response

No value

achieved

Response

containing

benefitsQ3 Reason for integrating

MBSE41 N/A 104

Q7 Value from consistent

model mgt.23 18 70

Q26 Benefit from

collaboration24 6 52

Totals 88 24 226


List of DE/MBSE Benefit Categories (Literature Review)

Category

List of Benefits

Quality

Reduce errors/ defects Improved risk analysis Improved capability

Improved traceability Improved system design More stakeholder involvement

Improved system quality Better requirements generation Strengthened testing

Reduce risk Increased accuracy of estimates Reduce cost

Increased rigor Improved predictive ability Better analysis capability

Increased effectiveness Improved deliverable quality

Velocity/ Agility

Improved consistency Increased productivity Higher level support for integration

Increased capacity for reuse Increased transparency Increased uniformity

Increased efficiency Increased confidence Increased precision

Reduce rework Increased flexibility Early V&V

Reduce time Better requirements management Reduce ambiguity

Reduce waste Ease of design customization Easy to make changes

User Experience

Higher level support for automation

Improved system understanding Reduce effort

Reduce burden of SE tasks Better data management/capture

Better manage complexity Better decision making

Knowledge Transfer

Better accessibility of info Improved architecture Improved collaboration

Better knowledge management/ capture

Better communication/ info sharing

Multiple viewpoints of model


Most Frequently-Reported Benefits from Survey Analysis by Question

Q3 Reason for integrating MBSE Q7 Value from consistent model mgt. Q26 Benefit from collaboration

Reduce cost (17) Increased capacity for reuse (18)Better communication/

information sharing (13)

Reduce time (17) Improved consistency (16) Improved system understanding (10)

Better accessibility of info (16) Improved system understanding (9) Better accessibility of info (6)

Increased efficiency (14) Reduce time (9) Improved consistency (5)

Improved consistency (13)Better communication/

information sharing (7)Reduce errors (5)

Increased traceability (11) Better accessibility of info (7) Reduce time (5)

Improved system understanding (10) Reduce cost (7) Increased capacity for reuse (5)

• Integration Benefits • Model Mgmt. Benefits • Collaboration Benefits


Top Benefits Reported from Survey Questions

0

2

4

6

8

10

12

14

16

18

20

Survey Questions on Benefits- Top Responses

Q3 Reason for integrating MBSE

Q7 Value from consistent model mgt.

Q26 Benefit from collaboration


All Benefit Categories Reported from Survey Questions

0

2

4

6

8

10

12

14

16

18

Red

uce

co

st

Red

uce

tim

e

Bet

ter

acce

ssib

ility

of

info

Incr

ease

d e

ffic

ien

cy

Imp

rove

d c

on

sist

ency

Incr

ease

d t

race

abili

ty

Imp

rove

d s

yste

m u

nd

erst

and

ing

Bet

ter

com

mu

nic

atio

n/

info

sh

arin

g

Hig

her

leve

l su

pp

ort

fo

r in

tegr

atio

n

Red

uce

err

ors

Bet

ter

man

age

com

ple

xity

Imp

rove

d s

yste

m q

ual

ity

Incr

ease

d c

apac

ity

for

reu

se

Imp

rove

d s

yste

m d

esig

n

Bet

ter

dec

isio

n m

akin

g

Imp

rove

d c

olla

bo

rati

on

Incr

ease

d e

ffec

tive

nes

s

Bet

ter

anal

ysis

cap

abili

ty

Bet

ter

req

uir

emen

ts g

ener

atio

n

Red

uce

ris

k

Imp

rove

d d

eliv

erab

le q

ual

ity

Bet

ter

dat

a m

gt./

cap

ture

Bet

ter

kno

wle

dge

mgt

./ c

aptu

re

Red

uce

rew

ork

Mu

ltip

le v

iew

po

ints

of

mo

del

Red

uce

am

big

uit

y

Hig

her

leve

l su

pp

ort

fo

r au

tom

atio

n

Earl

y V

&V

Red

uce

eff

ort

Incr

ease

d p

rod

uct

ivit

y

Imp

rove

d a

rch

itec

ture

Incr

ease

d r

igo

r

Stre

ngt

hen

ed t

esti

ng

Bet

er r

equ

irem

ents

mgt

.

Imp

rove

d r

isk

anal

ysis

Incr

ease

d p

reci

sio

n

Imp

rove

d c

apab

ility

Incr

ease

d f

lexi

bili

ty

Red

uce

was

te

Incr

ease

d u

nif

orm

ity

Mo

re s

take

ho

lder

invo

lvem

ent

Easy

to

mak

e ch

ange

s

Red

uce

bu

rden

of

SE t

asks

Imp

rove

d p

red

icti

ve a

bili

ty

Incr

ease

d t

ran

spar

ency

Q3 Reason for integrating MBSE

Q7 Value from consistent model mgt.

Q26 Benefit from collaboration


Analysis of Survey Question on Metrics

• Question 13 Please identify any metrics that have proven to be useful for measuring the performance of your MBSE activities was also analyzed against the literature review benefit categories

Adoption metrics: metrics related to the successful adoption and implementation of MBSE instead of measuring the value of MBSE itself

Non-

response

No

metrics

Response

containing

metrics

Q13 Metrics

for MBSE19 33 44

Adoption

metrics

Lit review

benefit

metrics

Unable to

categorize

Q13 Metrics for

MBSE17 72 11

Number of complete responses

Number of unique

comments


Most cited benefits and metrics categories from survey data

Top survey response metrics (Q13 only) Survey response benefits (Q3, Q7, and Q26)

Better requirements generation 7 Better requirements generation 7

Reduce errors 7 Reduce errors 19

Increased traceability 6 Increased traceability 17

Better requirements mgt. 6 Better requirements mgt. 3

Improved system design 5 Improved system design 9

Reduce cost 5 Reduce cost 25

Reduce time 5 Reduce time 31

Increased capacity for reuse 5 Increased capacity for reuse 30

Better analysis capability 4 Better analysis capability 6

Improved system quality 2 Improved system quality 14

Increased effectiveness 2 Increased effectiveness 6

Higher level support for automation 2 Higher level support for automation 3

Higher level support for integration 2 Higher level support for integration 14


Most cited benefits and metrics categories from survey data

Remaining survey response metrics (Q13 only) Top Survey response benefits (Q3, Q7, and Q26)

(not cited) 0 Improved consistency 34

(not cited) 0 Increased capacity for reuse 30

Better accessibility of info 1 Better accessibility of info 29

Improved system understanding 1 Improved system understanding 29

(not cited) 0 Better communication/ info sharing 29

(not cited) 0 Increased efficiency 20

(not cited) 0 Improved collaboration 13

Better knowledge mgt./ capture 1 Better knowledge mgt./ capture 8

Reduce ambiguity 1 Reduce ambiguity 8

Better manage complexity 1 Better manage complexity 7

Better decision making 1 Better decision making 7

(not cited) 0 Reduce rework 7

Improved deliverable quality 1 Improved deliverable quality 6

Better data mgt./ capture 1 Better data mgt./ capture 6

Reduce risk 1 Reduce risk 5

Increased uniformity 1 Increased uniformity 5

Multiple viewpoints of model 1 Multiple viewpoints of model 4

Strengthened testing 1 Strengthened testing 4

Reduce effort 1 Reduce effort 3

Improved capability 1 Improved capability 1


Challenges in Enterprise Adoption of MBSE

• Successful adoption of Model-based Systems Engineering (MBSE), like many other large-scale enterprise change initiatives, can present significant challenges - requires intentional focus on many aspects within an organization -not just the technical details of processes and tools.

―The Digital Engineering Work Group pain points relate to technical aspects such as tools, reference models, standards, and data, as well as other organization-level challenges such as implementation/deployment approach, IT infrastructure, and training/skills of the workforce.

―Cloutier (2019) reported the top five inhibitors to successful adoption of MBSE were: cultural and general resistance to change, availability of skills, the MBSE learning curve, lack of perceived value of MBSE, and lack of management support.

• This breadth of factors demonstrates the importance of a holistic, enterprise-wide perspective in designing and implementing the approach to adopt MBSE.

• Examining MBSE adoption from the lens of the Baldrige Criteria for Performance Excellence (CPE) can generate insight and more complete understanding of MBSE adoption.


List of enterprise success factors from the survey analysis

Category

List of Success Factors

Leadership Leadership support/commitment

Leadership understanding of MBSE

Communication Awareness of MBSE benefits/value

Communicating success stories/practices

Need for change

Resources Cost to use MBSE tools General resources for MBSE implementation

Workforce

General MBSE awareness and knowledge

People willing to use MBSE tools Teamwork

MBSE learning curve People in SE roles Training

Workforce knowledge/skills

Change Processes

Champions Competing priorities Legacy/current processes

Change management process design

Integration to support MBSE implementation

Vision and strategy for MBSE

Community of practice Demonstrating benefits/results

MBSE Processes

MBSE methods/processes MBSE tools Security of data and IP

MBSE terminology/ontology/libraries

Projects/programs to apply MBSE

Organizational Environment

Alignment with business strategy

Organizational culture Success metrics

Organizational characteristics Rewards/recognition Supportive infrastructure

External Environment

Alignment with customer requirements

Customer/stakeholder buy-in/engagement

External regulations Use in SE community


Analysis of Survey Responses: Obstacles

• There were 166 respondents providing comments to the question on obstacles, parsed into 303 unique response comments.

05

101520253035404550

Org

aniz

atio

nal

cu

ltu

re

Wo

rkfo

rce

kno

wle

dge

/ski

lls

Lead

ers

hip

su

pp

ort

/co

mm

itm

ent

Aw

aren

ess

of

MB

SE b

ene

fits

/val

ue

Ch

ange

man

agem

ent

pro

cess

de

sign

Inte

grat

ion

to

su

pp

ort

MB

SE im

ple

me

nta

tio

n

MB

SE m

eth

od

s/p

roce

sse

s

MB

SE t

oo

ls

Co

mp

etin

g p

rio

riti

es

De

mo

nst

rati

ng

be

ne

fits

/re

sult

s

Gen

eral

re

sou

rce

s fo

r M

BSE

imp

lem

enta

tio

n

Trai

nin

g

Pro

ject

s/p

rogr

ams

to a

pp

ly M

BSE

Lega

cy/c

urr

ent

pro

cess

es

Gen

eral

MB

SE a

war

en

ess

an

d k

no

wle

dge

Lead

ers

hip

un

de

rsta

nd

ing

of

MB

SE

Co

st t

o u

se M

BSE

to

ols

MB

SE le

arn

ing

curv

e

Sup

po

rtiv

e in

fras

tru

ctu

re

Alig

nm

en

t w

ith

cu

sto

me

r re

qu

irem

ents

Cu

sto

me

r/st

ake

ho

lde

r b

uy-

in/e

nga

gem

en

t

MB

SE t

erm

ino

logy

/on

tolo

gy/l

ibra

rie

s

Nee

d f

or

chan

ge

Alig

nm

en

t w

ith

bu

sin

ess

str

ate

gy

Exte

rnal

reg

ula

tio

ns

Org

aniz

atio

nal

ch

arac

teri

stic

s

Re

war

ds/

reco

gnit

ion

Secu

rity

of

dat

a an

d IP

Vis

ion

an

d s

trat

egy

fo

r M

BSE

Q27: Obstacles to Implementing MBSE Code

#

Comments

Obstacles

Organizational culture 44

Workforce knowledge/skills 30

Leadership support/commitment 25

Awareness of MBSE benefits/value 18

Change management process design 13

Integration to support MBSE

implementation13

MBSE methods/processes 13

MBSE tools 13

Competing priorities 11

Demonstrating benefits/results 11

General resources for MBSE

implementation11

Training 11

Projects/programs to apply MBSE 10

Legacy/current processes 9

General MBSE awareness and

knowledge8

Leadership understanding of MBSE 8


Analysis of Survey Responses: Enablers

• There were 156 respondents providing comments to the question on enablers, parsed into 223 unique response comments.

Code

#

Comments

Enablers


People willing to use MBSE tools 21


Champions 15


Training 13

People in SE roles 12

MBSE tools 11

Alignment with customer

requirements10

Change management process

design7

Communicating success

stories/practices6

Community of practice 6


implementation6

Use in SE community 6

0

5

10

15

20

25

30

35

40

45

50

Lead

ers

hip

su

pp

ort

/co

mm

itm

ent

Peo

ple

will

ing

to u

se M

BSE

to

ols

Wo

rkfo

rce

kno

wle

dge

/ski

lls

Ch

amp

ion

s

Dem

on

stra

tin

g b

en

efit

s/re

sult

s

Trai

nin

g

Peo

ple

in S

E ro

les

MB

SE t

oo

ls

Alig

nm

en

t w

ith

cu

sto

mer

re

qu

irem

ents

Ch

ange

man

agem

ent

pro

cess

des

ign

Co

mm

un

icat

ing

succ

ess

sto

ries

/pra

ctic

es

Co

mm

un

ity

of

pra

ctic

e

Inte

grat

ion

to

su

pp

ort

MB

SE…

Use

in S

E co

mm

un

ity

Ge

ner

al r

eso

urc

es

for

MB

SE…

MB

SE t

erm

ino

logy

/on

tolo

gy/l

ibra

ries

Ne

ed f

or

chan

ge

Pro

ject

s/p

rogr

ams

to a

pp

ly M

BSE

Aw

aren

ess

of

MB

SE b

ene

fits

/val

ue

MB

SE m

eth

od

s/p

roce

sses

Org

aniz

atio

nal

cu

ltu

re

Cu

sto

me

r/st

ake

ho

lde

r b

uy-

…

Team

wo

rk

Ge

ner

al M

BSE

aw

are

nes

s an

d k

no

wle

dge

Lead

ers

hip

un

de

rsta

nd

ing

of

MB

SE

Org

aniz

atio

nal

ch

arac

teri

stic

s

Alig

nm

en

t w

ith

bu

sin

ess

str

ateg

y

Co

st t

o u

se M

BSE

to

ols

Rew

ard

s/re

cogn

itio

n

Succ

ess

me

tric

s

Vis

ion

an

d s

trat

egy

for

MB

SE

Q28: Enablers to Implementing MBSE


Analysis of Survey Responses: Changes

• There were 153 respondents providing comments to the question on changes, parsed into 273 unique response comments.

Code

#

Comments

Changes

MBSE methods/processes 45

Training 31

MBSE tools 20

General resources for MBSE

implementation16


Change management process

design14


Organizational culture 12


implementation11


MBSE terminology/ontology/libraries 9

Supportive infrastructure 9

Awareness of MBSE benefits/value 7

Leadership understanding of MBSE 6

Vision and strategy for MBSE 6

0

5

10

15

20

25

30

35

40

45

50

MB

SE m

eth

od

s/p

roce

sses

Trai

nin

g

MB

SE t

oo

ls

Ge

ner

al r

eso

urc

es

for

MB

SE…

Lead

ers

hip

su

pp

ort

/co

mm

itm

ent

Ch

ange

man

agem

ent

pro

cess

des

ign

Wo

rkfo

rce

kno

wle

dge

/ski

lls

Org

aniz

atio

nal

cu

ltu

re

Inte

grat

ion

to

su

pp

ort

MB

SE…

Dem

on

stra

tin

g b

en

efit

s/re

sult

s

MB

SE t

erm

ino

logy

/on

tolo

gy/l

ibra

ries

Sup

po

rtiv

e in

fras

tru

ctu

re

Aw

aren

ess

of

MB

SE b

ene

fits

/val

ue

Lead

ers

hip

un

de

rsta

nd

ing

of

MB

SE

Vis

ion

an

d s

trat

egy

for

MB

SE

Alig

nm

en

t w

ith

cu

sto

mer

re

qu

irem

ents

Ne

ed f

or

chan

ge

Co

mm

un

icat

ing

succ

ess

sto

ries

/pra

ctic

es

Ge

ner

al M

BSE

aw

are

nes

s an

d k

no

wle

dge

Peo

ple

will

ing

to u

se M

BSE

to

ols

Cu

sto

me

r/st

ake

ho

lde

r b

uy-

in/e

nga

gem

en

t

Succ

ess

me

tric

s

Alig

nm

en

t w

ith

bu

sin

ess

str

ateg

y

Ch

amp

ion

s

Co

st t

o u

se M

BSE

to

ols

Pro

ject

s/p

rogr

ams

to a

pp

ly M

BSE

Rew

ard

s/re

cogn

itio

n

Team

wo

rk

Use

in S

E co

mm

un

ity

Co

mm

un

ity

of

pra

ctic

e

Lega

cy/c

urr

en

t p

roce

sses

MB

SE le

arn

ing

curv

e

Org

aniz

atio

nal

ch

arac

teri

stic

s

Peo

ple

in S

E ro

les

Secu

rity

of

dat

a an

d IP

Q29: Changes to Improve MBSE Implementation


Comparing Obstacles vs. Enablers vs. Changes

05

101520253035404550

Org

aniz

atio

nal

cu

ltu

re

Wo

rkfo

rce

kno

wle

dge

/ski

lls

Lead

ersh

ip s

up

po

rt/c

om

mit

men

t

Aw

aren

ess

of

MB

SE b

enef

its/

valu

e

MB

SE t

oo

ls

Ch

ange

man

agem

ent

pro

cess

des

ign

Inte

grat

ion

to

su

pp

ort

MB

SE im

ple

men

tati

on

MB

SE m

eth

od

s/p

roce

sses

Dem

on

stra

tin

g b

enef

its/

resu

lts

Trai

nin

g

Gen

eral

res

ou

rces

fo

r M

BSE

imp

lem

enta

tio

n

Co

mp

etin

g p

rio

riti

es

Pro

ject

s/p

rogr

ams

to a

pp

ly M

BSE

Lega

cy/c

urr

ent

pro

cess

es

Lead

ersh

ip u

nd

erst

and

ing

of

MB

SE

Gen

eral

MB

SE a

war

en

ess

and

kn

ow

led

ge

Co

st t

o u

se M

BSE

to

ols

Sup

po

rtiv

e in

fras

tru

ctu

re

MB

SE le

arn

ing

curv

e

Alig

nm

ent

wit

h c

ust

om

er r

equ

irem

ents

Cu

sto

mer

/sta

keh

old

er b

uy-

in/e

nga

gem

ent

MB

SE t

erm

ino

logy

/on

tolo

gy/l

ibra

ries

Nee

d f

or

chan

ge

Org

aniz

atio

nal

ch

arac

teri

stic

s

Alig

nm

ent

wit

h b

usi

nes

s st

rate

gy

Exte

rnal

reg

ula

tio

ns

Vis

ion

an

d s

trat

egy

for

MB

SE

Rew

ard

s/re

cogn

itio

n

Secu

rity

of

dat

a an

d IP

Peo

ple

will

ing

to u

se M

BSE

to

ols

Ch

amp

ion

s

Peo

ple

in S

E ro

les

Co

mm

un

icat

ing

succ

ess

sto

ries

/pra

ctic

es

Use

in S

E co

mm

un

ity

Co

mm

un

ity

of

pra

ctic

e

Team

wo

rk

Succ

ess

met

rics

Obstacles vs. Enablers vs. Changes


Framework for MBSE Adoption

• Insight from analysis of both obstacles and enablers, mapped to the Baldrige CPE, can be used to define a more comprehensive set of adoption practices for maturity in MBSE – e.g.:

― Leaders communicate a clear reason and need for MBSE adoption

― Leaders understand MBSE

― Leaders support and are committed to MBSE

― People understand the benefits of MBSE

― MBSE is aligned with the overall business strategy

― MBSE is used for the right projects/programs

― MBSE adoption is aligned with what customers need

― Customers and stakeholders buy-in to MBSE

― Data management processes support MBSE

― The IT infrastructure supports MBSE use

― Clear metrics are defined to track results & progress of MBSE

― Systems engineers have skills needed to support MBSE use

― Training is provided to develop needed skills

― People are rewarded/recognized for using MBSE

― The organizational culture is aligned with MBSE use

Baldrige CPE Framework

http://www.nist.gov/baldrige


Summary: Top DE Metrics Areas(>10 citations)

Category Most Cited Benefits Survey Lit Review

Quality

Reduce Cost 5.5% 5.5%

Reduce Defects/Errors/Rework 4.2% 1.3%

Increased Traceability 3.7% 8.4%

Higher Level of Support for Integration 3.1% 1.2%

Improved System Quality 3.1% 4.6%

Velocity/Agility

Improved Consistency 7.5% 6.0%

Reduce Time 6.8% 4.8%

Improved Capacity for Reuse 6.6% 5.5%

Increased Efficiency 4.4% 1.8%

Improved Collaboration 2.9% 0.8%

User ExperienceImproved System Understanding 6.4% 3.7%

Better Manage Complexity 2.0% 5.6%

Knowledge TransferBetter Accessibility of Information 6.4% 3.6%

Better Communication/ Information Sharing 6.4% 10.9%

Adoption

Methods/Processes 8.0% *

Roles/Skills, People Willing to Use 6.8% *

Leadership support/Commitment 5.5% *

Training/Tools, People Willing to Use 4.4% *

Change Management Process Design 3.1% *

*not assessed in lit review

44

Questions?

Thank you!


Summary: Top DE Metrics Areas(percent citation, >10 citations)

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

12.00%

Imp

rove

d c

on

sist

ency

Red

uce

tim

e

Incr

ease

d c

apac

ity

for

reu

se

Bet

ter

com

mu

nic

atio

n/

info

…

Imp

rove

d s

yste

m u

nd

erst

and

ing

Bet

ter

acce

ssib

ility

of

info

Red

uce

co

st

Incr

ease

d e

ffic

ien

cy

Red

uce

err

ors

Incr

ease

d t

race

abili

ty

Imp

rove

d s

yste

m q

ual

ity

Hig

her

leve

l su

pp

ort

fo

r…

Imp

rove

d c

olla

bo

rati

on

Bet

ter

man

age

com

ple

xity

Imp

rove

d s

yste

m d

esig

n

Bet

ter

kno

wle

dge

mgt

./ c

aptu

re

Red

uce

am

big

uit

y

Bet

ter

dat

a m

gt./

cap

ture

Bet

ter

req

uir

emen

ts g

ener

atio

n

Red

uce

rew

ork

Bet

ter

dec

isio

n m

akin

g

Imp

rove

d d

eliv

erab

le q

ual

ity

Incr

ease

d e

ffec

tive

nes

s

Bet

ter

anal

ysis

cap

abili

ty

Red

uce

ris

k

Incr

ease

d u

nif

orm

ity

Easy

to

mak

e ch

ange

s

Earl

y V

&V

Mu

ltip

le v

iew

po

ints

of

mo

del

Red

uce

was

te

Stre

ngt

hen

ed t

esti

ng

Bet

er r

equ

irem

en

ts m

gt.

Imp

rove

ris

k an

alys

is

Red

uce

eff

ort

Incr

ease

d p

rod

uct

ivit

y

Imp

rove

d a

rch

itec

ture

Hig

her

leve

l su

pp

ort

fo

r…

Incr

ease

d r

igo

r

Incr

ease

d p

reci

sio

n

Mo

re s

take

ho

lder

invo

lvem

ent

Incr

ease

d t

ran

spar

ency

Imp

rove

d c

apab

ility

Incr

ease

d f

lexi

bili

ty

Red

uce

bu

rden

of

SE t

asks

Imp

rove

d p

red

icti

ve a

bili

ty

Survey v. Lit review benefit occurrence in percentage

Survey

Lit review

benchmarking the benefits and current maturity of digital ... · systems engineering research...

Documents