ARL-HREDFAMU-FSU Simulation Group

Output Analysis Overview7 Dec 2005

Who we are

Dr. James Simpson, Principal Investigator

Associate Professor of Industrial Engineering

Florida State University

Graduate Assistants:– Lisa Hughes– Nicholas Done– Wayne Wesley– Michelle Zeisset

What we do

Bring operations research expertise

Not directly involved in product development so come from perspective of a potential user

Military background adds realism

Improve IMPRINT as a decision-making tool

Our history with IMPRINT

Phase Dates Project Description

I 7/02 – 3/03Output analysis approach for

VACP mission

II 5/03 – 3/04Suggested analysis for goal

oriented and advanced missions

III 10/04 – 9/05Assessment of maintenance

model

IV 9/05 – 4/06Analyses and reports for

maintenance model

• Efficient data reporting• Graphical and tabular reports

DataCompilation

Areas of concentration

• Multiple runs • Useful metrics

OutputVariable

Assessment

UserSupport

• GUI • Analysis tools

Study approach

Use small simple models

Compare observed results to expected results

Interpret results in context of a question a potential user may want to answer

Validate and demonstrate suggested improvements using realistic models

• Efficient data reporting• Graphical and tabular reports

DataCompilation

Areas of concentration

UserSupport

• Multiple runs Output

VariableAssessment

• GUI • Analysis tools

• Useful metrics

Need for multiple runs

Mean: 27.9

Std dev: 3.7

Min: 10.8

Max: 33.8

Combat model (Advanced operations module)

N = 1

fre

qu

en

cy

0

5

10

15

20

25

16 18 20 22 24 26 28 30 32 34

16 18 20 22 24 26 28 30 32 340

5

10

15

20

25

fre

qu

en

cy

N = 100

% of Time Overloaded (C/G)

How many runs?

Stryker model (Maintenance model)

Multiple run output:Frequency histogram

BFV (VACP operations module)

36.852.6

68.484.2

100115.8

131.6147.4

163.2179

194.8210.6

226.4242.2

258

Number of Times Overloaded

0

5

10

15

20

25

30

Fre

qu

ency

Which events led to results?

Function Network of BFV

M

R

P

P

P

0 : Start

1

7

2 3

4

10

11

14

15

16

12

8

5

6

20: END

M

R

P

P

P

0 : Start

1 2 3

4

8

20: END • 3 Function-level probability nodes act as switches

1

2

3

45

7

6

8

0

5

10

15

20

25

30

36.852.6

68.484.2

100115.8

131.6147.4

163.2179

194.8210.6

226.4242.2

258

Fre

qu

ency

F11

F10

F8

F4

Histogram of Number of Times Overloaded, Color-Coded by Trace Histogram of Number of Times Overloaded

Histogram broken down by function

BFV (VACP operations module)

Function Mission

Timemean / std dev

% of Time Overloaded mean / std dev

# of Times Overloaded mean / std dev

Overall 165 69 44 27 121 76

10 242 51 76 5 232 59

11 154 11 25 2 104 10

14 71 5 45 6 54 7

Tabular report by function

BFV (VACP operations module)

• Efficient data reporting• Graphical and tabular reports

DataCompilation

Areas of concentration

UserSupport

• Multiple runs • Useful metrics

OutputVariable

Assessment

• GUI • Analysis tools

Compiled per run report

Gunner-Driver

% of Time OverloadedMax V Max A Max C Max P Max OVERALL #V #A #C #P #OVERALL %V %A %C %P %OVERALL

1 24.8 10.6 24.1 5.2 62.9 169 16 153 0 42 70% 7% 64% 0% 18%2 28.2 15.5 36.2 5.8 76.6 1241 258 1202 0 367 76% 16% 74% 0% 22%3 24.8 15.5 30.4 6.2 71.3 1201 244 1113 0 344 79% 16% 73% 0% 23%4 24.8 11.6 26.1 6.2 66.7 191 33 175 0 48 68% 12% 63% 0% 17%5 24.8 15.5 30.4 6.8 71.3 1465 295 1397 0 415 69% 14% 66% 0% 20%6 22.8 15.5 30.4 6.2 71.3 1564 416 1490 0 470 78% 21% 75% 0% 24%

Max Workload Values # of Times Overloaded

94 22.8 15.5 30.4 5.8 73.9 348 136 370 0 144 77% 30% 82% 0% 32%95 28.2 15.5 30.4 6.2 71.3 1244 259 1191 0 339 71% 15% 68% 0% 19%96 28.2 15.5 30.4 6.2 68.7 2680 632 2576 0 811 78% 18% 75% 0% 24%97 24.8 15.5 30.4 6.8 71.3 1408 275 1308 0 389 76% 15% 70% 0% 21%98 28.2 15.5 30.4 6.2 77.7 1548 379 1531 0 477 71% 17% 70% 0% 22%99 22.8 11.6 25.1 6.2 61.1 444 98 413 0 122 78% 17% 72% 0% 21%

100 28.2 15.5 31.9 6.8 74.3 1362 220 1244 0 345 76% 12% 69% 0% 19%

Expected value 25.96 14.92 29.72 6.404 70.08888889 1233 268.2 1168 0.283 355.929293 75% 16% 71% 0% 21%Std dev 2.288 1.518 3.364 0.82 5.668385094 652.5 152.1 618.7 0.77 191.054659 4% 4% 4% 0% 3%

2 vs. 3 study (Goal orientation operations module)

Operator Overload Report

Maximum workload peak report

Workload Peak Summary Measures

Task i # times % times167 30 83.33%

E[max workload peaks] = 67.81168 3 8.33%

E[# of max workload peaks] = 1.47209 3 8.33%

E[# of ongoing tasks] = 8.67210 21 58.33%211 12 33.33%

Var[max workload peak] = 19.23212 2 5.56%

Var[# of max workload peaks] = 0.68219 36 100.00%

Var[# of ongoing tasks] = 0.43222 36 100.00%223 27 75.00%227 36 100.00%228 29 80.56%229 36 100.00%232 36 100.00%

multiple runs, 36 max workload peaks through 100 runs

Tasks contributing to max workload peaks

2 vs. 3 study (Goal orientation operations module)

DS GS ORGTotal

COR/PRE

Total MOS

95% C.I.

COR 20.6 0 24.5 45

PRE 0 0 0 0

COR 64.1 0 38.8 102.9

PRE 0 0 36.1 36.1

Total ORG Level

84.7 0 99.4

95% C.I.

UL = 106.1 LL = 63.3

UL = 0 LL = 0

UL = 124.6 LL = 74.2

UL = 48.8 LL = 41.2

UL = 148.8 LL = 129.2

Overall Total

184.1UL = 195.0 LL = 173.2

45

139

ORG Level

27R 10

27R 20

n = 5 replications

Simple study (Maintenance model)

Contingency table

1Average Occurrences/

Man-Hours

2Total Occurrences/ Man-Hours for MOS

3

Total Occurrences/ Man-Hours for MOS/ORG Level

4

95% Confidence Interval for total Occurrences/ Man-Hours for MOS/ORG Level

5

Overall Average and 95% Confidence Interval for Occurrences/ Man-Hours

COR Corrective

PRE Preventive

Key

• Efficient data reporting• Graphical and tabular reports

DataCompilation

Areas of concentration

UserSupport

• Multiple runs • Useful metrics

OutputVariable

Assessment

• GUI • Analysis tools

Study objective Preliminary output assessment System analysis modified and tailored to

the needs of user– Summary performance– Comparative model study– Model characterization– Sensitivity analysis/Model validation– Model optimization or enhancement

User support: output analysis

Analysis guidelines

Output system analysis

Type of study

Collect data over multiple runs

PerformanceSummary

Determine experimental levels for input

variables

MultipleFactorStudy

Collect data for each model over multiple runs

Single FactorStudy

Compute descriptive statistics

(i.e., mean, std., min, max)

Run experimental settings and collect

data

Use data to build statistical models

Use data to test for statistical differences

between models

Identify input variables and classify them

accordingly

Factor Low (-) High (+)MOUBF 1.00 hr 3.00 hrsMTTR 10:00 min 50:00 min

# of Maintainers 1 5# of Systems 1 10

# per Departure 1 10Length of Run 10 100Time Between

Departures 10:00 min 30:00 min

Example: multiple factor study

Simple study (Maintenance model)

Factor values may vary depending upon characteristics such as manufacturer or material type.

EXAMPLE: A certain component is made by two different companies. Component A has a MOUBF of 1 hour. Component B is a higher quality product and therefore has a higher MOUBF of 3 hours.

Significant Factors:– MOUBF– MTTR

Statistical Prediction Model: Total Direct Man hours = 68.7 - 1.4*(MOUBF) + 2.2*(MTTR)

+ 2.3*(# of Maintainers) + 3.16*(# of Systems)

+ 3.24*(Length of Run)

Prediction:

Example: multiple factor study

Simple study (Maintenance model)

– # of Maintainers– # of Systems

– Length of Run

MOUBF = 2 hr; MTTR = 20 min; # maint = 3; # sys = 7; run length = 30

Estimated total direct man hours = 236.12

Collaborative

Discussion

Workload level histogram

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

5% 10%15%20%25%30%35%40%45%50%55%60%65%70%75%80%85%90%95%100%105%110%115%120%125%130%135%140%145%150%155%160%165%170%175%180%185%190%195%200%>

200%

Workload Level

% o

f T

ime

overloaded

NOT overloaded

% of Time at Workload Levels for Driver 100 runs

21.05%

78.95%

2 vs. 3 study (Goal orientation operations module)