the autosimoa project
DESCRIPTION
AUTOMATING D.E.S OUTPUT ANALYSIS:. The AutoSimOA Project. HOW MANY REPLICATIONS TO RUN. Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC 07. A 3 year, EPSRC funded project in collaboration with SIMUL8 Corporation. http://www.wbs.ac.uk/go/autosimoa. Objective - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/1.jpg)
The AutoSimOA Project
Katy Hoad, Stewart Robinson, Ruth DaviesWarwick Business School
WSC 07
A 3 year, EPSRC funded project in collaboration with SIMUL8 Corporation.
http://www.wbs.ac.uk/go/autosimoa
![Page 2: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/2.jpg)
Objective
To provide an easy to use method, that can be incorporated into existing simulation software, that enables practitioners to
obtain results of a specified accuracy from their discrete event simulation model.
(Only looking at analysis of a single scenario)
![Page 3: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/3.jpg)
OUTLINE
IntroductionMethods in literatureOur AlgorithmTest Methodology & ResultsDiscussion & Summary
![Page 4: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/4.jpg)
Underlying Assumptions
Any warm-up problems already dealt with.
Run length (m) decided upon.
Modeller decided to use multiple replications to obtain better estimate of mean
performance.
N
jjXN
X1
1Response measure
of interest
summary statistic from each replication
Perform N replications
![Page 5: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/5.jpg)
QUESTION IS…
How many replications are needed?
Limiting factors: computing time and expense.
4 main methods found in the literature for choosing the number of replications N to perform.
![Page 6: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/6.jpg)
1. Rule of Thumb (Law & McComas 1990)
Run at least 3 to 5 replications.
Advantage: Very simple.
Disadvantage: Does not use characteristics of model output.
No measured precision level.
![Page 7: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/7.jpg)
2. Simple Graphical Method (Robinson 2004)
Cumulative mean graph
45
47
49
51
53
55
1 9 17 25 33 41 49 57 65 73 81 89 97 105
Number of replications (n)
Cum
ula
tive m
ean
Advantages: Simple Uses output of interest in decision.
Disadvantages: Subjective No measured precision level.
![Page 8: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/8.jpg)
3. Confidence Interval Method (Robinson 2004, Law 2007, Banks et al. 2005).
Advantages: Uses statistical inference to determine N.
Uses output of interest in decision.
Provides specified precision.
Disadvantage: Many simulation users do not have the skills to apply approach.
Cumulative mean graph
46
48
50
52
54
56
1 8 15 22 29 36 43 50 57 64 71 78 85 92 99 106
Number of replications (n)
Cum
ula
tive m
ean
![Page 9: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/9.jpg)
4. Prediction Formula (Banks et al. 2005)
• Decide size of error ε that can be can tolerated.• Run ≥ 2 replications - estimate variance s2.• Solve to predict N.
• Check desired precision achieved – if not recalculate N with new estimate of variance.
Advantages: Uses statistical inference to determine N. Uses output of interest in decision. Provides specified precision.
Disadvantage: Can be very inaccurate especially for small number of replications.
2
1,2
st
NN
![Page 10: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/10.jpg)
Run
Model START:
Load Input
Produce Output Results
Run Replication Algorithm
Precision criteria met?
Recommend replication number
Run one more
replication
YES
NO
AUTOMATE Confidence Interval Method: Algorithm interacts with simulation model sequentially.
![Page 11: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/11.jpg)
2,1 nt
n
nn
nX
nt
d
s2,1
100
is the student t value for n-1 df and a significance of 1-α,
nX
sn is the estimate of the standard deviation,
calculated using results Xi (i = 1 to n) of the n current replications.
Where
n is the current number of replications carried out,
We define the precision, dn, as the ½ width of the Confidence Interval expressed as a percentage of the cumulative mean:
is the cumulative mean,
ALGORITHM DEFINITIONS
![Page 12: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/12.jpg)
Stopping Criteria
• Simplest method:
Stop when dn 1st found to be ≤ desired precision, drequired . Recommend that number of replications, Nsol, to user.
• Problem: Data series could prematurely converge, by chance, to incorrect estimate of the mean, with precision drequired , then diverge again.
• ‘Look-ahead’ procedure: When dn 1st found to be ≤ drequired, algorithm performs set number of extra replications, to check that precision remains ≤ drequired.
![Page 13: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/13.jpg)
0
20
40
60
80
100
120
140
3 100
137
174
211
248
285
322
359
396
433
470
replication number (n )
f(kLim
it)
kLimit=0 kLimit=5
kLimit=10 kLimit=25
‘Look-ahead’ procedure
kLimit = ‘look ahead’ value. Actual number of replications checked ahead is
Relates ‘look ahead’ period length with current value of n.
100,100
100,)(
nkLimitn
nkLimitkLimitf
![Page 14: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/14.jpg)
23
25
27
29
31
33
35
37
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Replication number (n)
NsolNsol + f(kLimit)
f(kLimit)
Precision ≤ 5%X
X
95% confidence limits
Cumulative mean,
Replication Algorithm
![Page 15: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/15.jpg)
0.8
0.85
0.9
0.95
1
1.05
1.1
1.15
1.2
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
Replication number (n)
Precision
≤ 5%
Precision
> 5%
Precision ≤ 5%
f(kLimit)
Nsol2Nsol2 + f(kLimit)
Nsol1
![Page 16: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/16.jpg)
• 24 artificial data sets: Left skewed, symmetric, right skewed; Varying values of relative st.dev (st.dev/mean).
• 100 sequences of 2000 data values.
• 8 real models selected.
• Different lengths of ‘look ahead’ period tested:
kLimit values = 0 (i.e. no ‘look ahead’), 5, 10, 25.
• drequired value kept constant at 5%.
TESTING METHODOLOGY
![Page 17: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/17.jpg)
5 performance measures
1. Coverage of the true mean2. Bias3. Absolute Bias4. Average Nsol value5. Comparison of 4. with Theoretical Nsol
value
• For real models: ‘true’ mean & variance values - estimated from whole sets of output data (3000 to 11000 data points).
Microsoft Excel Worksheet
![Page 18: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/18.jpg)
Results
• Nsol values for individual algorithm runs are very variable.
• Average Nsol values for 100 runs per model close to the theoretical values of Nsol.
• Normality assumption appears robust.
• Using a ‘look ahead’ period improves performance of the algorithm.
![Page 19: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/19.jpg)
Mean bias significantly different to zero
Failed in coverage of true mean
Mean est. Nsol significantly different to theoretical Nsol (>3)
No ‘look-ahead’ period
Proportion of Artificial models
4/24 2/24 9/18
Proportion of Real models
1/8 1/8 3/5
kLimit = 5 Proportion of Artificial models
1/24 0 1/18
Proportion of Real models
0 0 0
![Page 20: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/20.jpg)
% decrease in absolute mean bias
kLimit = 0 tokLimit = 5
kLimit = 5 tokLimit = 10
kLimit = 10 tokLimit = 25
ArtificialModels
8.76% 0.07% 0.26%
RealModels
10.45% 0.14% 0.33%
Impact of different look ahead periods on performance of algorithm
![Page 21: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/21.jpg)
Number of times the Nsol value changes (out of 100 runs of the algorithm per model) because of the lengthening of the ‘look ahead’ period.
Model ID
kLimit = 0 to kLimit = 5
kLimit = 5 tokLimit = 10
kLimit = 10 to kLimit = 25
R1 0 0 0
R3 2 0 0
R5 24 0 1
R8 24 4 1
A5 30 1 3
A6 26 6 3
A15 1 0 0
A17 22 0 1
A21 25 2 1
A24 37 0 0
![Page 22: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/22.jpg)
Model ID
kLimit Nsol Theoretical Nsol (approx)
Mean estimate significantly different to the true mean?
A9 0 4 112 Yes
5 120 No
A24 0 3 755 Yes
5 718 No
R7 0 3 10 Yes
5 8 No
R4 0 3 6 Yes
5 7 No
R8 0 3 45 Yes
5 46 No
Examples of changes in Nsol & improvement in estimate of true mean
![Page 23: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/23.jpg)
DISCUSSION
• kLimit default value set to 5.
• Initial number of replications set to 3.
• Multiple response variables - Algorithm run with each response - use maximum estimated value for Nsol.
• Different scenarios - advisable to repeat algorithm every few scenarios to check that precision has not degraded significantly.
• Implementation into Simul8 simulation package.
![Page 24: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/24.jpg)
SUMMARY
• Selection and automation of Confidence Interval Method for estimating the number of replications to be run in a simulation.
• Algorithm created with ‘look ahead’ period -efficient and performs well on wide selection of artificial and real model output.
• ‘Black box’ - fully automated and does not require user intervention.
![Page 25: The AutoSimOA Project](https://reader036.vdocuments.site/reader036/viewer/2022081801/568144ff550346895db1cbab/html5/thumbnails/25.jpg)
ACKNOWLEDGMENTSThis work is part of the Automating Simulation Output
Analysis (AutoSimOA) project (http://www.wbs.ac.uk/go/autosimoa) that is funded by
the UK Engineering and Physical Sciences Research Council (EP/D033640/1). The work is being carried out in
collaboration with SIMUL8 Corporation, who are also providing sponsorship for the project.
Katy Hoad, Stewart Robinson, Ruth DaviesWarwick Business School
WSC 07