MONTE CARLO SIMULATION TO ASSESS PERFORMANCE VARIABILITY IN THE FRAM
Riccardo Patriarca, Giulio Di Gravio, Francesco Costantino
Sapienza University of Rome
Department of Mechanical and Aerospace Engineering
Via Eudossiana, 18 – 00184 Rome (Italy)
The 10th FRAMily meeting/workshop,
June 1-3 2016
University of Lisbon, Portugal
- Systems and places of work are well-designed and correctly maintained.
- Procedures are comprehensive, complete, and correct.
- People at the sharp end behave as they are expected to, and as they have been trained to. (WAD=WAI)
- Designers have foreseen every contingency and have provided the system with appropriate response capabilities.
OUR STARTING POINT
…and other Safety-II assumptions
Hollnagel, E. (2014). Safety-I and Safety-II: The
Past and Future of Safety Management. Farnham,
UK: Ashgate.
Monte Carlo simulation allows building models of possible results by substituting arange of values—a probability distribution—for any factor that has inherentuncertainty.
It then calculates results over and over, each time using a different set of randomvalues from the probability functions. Depending upon the number of uncertainties andthe ranges specified for them, a Monte Carlo simulation could involve thousands ortens of thousands of recalculations before it is complete.
Monte Carlo simulation produces distributions of possible outcome values.
JUST «TWO» WORDS
FRAM STEPS (the traditional ones, but…)
Step 1: Identification and description of system’s functions
Step 2: Identification of performance variability *
Step 3: Aggregation of variability *
Step 4: Management of variability *
*Steps modified by the innovative
formulation we propose
FRAM STEP 0
Mainly built for risk assessment, but applicable also for accident analysis (let’s talk about it)
The aim of this approach is proactively manage transient causes and links among functions, to
characterize WAD: Looking for combinations of failures and latent conditions that may
constitute a risk
FRAM STEP 1
Name of function
Description
Variability
AspectInput
Output
Precondition
Resource
Control
Time
Simple solution: Describe function variability based on two phenotypes (Timing and Precision)
FRAM STEP 2
𝑶𝑽𝒋 = 𝑽𝒋𝑻 ∙ 𝑽𝒋
𝑷
𝑉𝑗𝑇 represents the upstream output 𝑗 score in terms of timing
𝑉𝑗𝑃 represents the upstream output 𝑗 score in terms of precision
HP: The phenotpypes are statistically independent and thus we can evaluate their product to
summarize their combined effect on Output Variability (𝑂𝑉𝑗)
VARIABILITY SCORE
TIMING
On time 1
Too early 2
Too late 3
Not at all 4
PRECISION
Precise 1
Acceptable 2
Imprecise 3
Wrong 4
FRAM STEP 2
We can assign a score to each variable level, which allow the product. FOR EXAMPLE:
Is a SINGLE numeric score able to properly describe function’s variability?
BUT remember that resonance
IS NOT stochastic
but IS functional because
what people do IS NOT random…
FRAM STEP 2
Probability distributions may better define the variability …
FRAM STEP 2
For this purpose, Monte Carlo simulation helps evaluating the products
FRAM STEP 3 – AGGREGATING FUNCTIONS
𝑪𝑽𝒊𝒋 = 𝑶𝑽𝒋 ∙ 𝒂𝒊𝒋𝑻 ∙ 𝒂𝒊𝒋
𝑷
𝑎𝑖𝑗𝑇 represents the amplifying factor for the upstream output 𝑗 and the downstream function 𝑖, in terms of timing
𝑎𝑖𝑗𝑃 represents the amplifying factor for the upstream output 𝑗 and the downstream function 𝑖, in terms of precision
𝑎𝑖𝑗𝑇 (or 𝑎𝑖𝑗
𝑃 )
> 1 in case the upstream output has an amplyfing effect on the downstream function= 1 in case the upstream output has no effect on the downstream function< 1 in case the upstream output has a damping effect on the downstream function
Note that 𝑎𝑖𝑗𝑇 (or 𝑎𝑖𝑗
𝑃 ) may assume the following values:
We add damping/amplification coefficients to realte the output variability to the functions’ aspects
FRAM STEP 3 – DIFFERENT INSTANTIATIONS
𝑺𝑷𝑪𝟏 𝑺𝑷𝑪𝟐 … 𝑺𝑷𝑪𝒎
Function 1 𝑏11 𝑏1
2 𝑏1𝑚
Function 2 𝑏21 𝑏2
2 𝑏2𝑚
…
Function n 𝑏𝑛1 𝑏𝑛
2 𝑏𝑛𝑚
𝑏𝑗𝑘
= 1 in case the 𝑆𝑃𝐶𝑘 has a high impact on the j function
< 1 in case the 𝑆𝑃𝐶𝑘 has a moderate impact on the j function
= 0 in case the 𝑆𝑃𝐶𝑘 has no impact on the j function
𝑏𝑗𝑘 identifies the effect of the
𝑆𝑃𝐶𝑘 on the 𝑗 function.
To define a particular instantiation of the model, it is necessary to define a specific number 𝑚of variables, capable of identifying the scenarios to analyze, i.e. Scenario Performance
Conditions 𝑆𝑃𝐶𝑘, where 𝑘 = 1,… ,𝑚, and their potential effect
A particular combination of 𝑆𝑃𝐶𝑘 constitutes an operating scenario. It is possible to build the S matrix, which relates each scenario to the identified 𝑆𝑃𝐶𝑘, by the 𝑆𝑃𝐶𝑧
𝑘. 𝑆𝑃𝐶𝑧𝑘
represents the 𝑆𝑃𝐶𝑘 amplifying effect in the z scenario 𝑆𝑧, 𝑧 = 1,… , 𝑍
FRAM STEP 3 – DIFFERENT INSTANTIATIONS
𝑺𝑷𝑪𝟏 𝑺𝑷𝑪𝟐 … 𝑺𝑷𝑪𝒎
Instantiation 1 𝑆𝑃𝐶11 𝑆𝑃𝐶1
2 𝑆𝑃𝐶1𝑚
Instantiation 2 𝑆𝑃𝐶21 𝑆𝑃𝐶2
2 𝑆𝑃𝐶2𝑚
…
Instantiation Z 𝑆𝑃𝐶𝑍1 𝑆𝑃𝐶𝑍
2 𝑆𝑃𝐶𝑍𝑚
𝑆𝑃𝐶𝑧𝑘 =
𝑆𝑃𝐶𝑧𝑘′ 𝐻𝑖𝑔ℎ 𝑉𝑎𝑟𝑖𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑒𝑓𝑓𝑒𝑐𝑡 𝑜𝑓 SPCk
𝑆𝑃𝐶𝑧𝑘′′ 𝐿𝑜𝑤 𝑉𝑎𝑟𝑖𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑒𝑓𝑓𝑒𝑐𝑡 𝑜𝑓 SPCk
𝑆𝑃𝐶𝑧𝑘′′′ 𝑁𝑜 𝑉𝑎𝑟𝑖𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑒𝑓𝑓𝑒𝑐𝑡 𝑜𝑓 SPCk
𝑆𝑃𝐶𝑧𝑘 =
210
FRAM STEP 3 – DIFFERENT INSTANTIATIONS
The conditional variability 𝑒𝑗𝑧 of any output 𝑗, due to the operating conditions state in a
particular scenario 𝑧 is:
𝒆𝒋𝒛 =
𝒌=𝟏𝒎 𝑺𝑷𝑪𝒛
𝒌 ∙ 𝒃𝒋𝒌
𝒎
This equation has to be formally modified to consider that a function 𝑗 may be not influenced by
any 𝑆𝑃𝐶𝑘, i.e. 𝑏𝑗𝑘 = 0 for each 𝑘. In this case 𝑒𝑗
𝑧 = 1, confirming that the scenario does not
amplify the function variability 𝑒𝑗𝑧 = 𝑚𝑎𝑥 1;
𝑘=1𝑚 𝑆𝑃𝐶𝑧
𝑘 ∙ 𝑏𝑗𝑘
𝑚
𝑽𝑷𝑵𝒊𝒋𝒛 = 𝑽𝒋
𝑻 ∙ 𝑽𝒋𝑷 ∙ 𝒂𝒊𝒋
𝑻 ∙ 𝒂𝒊𝒋𝑷 ∙ 𝒆𝒋
𝒛
FRAM STEP 3 – OVERALL INDEX
The overall index for each coupling, which address its variability according timing and precision phenotypes, in an operating scenario z can be derived as:
function
variability
upstream/
downstream
link
scenario
Once assigned the variability following the distributions, it is possible to define criticalcouplings and paths, based on the 𝑽𝑷𝑵𝒊𝒋
𝒛 and then mitigating actions.
A coupling is considered critical if the cumulative distribution over a threshold is minor than a confidence level.
FRAM STEP 4
The distribution in this area (for example) represents
possible combinations of variability with critical
impact on performance (if minor than the confidence
level, the coupling becomes critical)
SIMPLE SOLUTION: IS IT ALWAYS ENOUGH?
T I M E
P R E C I S I O N
S P E E D
O B J E C T
D I R E C T I O N
F O R C E
Nominal
Too late or too early
- no impact on
mission success and
safety
Too late or too early
- low impact on
mission success
Too late or too early
- impact on mission
success and safety
1 2 3 4
Nominal
Acceptable - no
impact on mission
success and safety
Imprecise - low
impact on mission
success
Imprecise - impact
on mission success
and safety
1 2 3 4
Nominal
Too slow or too fast
- no impact on
mission success and
safety
Too slow or too fast
- low impact on
mission success
Too slow or too fast
- impact on mission
success and safety
1 2 3 4
Nominal Detectable FailedNot Detectable
Failed
1 3 4
Nominal
Not nominal
direction - no
impact on mission
success and safety
Imprecise direction -
low impact on
mission success
Wrong direction -
impact on mission
success and safety
1 2 3 4
Nominal
Too much or too
little - no impact on
mission success and
safety
Too much or too
litlle - low impact on
mission success
Too much or too
little - impact on
mission success and
safety
Score
Possible Output variability with regard to object
Score
Possible Output variability with regard to time
Score
Possible Output variability with regard to precision
Score
PH=1
Time
variability
PH=2
Precision
Variability
PH=3
Speed
Variability
PH=4
Possible Output variability with regard to speed
Object
Variability
PH=5
Direction
Variability
PH=6
Possible Output variability with regard to direction
Score
Possible Output variability with regard to force
Force
Variability
Each function variability can be described
by six phenotypes of variability
COMPLEX SOLUTION
𝑶𝑽𝒋 = 𝑽𝒋𝑻 ∙ 𝑽𝒋
𝑷 ∙ 𝑽𝒋𝑺 ∙ 𝑽𝒋
𝑶∙ 𝑽𝒋𝑫 ∙ 𝑽𝒋
𝑭
We develop the Phenotype Rule Block (PRB)
Multiplying the factors
is not statistically representative
COMPLEX SOLUTION
Similarly, for evaluating the coupling we develop the Coupling Rule Block (CRB), which allows defining
the Variability Priority Number (VPNij) of each coupling.
This approach required an IT support tool, and so we developed a VBA-based code
THAT’S «FUNNY», BUT HOW TO APPLY IT?
FRAM Model
Visualizer
MS ExcelPalisade @Risk
WALKTHROUGH
A brief description of how the tool works and how it could help our analysis
Just a summary…
Estimate distribution’s parameters
HOW TO DEFINE THE DISTRIBUTIONS’ PARAMETERS?
Chategorize functions into
H/T/O
Retrieve data from literature
Run Monte
Carlo
simulation
Isolate critical
paths
Develop
dedicate
report forms
for everyday
work
Analyze the
reports
FRAM model(the traditional
one)
Data from sharp-end reports
(simulation, judgments,etc.)
Define mitigating actions(improve training, change a procedure, etc.)
Downstream function Upstream function Scenarios
Name of
functionAspect
Name of
function
Description
of Aspect
Scenario
1
Scenario
2
Scenario
3
Scenario
4
Scenario
5
Scenario
6
Absence of
undesired
obstacles on APT
surface
ControlNOTAM
functioningNOTAM issued 0 0 0 0 0.004 0.004
Sector-sector
communication
InputADS-B
functioningADS-B data 0 0 0 0 0.004 0.004
Resource CoordinationCoordinated
personnel0 0.001 0.001 0.001 0.001 0.001
Pilot/ATCO
communication
Input
Communication
Pilot/ATCO
instruments
functioning
Pilot/ATCO
communication
link active
0 0.001 0.001 0.158 0.158 0.158
Start crossing InputPilot/ATCO
communication
Clarified
instructions0 0.275 0.275 0.275 0.275 0.770
EXAMPLE OF APPLICATION: RUNWAY INCURSION
EXAMPLE OF APPLICATION: APOLLO 11 POWERED DESCENT
http://history.nasa.gov/afj/
Presented at the 8th IAASS Conference on Advancement of Space Safety (18-20 May 2016, Melbourne, FL)
Railway safety
Ground Handling safety
Healthcare (perioperative care)
Industrial plants
…
OUR CURRENT RESEARCH THEMES…
Now have a look to our tool and our VBA code
A WALKTHROUGH THE CODE
FRAM Model
Visualizer
MS ExcelPalisade @Risk
One of the most serious accidents ever happened:
Deaths: >1’000’000
Destroyed more than 18 highly technolgical planes
What is this?
Have a look to this video!
AND NOW…A «SERIOUS» REFLECTION
« Great shot kid, that was one in a million »
Han Solo, Star Wars Ep. IV
Death star seems an ultrasafe system, so my questions are…(may be you need to look again the films)
If you were an imperial soldier, how would you prevent the death star’s destruction?
If a rebel, how to make that distruction just not a lucky case (due to Luke’s jedi skills)?
and moreover..
Do you think is it useful to «filter» the traditional functions?
Which way of filtering them is the most useful one?
Is a multi-layer representaion really useful?
Do you think is it useful an Excel-based representation?
Would you like to use this Excel tool? Do you think is there required any modifications?
AND SEVERAL FUTURE SERIOUS CHALLENGES