Insights for future HRA data collection based on research using HRA data

Katrina M. GrothUniversity of Maryland

Reminder on topic

My talk will center around some of my experiences from using HRA data, rather than discussing a specific data source. It will include a brief discussion of some data sources I've worked with (SACADA, HERA, and sources we gathered to help the IDHEAS expert elicitation) as well as the modeling pieces needed to use the data (e.g., PSFs, task breakdowns, causal models and factorization).

2

Conclusions: Data-informed insights for HRA Data collection1. New data sources enabled by international collaborations

offer great opportunity for HRA2. Models are essential for making data useful3. “Data” is not a synonym for “Numbers”4. Data quality > data quantity

Outline

Data sources I’ve used & Previous work Latest work: Insights from data use

4

Systems Risk and Reliability Analysis lab (SyRRA) Established 2017, Directed by Prof. Katrina Groth. Research into risk & reliability for complex systems

Human + system + environment + complex phenomena

5

Diverse, multi-source dataHuman decision makers

& engineering knowledgeEnhance safety,

reliability, resilience of complex systems

Causal & probabilistic models

Machine learning

Total Risk

0

0.001

0.002

0.003

0.004

0.005

0.006

1 2 3 4 5 6 7 8 9 10

Chart2

0.001

0.0015

0.00185

0.002

0.00175

0.00355

0.00305

0.0049

0.00425

0.0053

Total Risk

Sheet1

510

520

625

824

525

365

257

882

575

792

Sheet1

SI-1

SI-2

Sheet2

5100.00010.000050.001

5200.00010.000050.0015

6250.00010.000050.00185

8240.00010.000050.002

5250.00010.000050.00175

3650.00010.000050.00355

2570.00010.000050.00305

8820.00010.000050.0049

5750.00010.000050.00425

7920.00010.000050.0053

Sheet2

Total Risk

Sheet3

2004-2009 2011 2013 2015 2017 2019 …

Timeline of my HRA data work

2004-2009, UMDPhD on Data informed HRA

•2009 Dissertation on data-informed PSF modeling

•2012 Data-informed PIF hierarchy •2012 Data-informed BN for PIF dependency

2008, INLHERA data collection

2011, SNLU.S. HRA empirical

study

2012-2013, SNL Leveraging

Published Data to quantify IDHEAS

HEPs

2011-2014, SNLBayesian updating Bayesian Networks. Using SPAR-H + Halden data

2017, SNLCapturing

cognitive casual pathways in

HRA

2018, UMDSACADA data-use framework

•Submitted: A hybrid approach to HRA using simulator data, casual models, and cognitive science

2018-2019, UMD/UCLAPHOENIX + casual BNs

and quantification approach

HRA-relevant data sources I’ve explored Comprehensive simulator data w/detailed context & event descriptions

H2ERA (Halden HERA), SACADA, Briefly (intend to go deeper): OPERA & HuReX

Existing HRA models SPAR-H, IDAC, THERP, ASEP, CREAM, CBDT

Retrospective event data with detailed context HERA, HFIS, CORE-DATA, NUCLARR

Retrospective event data without detailed context (or limited to 1-2 factors) CORE-DATA, GRS data (or THERP-like tasks)

Single parameter studies & expert estimates (for a multitude of HRA quantities) Hundreds of published articles from a wide variety of high-consequence industries

(+Significant work aggregating dozens into comprehensive supporting documentation for the IDHEAS expert elicitation study)

Expert Survey for Zwirglmaier work; Engineering Data Compendium

Cognitive Models Dozens of cognitive models (NRC cognitive basis study; IDAC source documents;

Engineering Data Compendium;)

7

Insights from 15 years of HRA data use – Motivation for my work No HRA data sources were designed with causal modeling in

mind (except method-specific data collection, e.g. THERP). This makes modeling significantly harder.

No single HRA data set collects data on all of the factors, causes, and consequences relevant to modeling human performance.

90%+ of the work is in understanding the dataset and mapping dataset variables onto modeling variables. This has to be done for each dataset.

Many good HRA data collection projects have been started & stopped over the years – taking that data with them.

Thus, a “big picture” modeling paradigm is the only way to overcome the inherent limits of HRA data collection

Data-to-model process for “easy” reliability problems

9

Model Form(states & relationships)

Estimating Model Parameters (Quantification)

Final modelcapable of predicting reliabilityquantities of interest

Variables & definitions(For data collection & modeling)

Full scale data source 1

Repeat/update parameters (of final model at reasonable intervals)

Modernize at major intervals

Try to use the “easy” Data-to-model process is causing HRA to fail

10

Single data source (increasing in complexity over years, but few attempts to modernize)

No updating; instead we create a new model

50+ HRA methods exist, we’re not updating them, and we can’t use them together

Largely regression based; every model uses different variables; lacks causality

Data-to-model process for complex problems (like HRA)

11

Model Form(states & relationships)

Estimate Model Parameters (Quantification)

Final modelcapable of predictingHRA quantities of interest

Full scale data source 2

Partial data source 3

Modeling variables &definitions

Partial data source 2

Partial data source 1

Full scale data source 1

Modeling constructs (decomposition; causality)

Map of data collectionvariables & definitions onto modelingVariables &definitions

Repeat/update parameters of final model at reasonable intervals

Repeat/update data source designs at major intervals

Previous work created important pieces of model- and data-informed approach to HRA

Taxonomy of PIFs (Groth 2012) Application neutral, clearly defined, non-

overlapping set of factors for modeling

Bayesian Networks causal models (Groth et al 2009, 2013, 2014, 2017) To capture causal relationships &

uncertainty

Bayesian parameter updating algorithm (Groth et al 2014, 2017) To incorporate data into probability

assignments

Full hybrid BN + multi-data quantification approach: paper submitted to RESS in Nov 2018.

12

New: Identified requirements for data-informed HRA models Using HRA data to improve HRA models requires

new approaches Desirable characteristics of advanced HRA models

1. Using underlying causal model rooted in strong technical basis (combining psychological research, operating experience, simulator data)

2. Explicitly representing causal factors that affect performance (& are collected in data)

3. Support qualitative & quantitative HRA4. Framework should be both data-informed and model-informed.5. Flexibility to accommodate changes as our databases mature6. Ability to fuse information from multiple sources of data & models7. Generate detailed insights to improve human performance (beyond

quantifying)

13Groth, K. M (2018). A framework for using SACADA to enhance the qualitative and quantitative basis of HRA. Proceedings of the 14th Probabilistic Safety Assessment and Management Conference (PSAM 14), Los Angeles, CA.

Result of 2018 UMD SACADA research – published in PSAM 2018

New: Key elements of the multi-data-informed HRA framework

Framework for data-informed

HRA

Performance influencing

factor taxonomy

Human-machine team failure modes

Bayesian Network causal

modelsData and Bayesian parameter updating

Human-machine task

sequences

14

And model

Groth, K. M (2018). A framework for using SACADA to enhance the qualitative and quantitative basis of HRA. Proceedings of the 14th Probabilistic Safety Assessment and Management Conference (PSAM 14), Los Angeles, CA.

Result of 2018 UMD SACADA research – published in PSAM 2018

New: generalized set of HRA quantification elements Core types of variables:

Performance Influencing factors (PIFs)

Crew failure mechanisms Crew failure modes (proximate

causes) Macrocognitive functions

Core set of probabilistic relationships:

15

PIF1

FM1

PC1

MCF

⋯ PCNPC

FM2 FM3 FMNFMFMNFM−1⋯

PIF2 PIF3 PIFNPIFPIFNPIF−1⋯

Result of 2018 UMD SACADA research –draft submitted to RESS in Nov 2018

New: Full Algorithm for “A hybrid approach to HRA using simulator data, causal models, and cognitive science”

Block 1: Causal Factor Mapping (BN structure development) Create a causal map of the relationship between the PIFs, failure mechanisms, proximate causes of

failure, and MCFs by way of Bayesian Network. (Groth 2012; Ekanem &Mosleh 2013; Zwirglmaier, Straub, Groth 2017)

Simplify BN structure using node reduction (Zwirglmaier, Straub, Groth 2017) Block 2: Prior model quantification (BN parameterization)

Map existing HRA method PIFs to PIF taxonomy (Groth 2012) Use existing HRA method to get priors of the probability of MCF error Pr (MCFerror|PIFs) Expert elicit priors for PIFs, Pr (PIFs) (Groth, Swiler, 2014)

Block 3: Bayesian update model parameters Map simulation data source variables to PIFs Use simulation data source to update probability of MCF error Pr (MCFerror|PIFs) (Groth,

Swiler, Smith 2014) (Optional) Perform Bayesian update of the PIF probabilities Pr(PIFs) (Groth, Swiler, Smith

2014) Block 4: Use the BN for HRA activities

16

Result of 2018 UMD SACADA research –draft submitted to RESS in Nov 2018

New: Comprehensive causal maps for PHOENIX (w/ UCLA)

Full causal structures developed by Groth + UCLA using cognitive literature, IDAC, PIF hierarchy, PHOENIX CFMs, Zwirglmaier & Groth 2017 method I Phase CFMs (6 core CFMs, 3 mechanism-

like constructs, 20+ PIFs) D Phase CFMs ( 5 core CFMs; 2

mechanism-like constructs, 20+ PIFs) A Phase CFMs (3 core CFMs, 2 mechanism-

like constructs, 20+ PIFs) ~5 CFMs still need to be modeled. D phase &

Communications related.

Quantification is possible with future work (using algorithm in draft RESS paper) – needs customer

17

New: Draft mapping of SACADA elements to PIFs & FMs to enable quantification SACADA can be used to quantify: Pr(crew failure mode|PIFs)

See submitted RESS paper for full draft of the mapping

Need other data sources for for Pr(PIFs) UMD is now exploring how to use SACADA for modeling

dependency: Pr(HFE|HFE) and Pr(PIFs|PIFs)

18

Gaps & opportunities: Next steps for UMD Project: Develop the theoretical & mathematical foundations to

model “HRA dependency” and enable parameterization using multiple sources (starting with SACADA) Outcome 1: Terminology & definitions (both words and equations).

To resolve significant conceptual problems around:– “HRA dependency”– “Human failure mode / mechanism / proximate cause”– “Probability of Error”& uncertainty

» Overheard at PSAM “Probability of error is just number of failures out of number of opportunities. We just need to count things.”

– Conditional vs. marginal vs. joint probability

Outcome 2: Mathematical framework for modeling dependency (DBNs) Outcome 3: Mathematical framework for quantifying dependency using

simulator data.

19

NRC grant will support 1 year of research toward this broader objective; additional out-year support is needed.

Gaps & opportunities: Long-term UMD interests. Seeking partners & support: your data or model + UMD

algorithms Large projects envisioned

Extended work on HRA dependency (See previous slide) Multi-source data fusion algorithms for HRA

UMD algorithm to combine: SACADA + HuREX + EPRI + OSU + INL + literature + PIF studies +…

“Prior model” full quantification of causal BN version of PHOENIX.

20

Gaps, Opportunities, & Challenges for the HRA community 1. Strong need for consistent terminology & clear definitions of data fields

across data sources – can we start a pathway toward consensus? Implementation gaps

PIF terminology exists – needs to be implemented in data sources, either directly or via implementation of mapping

Clear definitions for each data entry field are needed – differentiates between high value databases and “Garbage In/Out” databases

Research gaps HRA still needs event/task decomposition terminology Need for assessment of types of data which exist & what part of the HRA modeling

problem they address.

2. We need to use all of the data: can we get data sharing agreements in place?

3. We need to discuss modeling & data together – we have models to handle HRA complexity, avoid the urge to oversimplify “and just count” Simulation, simplification, Bayesian methods, causal models, Bayesian Networks

Conclusions: Data-informed insights for HRA Data collection1. New data sources enabled by international collaborations

offer great opportunity for HRA.2. “Data” is not a synonym for “Numbers”

1. The best data sources must capture context & dependency; HFEs are not i.i.d events.

2. HRA also has other data: models, experts, literature;

3. Models are essential for making data useful1. The data sources should be designed to support modeling as much as

possible

4. Data quality >> data quantity We have methods for handling sparse data; but no method exists for the

“garbage in garbage out” problem Having a consistent HRA modeling framework >> getting the “right”

HEP for one single, repeatable simulator event

References1. Groth, Katrina M., Smith, Reuel, & Moradi, Ramin. A hybrid approach to HRA using simulator data, causal models, and

cognitive science. Submitted to RESS, November 2019. In Review.2. Groth, K. M (2018). A framework for using SACADA to enhance the qualitative and quantitative basis of HRA. Proceedings

of the 14th Probabilistic Safety Assessment and Management Conference (PSAM 14), Los Angeles, CA.3. Zwirglmaier, K., Straub, D., & Groth, K. M. (2017) Capturing cognitive causal paths in human reliability analysis with

Bayesian network models, Reliability Engineering & System Safety, 158, 117-129.4. Whaley, A.M. et al., “Cognitive Basis for Human Reliability Analysis,” US Nuclear Regulatory Commission, Washington

DC, NUREG-2114, Jan. 2016.5. Boring, R., Mandelli, D., Joe, J., Smith, C., & Groth, K. (2015). A Research Roadmap for Computation-Based Human

Reliability Analysis. Idaho National Laboratory, INL/EXT-15-36051.6. Chang, Y.J., Bley, D., Criscione L., Kirwan, B., Mosleh, A., Madary, T., Nowell, R., Richards, R., Roth, E. M., Sieben, S., &

Zoulis, A. (2014). The SACADA database for human reliability and human performance, Reliability Engineering & System Safety, 125, 117-133.

7. Groth, Katrina M., Smith, Curtis L., Swiler, Laura P., (2014). A Bayesian method for using simulator data to enhance human error probabilities assigned by existing HRA methods. Reliability Engineering & System Safety, 128, 32-40.

8. Groth & Swiler (2013). Bridging the gap between HRA research and HRA practice: A Bayesian Network version of SPAR-H. Reliability Engineering and System Safety, 115, 33-42.

9. Ekanem, N. J., & Mosleh, A. (2013). Human failure event dependency modeling and quantification: A Bayesian network approach, Proceedings of the European Society for Reliability Annual Meeting (ESREL 2013).

10. Groth, Katrina M., & Mosleh, Ali. (2012). Deriving causal Bayesian networks from human reliability analysis data: A methodology and example model. Proceedings of the Institution of Mechanical Engineers, Part O: Journal of Risk and Reliability, 226, 361-379.

11. Mosleh, A., Shen, S.-H., Kelly, D. L., Oxstrand, J. H., & Groth, K. (2012). A Model-Based Human Reliability Analysis Methodology. Proceedings of the International Conference on Probabilistic Safety Assessment and Management (PSAM 11).

12. Groth & Mosleh (2012). A data-informed PIF hierarchy for model-based Human Reliability Analysis. Reliability Engineering and System Safety, 108, 154-174.

23

Insights for future HRA data collection based on research using HRA dataReminder on topicConclusions: Data-informed insights for HRA Data collectionOutlineSystems Risk and Reliability Analysis lab (SyRRA) Timeline of my HRA data workHRA-relevant data sources I’ve exploredInsights from 15 years of HRA data use – Motivation for my workData-to-model process for “easy” reliability problemsTry to use the “easy” Data-to-model process is causing HRA to failData-to-model process for complex problems (like HRA)Previous work created important pieces of model- and data-informed approach to HRANew: Identified requirements for data-informed HRA modelsNew: Key elements of the multi-data-informed HRA frameworkNew: generalized set of HRA quantification elementsNew: Full Algorithm for “A hybrid approach to HRA using simulator data, causal models, and cognitive science”New: Comprehensive causal maps for PHOENIX (w/ UCLA)New: Draft mapping of SACADA elements to PIFs & FMs to enable quantificationGaps & opportunities: Next steps for UMDGaps & opportunities: Long-term UMD interests.Gaps, Opportunities, & Challenges for the HRA communityConclusions: Data-informed insights for HRA Data collectionReferences