a simulator study evaluating the efficacy of group … simulator study evaluating the efficacy of...

A Simulator Study Evaluating the Efficacy of Group-View Displays in Nuclear Control Rooms

by

Sean William Kortschot

A thesis submitted in conformity with the requirements for the degree of Masters of Applied Science

Mechanical and Industrial Engineering University of Toronto

© Copyright by Sean William Kortschot 2016

ii

A Simulator Study Evaluating the Efficacy of Group-View Displays in Nuclear Control Rooms

Sean William Kortschot

Masters of Applied Science

Mechanical and Industrial Engineering University of Toronto

2016

Abstract

Group View Displays (GVDs), have become prominent features in modern nuclear control room

(NCR) designs. Despite their widespread implementation, an operating experience review by

Myers & Jamieson (2014) revealed that the efficacy of these GVDs has yet to be thoroughly

evaluated. This thesis addresses this gap by presenting the analysis, design, and experimental

results of a full-scale nuclear simulator study designed to evaluate the purported benefits of

GVDs. The results of this study indicate that the current state of the art in NCRs can be improved

upon. We found evidence suggesting that the dominant GVD solution in NCRs was

outperformed in terms of communication by the experimental GVD alternative. Furthermore, we

demonstrated marked situation awareness improvements fostered by the ecological display

framework when compared against the advanced displays currently in use in NCRs. The

implications, limitations, and recommendations for future work from this study are discussed.

iii

Acknowledgments

I would like to thank CANDU Energy Inc. for their financial support and assistance throughout

this project . Specifically, I would like to thank Robert Leger who provided guidance and

assistance from start to finish. This work would not be possible without the help of Rick Bodner,

Brandon Elliott, Richard Brown, and the rest of the team that helped us set up the simulator and

interfaces at CANDU. I would like to also thank Raymond Dufrense, Radik Ixanov, and

Christian Runkowski, whose expertise was unparalleled.

I want to extend my gratitude to my family, who supported me through this entire process.

Special thanks to Cole Wheeler, whose friendship made the long hours at the simulator

enjoyable. Thanks to Billy Myers for laying down the groundwork of this project, Antony

Hilliard who listened to endless questions about everything from CWA to statistical analysis, and

to the rest of CEL. Special thanks to Maya Whitehead, for your friendship, insights, and

conversation.

I would also like to thank Birsen Donmez and Olivier St-Cyr for your assistance and guidance as

my committee members. I would particularly like to thank my supervisor, Greg Jamieson, for

your endless support and guidance. Your assistance and mentorship throughout my degree

showed the structure and patience required for good research.

iv

Table of Contents

ACKNOWLEDGMENTS.........................................................................................................................III

TABLEOFCONTENTS...........................................................................................................................IV

LISTOFTABLES..................................................................................................................................VII

LISTOFFIGURES................................................................................................................................VIII

LISTOFAPPENDICES...........................................................................................................................XI

INTRODUCTION.............................................................................................................................11

1.1 PLANTSAFETYANDEARLYDISPLAYS.......................................................................................................1

1.2 DISPLAYEVOLUTION............................................................................................................................1

1.2.1 DigitizationofControlRooms..................................................................................................2

1.2.2 InterfaceAdvancement...........................................................................................................3

1.3 GROUPVIEWDISPLAYS........................................................................................................................5

1.3.1 LargeScreenDisplays..............................................................................................................5

1.3.2 RedundantDisplays.................................................................................................................6

1.4 SELECTEDLITERATUREONGVDS...........................................................................................................8

1.5 RESEARCHQUESTIONS.........................................................................................................................8

1.6 DOCUMENTSTRUCTURE......................................................................................................................9

PROJECTSCOPE............................................................................................................................102

2.1 THECANDU6®NUCLEARPOWERPLANT.............................................................................................11

COGNITIVEWORKANALYSIS........................................................................................................123

3.1 STAGESOFANALYSIS.........................................................................................................................13

3.2 WORKDOMAINANALYSIS..................................................................................................................14

3.2.1 AnalyticalTool–TheAbstractionHierarchy.........................................................................14

3.2.2 ScopeofAnalysis...................................................................................................................15

3.2.3 ProductsofAnalysis..............................................................................................................17

3.2.4 SummaryofWorkDomainAnalysis......................................................................................22

3.3 CONTROLTASKANALYSIS...................................................................................................................23

3.3.1 AnalyticalTool–TheDecisionLadder...................................................................................23

v



3.3.4 SummaryofConTA................................................................................................................28

3.4 STRATEGIESANALYSIS........................................................................................................................28

3.4.1 AnalyticalTool–InformationFlowMap...............................................................................28



3.4.4 SummaryofStrategiesAnalysis............................................................................................33

3.5 SUMMARYOFCOGNITIVEWORKANALYSIS...........................................................................................33

DISPLAYDEVELOPMENT..............................................................................................................344

4.1 DISPLAYCOMPARISON.......................................................................................................................34

4.1.1 InformationalContent...........................................................................................................35

4.1.2 DisplayNavigation................................................................................................................35

4.1.3 DisplayFidelity......................................................................................................................35

4.1.4 DisplayAppearance...............................................................................................................35

4.2 DEVELOPMENTOFNOVELDISPLAYS.....................................................................................................36

4.2.1 Existingformusage...............................................................................................................37

4.2.2 DevelopmentofNovelForms................................................................................................38

4.2.3 DisplayFinalization................................................................................................................42

EXPERIMENTALMETHOD.............................................................................................................455

5.1 CONTROLROOMMOCK-UP...............................................................................................................45

5.2 DESIGN...........................................................................................................................................45

5.2.1 ExperimentalTasks................................................................................................................47

5.2.2 ExperimentalScenarios.........................................................................................................49

5.3 PARTICIPANTS..................................................................................................................................52

5.3.1 PlantRepresentation.............................................................................................................53

5.3.2 DisplaySimplification............................................................................................................55

5.3.3 ParticipantTrainingPackage................................................................................................56

5.4 MEASURES......................................................................................................................................56

5.4.1 SituationAwareness..............................................................................................................56

5.4.2 Communication.....................................................................................................................59

5.4.3 DiagnosticPerformance........................................................................................................60

vi

5.5 PROCEDURE.....................................................................................................................................61

5.5.1 Training.................................................................................................................................61

5.5.2 ExperimentalTrials................................................................................................................63

5.5.3 Post-TrialInterview&Debriefing..........................................................................................64

EXPERIMENTALRESULTS..............................................................................................................646

6.1 SITUATIONAWARENESS.....................................................................................................................64

6.1.1 InferredModel.......................................................................................................................69

6.1.2 DifferencesofLeastSquaresMeans......................................................................................69

6.2 COMMUNICATION.............................................................................................................................71

6.3 DIAGNOSTICPERFORMANCE...............................................................................................................72

DISCUSSION.................................................................................................................................747

7.1 SITUATIONAWARENESS.....................................................................................................................75

7.2 COMMUNICATION.............................................................................................................................77

7.3 DIAGNOSTICPERFORMANCE...............................................................................................................78

SUMMARY&CONCLUSIONS........................................................................................................798

8.1 SUMMARY.......................................................................................................................................79

8.2 CONCLUSIONS..................................................................................................................................81

8.2.1 Contributions.........................................................................................................................81

8.2.2 Limitations.............................................................................................................................82

8.2.3 FutureResearch.....................................................................................................................83

REFERENCES......................................................................................................................................85

APPENDICES......................................................................................................................................92

vii

List of Tables

Table 1. Relative advantages and disadvantages of GVD alternatives ........................................... 7

Table 2. Summary of Research Questions .................................................................................... 10

Table 3. Five abstraction levels of the AH (adapted from Rasmussen, 1985) .............................. 14

Table 4. Experimental Design ....................................................................................................... 46

Table 5. Example of one team's experimental schedule. .............................................................. 49

Table 6. List of selected faults for experimental scenarios ........................................................... 52

Table 7. Summary of measures ..................................................................................................... 61

Table 8. Number of PO measure queries for each Scenario X Setup ........................................... 65

Table 9. Type III Test for Fixed Effects ....................................................................................... 69

Table 10. Type III Test for Fixed Effects ..................................................................................... 71

Table 11. Type III Test for Fixed Effects ..................................................................................... 73

Table 12. Summary of main findings. .......................................................................................... 75

viii

List of Figures

Figure 1. Large screen displays used in the Qinshan control room. ............................................... 6

Figure 2. Redundant display configuration (Kiran Infosystems, 2014). ......................................... 7

Figure 3. Simplified diagram of the CANDU 6® main systems. .................................................. 12

Figure 4. Overall AH for the CANDU 6 nuclear power plant. Green boxes indicate the systems

included in this report. .................................................................................................................. 17

Figure 5. AH for HTS. .................................................................................................................. 18

Figure 6. AH for the P&IC. .......................................................................................................... 19

Figure 8. AH for the MSS. ............................................................................................................ 21

Figure 9. AH for the turbine system. ............................................................................................ 22

Figure 10. Typical decision ladder structure (Jenkins et al., 2010). ............................................. 24

Figure 11. DL for active search for fault without an alarm. ......................................................... 26

Figure 12. DL representing an active search for a deviation with the guidance of an alarm. ...... 28

Figure 13. IFM depicting the strategies used for searching for a fault without an alarm. ............ 30

Figure 14. IFM depicting the different strategies for searching for a fault with an alarm ............42

Figure 15. IFM depicting the different strategies within the task of online diagnosis of a fault. . 32

Figure 16. Unusable graphical forms ............................................................................................ 36

Figure 17. Mass-Energy-Saturation Ecological form ................................................................... 37

Figure 18. Rearrangement of existing charts. ............................................................................... 38

Figure 19. Mass-balance plots ...................................................................................................... 39

ix

Figure 20. Delta plot showing current inflow balancing with current outflow and the level at the

setpoint. ......................................................................................................................................... 40

Figure 21. Four quadrants of the delta plot ................................................................................... 41

Figure 22. Delta plot with trend information ................................................................................ 42

Figure 23. Ecological Critical Safety Parameter screen. .............................................................. 43

Figure 24. Ecological Plant Operations screen. ............................................................................ 43

Figure 25. Ecological Steam Generator screen. ............................................................................ 44

Figure 26. Ecological Primary Heat Transfer screen. ................................................................... 44

Figure 27. Schematic of the CANDU mock-up ............................................................................ 45

Figure 28. GVD configurations ................................................................................................... 46

Figure 29. Display types .............................................................................................................. 47

Figure 30. Operators performing primary monitoring task in the LSD configuration. ................ 48

Figure 31. Operators performing their secondary distractor task in the redundant display

configuration. ................................................................................................................................ 49

Figure 32. Scenario timelines. ...................................................................................................... 52

Figure 33. Final diagram presented to participants. ...................................................................... 55

Figure 34. Typical process overview query. ................................................................................. 57

Figure 35. Example of how recently was defined. ........................................................................ 58

Figure 36. Visible vs. Inferred means. .......................................................................................... 66

Figure 37. PO measure means for each scenario across the four conditions ................................ 67

Figure 38. Each group's PO measure scores across the four setups. ............................................ 67

x

Figure 39. Individual subjects' PO measure scores over the different setups. .............................. 68

Figure 40. LS Mean score for each experimental setup. ............................................................... 69

Figure 41. Multiple comparisons diffogram showing no significant contrasts. ........................... 70

Figure 42. Communication scores across the four experimental setups. ...................................... 72

Figure 43. Diagnostic performance mean estimates by experimental setups. .............................. 73

Figure 45. Difference in performance between the two display configurations. .......................... 74

Figure 46. Differences between advanced and ecological graphics. ............................................ 77

xi

List of Appendices

Appendix A. Information Requirements ....................................................................................... 92

Appendix B. Experimental Schedule ............................................................................................ 95

Appendix C. Process Overview Queries ....................................................................................... 96

Appendix D. Selected Statistical Outputs ................................................................................... 100

Appendix E. Selected SAS Code ................................................................................................ 101

1

Introduction 1Nuclear power generation represents a viable, stable method for producing carbon-free energy

and currently accounts for over 11% of global power supply (World Nuclear Association, 2016).

Unfortunately, accidents in the past such as Three-Mile Island, Chernobyl, and Fukushima, as

well as the initial war-related motives behind its development have tainted the global perception

of nuclear energy (Nuclear Energy Agency, 2010). In order to move past this public perception

and ensure the safe operation of this carbon-free resource, designers and engineers must

minimize the risk of operating the plants without compromising their functional purpose of

large-scale power production.

1.1 Plant Safety and Early Displays

From the beginning of nuclear power generation in the early 1950s, engineers were conscious of

the potential risks involved with fission power (World Nuclear Association, 2014). This

motivated the design and development of robust physical systems capable of containing the

radiation inherent to nuclear power generation. To ensure the proper operation of these physical

systems, engineers also needed to develop equally robust control and monitoring systems (IAEA,

2009).

The monitoring systems used to run early generation nuclear power plants were quite crude by

today’s standards, being composed of hardwired dials and gauges (IAEA, 2009). The human-

factors issues with these systems were first noticed after the Three-Mile Island incident in 1979,

wherein human error in monitoring allowed for coolant to boil away thereby causing the reactor

core to remain exposed for approximately fourteen hours (Joyce & Lapinsky, 1983).

Investigators concluded that these monitoring errors were largely the result of ineffective

displays, stating that the information on these displays was presented too disparately for the

operators to assemble meaningful mental models of the plant state.

1.2 Display Evolution

The findings from Three-Mile Island led to design recommendations for future safety parameter

displays. Investigators determined that to safely monitor plant operations a display must facilitate

(Joyce & Lapinsky, 1983):

2

1. Developing accurate and complete mental models of plant parameters,

2. Integrating information from dispersed areas of the control room,

3. Remembering that gathered information for later comparisons, and

4. Integrating all of this information to update the mental model of the plant.

Designers began to realize that presenting the values of individual parameters in separate

locations was not conducive to operators’ understanding of the behaviour of the plant (Joyce &

Lapinsky, 1983). In other words, simply ensuring that all of the requisite information is available

on the display is not sufficient. This can result in unnecessary taxation of the operators’ mental

resources, thereby inhibiting adequate situation awareness.

To ensure the presentation of sufficient content and suitable configuration of that content, the

Electric Power Research Institute put forth a three-step procedure for the development of

displays in control rooms (Beltracchi, 1988):

1. Task and functional analysis to ensure that the information required to perform each task

and subtask was available,

2. Synthesis, which is the process of determining appropriate display formats for the

information from the analysis phase, and

3. Evaluation, which is the rigorous assessment of the effectiveness of the display formats

developed in the synthesis phase

Of the three-phase program, the most innovative was the synthesis stage. Prior to the

implementation of Cathode-Ray Tube (CRT) displays, the constraints imposed by the hardwired

dials and gauges limited display synthesis. Advancing CRT and digital display technology

presented new opportunities in the control room to not only display information, but to make that

information more accessible to operators (Murch, 1984). They opened up the possibility of using

colour, active bar charts, and trends to relate meaningful information to operators (Beltracchi,

1988). Furthermore, they allowed for active-mimic displays to be used to show processes

occurring in real-time within the plant.

1.2.1 Digitization of Control Rooms

As display technology continued to develop, nuclear control rooms (NCRs) moved further away

from analog displays and control systems in favour of digital solutions (IAEA, 2009). As

3

discussed in the previous section, the first stage of this transition was the implementation of CRT

displays within the analog panels. The next phase of this transition saw the continued digitization

of many processes in the control room.

Digitized processes include emergency operating procedures, advanced alarm systems, graphic

display systems, and intelligent operator support (Roth & O’Hara, 2002). This shift was

motivated by the belief that reallocating these processes to the computer would lead to improved

control system performance in terms of accuracy, computation, increased capacity for data

handling, and storage (Lin, Yenn, & Yang, 2010). These improvements were meant to take some

of the load off of the operators allowing them to focus on important plant behaviours and thus

created more usable systems.

The transfer of many control room processes from analog to digital necessitated a parallel

transition in display technology. Operators needed to be able to monitor and control these newly

digitized processes, which led to the implementation of software interfaces, touch screen

controls, overview displays, and individual operator consoles (O'Hara, 2004).

1.2.2 Interface Advancement

With the newly ubiquitous software interfaces, designers found new areas of latitude for the

development and growth of display technology. They already knew that simply presenting the

necessary information to operators was insufficient (Joyce & Lapinsky, 1983) and therefore

focused their attention on improving the way that this information was being presented.

Traditional displays marked the first generation of software interfaces in NCRs. These displays

employed a mimic framework, wherein information presentation is based on the physical layout

of the plant (Lau, Jamieson, Skraaning, & Burns, 2008). This was the same framework that was

used on the hardware interfaces wherein dials and gauges would be mapped to the hardware

panel in accordance with the physical layout of the plant. Essentially, traditional displays

digitized the methods used on the hardwired panels without exploring the potential benefits

presented by the new technology.

The next stage in the evolution of display technology was initiated by the realization that

traditional displays were not taking full advantage of the available technology. To move past

this, the traditional mimic displays were supplemented with novel configural graphics such as

4

trend charts and other plots. The combination of these two graphical methods represented the

central characteristic of advanced displays (Kim & Kim, 2014). Advanced displays employ the

original mimic-style interfaces, but add configural graphics and new visualization methods to

illustrate how parameters change over time and in relation to one another (Bennett & Malek,

2000; Lau et al., 2008).

Mimic displays are based on the belief that they will help to develop and maintain operators’

mental models of plant operations (Jamieson & Miller, 2007). These have been found to be

useful for novice operators, who are still developing their mental models (Butcher, 2006; Gary &

Wood, 2011). However, as operators gain more experience and further their understanding of

their respective systems, their need for a mental-model-facilitating display is reduced since their

mental models have already become fully developed (Jenkins, Stanton, Salmon, Walker, &

Rafferty, 2010; Rasmussen, 1993; Zhang, 2008). This therefore compromises the utility of the

mimic aspect of advanced displays, especially in the nuclear domain where plants are run by

exclusively expert operators with fully formed mental models (Lee & Seong, 2009; Vicente,

Roth, & Mumaw, 2001).

This shortcoming of mimic-displays for expert operators gave rise to Ecological Interface

Design (EID), whose foundation is built on grouping functionally related information in cohesive

graphical forms to support the presentation of psychologically relevant information (Vicente &

Rasmussen, 1992). By doing this, ecological interfaces reduce the need for mimic-displays

unless some aspect of that mimicry is tied to the functional purpose of the system (e.g., process

sequences; Bennett & Malek, 2000). Ecological interfaces have demonstrated marked

improvements for both situation awareness and operator performance in simulator studies in the

nuclear domain (e.g., Burns et al., 2008), the hydrocarbon industry (e.g., Tharanathan et al.,

2012), and several others. Furthermore, ecological interfaces have demonstrated performance

improvements during unanticipated events (Lau et al., 2008). In spite of these performance

benefits, EID has yet to be fully integrated into NCRs (Vicente, 2002). In order to shift display

frameworks away from the status quo, research needs to continue evaluating the demonstrated

benefits of EID within full-scale simulator studies.

5

1.3 Group View Displays

Although the need for research evaluating the merits of different display frameworks has been,

and continues to be important, studying the configuration of the displays on which the

frameworks are presented is equally pressing. One of the main areas of this development focuses

on the different methods for presenting information to multiple operators within the control room

(Myers & Jamieson, 2014).

Group View Displays (GVDs) represent a recent trend in control room display technology that

was only made possible through the aforementioned digital evolution. GVDs are a class of

display configurations that enable the same information to be viewed by multiple operators in

different areas in the control room through a common point of reference (O’Hara, Brown, Lewis,

& Persensky, 2002). The motivation behind GVDs revolves around their believed improvements

towards operator situation awareness, communication between operators, and overall

performance (Roth et al., 1998). Although they come in multiple forms, the two most prominent,

are Large-Screen Displays and redundant displays.

1.3.1 Large Screen Displays

Large-Screen Displays (LSDs) are large (~100 X 100” or greater), wall mounted displays that are

typically positioned at the front of control rooms (Figure 1; Myers & Jamieson, 2014). They

achieve the GVD objective by locating the common point of reference at the same position in the

control room. This method therefore allows for the different operators viewing the LSD to be

looking not only at the same information, but also at the same geographic location within the

control room.

6

Figure 1. Large screen displays used in the Qinshan control room.

A key advantage of LSDs is that they are visible from almost anywhere in a control room (Roth

et al., 1998). Therefore, if an operator needs to check something at the panel, they don’t lose

sight of the GVD. However, the increased size of LSDs also introduces necessary tradeoffs.

LSDs require operators to be positioned further away, which increases the overall footprint of the

control room and therefore also the cost (Myers & Jamieson, 2015).

Although LSDs represent only one form of GVD, they have become the dominant technology

implemented in NCRs to achieve the GVD purpose (Myers & Jamieson, 2014). While the

posited benefits of LSDs are intuitive, there are viable alternatives such as redundant displays

that may be able to yield the same, or better results in terms of situation awareness,

communication, and operator performance.

1.3.2 Redundant Displays

Redundant displays are the primary GVD competition to LSDs in NCRs (Myers & Jamieson,

2014). Redundant displays are individual multi-display arrays that are located in front of each

operator’s work station. The arrays typically have a predetermined subset of screens dedicated to

the GVD and the content of these screens is identical for each operator. Therefore, the operators

still have common points of reference with which to look at matching information, but those

common points of reference are replicated in different spatial locations in the NCR. For example,

if an array consisted of four displays configured in a 2 X 2 arrangement, the top two displays

may fulfill the GVD function of the array, meaning that they would be the same for each

7

operator. This means that for both operators, their top two panels will always be identical,

thereby enabling the individual operators to always have a common point of reference with

which they can discuss information about the plant. Essentially, redundant displays relocate the

LSDs from the front wall of the NCR to the individual operator work stations. Figure 2 illustrates

this concept.

Figure 2. Redundant display configuration (Kiran Infosystems, 2014).

Redundant displays allow for smaller control rooms relative to LSDs since the operators do not

need to be located as far away from their screens. However, redundant displays are typically not

visible from all locations in the control room. This means that an operator may lose sight of the

GVD if he or she needs to check something at one of the panels. A summary of some of the

benefits and pitfalls of both GVD options is presented in Table 1.

Table 1. Relative advantages and disadvantages of GVD alternatives

GVD Type Advantages Disadvantages Large Screen Display Visible from any location in

the control room Increases the footprint of the control room

Redundant Display Allows for a smaller control room since

Not visible from any point in the control room

8

1.4 Selected Literature on GVDs

In two reports, Roth et al. (1997, 1998) assessed the benefits of LSDs under different display

frameworks. They found that the SA, workload, and performance benefits resulting from the

addition of functional information outweighed the costs of sacrificing extra screen space.

Although these results shed light on the possible benefits of functional information when

presented on wall-mounted displays, there is still a great deal to be learned with respect to

display frameworks and GVD alternatives. For example, these studies did not assess the

behaviour differences evoked from GVD alternatives, as they only studied LSDs. Furthermore,

they did not examine the interaction between display framework and GVD type to determine if

certain pairings between GVD and framework are superior to others. Finally, the statistical

evidence provided to support their claims regarding LSDs and the presentation of functional

information is scarce. For example, no F or t values are presented, nor are the details of their

functional displays. In spite of these limitations, their evidence converges with the body of

literature supporting functional displays (e.g., Burns et al., 2008) and provides a foundation for

new questions to be asked.

1.5 Research Questions

The literature described above describes the various options that are available to monitor a

nuclear power plant. This highlights two key areas of concern with respect to the current state of

the art in nuclear control rooms. The first is that the favouring of LSDs over redundant displays

appears to have no empirical backing (Myers & Jamieson, 2014). The second is that in spite of

the widespread support for ecological displays, advanced displays still represent the dominant

display framework used in nuclear control rooms (Lau et al., 2008).

This project aimed to address these two gaps in the literature with the underlying motivation of

assessing whether or not there is any empirical evidence supporting the current state of the art in

nuclear control rooms. To achieve this goal, we had two research questions:

1. Is the widespread favouring of LSDs over redundant displays rooted in improvements in

SA, communication, and overall performance?

9

2. Is the widespread implementation of advanced displays over other display frameworks

rooted in improvements in SA, communication, and overall performance?

We therefore set out to conduct a full-scale simulator study that would answer both of these

questions. The experiment included both LSDs and redundant displays, and both advanced and

ecological display frameworks. This 2 X 2 study included the necessary conditions for

addressing the gaps in the literature and allowed us to assess the empirical validity of the current

state of the art in nuclear control rooms.

1.6 Document Structure

This document outlines the analysis and design leading up to a summative experiment. It then

details the experimental design, training, method, and results of that experiment. Because a

comprehensive literature review (Myers & Jamieson, 2014) had already given rise to the research

questions, a detailed literature review is not included. This document therefore consists of five

main sections: analysis, design, experimentation, results, and discussion.

Chapter 2 provides details on the project’s scope. It summarizes some of the methods and

constraints that were involved in answering the two experimental questions mentioned above. It

also provides a brief overview of the domain of our analysis and experiment, the CANDU 6®

nuclear power plant.

Chapter 3 describes the Cognitive Work Analysis (CWA) that was conducted on the system. s

Although the individual phases of CWA are conducted for distinct purposes, each contributes to

attaining a more comprehensive understanding of the system. Our CWA analyzed the entirety of

a system and therefore proved to be a useful method for establishing the requirements necessary

for achieving our experimental goals within that system.

Chapter 4 details the design process and outcomes for the novel ecological interfaces. As

described above, we needed to create novel ecological interfaces against which to compare the

existing advanced displays. This stage was undertaken in service of the second experimental

question rather than to advance ecological interface design. We therefore relied as heavily as

possible on existing ecological forms as well as the current graphical suite in the CANDU

developer software.

10

Chapter 5 describes the experiment that was aimed at answering the two main research

questions. This includes descriptions of the experimental design, participant training, display

implementation, scenario design and implementation, and conducting the experiment.

Chapter 6 details the results of the study. This section encompasses the statistical methodology

and findings from the measures that were taken during experimentation.

The final portion of this document discusses the findings, limitations, and implications of this

study. Theories are presented that attempt to explain these findings with respect to the relevant

limitations.

Project Scope 2This research aimed to answer the two research questions through a full-scope simulator

experiment. We ran operators through four different display setups, each representing a unique

combination of display configuration and display type1. The four setups are summarized in Table

2 below.

Table 2. Summary of Research Questions

Question 1

Display Configuration

LSD Redundant

Question 2 Display Type

Advanced LSD-ADV RED-ADV ADV Ecological LSD-ECO RED-ECO ECO

LSD RED

The first research question asked what display configuration yields higher levels of operator

situation awareness, communication, and performance. To evaluate this we compared these three

dimensions across LSD and the redundant display configurations. Unfortunately, due to time and

resource limitations, we did not have a baseline condition wherein no GVD was present.

Including a baseline without any shared information screens would have contributed towards

1 The terms presented here to represent the different conditions will be used throughout the document. Display

Configuration refers to the LSD vs. redundant dimension. Display Type refers to the advanced vs. ecological dimension. Display Setup refers to the unique coupling of display configurations and display types (e.g., LSD-ADV, LSD-ECO, RED-ADV, & RED-ECO).

11

evaluating the GVD concept as a whole rather than evaluating the benefits of different

alternatives. Although this was primarily due to time constrictions, this was also motivated by

the fact that it has become an expectation that NCRs will satisfy the GVD requirement (O’Hara

et al., 2002). GVDs have thus become the standard not only in nuclear, but in many other process

control domains as well (Myers & Jamieson, 2014). Therefore, although this baseline condition

would have been helpful for either affirming or refuting the GVD concept, it would not represent

a state in the control room that exists in today’s NCR climate.

The second research question, asking what display type best elicits the postulated benefits of

GVDs, needed to be answered through a comparison of two distinct display frameworks. The

baseline condition for this question was the advanced displays that currently represent the state

of the art in the NCR (Lau et al., 2008). The comparison chosen to compare these against was

ecological displays. These were chosen because they fundamentally differ in the way that they

present information by focusing on depicting functional relationships between parameters

wherever possible. Therefore, ecological displays represent a framework that is sufficiently

different from advanced displays by simply altering the form rather than the content of the

information presented. This is particularly important for a valid comparison study because

maintaining the informational content across conditions allows for any performance differences

to be solely attributed to the different frameworks rather than different access to information

(Christoffersen, Hunter, & Vicente, 1996; Maddox, 1996).

2.1 The CANDU 6® Nuclear Power Plant

We conducted our analysis on a CANDU 6® advanced nuclear reactor. The reactor uses heavy

water (D2O) coolant, which allows for natural uranium to be used for fuel as opposed to

enhanced uranium. This enables online refueling, which reduces the frequency with which the

system has to be taken offline. A simplified block diagram of the system and its constituent

subsystems is presented in Figure 3.

12

Figure 3. Simplified diagram of the CANDU 6® main systems.

Cognitive Work Analysis 3Cognitive Work Analysis (CWA) is a framework used to systematically analyze complex

sociotechnical systems like the CANDU 6® nuclear power plant (Rasmussen, Pejtersen, &

Goodstein, 1994; Vicente, 1999). CWA analyzes the different classes of constraints that shape

13

work within a particular domain (Elix & Naikar, 2008). These classes of constraints represent the

key factors involved in governing the operations of a particular system.

Our CWA was conducted for three reasons. The first purpose was to understand and analyze the

system in order to develop tools and products that can improve the monitoring and operations of

those systems (Elix & Naikar, 2008). The second reason was to provide a structured approach for

the development of our own understanding of the system. Since both myself and my research

partner were new to the nuclear domain, a framework for studying the system that followed a

logical and proven structure was necessary (Jamieson & Miller, 2007). The third reason, which is

tied to the second, was to determine a suitable scope for the rest of the project. By fostering a

comprehensive understanding of the system, we were able to effectively use CWA to determine

what would be required in the design and experimentation phases in order to answer the research

questions.

3.1 Stages of Analysis

Although CWA has a canonical structure that consists of five phases, it also allows for

customization according to the specific demands of the analysis being conducted (Elix & Naikar,

2008; Vicente, 1999). This allows researchers to tailor the framework to fit the specific needs of

the project.

This study employed the first three phases of CWA: Work Domain Analysis (WDA), Control

Task Analysis (ConTA), and Strategies Analysis (StrA). These phases progress by initially

analyzing the physical and functional constraints of the system in WDA, then the control tasks

that operators use when operating according to these constraints in ConTA, and finally by

decomposing the different strategies that can be employed while performing these tasks in StrA

(Vicente, 1999). The final two phases, Social Organization and Cooperation Analysis, and

Workers Competencies Analysis were not formally conducted.

There were two clear task classes within our experimental scope: 1) Plant automation in response

to changing scenario characteristics, and 2) Operator observation and diagnosis. Since no control

actions were being performed, it was not necessary to analyze the functional allocation of these

tasks. This rendered Social Organization and Cooperation Analysis unnecessary for this

particular project. Workers Competencies Analysis was not omitted on the basis of it being

14

unnecessary, but rather it was integrated in all other phases of the CWA. We knew that the

eventual experimental participants would be students who were unfamiliar with the nuclear

domain. We therefore used the first three phases of CWA to determine exactly what areas of the

system to focus on in design and experimentation, based on the anticipated competencies of our

eventual participants. This method was also used during the experimental and display design

phases of the study, thereby encompassing an informal version of Workers Competencies

Analysis.

3.2 Work Domain Analysis

The first stage of CWA is Work Domain Analysis (WDA). WDA identifies purposive,

functional, and physical constraints of a particular system (Vicente, 1999). These constraints are

identified through the use of abstraction hierarchies.

3.2.1 Analytical Tool – The Abstraction Hierarchy

Abstraction hierarchies (AHs) trace the high level functional purposes of a system down to its

physical components through five levels of abstraction (Rasmussen, 1985). The elements in the

AH are connected via means-ends relationships, with elements at lower levels being the means to

achieving the ends that they are connected to at higher levels. By doing this, the AH illustrates

the role that each component serves within the higher-level system purposes. Descriptions of

each level as well as examples within the CANDU® plant are presented in Table 3.

Table 3. Five abstraction levels of the AH (adapted from Rasmussen, 1985)

Level Definition Example Abstraction Functional purpose

Objectives of the system Generate electricity How an element

is achieved

ê é

What an element achieves

Abstract function

Underlying laws and principles governing the system

H2O enthalpy

Generalized function

Standard functions and processes Heat transportation

Physical function

Functional descriptions of individual components

Heat transport system

Physical form The physical characteristics of individual components

Figure 8 configuration

15

The physical form level of the AH describes the physical characteristics of the individual

components in terms of appearance and configuration within the plant. The main concern with

respect to nuclear operation at the physical form level is the configuration of the systems within

the plant, especially for the novice operators who are still developing their mental model of the

operations (Butcher, 2006; Gary & Wood, 2011). Therefore, rather than including typical

physical form information in the AHs, we used a physical form schematic of the plant (see

Figure 3) to represent the bottom layer of each of the systems that were analyzed.

We used a somewhat unique approach to the abstract function level of our analysis by adopting

the Source, Store, Sink nomenclature. This is adopted from Reising and Sanderson (2002) and

describes the mass and energy balances of a system in terms of where the energy or mass came

from (the source), where mass or energy is contained in a system (the store), and how that mass

or energy is removed from a system (the sink). This nomenclature is well-suited for closed

systems like the CANDU 6® power plant since virtually all mass and energy in and out of the

system can be accounted for.

3.2.2 Scope of Analysis

The purpose of the CWA and its constituent stages was to answer the two research questions.

Therefore, the scope of the WDA needed to include the systems that would be necessary for

achieving this goal. This afforded the freedom of removing certain systems from the analysis and

thus also from the final representation of the plant.

A system was deemed eligible for removal if its removal did not impact the understanding of the

rest of the system. This meant that all systems in the primary and secondary loops of the system

had to be included in the plant representation (see Figure 3). This is because understanding each

of these systems, particularly at the abstract function level, requires an understanding of the input

and output systems that are connected to it.

We therefore focused our analytic efforts on the systems that nuclear operators tend to pay the

most attention to. These are outlined in Davey, (2000) as:

1. The nuclear reactor

2. The heat transport system (HTS)

3. The pressure and inventory control (P&IC)

16

4. The four steam generators (SG)

5. The main steam system (MSS)

6. The turbine

7. The feedwater system

The main systems that we eliminated from analysis were the moderator system, the refueling

system, and most importantly, the power grid. Again, the accurate understanding of how these

systems operated was unnecessary for understanding how the rest of the systems work together

to achieve the overarching goals of the plant.

Once we had identified the seven key systems in the plant, we determined the level of specificity

at which these systems would be represented. The most influential constraint when considering

this was the skills and competencies of our eventual experimental operators, which illustrates the

integration of Workers Competencies Analysis in the WDA. Due to limitations with respect to

access to either current or former nuclear operators, students from Lambton College’s Chemical

Production and Power Engineering courses represented the eventual participants. These students

had experience with other process control systems, but had no experience in the nuclear domain.

Because of this, and because we would have a limited window in which to train the operators,

the system did not need to be represented at its most detailed level.

We therefore reduced the plant to a level of specificity whereby all of the core functionalities of

each of the seven constituent systems was captured. For example, if a valve was integral to the

operation of a system, it was included in the analysis. However, if there were multiple

redundancies in the design of these valves, some were removed from our representation. The

diagram presented in Figure 3 illustrates the level at which the plant was represented in the

WDA.

It is important to note that as we approached the actual experiment, the PFn diagram was

significantly simplified to meet the specific constraints of our participants and limited training

window. Therefore, the WDA and resulting AHs presented in this section are more detailed than

what the participants ended up being trained on.

Although this analysis can be considered excessive, it contributed to the successful completion of

the project in two key ways. Firstly, initial over-analysis affords the freedom to selectively

simplify in subsequent stages. Secondly, the initial over-analysis bolstered our own

17

understanding of the dynamics of the system, which allowed us to assess what aspects of the

plant were necessary in the training regimen.

The following sections present the AHs for the individual systems that I analyzed. The

functionality of each system is briefly outlined in sufficient detail to contextualize later

descriptions of the experiment. Rather than presenting the lowest level of the AH, the physical

form level, diagrams are shown to illustrate the general structure of the systems.

3.2.3 Products of Analysis

The following sections consist of the final products from the WDA that I conducted for several

of the main subsystems of the CANDU 6® plant. Figure 4 presents the overall abstraction

hierarchy for the CANDU 6® plant. The systems highlighted in green represent the systems that I

analyzed while those with grey coloring show the systems that my partners analyzed. Because a

description of the work conducted on these systems is not necessary for understanding the

remainder of the paper, I have not included their AHs in this section.

Figure 4. Overall AH for the CANDU 6 nuclear power plant. Green boxes indicate the

systems included in this report.

18

3.2.3.1 Heat Transport System

The Heat Transport System (HTS) is responsible for circulating heavy-water coolant (D2O)

through the reactor and taking it to the steam generators where it transfers its heat to H2O. Once

it has transferred its heat to the H2O it is circulated back through the reactor where it regains its

heat. This process is continuously repeated while the plant is running. Figure 5 presents the AH

for the HTS.

Figure 5. AH for HTS.

3.2.3.2 Pressure & Inventory Control

The Pressure and Inventory Control (P&IC) system maintains the heavy water system pressure

at a setpoint throughout a variety of operating modes. The P&IC is principally controlled through

the pressurizer, which is a large tank connected to the primary circuit of the HTS with a large

pipe whose direction of flow is determined by pressure differential between the HTS and the

pressurizer. If the pressure in the HTS drops below setpoint, heaters in the pressurizer activate,

19

increasing the heat and therefore pressure in the pressurizer. This forces inventory into the HTS

thereby increasing the pressure in the HTS. If the HTS is above its setpoint, vents above the

pressurizer open, releasing pressure to a secondary tank. This causes inventory to flow from the

HTS to the pressurizer thereby reducing the pressure in the HTS. Figure 6 presents the AH for

the P&IC system.

Figure 6. AH for the P&IC.

3.2.3.3 Steam Generator

There are four steam generators (SGs) in the CANDU 6® plant. These serve as the link between

the primary and the secondary sides of the system by acting as a heat transfer. Hot D2O flows

into the primary side of the SGs where it goes through an inverted U-tube bundle. Liquid H2O

surrounds the tube-bundle, receiving the heat energy and vapourizing. The H2O steam then

leaves the SGs and flows towards the turbine. Figure 7 presents the AH for the SG.

20

Figu

re 7

. AH

for

the

SG.

21

3.2.3.4 Main Steam System

The Main Steam System (MSS) is responsible for transferring dry steam leaving the SGs to the

turbine. It has a series of valve-sets called the atmospheric discharge valves (ASDVs) and the

main steam safety valves (MSSVs), which vent steam to the atmosphere in conditions where the

pressure in the steam lines or the turbine is higher than the setpoint. A third type of valve, the

condensate steam discharge valve (CSDV), reroutes steam to the condenser, thereby bypassing

the turbine. This valve is used when the turbine is producing excess energy or in shutdown

scenarios. The electricity being generated by the turbine will decrease if any of these valves are

opened. Figure 8 presents the AH for the MSS.

Figure 8. AH for the MSS.

3.2.3.5 Turbine System

The turbine receives hot, dry steam from the MSS and converts it to rotational energy. The

turbine system in the CANDU 6® plant has one high-pressure (HP) turbine and two low-pressure

(LP) turbines. The HP turbine is the first in the sequence and takes a large portion of the energy

out of the steam. Steam leaves the HP turbine and is directed to the first LP turbine. Before it

22

reaches the LP turbine it goes through a reheater, which is powered by extraction steam taken

directly out of the steam generators. The steam passes through the first LP turbine and the

process is repeated before it reaches the second LP turbine. Upon leaving the second LP turbine

the steam has lost the vast majority of its energy. It is then fed into the feedwater system where it

is condensed and recycled back into the steam generators. Figure 9 presents the AH for the

turbine system.

Figure 9. AH for the turbine system.

3.2.4 Summary of Work Domain Analysis

Work Domain Analysis decomposed the CANDU 6® plant, illustrating how each subsystem and

their constituent components contribute to the plant’s functional purpose of consistent and

reliable electricity generation. Not only did it provide the basis for developing a comprehensive

set of information requirements (see Appendix A), but it also acted as a filter, revealing

important insights with respect to what systems were necessary for inclusion, and what systems

could be removed. This filtering limited the potential for unnecessary design and training.

23

3.3 Control Task Analysis

The aim of Control Task Analysis (ConTA) is to identify the recurring tasks that operators need

to perform within a work domain (Vicente, 1999). These tasks are examined independently of

who they are performed by and are analyzed according to the sequence of data-processing

activities and their resulting states of knowledge. Therefore, ConTA identifies and describes the

cognitive activities that occur while an operator performs a task.

3.3.1 Analytical Tool – The Decision Ladder

Rasmussen (1974) developed ConTA to trace the cognitive activities that transform an initial call

to action, to a comparison between the current system state against the targeted system state, to

the system manipulation that will achieve that target. The tool that he developed to do this was

the Decision Ladder (DL). DLs are structured flow diagrams that alternate between cognitive

activities and their resulting states of knowledge. The typical structure of a DL is presented in

Figure 10.

24

Figure 10. Typical decision ladder structure (Jenkins et al., 2010).


ConTA identifies the recurrent control tasks that are performed within an operating environment

(Kilgore, St-Cyr, & Jamieson, 2008). Unfortunately, because novice operators would be used

during experimentation, control actions were omitted from the experimental design. This was

largely based on subject matter expert recommendation. Including control actions would have

complicated the scenarios too much to elicit accurate performance data for our novice operators

25

while also increasing the complexity of the training regimen. Therefore, a new application of

ConTA, within this study’s operating environment, needed to be found.

There were two classes of tasks within the scope of the present study. The first class includes

search behaviour while the second includes diagnostic processes. Since control actions fell

beyond the present study’s scope, the DLs essentially stop after the comparison between the

current state and the target state thereby removing the bulk of the second leg of the DL. This

stretch of the second leg of the DL, which covers definition of task, task, planning of procedure,

procedure, and execution is most helpful in domains with loosely defined tasks (Jenkins et al.,

2010). In the nuclear domain, where every task is regulated and every procedure is carefully

defined, this portion of the DL provides less value. Therefore, omission of this leg was deemed

suitable given the characteristics of the present study.

Within the high level class of search tasks, there were two categories: 1) Active search without

an alarm, and 2) active search with an alarm. Examining these two tasks independently revealed

the differences and similarities between them. This stage was informative for the experimental

design in that it revealed the underlying behaviours involved in plant operations which could

therefore be examined.

Diagnostic behaviour, the second type of action our participants would need to perform, was

omitted from this phase of the CWA. The decision ladder did not appear to be a suitable tool for

modeling this task, and therefore we focused more heavily on applying strategies analysis to

characterize diagnosis.


3.3.3.1 Active Search for Deviation (without alarm)

Searching for a system fault without the instigation and guidance of an alarm requires operators

to scan their displays to determine if any parameter or group of parameters had significantly

deviated from setpoint. Although there are typically advanced alarm systems integrated into the

CANDU instrumentation and control systems, there are still instances where the alarms either

fail to indicate the specific system where there is a deviation, or fail to indicate the breadth of the

problem. One of the reasons that this can occur is because alarms are presented in a list format,

which requires the operator to perceive individual alarms piece them together to determine the

26

problem. Therefore, regardless of the presence of an alarm system, a critical task within nuclear

monitoring involves searching for symptoms within the full array of parameters. Figure 11

illustrates this process in a decision ladder.

Figure 11. DL for active search for fault without an alarm.

As the above figure illustrates, many stages of the second leg of the decision ladder are bypassed

since there is no new procedure development during operation at a power plant. Instead,

operators would detect something and immediately jump to the appropriate procedure to respond

to that issue.

27

3.3.3.2 Active Search for Deviation (with alarm)

We analyzed this task for the purpose of contrasting the task of searching for a fault without the

aid of alarms. The analysis revealed that the only major differences between the two tasks are at

the activation and alert stages. In this task, the operators are called to action via the aid of the

alarm system rather than simply noticing a significant deviation in a parameter manually. After

this, the two tasks converge since the alarm system provides minimal guidance with respect to

how best to respond to a situation. The decision ladder for active alarm search is presented in

Figure 12.

28

Figure 12. DL representing an active search for a deviation with the guidance of an alarm.

3.3.4 Summary of ConTA

Understanding control tasks represents a key step towards understanding the nature of operations

within a complex sociotechnical system. Our ConTA helped us to identify the two main classes

of control tasks within our experimental scope: Search and diagnosis. The ConTA highlighted

the areas of operation that require the most support from an interface. This therefore informed

the display and the experimental design process by showing not only what areas to focus on

designing for, but by also revealing what areas are the most important to evaluate in an

experiment.

3.4 Strategies Analysis

The third and final stage of our CWA was Strategies Analysis (StrA). StrA illustrates the

different strategies that can be used to accomplish a control task (Vicente, 1999). Strategies, as

defined by Rasmussen (1981), are a series of actions that transform an initial state of knowledge

into a final state. Because there are a multitude of strategies for any given task, and each strategy

can be influenced by many factors, it is difficult to identify highly specific strategies. Instead,

StrA aims to identify generic strategy classes for the purpose of facilitating the design of displays

capable of supporting these strategies (Kilgore et al., 2008).

3.4.1 Analytical Tool – Information Flow Map

Information Flow Maps (IFMs) are graphical representations of the different strategies that an

operator can take to achieve a certain task (Vicente, 1999). IFMs have been presented in many

different ways in the past. Some studies have presented each distinct strategy as an individual

IFM (e.g., Kilgore et al., 2008). Others have shown all strategies on a single IFM to illustrate

where the individual strategies diverge from one another (e.g., Cornelissen, Salmon, Jenkins, &

Lenne, 2013). While there are merits to each method, unified IFMs are more capable of

illustrating the points where strategies can diverge from one another.

We adopted the “question notation” from the Jenkins et al. (2010) work on decision ladders. This

notation describes how a situation is assessed according to the presence or absence of a set of

variables. For example, a question may be “Is an alarm currently active?” From this point the

29

different strategies would diverge according to the presence or absence of an alarm. This

notation therefore illustrates not only the different strategies, but also the factors that influence

the selection of those strategies.


We performed StrA on both of the search strategies that were analyzed in ConTA. As stated

earlier, we also used StrA for online diagnosis of operational faults. Due to the ambiguous nature

of the tasks being described, the strategies are highly interwoven and operators can often switch

between strategies during a task. For example, in a search task, the strategies can diverge

according to the level of specificity at which an operator views the system. However, during

holistic perception of a system, an operator may focus in on one area, thereby switching to a

more localized strategy. These relationships are depicted in unified IFMs.


3.4.3.1 Search for Fault (without alarm)

The strategies identified in the IFM for searching for a fault with the assistance of an alarm are

largely informed by the strategies identified in Mumaw, Roth, Vicente, & Burns (2000). The

strategies identified in this paper were derived from field observations in nuclear power plants.

They therefore represent strategies that expert operators would use and are categorized in three

broad classes of strategy, each with several sub-strategies:

1. Strategies used to maximize information extraction

2. Strategies used to create information

3. Strategies used to offload cognitive demands

Of the three classes identified in Mumaw et al. (2000), the most pertinent to this study are those

that maximize information extraction. These strategies were built into a unified IFM presented in

Figure 13.

30

Figure 13. IFM depicting the strategies used within the task of searching for a fault without

an alarm. Note that the legend listed here applies to all IFMs.

3.4.3.2 Search for Fault (with alarm)

The main difference caused by the presence of an alarm is that it provides preliminary guidance

towards the source. However, it does not necessarily identify the underlying cause of the alarm,

nor does it detail the full scale of the problem. Therefore, the strategies illustrated in the IFM

presented in Figure 14 details how an operator would use the aid of an alarm to answer these

questions.

31

Figu

re 1

4. IF

M d

epic

ting

the

diff

eren

t str

ateg

ies w

ithin

the

task

of s

earc

hing

for

a fa

ult w

ith a

n al

arm

.

32

3.4.3.3 Online Diagnosis of Fault

This strategy depicts the process of forming and testing hypotheses about the potential cause of a

fault. It is assumed that the input to this strategy is the successful assessment of the full scope of

the manifestation of the fault. As Figure 15 illustrates, there are two main strategies used in this

task and they differ in accordance with the level of specificity at which the operator observes the

system.

Figure 15. IFM depicting the different strategies within the task of online diagnosis of a

fault.

33

A critical aspect of this IFM is representation of the interconnectedness of the strategies within

the search box. In this task, operators would likely be switching rapidly between holistic and

localized processing. This is depicted by essentially resetting the search process after the

operator asks themselves the question of whether or not they have sufficient evidence to

formulate a preliminary hypothesis of the potential cause for the event.

Once they have that hypothesis, they need to weigh costs and benefits of the urgency with which

they respond. If they perceive a fault to be both time sensitive and safety critical upon initial

hypothesis formation, then they would be very likely to resort to an emergency operating

procedure rather than waiting to confirm their hypothesis. This actually represents a secondary

area where there are multiple strategies in spite of the fact that only one is shown. This is

because this safety-first strategy is the only strategy that exists in the nuclear domain (Meshkati,

1998).

3.4.4 Summary of Strategies Analysis

Although StrA is often underemphasized in CWA, it served a critical role in our project. The

graphical representations of the various tasks provided an illustration of the structure of nuclear

operations during loosely defined tasks. We were able to use these graphical representations to

build interfaces capable of supporting the strategies that they depict.

Our StrA also helped with the experimental design. StrA highlighted many of the behaviours that

are most important to nuclear operation, such as checking on referent systems, confirming initial

hypotheses, and many more. These findings were built into our experiment to ensure that the

tests would be evaluating key areas of operations.

3.5 Summary of Cognitive Work Analysis

Conducting the CWA on the CANDU 6® nuclear power plant represented a critical step towards

our goal of evaluating the efficacy of group view displays. CWA fostered our own understanding

of the system and operations, which directly led to the development of our experiment and

training materials for our participants. Furthermore, our CWA helped to develop the content and

architecture of the novel ecological interfaces against which we would be evaluating the existing

advanced design interfaces.

34

Display Development 4The existing interfaces in the CANDU 6® plants employ an advanced display methodology,

which combines mimic-elements with graphical trends and relatively simple configural graphics

(Lau, Jamieson, et al., 2008; Myers & Jamieson, 2014). To answer the experimental question that

asked what display type yields higher levels of the postulated benefits of GVDs, comparison

displays needed to be developed.

Ecological displays were selected as the comparison condition because they differ on a

philosophical level from advanced displays. They focus on depicting the functional relationships

between parameters (Vicente & Rasmussen, 1992). Physical relationships can still be depicted in

instances where a mimic diagram reflects a functional purpose of a system, as is the case in many

chemical processes where a specific sequence of processes is necessary to achieve the functional

purpose of a system (e.g., Paulsen display from Jamieson & Vicente, 2001).

As NCR display technology continues to develop, the displays focus more and more on depicting

information in a way that is psychologically relevant and accessible to operators. Traditional

displays focused entirely on mimicking the physical layout and hardware interfaces of the plant.

Advanced displays improved upon this by adding some functional information to make the

behaviour of parameters in the plant more accessible to operators. Ecological displays further

this abstraction by focusing on depicting the functional relationships between parameters in a

way that allows for the behaviour governing a system’s dynamics to be understood (Vicente &

Rasmussen, 1992).

4.1 Display Comparison

The comparison that we were making regarded the way that information is presented on an

interface. Issues such as informational content, information salience, and control methodology

were beyond the scope of this study and therefore needed to be controlled for. To do this, several

assurances were taken into account regarding the similarities and differences between the two

display type conditions.

35

4.1.1 Informational Content

Our principal concern in developing the comparison displays was maintaining the informational

content across conditions. Maddox (1996) challenged the purported benefits of ecological

displays by attributing the differences in the Christoffersen et al. (1996) results to the operators

simply having access to more information. In order to head off such criticisms, we needed to

ensure that information content was identical between the existing advanced displays and the

novel ecological displays.

4.1.2 Display Navigation

A second assurance had to be taken into account with respect to the way that operators navigate

through the displays. Again, we did not want potential differences between the conditions to be

attributed to anything other than the different display frameworks. We therefore implemented a

universal menu bar, which had tabs for each of the pages of the interface and made sure that the

controls to navigate between these tabs were identical between the two display conditions.

4.1.3 Display Fidelity

A critical aspect for comparing any novel design to an existing design is the degree of fidelity of

the two designs. Fidelity can come in many forms in display design, but the main areas of

concern were graphics, refresh rate, and accuracy. If, for example, the graphics on the novel

displays took longer to refresh than on the existing displays, there could have been far reaching

usability issues.

4.1.4 Display Appearance

The final assurance that had to be made came with respect to the appearance of the displays,

namely, in the salience of the colour indicators. The existing advanced displays use a light grey

background with black plots and predominantly green, white, or orange trend lines. Again, if we

were to use a completely different colour scheme, there may have been unforeseen consequences

with respect to situation awareness, diagnostic performance, or communication. To prevent any

potential confounds, we maintained the colour scheme for the novel displays.

36

4.2 Development of Novel Displays

The display development stage of the project was only done in service of assessing the

differences in task performance, situation awareness, and communication between ecological and

advanced displays. In other words, we did not develop these displays to advance the field of

ecological interface design. Therefore, we used existing ecological forms and principles

wherever possible rather than creating our own.

Unfortunately, limitations in the system used to build the display rendered certain forms

unfeasible. The principal limitations in the software were:

1. Unable to anchor lines to parameter values,

2. Unable to overlay anything on top of charts,

3. Unable to use dynamic angled lines, and

4. Complicated and time consuming coding process

Many ecological forms rely heavily on anchoring connections between parameter values (see

Figure 16). This meant we were unable to use connected bar charts, polar stars, and several other

ecological forms. Furthermore, the software prevented the use of dynamic angled lines, meaning

that certain forms such as the one presented in Figure 17 were unusable as well.

Figure 16. a) Connected bar chart from Lau et al. (2008); b) Polar star from Jamieson &

Miller (2007).

37

Figure 17. Chart depicting relationship between mass, energy, and saturation. The angled

line pivots in response to changes in mass. The three states depicted here are shown in the

legend in the bottom corner.

4.2.1 Existing form usage

While it was evident that certain forms were simply impossible to implement given the

restrictions imposed by the developer software, there were also issues pertaining to forms that

were technically possible. These issues stemmed from the unique programming language used

for this software, resulting in time constraints with respect to creating any novel forms. We

therefore attempted to reconfigure the existing forms within the software’s graphical suite

according to ecological principles.

We relied heavily on multivariate display forms, which compare the values of functionally

related variables to assess for any meaningful differences (Burns & Hajdukiewicz, 2004). These

forms were useful because they can emerge from simply rearranging existing bar charts on the

advanced displays. The existing advanced displays often presented values according to their

physical layout even for variables that had to be equal to one another. This meant that their

values were often presented in isolated locations on the screen. Therefore, in transforming these

38

into ecological displays, we simply rearranged the values to place them beside one another to

allow for easy comparison (see Figure 18).

Figure 18. Rearrangement of existing charts.

4.2.2 Development of Novel Forms

As described above, the purpose of this stage of the project was not to advance ecological

interface design. We therefore focused on utilizing existing tools wherever possible, both from

the collection of forms in CANDU’s development software and from the current ecological suite.

However, the aforementioned software limitations and perceived limitations with existing

ecological graphics necessitated the development of a novel graphical form.

As stated in Section 4.1, we were only interested in comparing the impact of displaying identical

information in different ways. Because the configuration contrasts were the only differences of

interest, we needed to ensure that the ecological displays depicted as many useful functional

relationships as possible. This means that wherever possible, the information on the displays

should reflect the underlying functional properties of the system.

Current advanced displays tend to focus on mass and energy balances in the system. They

therefore predominantly depict the abstract function level of the abstraction hierarchy. In order to

develop comparison displays, the ecological forms needed to also focus on these balances. The

ecological form that illustrates the abstract function level is presented in Figure 19.

39

Figure 19. a) Mass balance plot with more outflow than inflow. b) Mass balance plot with

more inflow than outflow.

There are three main problems with this plot. The first problem is that we were unable to anchor

the comparison line between inflow and outflow values. This isn’t necessarily a problem with the

form, but it was a problem for our project. This inability to anchor lines is particularly

problematic for this form since the values that need comparing are located relatively far apart

thereby rendering the connecting line essential for determining slight imbalances.

The second problem is that the point where the line connecting current inflow and current

outflow intersects with the line indicating the tank’s level is meaningless. This fails to take full

advantage of the relationship between these highly related values. Furthermore, if an operator is

accustomed to the relationships depicted in an interface being meaningful, this can serve as a

distraction.

The third problem is the lack of time-based information to indicate how the balance of these

three parameters lead to different tank states. For example, in Figure 19b the inflow is greater

than the outflow, which means that the volume in the tank should be increasing. If there was a

leak, the volume may stay the same because the extra inflow may be exiting the tank somewhere

before the sensor that measures the outflow. Without some trend information it would be very

difficult for an operator to assess this. A more problematic issue could arise if the inflow and the

outflow were balanced since the operator would likely take this to mean that the tank is stable.

40

Similarly to the previous example if there was a leak in the tank, the volume would gradually

decrease over time in spite of the balanced inflow and outflow.

To graphically address the first problem we used delta flow instead of presenting inflow and

outflow separately. Figure 20 shows the Delta Plot, which presents the current level along the Y-

Axis, and delta flow along the X-Axis. The vertical line at the 0 point along the X-Axis

represents the point where the current inflow and current outflow are equal. Therefore, if the

indicator is along this line, level in the tank should not be changing. The horizontal line in the

plot represents the current setpoint of the system, which can be set anywhere along the vertical

axis as determined by the system’s requirements.

Figure 20. Delta plot showing current inflow balancing with current outflow and the level

at the setpoint.

By intersecting these two lines, four quadrants are formed. These four quadrants represent the

four principal states of any closed system with respect to mass or energy balance. Quadrants Q2

and Q4 indicate that the level in the tank is moving towards the setpoint while Quadrants Q1 and

Q3 indicate that the level in the tank is moving away from the setpoint (see Figure 21).

Therefore, by looking at the location of the single point, an operator is able to determine if the

system is in an acceptable state or not.

41

Figure 21. Four quadrants of the delta plot

Although this form addresses the problem in the mass balance display presented in Figure 19 of

relating current level to inflow and outflow, it does not reveal how the individual parameters

making up this aggregated information relate to one another over time. This trend information

was therefore added as a secondary component of the graph, which allows the operator to see the

inflow, outflow, delta, and volume separately. Furthermore, it allows the operator to see how

these parameters have changed over time in relation to one another.

The outflow is presented on the bottom of the trend, sharing a common zero-point with the

inflow, but protruding in opposite directions (see Figure 22). The orange bar in the middle

illustrates the difference between these two values, thereby showing the current delta flow in or

out of the tank. The value of this orange bar is reflected about the angled line below the delta

plot, thereby providing the X-coordinate for the point on the delta plot. There is also a tail on the

point in the delta chart to show activity in the past ten seconds. The tail presented in Figure 22b

shows a recent increase in volume with changes in the delta flow occurring around eight seconds

ago.

42

Figure 22. a) Delta trend plot showing balanced state over the past 10 minutes. b) Delta

trend plot showing an unbalanced state with 76 kg/s greater inflow than outflow along with

the resulting changes in level.

4.2.3 Display Finalization

The delta chart proved to be very useful for representing multiple aspects of the CANDU 6®

plant at the abstract function level. In areas where the delta chart was unnecessary, the trend

portion of the chart was used, which included the mirrored bar charts showing either the energy

or mass input and output as well as the delta between those values.

Four screens were developed for the experiment, which covered the majority of the plant

parameters that would be involved in the experimental scenarios. Figure 23, Figure 24, Figure

25, and Figure 26 show the four ecological screens that were used during experimentation.

43

Figure 23. Ecological Critical Safety Parameter screen.

Figure 24. Ecological Plant Operations screen.

44

Figure 25. Ecological Steam Generator screen.

Figure 26. Ecological Primary Heat Transfer screen.

45

Experimental Method 5

5.1 Control Room Mock-UP

The experiment was conducted in a CANDU® Main Control Room (MCR) mock-up. The mock-

up is a scale representation of an actual MCR in a CANDU® plant. Some of the main features of

the MCR mock-up are (Candu Energy Inc., 2014):

• 19 touch screen panel displays

• Two 100” large screen display units

• Individual reconfigurable operator workstations

• Full-scope simulator control logic

The simulator control logic allowed for accurate representations of the plant operations during

experimentation while the high-fidelity nature of the rest of the MCR mock-up ensured that the

operating environment would be as close to real-world operating environments as possible.

Figure 27 shows a schematic of the MCR mock-up.

Figure 27. Schematic of the CANDU mock-up. Note that although this is not to scale, the

relative positions of relevant elements in the mock-up are accurate.

5.2 Design

The experiment’s aim was to evaluate the two research questions, which were what display

configuration and what display type yielded the highest levels of situation awareness,

46

communication, and diagnostic performance. Furthermore, we were interested in evaluating

whether certain interactions between display configuration and display type facilitated higher

levels of the three primary metrics. Table 4 illustrates this experimental design. The two

principal comparisons are shown in Figure 28 and Figure 29 respectively.

Table 4. Experimental Design

Question 1

Display Configuration

LSD Redundant

Question 2 Display Type

Advanced LSD-ADV RED-ADV ADV Ecological LSD-ECO RED-ECO ECO

LSD RED

Figure 28. a) LSD configuration; b) Redundant configuration.

47

Figure 29. a) Advanced display; b) Ecological display. Both images present identical

informational content.

5.2.1 Experimental Tasks

Experimentation consisted of two-person operating teams with each operator having a set of

predefined tasks. Both operators’ primary task throughout the entirety of the trial was

monitoring. This encompassed both observation of the plant parameters to determine if anything

was wrong as well as the diagnosis of a fault in cases when they did notice systematic deviations.

Since responsive control actions were outside of the scope of this experiment, once an operator

noticed a deviation, their principal role shifted to determining the underlying cause of the

deviation. Figure 30 shows a participant performing his primary monitoring task in the LSD

configuration.

48

Figure 30. Operators performing primary monitoring task in the LSD configuration.

In addition to monitoring the plant, the operators performed distractor tasks. These were included

in an attempt to portray operator activities as accurately as possible. In real-world operation,

operators are responsible for performing a number of activities beyond simply monitoring the

systems. Examples of these tasks include logging current values and testing systems (Davey,

2000). Each participant in an experimental team had to perform a different distractor task. One

operator was responsible for testing shutdown system one, which involved them approaching the

panels, reading and recording the levels of a variety of indicators, followed by testing the correct

functioning of the control logic. The second operator was responsible for checking the liquid

zone levels in the reactor. This task allowed them to remain at their workstations, but required

them to allocate one of their screens to the liquid zone levels display, thereby taking one of their

available monitoring screens offline for the duration of their distractor task. Figure 31 shows the

operators performing their individual distractor tasks.

49

Figure 31. Operators performing their secondary distractor task in the redundant display

configuration.

5.2.2 Experimental Scenarios

Each experimental team participated in four trials. Each trial involved a unique pairing between

scenario and display setup. Scenario refers to the type of fault that was introduced into the

system and display setup refers to the pairing between display configuration and display type.

There were two display configurations (LSD, RED) and two display types (ADV, ECO), and

thus four possible display setups. Therefore, the experimental teams had a different scenario and

a different display setup in each of their four trials such that there was no repetition of either. An

example of one team’s schedule is presented in Table 5.

Table 5. Example of one team's experimental schedule.

Display Setup Trial Scenario Configuration Type

1 3 LSD ECO 3 2 LSD ADV 2 4 RED ADV 4 1 RED ECO

50

The randomization of the pairings between scenario and display setup was controlled such that

each setup was paired with each scenario an equal number of times. Furthermore, the ordering of

these pairings was balanced to counteract the expected learning effects. The full schedule for all

eight teams can be found in Appendix B.

The scenarios were similar to one another in structure, but differed in the type of fault that was

introduced into the system. In order to accurately assess the performance differences caused by

the different experimental conditions, four considerations were taken into account when

designing and selecting the experimental scenarios.

The first consideration related to the novice operators. The scenarios needed to be complex

enough to challenge the operators but not so complex that they would be overwhelmed.

Candidate scenarios were therefore selected from normal operations rather than emergency, and

revolved around a set of parameters that the participants would have encountered in their college

program. This meant focusing on pressures and flows instead of the electrical grid or nuclear

processes.

The second criterion for scenario selection was that the scenarios needed to call into question the

postulated benefits of GVDs: 1) That they increase operator situation awareness, 2) That they

increase operator communication and collaboration, and 3) That they improve overall

performance (Roth et al., 1997, 1998). In an effort to meet these requirements, scenarios needed

to require a holistic understanding of the behavior of different systems. This was assessed by

examining the degrees of separation between the actual fault and the manifestation of that fault.

For example, a leak in the heat transport system will reveal itself in the D2O storage tank, which

is several tanks removed from the actual source of the problem (see Figure 3). In order to

diagnose this problem, an operator would need a sufficient understanding of the overall behavior

of the plant, which requires situation awareness of multiple related systems. The second and third

claims, pertaining to communication, collaboration, and performance, can be slightly more

difficult to ensure in an experimental scenario since these are often dependent on the individuals

in a team and the dynamic of that team as a whole. However, it has been shown that performance

in problem-solving situations, especially those in complex, high-risk environments, is positively

correlated with communication behaviour (Johansson & Persson, 2009). Therefore, having a

51

diagnosis stage of the trials should call into question the communication and performance claims

about GVDs.

The third scenario inclusion criterion was that no control actions will be required. This criterion

is largely based on subject matter expert recommendation and experimental constraints.

Including control actions would likely have complicated the scenarios too much to elicit accurate

performance data for our novice operators. Requiring control actions would also increase the

complexity of the training regimen. Furthermore, since the experiment’s aim was to evaluate the

efficacy of GVD display types and configurations, including control actions that would be made

at the panel would detract from the goals of the experiment. Therefore, control actions were

omitted from the scenarios.

Finally, the scenarios needed to be relatively similar to one another in difficulty. Differences in

scenario difficulty would likely result in significant performance differences independent of the

experimental manipulations. This is very difficult to control for in simulator studies, but by

focusing the scenarios on similar types of faults we were able to provide some element of

control. Notwithstanding these efforts, the scenario was specified as a random factor in statistical

analysis.

We selected our scenarios both through consultation with operational subject matter experts and

by reviewing the parameters used to initiate a setback or stepback on the reference CANDU 6®

design. Furthermore, we consulted a list of Enhanced CANDU 6® design basis events that would

lead to an Anticipated Operational Occurrence (AOO) plant state. An AOO is defined as a

deviation from normal operation that is expected to occur once or several times during the

operating lifetime of the NPP but that is unlikely to cause any significant damage to items

important to safety, nor lead to accident conditions. Past research has demonstrated the benefits

of functional information in unexpected operating scenarios (e.g., Burns et al., 2008). We

believed that these AOOs meet this criteria for the novice operators since any changes in plant

state would be unexpected.

The structure of each of the scenarios involved a period of steady-state, a fault introduction, a

freeze, and a period after the freeze where the fault continued to manifest itself. Freezes are

pauses in the simulation where the participants complete a measure to assess situation awareness.

52

This measure is discussed in detail in Section 5.4.1. The list of faults that were used in our

scenarios is presented in Table 6 and their timelines follow in Figure 32.

Table 6. List of selected faults for experimental scenarios

Fault Primary Symptom* D2O/ H2O Minor loss of coolant accident Loss of inventory in the D2O storage

tank D2O

Partially closing feedwater control valve

Steam Generator level decrease H2O

Inadvertent opening of MSSVs Decrease in Turbine power H2O Spurious closure of P&IC bleed path

Increase in flow into P&IC D2O

*Only the main indicator of the fault is listed here. The faults will impact many areas of the plant beyond what is listed

Figure 32. Scenario timelines.

As Table 6 indicates, two of the faults occur on the heavy water side of the plant, while two of

them occur on the light water side. We believe that this distributed sampling of faults ensured

good representativeness of different plant operations while also standardizing the difficulty of

each scenario to allow for fair comparison between scenarios.

5.3 Participants

Due to access limitations, licensed nuclear operators were not available for the experiment. We

therefore recruited students from Lambton College’s Process-Operations Program to participate

in the study. A total of 16 process operations students were recruited. Participants were paid

53

$250 for their participation with all expenses covered on top of this. The reason for the large

payment was due to the limited pool that we were able to draw from, and because we required

them to spend roughly six hours in transit, a night in a hotel, and a full day at the simulator

facility.

The participants were organized into eight teams of two, with one team being used for the pilot

and the other seven in the actual study. Some participants knew each other prior to

experimentation and formed a team beforehand. The others were put into teams by the

experimenters using their dates of availability as the sole selection criterion. Although this

presents a potential confound for our results, this was the only feasible way to recruit the

participants since they were active students who had class schedules to balance.

The students had simulator experience and a level of familiarity with other process systems such

as petrochemical or refinery plants. None of the students had any experience with nuclear

operations and were therefore considered novice operators during experimentation. This was

problematic from an ecological validity standpoint because nuclear plants are exclusively

operated by expert operators in the real-world (Juhasz & Soos, 2007).

The challenge therefore became determining the most effective way to capitalize on their

previous experience and training. Due to the time constraints on training, the only feasible way

to meet this challenge was to reduce the complexity of how the plant was represented during

experimentation. This decision was made on the basis that mimicking the expertise of real-world

operators in a simplified plant is more valid than putting novice operators in a system that they

are unqualified to monitor. Although this point can be contested, the simplification of the plant

was performed systematically to ensure that the processes governing plant operations were

accurately represented.

5.3.1 Plant Representation

The first step in reducing the complexity at which the plant was represented was to determine the

areas of operations that were integral to plant functionality. Integral, as used in this context,

refers to the areas that make up the two main circuits in the plant. Because of the closed nature of

these circuits, the understanding of any one system hinges on an understanding of both the inputs

and the outputs for that system. For example, although the reactor is obviously a critical system

54

within the plant, understanding the processes within that reactor is not actually necessary to

understand the behaviour of the rest of the plant at the depth required for our trials. In essence,

one only needs to understand the output of the reactor. Similarly, one does not actually need to

understand how the electricity being produced in the system is fed into the grid, but rather that

the power grid is the output system for electricity generated in the plant.

Any system where understanding both the input and the output were required for a complete

understanding of the behaviour of the plant was considered to be integral. For example, the heat

transport system receives heat energy from the reactor and transfers it to the steam generator

where it converts liquid water to steam. Since understanding this critical energy balance is

necessary for understanding the overall behaviour of that plant, the participants needed to be

trained on this system.

By removing the systems deemed superfluous to understanding plant dynamics, the participants

were able to focus their efforts during training and experimentation on the systems that played

key roles in the scenarios. This allowed them to gain a deeper understanding of the relevant

systems, which essentially increased their relative level of expertise. Figure 33 shows the final

diagram that was given to participants.

55

Figure 33. Final diagram presented to participants which illustrates the level of complexity

of plant representation.

5.3.2 Display Simplification

The interfaces also needed to be simplified to reflect the tailored representation of the plant.

Therefore, any display components relating to parameters beyond the scope of our experimental

scenarios were removed. Again, this allowed the participants to focus their attention on the areas

of the interfaces that were relevant during experimentation in a similar fashion to what an expert

operator would do. Since the participants did not have experience discerning what parameters

were relevant given a current operational state, this stage was necessary for maximizing the

relative expertise levels of these operators.

REACTOR

Deaerator

Deaerator Storage Tank

Condenser

HotwellHotwell

LP Feedheater

Condensate Storage Tank

Condenser

Emergency Stop Valve

Governor Valve

Release Valves

SG Isolation Valves

MSSVs

CSDV

ASDVs

PZR

D2O Storage

Tank

Deg.Cond

Outlet Headers

SGFP

SGFP

SGFP

Turbine GeneratorLegend

D2O PathH2O PathNormally closed valveNormally open valve

Pump

Cooler

Fail Open Valve

Fail Close Valve

HP Feedheater

SG1

SG4SG2

SG3

SG1

SG4

SG2

SG3

56

5.3.3 Participant Training Package

Once the appropriate level of system complexity was determined, I developed a training manual

focused on the areas of the plant that the experimental scenarios would focus on. The training

package was based on an existing introductory training package developed for CANDU

(Bereznai & Harvel, 2011). The pre-existing package was 258 pages long and consisted of

training on everything from the fuel handling systems to the control systems. Due to the limited

time window in which the participants had access to the materials, I selected the content that was

relevant to the simplified representation of the plant in the context of the experimental scenarios.

This allowed for the 258 page training manual to be tailored to 27 pages of essential

experimental training material.

The training package covered only the systems that would be relevant to the experiment,

focusing equally on those systems. The material took the eventual scenarios into account to

ensure that any participant would have the requisite knowledge for adequate performance in that

scenario. However, no extra information or focus was placed on the specific areas that caused the

faults in the different scenarios. This ensured that the participants would not have clues as to

what the cause of a fault in any of the scenarios would be. Details on the administration of the

training package will be covered in Section 5.5.1.

5.4 Measures

5.4.1 Situation Awareness

Situation Awareness (SA) was evaluated using the Process Overview (PO) measure (Lau et al.,

2011). The process overview measure was designed specifically for measuring SA in process

control plants. It uses context-dependent, top-down queries to assess operators’ overall

comprehension of the behaviour in the plant. The PO measure was selected over other measures

of situation awareness, such as the Situation Awareness Global Assessment Technique (SAGAT;

Endsley, 1995) because it was designed specifically for process control. SAGAT is based on

Endsley’s 3-level model of SA, which views SA as an information processing activity that is

assumed to generalize across different domains. SAGAT therefore uses the same approach to SA

measurement irrespective of the domain. In process control, SA is viewed as a creative problem

57

solving activity that is heavily influenced by domain context (Lau et al., 2011). Since SA is

characterized differently in process-control, the PO measure is tuned to this situational context.

To use the PO measure, we developed queries that ask questions about the behavior of specific

aspects of the plant in the given scenario. An example of the structure of a PO query is presented

in Figure 34. Each trial had two sets of 12 PO queries administered to each participant

individually. Participants were not allowed to communicate to one another nor did they have

access to the displays while completing these tests. The first set was presented to participants at

the freeze in the trial and the second was presented at the end of each trial (see Figure 32). The

queries asked about the recent behaviour of various parameters relevant to that particular

scenario. Although there can be some ambiguity with respect to how recently is defined (Lau et

al., 2011), we defined it to our participants for the first set (i.e., the set presented at the freeze) as

since the beginning of the trial. For the second set of PO queries, recently was defined as the

time since the freeze point. Therefore, if a particular value is currently lower than it was at the

beginning of that portion of the trial (i.e., either from the start until the freeze, or from the freeze

until the end), irrespective of the behaviour in the middle, the correct answer to that query is

decreased. An example of this is presented in Figure 35.

Figure 34. Typical process overview query.

58

Figure 35. Example of how recently was defined.

Each set of queries consisted of 12 questions. Eight of these were visible, meaning that the value

that the query was inquiring about was displayed somewhere on the display. The other four

questions were inferred. This means that their value was not visible anywhere in either the

advanced or the ecological displays. Correctly answering these was based on the participants’

understanding of the recent behaviour of related systems in the plant.

The queries should inquire about parameters in the plant that are integral to the present state (Lau

et al., 2011). For example, if a scenario involves a leak in the heat transport system, the queries

should focus on the heavy water side of the plant rather than ask questions about what is

happening in the steam system. To select the components and the queries, we consulted with a

SME. The SME was a former nuclear operator who was familiar with multiple CANDU® plants.

Although the PO measure is typically scored during experimentation by an SME, we were able

score the queries after the data collection. Most studies employing the PO measure involve

control actions, which can impact the plant’s state and therefore change the values of the

respective parameters and thus the answers to the respective PO queries. Since our study did not

have control actions, the behaviour of parameters in each scenario was identical. Therefore, all

scenarios could be scored using a reference key that represented the objective answers to each

query. This also circumvented any potential issues with interrater reliability (Lau, Jamieson, &

Skraaning, 2014). Further description of the scoring and coding of PO measure queries follows

in the results section. The list of queries is presented in Appendix C.

59

5.4.2 Communication

There were two sources of communication data. The data presented in this report was derived

from subjective ratings of communication during each trial. These ratings were based off of

experimental logs that were kept by the experimenters during testing. The logs included notes on

operator communication, performance, and overall monitoring behaviour. Ratings were therefore

meant to indicate a general communication metric. They were loosely defined and intended to

simply provide an initial impression of how well the operators communicated with one another

throughout a trial. All recording was done discretely from the experimenter recording station (see

Figure 27) to prevent any interference or to influence the communication behaviour of the

participants. These scores were done on a 7-point scale. This method of recording

communication served as a preliminary assessment of communication patterns within the various

setups. Although all efforts were made to accurately reflect the communication in each trial, this

data is likely subject to biases.

The second measure of communication employed Behaviorally Anchored Rating Scales

(BARS). Montgomery, Gaddy, and Toquam (1991) used BARS to rate team interaction in

nuclear power plant control rooms along the following five dimensions: communication, task

coordination, team spirit, maintaining task focus in transition, and adaptability. Using these five

dimensions in conjunction with an extensive rater-training system, they found interrater

reliability of 0.89, which is considered very high by interrater reliability standards (Gwet, 2014).

We recruited three students from the Mechanical and Industrial Engineering programs at The

University of Toronto to act as raters. Montgomery et al. (1991) state that non-expert raters can

still achieve high interrater reliability readings, provided they have sufficient training. We used

example scripts to train them on the five dimensions. These included both positive and negative

examples of behaviour along each of these dimensions to provide benchmarks. The raters were

trained in the system at a level sufficient to detect communication patterns. For example, the

raters needed to know when an operator noticed a problem. In order to do this, the raters needed

to know what the problem was and examples of things that the operators may be saying when

they notice the problem. The raters were paid $20 per hour to perform the rating task.

The raters used recordings from each trial to score the BARS. Raters were blind to the

experimental setups, but were aware of the scenarios in order to determine whether or not the

60

communications between operators were focused on the actual fault from the respective scenario.

Each team completed four trials, which amounted for a total of 75 minutes of recordings per

team and thus 525 total minutes for all 7 teams combined. This therefore took several weeks for

the raters to work through. Unfortunately, time restraints with respect to receiving the rating data

prevented them from being included in this report. This information will be included in future

publications.

5.4.3 Diagnostic Performance

To measure operator diagnostic performance, we adapted the method used by Lang, Roth, Bladh,

& Hine (2002), who used two principal metrics for evaluating performance: Detection and

diagnosis. In their study detection performance was determined by whether or not the crews

detected the target event (e.g., loss of coolant accident). This was done online and recorded

through observation by raters. Their second metric was diagnosis, was also done through

observation by raters who noted whether one or more of the team members explicitly stated the

cause of the event during the trial. We adapted this to make it post hoc, which better met the

needs of our novice participants. We used a semi-structured interview after each trial. The

interview consisted of the following questions:

1. Did you notice a problem? 2. Was the problem on the primary or the secondary side of the plant?

3. What were the main symptoms of the problem? 4. What systems did those symptoms manifest in?

5. What would you say was the exact cause of the problem?

Formalizing the measure in the form of a semi-structured interview following each trial

prevented us from relying entirely on the participants to explicitly state the cause of the fault.

Although this method is less formal than some of the other operator performance measures, it

was best suited for our needs since our participants did not engage in any control actions.

We also recorded temporal performance metrics using the same team of raters from the BARS

ratings. Although time restrictions prevented this data from being used in this report, it is

worthwhile detailing how temporal performance will be evaluated. These were metrics

representing the sequence of actions that an operator would go through when they first notice a

fault: 1) Detection, 2) problem solving, and 3) diagnosis. The raters were given criteria to

61

determine what constitutes each of the stages. Detection was defined as a participant verbalizing

that they noticed a problem. Problem solving was defined as the point in the trial when

participants begin verbally attempting to determine the cause of a problem. Accurate diagnosis

was defined as the point in the trial when the team verbalizes the actual cause of the problem

Again, since these definitions are slightly ambiguous, interrater reliability will be evaluated.

Using this data will eventually allow us to answer more questions relating to process monitoring

and control. While the detection time data may provide valuable insight towards the benefits of

the different experimental setups, finding significant relationships between experimental setup

and problem solving time and diagnosis can be extremely valuable for understanding the types of

behaviour that the different displays and configurations facilitate.

Table 7. Summary of measures

Construct Measure Description Included?

Situation Awareness

Process overview measure

A probe-based query system developed specifically for process control situation awareness

Yes

Operator Performance

Lang et al., (2002) method

Semi structured interviews to determine participants’ ability to accurately detect and diagnose a fault

Yes

Temporal metrics Latency of fault detection, troubleshooting, and accurate diagnosis No

Team Communication

Subjective ratings

Preliminary rating of communication based on experimental logs Yes

Behaviorally anchored rating scales

External raters judge operator interaction along a predetermined set of dimensions No

5.5 Procedure

5.5.1 Training

Participants received the training manual described in Section 5.3.3 four days before their

scheduled date at the simulator. They were instructed to read it over and make note of any

questions that they had. Finally, they were also told that they would be receiving a test on the

content of the training package to help motivate them to study the package closely.

62

Upon arrival at the simulator facility, participants were greeted and asked to sign non-disclosure

agreements and letters of informed consent. Once this was completed, the experimenters would

perform a walk-through of the schematic of the system to make sure that everyone had a basic

level of understanding of how the system worked. During this time, participants were

encouraged to ask any questions that may not have been addressed in the training package.

Once the systems walkthrough was completed, participants were given a brief test that evaluated

their understanding of certain aspects of the system. This test was included for two main reasons.

The first, was that since the participants were informed of this test before signing up for the

study, they would likely be motivated to read the training manual carefully than if they didn’t

think they were going to be evaluated. The second reason was to determine if there were any

areas of plant operations that the participants still did not understand. The tests were reviewed

with the participants upon completion and any points of confusion were discussed.

After the test debriefing, participants were trained on how to read both the ecological and

advanced display elements. They were introduced to the elements as standalone graphical forms

rather than being introduced to the full displays right away. This allowed them to focus on the

individual forms rather than potentially becoming overwhelmed by the breadth of the displays.

The delta plots, a prominent feature on the ecological interfaces, required more instruction than

did the elements used in the advanced displays. However, it was important for the participants to

spend equal amounts of time with both displays to prevent any preconceived biases from

developing. We therefore spent the same amount of time on both display types.

Once participants had been trained on the individual graphical forms, they were introduced to the

full displays. The experimenters guided the participants through both the ecological and

advanced display types in both the LSD and redundant configurations. Again, the ordering of this

training was randomized and counterbalanced to prevent any bias. After the participants had

been sufficiently introduced to the interfaces they were given 10-20 minutes to freely explore

them.

Following the interface exploration, participants began test trials. The trials were ordered in

terms of the severity of their symptom manifestation. Prior to the first trial, participants were

informed that changes in system state during actual experimentation would be likely to be much

more subtle than the ones in the test trials. The first test trial induced a reactor trip in less than a

63

minute. This was to illustrate the types of parameter behaviours that the participants should

expect in the actual trials. The system was then reset at the second test trial began. The test trials

were designed to be similar to the actual experimental trials, but without having overlap in the

potential causes for the symptoms. Because of this, we were limited in the number and scope of

test trials.

During the second test trial the participants were introduced to the freeze, which are simulator

pauses, during which time the PO measure was administered. Participants were instructed to

leave their workstations during the freeze and to approach a conference table to fill out the PO

measure (see Figure 27). During their completion of the PO measure the operators were not

given access to any displays. They were informed ahead of time that the PO measure relied on

their ability to maintain an accurate and up-to-date mental model of the operations of the plant.

We then reviewed their responses to the queries and discussed the answers to ensure that all

parties had the same concept of how to correctly interpret the queries.

Training was finished when the experimenters determined that the participants had attained a

sufficient level of proficiency with the system. This varied from team to team as some

participants differed significantly in the speed with which they became comfortable with the

system. There was no formal assessment to ensure an adequate level of proficiency prior to the

commencement of the experimental trials, but due to time restrictions, this was the only feasible

method of experimentation. Both subject and group were specified in all statistical analyses as

random variables in an effort to account for this limitation.

5.5.2 Experimental Trials

Prior to each trial, both participants in a team were instructed to activate their voice recorders

and to leave the recorders running until the interview following completion of the trial was

finished. Participants were instructed to go to their respective workstations while the

experimenters ensured that the necessary forms were in place. These forms included those

required to complete both of the distractor tasks, as well as the process overview measures.

The trials began with one of the experimenters giving the participants a countdown, after which

the trial was active. Once the trials were active the experimenters took their position at the shift

supervisors desk (see Figure 27) and began recording log notes.

64

Trials ran uninterrupted until the freeze. The simulator was programmed to automatically pause

the operations at the freeze. When the freeze occurred the participants left their workstations and

walked to the conference table at the back of the room (see Figure 27). Each participant filled out

the PO measure for that portion of the trial with no set time limit. During this time they were not

allowed to communicate with one another nor did they have access to the displays. When they

were finished they returned to their workstations, at which point one of the experimenters gave

them a second countdown marking the commencement of the second leg of the trial.

Again, that experimenter returned to the shift supervisor desk and resumed taking log notes for

the remainder of the trial. The trial ran uninterrupted until it was complete. When complete, the

simulator was programmed to restore its baseline conditions and the participants filled out the

second PO measure at the conference table.

5.5.3 Post-Trial Interview & Debriefing

After each participant was finished with their second PO measure (i.e., the set of queries at the

end of the trials) they were given individual post-trial semi-structured interviews. The purpose of

these interviews, as described in Section 5.4.2, was to assess diagnostic performance (Lang et al.,

2002). The interview consisted of five questions, which gradually got more specific as to the

nature of the fault from the preceding trial.

Once the interview was complete the participants were instructed to stop and save their voice

recordings. They were then debriefed on the actual cause of the problem. Since the problems

were unique, this did not present learning effect confounds.

This process was repeated for each of the four trials. Participants were never told the true nature

of the testing because of concerns surrounding the close-knit community at Lambton College and

the potential interference that could occur if a team told another team the motivations behind the

study.

Experimental Results 6

6.1 Situation Awareness

The data used to measure situation awareness (SA) came from process overview (PO) queries

that were given to the operators at both the freeze and endpoint of each trial. As stated above,

65

each set of queries consisted of 12 questions pertaining to characteristics of the plant that were

relevant to that particular scenario. Of the 12 questions, four were inferred, meaning that the

parameters they were inquiring about were not actually visible on any of the display screens.

These therefore required the participants’ deeper understanding of the behaviour of the plant.

The other eight queries were visible, meaning that the information for the parameter inquired

about was available when they were at their operating stations.

A total of 1344 process overview responses were recorded, 896 of which were visible and 448

inferred. A summary of the data collected is presented in Table 8. Due to one of the experimental

teams serving as the crew used in the pilot study, their data was not usable in this analysis. This

resulted in only seven teams being used for actual experimentation. Therefore, some Setup by

Scenarios were only tested by one team, resulting in 48 total PO queries for those conditions

rather than the 96 for all others.

Table 8. Number of PO measure queries for each Scenario X Setup

Setup LSD-ADV LSD-ECO RED-ADV RED-ECO Total

Scen

ario

HTS Leak 48 96 96 96 336 FW Valve Closure 96 48 96 96 336 MSSV Open 96 96 48 96 336 P&IC Path Closure 96 96 96 48 336

Total 336 336 336 336 1344

A mixed linear regression was performed on the visibility of the queries. As expected, the means

on the inferred queries (m = 0.44, SE = 0.06) were significantly lower than the visible queries (m

= 0.75, SE = 0.05; F(1323) = 111.52, p < .0001). This significant performance difference held

across all setups and scenarios (see Figure 36). The results of these t-tests are presented in

Appendix D.

66

Figure 36. Visible vs. Inferred means.

As the above figure illustrates, the participants’ responses on the inferred queries were

considerably lower than they were for the visible queries. The initial mixed generalized linear

model using the PROC GLIMMIX procedure in SAS University Edition utilized both the

inferred and visible queries. However, it failed to converge. Although there are many reasons for

this, the most likely is that there was either complete or semi-complete separation in the data

preventing the maximum likelihood estimates from being calculated (Allison, 2008). This failure

is likely the result of guess behaviour on the inferred queries counteracting any predictable form

in the data. Because the recommended number of queries per PO measure was eight, as per

recommendation from the developer of the PO measure (Lau, n.d.), subsequent analyses were

able to omit inferred queries from the models while maintaining an adequate sample size.

Therefore the pool of responses used to evaluate SA performance consisted of 224 query

responses from each experimental setup and scenario, all of which were visible.

A mixed generalized linear model was used to investigate the two experimental questions using

the PROC GLIMMIX procedure in SAS University Edition. This method examined the

probability of any given query being answered correctly in the various experimental setups,

irrespective of the scenario, group, or subject. Ideally, the scenarios, groups, and subjects would

67

be identical in terms of performance. However, as Figure 37, Figure 38, and Figure 39 illustrate,

there was a large amount of uncontrolled variance resulting from these variables. These were

thus specified as random variables, and were controlled for in the model.

Figure 37. PO measure means for each scenario across the four conditions. SCN refers to

the scenario (see Table 6).

Figure 38. Each group's PO measure scores across the four setups. GID refers to the group

identifier.

68

Figure 39. Individual subjects' PO measure scores over the different setups. SID refers to

the unique identifier for each of the 14 subjects.

After the random effects described by the three above figures were controlled for, the final model

used in the PROC GLIMMIX procedure examined a binomial distribution through a logit link

function. The SAS code used in this model is presented in Appendix E. A Type III test for fixed

effects revealed a significant effect of display type on PO measure scores, showing that the

likelihood of any given question being answered correctly was significantly higher in the

ecological displays (m = 0.80, SE = 0.08) compared to the advanced displays (m = 0.72, SE =

0.06; F(873) = 5.76, p < .02). Neither display configuration nor the interaction between display

configuration (F(873) = 0.07, p > .05) and display type (F(873) = 0.92, p > .05) were found to be

significant. Table 9 and Figure 40 illustrates these findings.

69

Table 9. Type III Test for Fixed Effects

Effect Num DF Den DF F Value Pr > F Configuration 1 873 0.07 0.79 Display Type 1 873 5.76 0.017 Interaction 1 873 0.92 0.34

Figure 40. LS Mean score for each experimental setup. LS Means have been transformed

to be on an interpretable scale (i.e., 0 < LSM < 1). Red bars indicate the 95% confidence

interval.

6.1.1 Inferred Model

A separate generalized linear mixed model was performed using only the inferred data to see if

there were any setups that yielded higher scores on inferred queries. As expected, there were no

significant effects found for either display configuration (F = 0.36, p > .05), display type (F =

0.41, p > .05), or the interactions between configuration and type (F = 1.89, p > .05).

6.1.2 Differences of Least Squares Means

Testing the differences of least squares means is a process that can be requested within the

PROC GLIMMIX procedure in SAS. This test conducts pairwise comparisons to evaluate

whether specific setups (i.e., Configuration X Display type) significantly differ from one another.

70

This can reveal whether certain display types perform better when coupled with certain display

configurations. Two differences were initially found to be significant: LSD-ECO vs. RED-ADV

(t(873) = 1.98, p < .05) and RED-ADV vs. RED-ECO (t(873) = -2.46, p < .02). However, after a

simulated adjustment for multiple comparisons, these differences were found to be not

significant. A full table showing the multiple comparisons is presented in Appendix D. The

differences are presented in a diffogram in Figure 41. Diffograms plot the mean of each setup

against the mean of every other setup to illustrate if the difference is significant (High, 2014).

When two means are very similar, their point of intersection will fall along the dashed angled

line projecting from the lower left corner to the upper right corner on the diffogram. When two

means are significantly different, their point of intersection in the diffogram will be further away

from this line. As Figure 41illustrates, no differences were found to be significant.

Figure 41. Multiple comparisons diffogram showing no significant contrasts.

71

6.2 Communication

A mixed linear regression was performed using PROC MIXED in SAS that specified the

scenario and group as random effects. As stated earlier, due to time constraints, the only data

used in this analysis are from subjective ratings given by the experimenters. Although all efforts

were made to provide unbiased ratings, the results presented in this section are considered

preliminary until the data from the BARS has been coded and analyzed.

A square root transformation was performed on the communication scores to normality

assumption. This compressed the scale such that the maximum score, which was originally 7,

became 2.65 with the minimum remaining at 0.

The mixed linear regression revealed a significant main effect of display configuration on

communication, indicating that the redundant displays (m = 2.10, SE = 0.17) elicited higher

communication scores compared to LSDs (m = 1.92, SE = 0.16; F(15) = 5.8, p < .05). There was

no significant effect found for display type (F(15) = 0.04, p > .05) or for the interaction between

configuration and display type (F(15) = 0.01, p > .05; see Table 10). The least squares means

estimates for the four experimental setups are presented in Figure 42. As the figure illustrates,

both of the redundant conditions yielded higher communication scores than both of the large

screen conditions. The SAS code used for this test is presented in Appendix E.



72

Figure 42. Communication scores across the four experimental setups. Red bars indicate

the 95% confidence interval.

6.3 Diagnostic Performance

The data used to evaluate diagnostic performance was based on the method described in 5.4.2. A

mixed linear regression was performed using PROC MIXED in SAS, which once again

identified the scenario, group, and subject nested within group as random variables. The SAS

code used for this mixed linear regression is presented in Appendix E.

The mixed linear regression revealed no significant main effects. This indicates that there were

no significant differences between diagnostic performance mean estimates for GVD

configurations (F(36) = 2.61, p > .05), display types (F(36) = 0.31, p > .05), or for the interaction

between configuration and display type (F(36) = 0.73, p > .05; see Table 11 and Figure 43).

Furthermore, there were no significant pairwise contrasts.

73



Figure 43. Diagnostic performance mean estimates by experimental setups. Red bars

indicate the 95% confidence interval.

To further investigate diagnostic performance, a mixed linear regression using the was

performed which examined the relationship between each trial’s performance score and how well

they did on that trial’s process overview queries. For each trial, each participant’s process

overview average was calculated and paired with their performance score on that trial. The

model revealed that process overview scores significantly predicted performance scores (F(45) =

6.46, p < .02).

It should be noted, that although the difference between configurations was not significant (F(36)

= 2.61, p = .11, the LSD display configuration achieved much higher performance estimates (m =

0.63, SE = 0.11) in the final model compared to the redundant display configuration (0.46, SE =

0.11). This difference was particularly large when comparing performance in the LSD-ECO

74

setup (m = 0.65, SE = 0.13) to performance in the RED-ECO setup (m = 0.38, SE = 0.13),

showing a 27% increase when the ecological displays were presented on the LSD compared to

redundant configurations. Although these results initially appeared systematic, an adjustment for

multiple comparisons rendered them insignificant (t = 1.81, p = 0.28). Although insignificant,

these sizable differences should warrant further evaluation in future studies. Figure 44 presents

the results.

Figure 44. Difference in performance between the two display configurations. Red bars

indicate the 95% confidence interval.

Discussion 7This study aimed to evaluate the efficacy of two GVD alternatives, and two display types within

a nuclear control room mock up. We evaluated how well different GVD and display framework

alternatives fostered SA, communication, and diagnostic performance. Although there is still

much to be learned about the different GVD and display alternatives in nuclear control rooms,

our study sheds light on the benefits of some of the foremost candidates.

Going into this study we had two principal research questions:

75

1. Is the widespread favouring of LSDs over redundant displays rooted in improvements in

SA, communication, and overall performance?

2. Is the widespread implementation of advanced displays over other display frameworks

rooted in improvements in SA, communication, and overall performance?

These questions focus on the three claimed benefits of GVD implementation for both GVDs and

display frameworks. Our findings are summarized in Table 12 and will be discussed according to

these three claims.

Table 12. Summary of main findings.

Dimension Main Findings

Display Configuration Display Type Situation Awareness

No significant difference for display configuration

Higher levels elicited by the ecological displays

Communication Significantly higher communication scores yielded by the redundant display configuration

No significant difference for display type

Performance No significant difference for display configuration

No significant difference for display type on communication

7.1 Situation Awareness

The process overview (PO) measure evaluated participants’ situation awareness (SA) under the

different experimental conditions. Participants scored significantly higher on the PO measure

under ecological conditions compared to advanced conditions, indicating that this display type

fostered a better understanding of the recent behaviour of the plant. There were no differences in

SA between the LSD and redundant configurations.

These results support previous research that has demonstrated the benefits of functional displays

on operator SA (Burns et al., 2008; Lau et al., 2008; Roth et al., 1998; Tharanathan et al., 2012).

As expected, the advanced displays performed well, scoring a 73% probability that any given PO

query would be answered correctly. This score is actually the exact score that Burns et al. (2008)

found under similar conditions using the PO measure. Contrary to the Burns et al. (2008) finding,

however, our participants demonstrated better PO scores in similar conditions using the

ecological displays than did their expert operators who scored 61% in similar operating

76

conditions. Of course, these results cannot be directly compared since their study employed

expert operators, different displays, and a different methodology, but it does serve as a helpful

reference point. Therefore, while both displays performed at a sufficient level, the significant

improvement elicited by the ecological displays indicates that improvements can be made to the

current advanced displays.

The higher levels of SA elicited by the ecological interfaces can be explained in a variety of

ways. Since there was no significant effect of display configuration, these differences in SA are

solely the result of display type. Furthermore, by employing the array of controls ensuring that

the only thing separating the advanced from the ecological displays was their driving framework

(see section 4.1), we can safely attribute these SA differences to differences between the display

frameworks.

The most likely explanation, given the control over the different conditions, revolves around the

integration of functionally related values into consolidated graphical forms in the ecological

displays. Instead of having to recall individual parameters when filling out the PO measure,

participants were able to simply recall the behaviour of a unified form. For example, the delta

plot (see section 4.2.2), which plots delta flow (i.e., inflow – outflow = delta flow) against

current level, was a prominent feature in the ecological displays for showing both energy and

mass balances. This plot used a single point with a trendline to represent the majority of the

necessary information pertaining to a system of interest. Participants could theoretically use this

plot to answer a variety of PO queries by simply remembering the behaviour of this single point

on that chart. The advanced displays presented the same information as individual values for

input, output, and level. This would therefore require them to recall three elements in the display

rather than one (see Figure 45). This contrast captures some of the essential differences between

the frameworks and is the likely factor driving the contrast in SA across the two display types.

77

Figure 45. Differences between advanced and ecological graphics.

7.2 Communication

Participants demonstrated significantly improved communication in the redundant display

configurations compared to the LSD configurations. Although there is an assumed benefit of

GVDs on communication (Roth et al., 1998), there were little expectations regarding the

difference between GVD alternatives since these had yet to be evaluated. Communication scores

were significantly higher in both the ecological and advanced display conditions for redundant

configurations. Since the screen content and navigation functionality of the two configurations

was identical, this finding suggests that the difference found between GVD types was systematic

and likely the direct result of differences in the location of the shared informational content in the

control room.

There are many possible explanations for this effect. I believe that the most likely reason stems

from differences in awareness of the behaviour of the other operator in a team. When looking at

the LSDs, operators were facing the same location. This allowed them to maintain sight of each

other which may have resulted in an implicit assumption of team-level situation awareness.

Because of this assumption, operators may have felt less need to communicate verbally with one

another. In the redundant conditions, the operating partners were out of each other’s peripheral

vision. This may have forced them to verbalize what they perceive to be meaningful behaviour in

the system because they were not assuming that their operating partner is seeing the same

information. These hypotheses will be discussed in further detail in the Future Research section

(see section 8.2.3).

78

There were significant limitations in the communication data. The communication metric was

based on ratings given by the experimenters during and after experimentation. Although we

attempted to be as unbiased as possible with our ratings, they were likely influenced by implicit

preconceptions. The same team of raters that recorded the time-based performance metrics was

used to rate each trial using behaviourally BARS. This method will yield much more reliable and

valid data describing the communication behaviour of the participants in the different

experimental trials. However, due to time restrictions, this data has not been included in the

present report.

7.3 Diagnostic Performance

Contrary to expectations, our results revealed no significant differences in diagnostic

performance between either display types or the display configurations. This unexpected result

was compounded by the fact that there was a significant positive correlation linking increased

SA to improved diagnostic performance. This finding suggests that although there was no direct

evidence linking improved performance to display type or configuration, there may be an

indirect link. In other words, since the ecological displays yielded better SA, and better SA

correlated with better performance, ecological displays may have indirectly improved diagnostic

performance.

The insignificant findings are likely due to unexpectedly low performance scores in the

redundant-ecological setup. This experimental setup yielded a mean estimate of just under 40%,

while the others were at 53%, 60%, and 63%, respectively. Since these performance metrics are

based on the averages of just 56 data points, they are more significantly impacted by variation in

the data. This surprising result should be examined in future studies.

There were two main limitations in the diagnostic performance data. The first is that the sole

indicator of performance presented in this thesis stems from an adaptation of a Lang et al. (2002)

method, which collapses all possible performance metrics into a single data point. We also

recorded time-based metrics for each group, including the time that they detected the fault, the

time after that detection that they began trouble-shooting the fault. Due to time limitations with

respect to receiving this data, it has not been included in the present report. However, we believe

that it will likely reveal interesting results linking the different display types to impacts on

performance.

79

The second limitation in performance data is the result of control actions being omitted from the

experiment. Unfortunately this was an unavoidable limitation in our study stemming from the

use of novice participants. Although this was inherent to our study, future work should look at

the same experimental questions with the addition of control actions.

Summary & Conclusions 8

8.1 Summary

This research was motivated by a finding from an operating experience review conducted by

Myers & Jamieson (2014) that there had yet to be a formal evaluation of the postulated benefits

of GVDs in nuclear control rooms. These claims are that GVDs will improve (Roth et al., 1998):

1. Operator situation awareness, 2. Communication within operating crews, and

3. Overall performance

This study therefore focused on developing and conducting an experiment capable of evaluating

different GVD alternatives along these three dimensions. Furthermore, we evaluated the

structure of the information presented on those GVDs to determine if ecological displays

outperformed the advanced displays along the same three dimensions.

We began the process of answering these questions by conducting a three-phased CWA. The

three phases we completed were work domain analysis, control task analysis, and strategies

analysis. Because the competencies of our experimental participants were known prior to

analysis, worker competencies analysis was interwoven into the three overt phases. Furthermore,

we were able to use these limited worker competencies to restrict operator actions to monitoring

and diagnosis. This removed the need for the social organization and cooperation analysis in our

CWA. This CWA provided us with the foundation on which to develop novel displays and

design a comprehensive experiment evaluating how these displays interact with different GVD

alternatives.

After completing the CWA, we developed a set of novel ecological displays that mirrored the

existing advanced displays in terms of informational content, navigation, and fidelity. They

differed significantly, however, in terms of how that content was presented to the operators,

focusing on depicting functional rather than physical relationships. While we attempted to use

80

existing ecological forms and elements that already existed in the display software’s graphical

suite, limitations in that software as well as perceived limitations in some ecological forms

necessitated the development of a novel ecological form.

This form depicts the abstract function level of the abstraction hierarchies from the work domain

analysis stage of our CWA. The abstract function level of the hierarchies describes the

underlying principles that drive system behaviour. Our analysis described these principles in

terms of their mass and energy balances focusing on the source, store, and sink of that mass or

energy. The form that we created illustrates these balances by presenting input, output, and how

they relate to the current level in a unified display. Furthermore, it shows how these individual

parameters change over time, both individually and in relation to one another.

Once the ecological displays had been created, we ran an experiment evaluating them against

advanced displays across two GVD configurations, LSD and redundant. This resulted in a 2X2

experimental design. Each operational team participated in each of the four possible display

setups (i.e., Configuration X Display Type) under a unique experimental scenario such that no

setup or scenario was repeated within an operating team.

The experimental data was categorized and recorded along the three posited benefits of GVDs.

Situation awareness data was recorded through the use of the process overview measure at two

locations in each trial. The communication scores that are presented here are based on ratings

given by the experimenters and are based on logs recorded during the experimentation. Finally,

the performance metrics were based on post-trial interviews wherein participants had to identify

the location and cause of a fault from the preceding scenario.

Results revealed marked SA improvement yielded by the ecological interfaces compared to the

advanced interfaces. There was a significant difference in communication between the display

configurations, with redundant displays outperforming the LSDs. Finally, although no significant

effects were found for diagnostic performance for either display configuration nor display type,

there was a significant correlation between diagnostic performance and SA. Therefore, we

believe that there may be an indirect benefit of ecological displays over advanced displays on

performance.

81

While the results describing the differences in display type are illuminating, the insignificant

difference between the LSD and redundant displays is also interesting. To our knowledge, no

previous study has examined the differences in SA, communication, and performance across

different GVD configurations. Our findings suggest that LSDs do not foster greater levels of SA,

communication, or performance compared to redundant displays. In fact, due to the increased

footprint required for LSD implementation, our results suggest that redundant displays may be

preferred in future control room designs.

8.2 Conclusions

8.2.1 Contributions

Although considerable improvements have been made to nuclear control rooms since the Three-

Mile Island accident, many of these improvements are driven by assumed benefits of new

technology rather than by empirical evidence. This thesis addresses this issue for the next stage

of control room evolution by systematically evaluating the configuration and framework by

which overview information is presented to operators. Our results suggest that the recent trend

towards the implementation of LSDs may not be the best option compared to redundant displays.

Not only does this result have direct implications for GVD technology, but it also illustrates the

need for empirical evaluation of safety-critical control room improvements.

This study did not find any evidence to suggest that the increased footprint required by LSDs is

warranted by SA, communication, or performance improvements. Further, we found evidence

suggesting that team communication is better in redundant conditions compared to LSD. In real-

world operating scenarios and in particular, unanticipated operating occurrences, this increased

communication may represent a critical aspect of performance and team situation awareness

(Juhasz & Soos, 2007). This suggests that although we did not find performance or SA benefits

from the redundant displays, there could be an indirect link that is formed by the communication

benefits of redundant displays. Furthermore, since both communication and performance are

multi-faceted constructs, it is possible that we simply did not measure the aspects of them that

would reveal correlation.

Our study also provides further support to the body of literature that has demonstrated SA

improvements elicited by ecological displays (e.g., Burns et al., 2008). We have shown these

82

benefits in a complex, full-scope environment, which further supports the ecological validity of

this framework.

In addition to substantiating the literature supporting EID and CWA, this study also advanced

EID by creating a novel, widely applicable graphical form. Although this design work was a

byproduct of the larger project, it resulted in a tool that is capable of displaying mass or energy

balances across domains. The form allows for easy problem detection, future state anticipation,

and online diagnosis. Operators are able to assess whether a system is balanced, unbalanced,

leaking, or blocked by looking at the behaviour of a single point on the delta plot. Since this

form was completely new to the participants, it is reasonable to assume that with more time to

adapt to the chart, further SA and performance benefits would accrue. The fact that the

ecological displays, which relied heavily on a form that the participants would have had very

little experience with, outperformed the advanced displays, who used forms the participants

would have familiarity with, on the first and only day of each team’s testing is a testament to the

potential utility of this tool in future applications.

8.2.2 Limitations

Several of the limitations of this study have been discussed in detail in the preceding sections,

but this section will summarize the major areas of concern. The principal limitation was the fact

that we used novice participants. Fortunately, we were aware of this prior to the analysis and

design phases of the study and were therefore able to accommodate their competencies in the

early stages of the study. However, studies in a domain that is exclusively operated by experts

should better reflect that user base. While we made all efforts to simplify the system and bring

our participants up to a reasonable level of expertise on that system, the results are still based on

a novice user base. As stated above, it should be noted that the participants scored similar levels

on the PO measure compared to previous simulator studies that have used expert operators (e.g.,

Burns et al., 2008). It should also be noted that if we had used expert operators, they would likely

have had experience with the existing advanced displays as well as the LSDs. This would have

likely biased the results. Therefore, using novice participants on a simplified system was not

without some benefits.

Another limitation related to our participants revolved around the amount of time we had with

them. Since we were drawing from a very select pool of active students, they were only able to

83

miss one day of classes. This meant that their training was only able to consist of a training

manual and one morning of in-person training on the systems. Furthermore, participants had a

very limited amount of time with which to familiarize themselves with the control interfaces and

navigation.

A third limitation related to our participants, was the limited sample size. This was significant

from a statistical analysis perspective, especially in terms of performance and communication

where there was less data per participant. Unfortunately, due to the nature of simulator research

in nuclear domains, a limited sample size is a common hindrance. Therefore, although our study

would have benefited from more participants, our sample size was similar to other studies in the

nuclear domain (e.g., Burns et al., 2008). This means that our methods and results are

comparable to other studies within the nuclear domain.

A final limitation in this report comes from the incomplete data presented in both the

communication and diagnostic performance sections. Although the main focus of our analysis

was on SA, this affected the validity of our conclusions regarding communication and

performance. This limitation will be amended once the BARS and performance timing data have

been coded and analyzed.

8.2.3 Future Research

Our study provided evidence suggesting that the current status quo in nuclear control rooms can

be improved upon. Future studies should continue this thread of research by not only examining

the benefits of different GVD alternatives, but also compare performance, situation awareness,

and communication between conditions with GVDs against conditions without GVDs.

Furthermore, studies should also longitudinally examine the same experimental question using

expert operators with the addition of control actions.

Future research should examine more thoroughly the pairings between GVD and display type. It

is possible that different display types perform better in different display configurations, as

demonstrated by the nearly significant contrast between the redundant-advanced and redundant-

ecological setups. This could demonstrate that the redundant displays are most effective when

combined with ecological displays.

84

In order to accurately capture the reason for communication differences between GVD

alternatives, future studies should examine the effect of different redundant display

configurations. We hypothesized that the improved communication scores in redundant

configurations was the result of not being able to see one’s operating partner. If this is correct,

then there should be a decrease in communication as the redundant displays are moved closer

together. Future studies should also assess the impact of these metrics on performance and team-

level situation awareness. It is possible that there are negative effects of assuming team-wide

situation awareness and that it is actually better to force the operators to verbally communicate

with one another. It should be noted that communications within nuclear are highly regulated,

and therefore with expert operators there may be less variation resulting from GVD

configurations. Therefore, future studies should also look at communication differences elicited

by GVD alternatives in fields with less regulated communication behaviour.

Again, despite the fact that design was not one of the principal concerns of this study, the tool

that was created proved to be very useful. Future research should examine the utility of this tool

over time. The tool appears to be a useful method of displaying information at the abstract

function level and we believe it can be used in a wide array of domains, particularly, in domains

where control actions are necessary.

85

References

Allison, P. D. (2008). Convergence Failures in Logistic Regression. SAS Global Forum, (5), 1–

11.

Beltracchi, L. (1988). An Expert Display System and Nuclear Power Plant Control Rooms. IEEE

Transactions on Nuclear Science, 35(2), 991–1000.

Bennett, K. B., & Malek, D. A. (2000). Evaluation of alternative waveforms for animated mimic

displays. Human Factors, 42(3), 432–450. doi:10.1518/001872000779698114

Bereznai, G. T., & Harvel, G. (2011). Workshop on NUCLEAR POWER PLANT SIMULATORS:

INTRODUCTION TO CANDU Systems and Applications.

Burns, C. M., & Hajdukiewicz, J. R. (2004). Ecological Interface Design. Boca Raton, Florida:

CRC Press.

Burns, C. M., Skraaning, G., Jamieson, G. a, Lau, N., Kwok, J., Welch, R., & Andresen, G.

(2008). Evaluation of ecological interface design for nuclear process control: situation

awareness effects. Human Factors, 50(4), 663–679. doi:10.1518/001872008X312305.

Butcher, K. R. (2006). Learning from text with diagrams: Promoting mental model development

and inference generation. Journal of Educational Psychology, 98(1), 182–197.

doi:10.1037/0022-0663.98.1.182

Candu Energy Inc. (2014). Main Control Room Mock-up Candu Energy.

Christoffersen, K., Hunter, C. N., & Vicente, K. J. (1996). A Longitudinal Study of the Effects of

Ecological Interface Design on Skill Acquisition. Human Factors, 38(3), 523–541.

Cornelissen, M., Salmon, P. M., Jenkins, D. P., & Lenne, M. G. (2013). How can they do it? A

structured approach to the strategies analysis phase of Cognitive Work Analysis.

Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 55(1), 340–

344. doi:10.1177/1071181311551070

Davey, E. (2000). Process Monitoring during Normal Operations at Canadian Nuclear Power

86

Plants. Proceedings of the Human Factors and Ergonomics Society Annual Meeting,

44(22), 811–814. doi:10.1177/154193120004402282

Elix, B., & Naikar, N. (2008). Designing safe and effective future systems: A new approach for

modelling decisions in future systems with Cognitive Work Analysis. Proceedings of the

8th International Symposium of the Australian Aviation Psychology Association.

Endsley, M. R. (1995). Toward a Theory of Situation Awareness in Dynamic Systems. Human

Factors, 37(1), 32–64. doi:10.1518/001872095779049543

Gary, M. S., & Wood, R. E. (2011). Mental Models, Decision Rules, and Performance

Heterogeneity. Strategic Management Journal, 32, 569–594. doi:10.1002/smj

Gwet, K. L. (2014). Handbook of Inter-Rater Reliability (4th ed.). Gaithersburg, MD: Advanced

Analytics, LLC.

High, R. (2014). Plotting Differences among LSMEANS in Generalized Linear Models, 1902–

2014. Retrieved from

https://sasglobalforum.activeevents.com/connect/fileDownload/session/7F75189251321B22

63401619B4AB46FB/1902_High_FinalPaper.pdf\nhttp://support.sas.com/resources/papers/

proceedings14/1902-2014.pdf

IAEA. (2009). Instrumentation and Control ( I & C ) Systems in Nuclear Power Plants : A Time

of Transition.

Infosystems, K. (2014). RGB Spectrum – Kiran Infosystems. Retrieved March 4, 2016, from

http://www.kiraninfo.com/rgb-spectrum/

Jamieson, G. A., & Vicente, K. J. (2001). Ecological interface design for petrochemical

applications: Supporting operator adaptation, continuous learning, and distributed,

collaborative work. Computers and Chemical Engineering, 25(7-8), 1055–1074.

doi:10.1016/S0098-1354(01)00678-0

Jamieson, G., & Miller, C. (2007). Integrating task-and work domain-based work analyses in

ecological interface design: a process control case study. Systems, Man and …, 37(6), 887–

905. Retrieved from http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4344953

87

Jenkins, D. P., Stanton, N. A., Salmon, P. M., Walker, G. H., & Rafferty, L. (2010). Using the

Decision-Ladder to Add a Formative Element to Naturalistic Decision-Making Research.

International Journal of Human-Computer Interaction, 26(2-3), 132–146.

doi:10.1080/10447310903498700

Johansson, B. J. E., & Persson, P. A. (2009). Reduced uncertainty through human

communication in complex environments. Cognition, Technology and Work, 11(3), 205–

214. doi:10.1007/s10111-007-0108-6

Joyce, J. P., & Lapinsky, G. W. (1983). A History and Overview of the Safety Parameter Display

System Concept. IEEE Transactions on Nuclear Science, NS-30(1), 744–749.

Juhasz, M., & Soos, J. K. (2007). Impact of non-technical skills on NPP Teams’

performance: Task load effects on communication. Human Factors and Power Plants and

HPRCT 13th Annual Meeting, 2007 IEEE 8th, 225–232. doi:10.1109/hfpp.2007.4413210

Kilgore, R. M., St-Cyr, O., & Jamieson, G. A. (2008). From Work Domains to Worker

Competencies: A Five-Phase CWA. Applications of Cognitive Work Analysis, 15–48.

doi:10.1201/9781420063059.ch2

Kim, D. Y., & Kim, J. (2014). Journal of Nuclear Science and Technology How does a change in

the control room design affect diagnostic strategies in nuclear power plants ? Journal of

Nuclear Science and Technology, (January 2015), 37–41.

doi:10.1080/00223131.2014.923792

Lang, A. W., Roth, E. M., Bladh, K., & Hine, R. (2002). Using a Benchmark-Referenced

Approach for Validating a Power Plant Control Room: Results of the Baseline Study.

Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 46(23), 1878–

1882. doi:10.1177/154193120204602302

Lau, N. (n.d.). Re: A few More Questions.

Lau, N., Jamieson, G. A., & Skraaning Jr., G. (2014). Inter-rater reliability of query/probe-based

techniques for measuring situation awareness. Ergonomics. Taylor & Francis.

doi:10.1080/00140139.2014.910612

88

Lau, N., Jamieson, G. A., Skraaning Jr., G., & Burns, C. M. (2008). Ecological Interface Design

in the Nuclear Domain: An Empirical Evaluation of Ecological Displays for the Secondary

Subsystems of a Boiling Water Reactor Plant Simulator. IEEE Transactions on Nuclear

Science, 55(6), 3597–3610. doi:10.1109/TNS.2008.2005725

Lau, N., Skraaning Jr., G., Eitrheim, M. H. R., Karlsson, T., Nihlwing, C., & Jamieson, G. A.

(2011). Situation Awareness in Monitoring Nuclear Power Plants – The Process Overview

Concept and Measure. Toronto, Ontario.

Lau, N., Veland, Ø., Kwok, J., Jamieson, G. A., Burns, C. M., Braseth, A. O., & Welch, R.

(2008). Ecological Interface Design in the Nuclear Domain : An Application to the

Secondary Subsystems of a Boiling Water Reactor Plant Simulator, 55(6), 3579–3596.

Lee, H.-C., & Seong, P.-H. (2009). A computational model for evaluating the effects of

attention, memory, and mental models on situation assessment of nuclear power plant

operators. Reliability Engineering & System Safety, 94(11), 1796–1805.

doi:http://dx.doi.org/10.1016/j.ress.2009.05.012

Lin, C. J., Yenn, T.-C., & Yang, C.-W. (2010). Evaluation of Operators’ Performance for

Automation Design in the Fully Digital Control Room of Nuclear Power Plants. Human

Factors and Ergonomics in Manufacturing, 20(1), 10–23. doi:10.1002/hfm

Maddox, M. E. (1996). Critique of “A longitudinal study of the effects of ecological interface

design on skill acquisition” by Christoffersen, Hunter, and Vicente. Human Factors, 38(3),

542–546. Retrieved from

http://go.galegroup.com.myaccess.library.utoronto.ca/ps/i.do?id=GALE%7CA19027801&si

d=googleScholar&v=2.1&it=r&linkaccess=fulltext&issn=00187208&p=AONE&sw=w

Meshkati, N. (1998). Lessons of Chernobyl and Beyond: Creation of the Safety Culture in

Nuclear Power Plants. Proceedings of the Human Factors and Ergonomics Society Annual

Meeting, 42(10), 745–749. doi:10.1177/154193129804201019

Montgomery, J., Gaddy, C., & Toquam, J. (1991). Team interaction skills evaluation criteria for

nuclear power plant control room operators. In Proceedings of the Human Factors Society

35th Annual Meeting (Vol. 15, pp. 918–922).

89

Mumaw, R. J., Roth, E. M., Vicente, K. J., & Burns, C. M. (2000). There Is More to Monitoring

a Nuclear Power Plant than Meets the Eye. Human Factors: The Journal of the Human

Factors and Ergonomics Society, 42, 36–55. doi:10.1518/001872000779656651

Murch, G. M. (1984). Physiological priniciples for the effective use of color. IEEE Computer

Graphics and Applications, 4(11), 49–54.

Myers, W. P., & Jamieson, G. A. (2014). Operating Experience Review of Large-Screen

Displays in Nuclear Power Plant Control.

Myers, W. P., & Jamieson, G. A. (2015). RETHINKING GROUP-VIEW DISPLAY

ALTERNATIVES, 2570–2577.

Nuclear Energy Agency. (2010). Public attitudes to nuclear power.

O’Hara, J. (2004). Plant Modernization Programs. Nuclear Plant Journal, 22(2), 47–48.

O’Hara, J. M., Brown, W. S., Lewis, P. M., & Persensky, J. J. (2002). Human-System Interface

Design Review Guidelines Human-System Interface Design Review Guidelines.

Washington, DC.

Rasmussen, J. (1974). The human data processor as a system component: Bits and pieces of a

model (Report No. Risø-M-1722). Roskilde, Denmark.

Rasmussen, J. (1985). The role of hierarchical knowledge representation in decisionmaking and

system management. IEEE Transactions on Systems, Man, and Cybernetics, SMC-15(2),

234–243. doi:10.1109/TSMC.1985.6313353

Rasmussen, J. (1993). Diagnostic reasoning in action. IEEE Transactions on Systems, Man and

Cybernetics, 23(4), 981–992. doi:10.1109/21.247883

Rasmussen, J., Pejtersen, A. M., & Goodstein, L. P. (1994). Cognitive systems engineering. New

York: Wiley.

Roth, E. M., Brockhoff, C. S., Rusnica, L. A., Kenney, S., Kerch, S. P., Thomas, V. M., &

Sugibayachi, N. (1997). Rapid prototyping and simulator evaluation of a wall panel

90

overview display. In IEEE 6th Annual Human Factors Meeting (Vol. 18, pp. 14–19).

Orlando. doi:10.1017/CBO9781107415324.004

Roth, E. M., Lin, L., Thomas, V. M., Kerch, S., Kenney, S. J., & Sugibayashi, N. (1998).

Supporting situation awareness of individuals and teams using group view displays.

Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting, 1, pp.

244–248.

Roth, E., & O’Hara, J. (2002). Integrating digital and conventional human-system interfaces:

Lessons learned from a control room modernization program. Retrieved from

http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:Integrating+Digital+and+

Conventional+Human-

System+Interfaces:+Lessons+Learned+from+a+Control+Room+Modernization+Program#0

Tharanathan, a., Bullemer, P., Laberge, J., Reising, D. V., & Mclain, R. (2012). Impact of

Functional and Schematic Overview Displays on Console Operators’ Situation Awareness.

Journal of Cognitive Engineering and Decision Making, 6(2), 141–164.

doi:10.1177/1555343412440694

Vicente, K. (1999). Cognitive Work Anlaysis: Toward Safe, Productive, and Healthy Computer-

Based WOrk. Mahwah, N.J.: Lawrence Erlbaum Associate.

Vicente, K. (2002). Ecological interface design: progress and challenges. Human Factors, 44(1),

62–78. doi:10.1518/0018720024494829

Vicente, K., & Rasmussen, J. (1992). Ecological interface design: Theoretical foundations. IEEE

Transactions on Systems, Man, and Cybernetics, 22(4), 589–606. doi:10.1109/21.156574

Vicente, K., Roth, E. M., & Mumaw, R. J. (2001). How do operators monitor a complex,

dynamic work domain? The impact of control room technology. International Journal of

Human-Computer Studies, 54(6), 831–856. doi:http://dx.doi.org/10.1006/ijhc.2001.0463

World Nuclear Association. (2014). History of Nuclear Energy. Retrieved January 8, 2016, from

http://www.world-nuclear.org/info/Current-and-Future-Generation/Outline-History-of-

Nuclear-Energy/

91

World Nuclear Association. (2016). Nuclear Power Today | Nuclear Energy. Retrieved January

8, 2016, from http://www.world-nuclear.org/info/Current-and-Future-Generation/Nuclear-

Power-in-the-World-Today/

Zhang, Y. (2008). The influence of mental models on undergraduate students’ searching

behavior on the Web. Information Processing and Management, 44(3), 1330–1345.

doi:10.1016/j.ipm.2007.09.002

92

Appendices

Appendix A. Information Requirements

HEATTRANSPORTSYSTEMLevel Box InformationRequirements

Reactivity PoweroutputTemperature???SteamGenerators Levelcontrol Pressurewithin Steamwithin H2OIN/OUT D2OIN/OUT Steamout BlowdownRate

D2OCoolant TotalmassReactorInletheaders Pressureatheaders Volumethrough ratethroughReactoroutletheaders Pressureatheaders Volumethrough Ratethrough

FlowthroughPressurewithinHeatgeneration EnthalpyinsystemFuelCooling Efficiencyofexchangesteamgeneration QuantityofsteamgeneratedCoolantCirculation Massdistribution TotalvolumeCavitationprevention Pressuredistribuiton

Fuelin PoweroutFlowpumpenergy Massthrough

Massout HeatoutTotalmassinsystem

TotalSystemEnthalpy Totalsystempressure(~10Mpa)CoolantCirculation VolumeincirculationMassfromP&IC InputRateMasstoP&IC OutputRate

AveragetemperaturePhysicalFunction

GeneratlizedFunction

AbstractFunction

Reactivity

Averagetemperature

CoolantflowSteamGeneratorSink

HTPumps

Reactor

Pressure&InventoryControlLevel Box InformationRequirements

DegasserCondenser Level BleedRate FeedRate Steambleedrate Pressure TemperatureOpen/Close Flowthrough PressureatTemperature Tempbefore/after Flowthrough

steamD2Obleedvalves Open/Close FlowthroughShutdownCoolingSystem

PressureInside Pressureoutside Levelinside Steamlevelinside Inputrate outputrateLiquidD2Obleedvalves Open/Close FlowThrough

FeedrateOpen/Close Flowthrough

D2OStoragetank Level Input OutputPurificationcircuit FissionproductlevelsinD2O

LiquidD2OtemperaturereductionCondenseD2OSteam SteamD2Olevel

SteamD2OcoolingFlowthrough

D2OtemperatureincreaseMaintainPZRpressureatsetpoint PZRPressureLCwithincondenserPressureabsorptionfromD2OswellCoolantinsertionExcesscoolantremovalAcceptexcessinventoryPreventdrainingPreventliquidoverflowRemoveradioactivefissionproductsformD2O

Pressureloss HeatlossHeatloss

EnergyinLiquidD2OEnergyinsteamD2O

HeatinCoolantinflow

MassrecycleMassfromHTS Volumein

DegasserCooler

FeedValves

Refluxflow

Cooling

Heaters

GeneralizedFunction

AbstractFunction

Physicalfunction

TempbeforeandafterTotalsystempressure

VolumeinstoragetankRadioactivityofD2O

LiquidD2OsteamlevelD2Otemperature

Tempbeforeandafter

Temperaturebefore/after

VolumeofD2Oincondenser

Feedpumps

Pressurizer

refluxvlaveheaters

VolumeofcoolantinsystemVolumeofcoolantinsystem

Volumeinstoragetank

D2OstorageD2OtoHTS

TemperaturebroughtinfromD2O

RadioactivityofD2OcontentsD2Opurity

PressureofD2OTemperatureinD2O

TotalvolumeinstorageTotalvolumecyclinginthesystem

VolumeofinventorycirculatingintheHTS

VolumeinsystemVolumeinsystemVolumeinstoragetank

TemperatureinD2O

Spray

PressureofD2O

93

Table2.3.BOILERANDSTEAMGENERATORLevel Box InformationRequirements

Percentfull TemperaturePressurewithin

Levelwithin Presurewithin

Feedwaterinlet Pressureat Flowthrough

Temperature

Capacity

Steamscrubbers Integrity Moistersaturation

Cycloneseparators On/off Rotationrate

Downcomerannulus Level

Level PercentFull

Heavywaterinlet Pressureat Flowrate

U-Tube Pressureat Flowrate

Heavywateroutlet Pressureat FlowRate

HeatExchange Heatexchangerate

Lightwatervaporization TemperatureofD2O TemperatureofH2O PressureinBSG

Lightwatercirculation Overalflowrate Pressurewithintheboiler TemperatureofH2OTemperatureatreheaterdrainsinlet

Impurityremoval Removalrate

Moistureremoval Moistercontentbefore/after

Waterrecycling Flowthroughrecyclingcomponents

Steamrelease Outflowrate Presssureatsteamoutletnozzle

Heavywatercirculation Overalflowrate

Inputheavywaterenergy

Inputlightwaterenergy

Heavywaterenergy(withinboiler)

Lightwaterenergy(withinboiler)

Steamenergy(withinboiler)

Outputsteamenergy

Outputheavywaterenergy

Inputlightwatermass Inputrate

Lightwatermass(withinboiler)

Steammass(withinboiler)

Waterextractedfromsteammass

Outputsteammass

Outputlightwatermass

PressurePrinciples Pressure-temperature-boilingpointrelationship

Inputheavywatermass

Heavywatermass(totalinsystem)

Outputheavywatermass

DeliverrequiredquantityofsteamtoMainsteamsystemTargetsteamquantity Currentsteamquantity

RemoveheatfromHTS Targetheatremoval Currentheatremoval

PreventoverpressurizationofSG Pressurelimit Currentpressure

Abstract

Function

EnergyBalance

Lightwatermassbalance

Heavywatermassbalance

Functional

Purpose

Physical

Function

Generalized

Function

Steamoutlet

Blowdown

Preheater

Tubebundle

Risersection

94

Level Box InformationRequirements NotesGenerator Generatoroutput 680MWeLPTurbine Flowrate,output kg/h,MWHPTurbine Flowrate,output kg/h,MWEmergencyStopValve Open/closeGovernorvalve %openSafetyValve Open/close Combinedreliefcapacityof3/4is115%ofsteamflowfromeachSGASDVs Open/closeCSDVs Open/closeCondenserandhotwell CondenserlevelMoistureseparatorHPThrottlevalves steamflowthrough 957kg/stotheturbinethrottlevalvesSteamreheater livesteamflowtothesteamreheater 90kg/slivesteamtothesteamreheaterConvertlatentheatofsteamtomechanicalenergyEntropy(T,Pofsteamenteringtheturbines) steamenteringat250˚Cand4000kPa(density-.;05m^3/kg),leavingat35˚Cand5kPa(Density-25.2m^3/kg)Steamquantitycontrol flowthroughthegovernorvalvesandASDVs PositionofgovernorvalvessetbycombinationofspeedersettingandfrequencyerrorRapidlycutflowtoturbine Emergencystopvalvesandinterceptvalvesclose-openstatusCondensatecreation Pincondenser ~5kPa(absolutepressure),35˚C,10%moistureCondensatecollection T,Pofsteam Condensedsteamat5kPaiscollectedinthehotwellMoistureremoval %moisture/moistureremovalrate Steamleavesturbineat900kPa,170˚Cand10%moisture

Designedtoachievefullliftat4%abovesetpressuresSteamheatenergy energytransferfromtheteamgeneratorstothehpturbineSystempressure systempressureacrosscomponentsElectricaloutput enthalpy(energytransferfromtheLPturbinestothegenerator)

Atm.PressuredischargeenergyreleasetotheatmospherethroughASDVsSteamfromSG SteammassenteringfromSGDryvapormass DryvapormassWetVaporMass WetVaporMassPooledcondensate CondenstaemassFeedwatersystem MassinfeedwatersystemAtmosphericdischargesteamflowthroughtheASDVsifany

Maintainpressure:temp.ratio.Steaminjectionheat Flowin/outofreheaterMitigatepotentialfordamagetothesystemASDVs Potentiallyfaultysystems&componentsConvertheatintomechanicalenergyforthegeneratorGeneratoroutput,turbinespeedanderrorProvideCondensatetotheLPfeedwatersystemOutputrate

EnergyBalance

MassbalanceAbstractFunction

Functionalpurpose

total=1047kg/s

PhysicalFunction

Generalizedfunction

Table2.4.STEAMSUPPLYSYSTEM

Level Box InformationRequirements NotesGlandseal Integrity PotentiallygeneralIRforsystemintegrity?Diaphragm Wear,integrityCasing(upper)Casing(lower)ShaftBlades RotationalcapacityBearingsFixedbladesLowpressureoutlets PressureExtractionsteam(pump) PumpcapacityCondenser Coolingarea(sqm)Frictionprevention TempofbearingsLeakageprevention Pressuredrop(deviationfromnormalvalues)Steam-drivenshaftrotation RPM(revolutionperminute)Heattransfertohighvelocitykineticenergy P,TofheatTransfersteamtolowpressureturbines Outletpressure OutputrateExhaustlowpressuresteamtocondenser PressureRemovemoisturecontentfromturbines Moisture%Vibration mm/s

Shaftrotation TorqueLeakage Energybalance(conservationofenergy)/mass,energyHeatenergy InflowSteamenergy SteamtransferratetoturbinesRecycleenergyRotationalenergy MomentumSteaminflow SteamflowratefrombalanceheaderRecycledsteam Flowrate(kg/s)Condensate(wetsteam)Transferratetothecondenser

Content Moisture MoistureremovalrateConvertsteampressuretorotationalenergyCurrentMWoutputvssetpointPreventsystemdamage SystempressureMaximizeefficiencyofrotation CurrentRPMvsgoal

PhysicalFunction

GeneralizedFunction

Massbalance

Energybalance

AbstractFunction

Table2.5.TURBINECONTROL

Functionalpurpose

95

Appendix B. Experimental Schedule

TRIALNO.Scenario Setup Scenario Setup Scenario Setup

1 1 1 2 3 3 12 2 2 3 4 4 23 3 3 4 1 1 34 4 4 1 2 2 4

TRIALNO.Scenario Setup Scenario Setup Scenario Setup Scenario Setup Scenario Setup

1 4 3 4 4 1 2 2 4 3 22 1 4 1 1 2 3 3 1 4 33 2 1 2 2 3 4 4 2 1 44 3 2 3 3 4 1 1 3 2 1

SCENARIOS Setups1 HTSLeak 1 QinshanLSD2 FWvalveclosure 2 QinshanRedundant3 SpuriousMSSVOpening 3 EIDLSD4 PZRBleedpathClosure 4 EIDRedundant

Team4 Team5 Team6 NOV.5 NOV.6

OCT.26 PILOT OCT.28 Team2 Team3

96

Appendix C. Process Overview Queries

HTSLeak

Time 11m

Decreased Increased Inferred?

X

X

X

X *

X

X

X

X *

X *

X *

X

X

Time 20:00

Decreased Increased

X

X

X

X

X *

X

X

X

X *

X *

X *

X

Stayedthe

Same

Recently,themass(kg)leavingtheprimaryheattransportcircuithas

Recently,thedifferencebetweenreactorpowerandturbinepower(%FP)has

Recently,themassinsidetheD2Ostoragetank(kg)has

Recently,themassinsidetheperssurizer(kg)has

Recently,thedegasser-condenserlevel(m)has

Recently,thepressurizerpressure(MPa)has

Recently,theHTSpressure(MPa)has

Recently,theheatouputfromthepressurizer'sheaters(kj/s)has

Recently,theturbineoutput(MW)has

Recently,thesteamgeneratorpressure(MPa)has

Recently,thepressurizerlevel(m)has

Recently,theD2OStorageTanklevel(m)has

Stayedthe

Same

Recently,theD2OStorageTanklevel(m)has

Recently,thepressurizerlevel(m)has

Recently,themassinsidetheperssurizer(kg)has

Recently,theheatouputfromthepressurizer'sheaters(kj/s)has

Recently,thedifferencebetweenreactorpowerandturbinepower(%FP)has

Recently,themass(kg)leavingtheprimaryheattransportcircuithas

Recently,thesteamgeneratorpressure(MPa)has

Recently,theturbineoutput(MW)has

Recently,themassinsidetheD2Ostoragetank(kg)has


Recently,thepressurizerpressure(MPa)has

Recently,thedegasser-condenserlevel(m)has

97

FEEDWATERVALVECLOSURETime 8:45

Decreased Increased

X

X

X

X

X

X

X

X

X

X

X

Recently,thevalveopeningtosteamgenerator1(%)has X

Time 17:00Decreased Increased

X

X

X

X

Recently,thevalveopeningtosteamgenerator1(%)has X

X

X

X

X

X

X

X

Recently,thelevelinsteamgenerator4(m)has

StayedtheSame

Recently,theTurbineOutput(MW)has

Recently,thedifferencebetweenthesteamgeneratorlevels(m)has


Recently,theaveragesteampressure(KPa)has

Recently,thefeedflowintosteamgenerator4(kg/s)has

Recently,theaveragesteamflowleavingthesteamgenerators(kg/s)has

Recently,thesteamgeneratorfeedpumppressurehas

Recently,thePressurizer(m)levelhas


Recently,theaveragemassinthesteamgeneratorshas

Recently,thePressurizerlevel(m)has


Recently,theaveragemassinthesteamgeneratorshas


StayedtheSame


Recently,thediffernecebetweenthesteamgeneratorlevels(m)has


Recently,thesteampressure(KPa)has

Recently,thefeedflowintosteamgenerator4(kg/s)has

Recently,thesteamflowleavingthesteamgenerators(kg/s)has

Recently,theaveragesteamgeneratorfeedpumppressurehas

98

MSSVOpeningTime11m Decreased Increased

X

X

X

X

X

X

X

X

X

X

X

X

Time20m Decreased Increased

X

X

X

X

X

X

X

X

X

X

X

X

StayedtheSame


Recently,theaveragelevelinthesteamgenerators(m)has

Recently,theaverageflowoffeedwaterintothesteamgenerators(kg/s)has

Recently,themassinthecondensatestoragetank(kg)has

Recently,thepressureofthesteam(MPa)leavingtheturbineshas

Recently,thelevelinthecondensatestoragetank(m)has



Recently,thesteambeingdischargedtotheatmosphere(Kg/s)has

Recently,thesteamflowleavingsteamgenerator1(kg/s)has

StayedtheSame


Recently,theaveragelevelinthesteamgenerators(m)has

Recently,theaveragepressureinthesteamgenerators(MPa)has

Recently,thepressureofthesteam(MPa)leavingtheturbineshas

Recently,theaveragepressureinthesteamgenerators(MPa)has

Recently,thedifferencebetweenreactorpowerandturbinepower(%)has

Recently,thedifferencebetweenreactorpowerandturbinepower(%)has

Recently,theaverageflowoffeedwaterintothesteamgenerators(kg/s)has

Recently,thesteamflowleavingsteamgenerator1(kg/s)has

Recently,thesteambeingdischargedtotheatmosphere(Kg/s)has



Recently,thelevelinthecondensatestoragetank(m)has

Recently,themassinthecondensatestoragetank(kg)has

99

PZRBleedClosureTime 10:30

Decreased Increased

Recently,theD2OStorageTanklevel(m)has X

Recently,thedegasser-condenserlevel(m)has X

Recently,theflowleavingthepressurizer(kg/s)has X

Recently,theheatoutputfromthepressurizer'sheaters(Kj/s)has X

Recently,theHTSpressure(MPa)has X

Recently,themassinsidetheD2OStorageTank(kg)has X

Recently,themassinsidethepressurizer(kg)has X

Recently,thepressurizerlevel(m)has X

Recently,thepressurizerpressure(MPa)has X

Recently,thesteamgeneratorlevel(m)has X

Recently,thesteamgeneratorpressure(KPa)has X

Recently,theturbineoutput(MW)has X

Time 18:00Decreased Increased

Recently,theturbineoutput(MW)has X

Recently,thesteamgeneratorpressure(KPa)has X

Recently,thesteamgeneratorlevel(m)has X

Recently,thepressurizerpressure(MPa)has X

Recently,thepressurizerlevel(m)has X

Recently,themassinsidethepressurizer(kg)has X

Recently,themassinsidetheD2OStorageTank(kg)has X

Recently,theHTSpressure(MPa)has X

Recently,theheatoutputfromthepressurizer'sheaters(Kj/s)has X

Recently,theflowleavingthepressurizer(kg/s)has X

Recently,thedegasser-condenserlevel(m)has X

Recently,theD2OStorageTanklevel(m)has X

StayedtheSame

StayedtheSame

100

Appendix D. Selected Statistical Outputs

a) GLIMMIX output comparing visible to inferred queries.

0 refers to visible queries, while 1 refers to inferred.

b) LS mean comparisons for PO scores.

101

Appendix E. Selected SAS Code

a) Mixed Generalized Linear model for SA proc GLIMMIX data=subresults plots=anomplot plots=controlplot plots=diffplot ic=Q; class LR AE LRAE SET CON GID SID trial quesno; model score = LR AE LR*AE/ dist=binomial link=logit oddsratio solution; random con; random GID; random SID(GID); random trial; lsmeans AE LR AE*LR/cl ilink adjust=sim plot=diffplot plot=meanplot(plotby=LR join ilink) ; ods output lsmeans=lsmeans; output out=glimresid (drop=TID QQ QUESNO INF FLAG CQQ) pred=pred2 / allstats; run;

b) Mixed Linear Model for Communication

proc mixed data=commdat plots=none cl; class set trial lr AE con GID; model comsqrt = LR|AE; random con/ s; random GID /s; random trial/ s; lsmeans LR|AE/pdiff cl; ods output lsmeans=lsmeansout2; run;

c) Mixed Linear Model for Performance

proc mixed data=perfdata2 plots=none; class TID GID SID LR AE LRAE CON; model IDSCORE=LR|AE; Random con; Random GID; Random SID(GID); lsmeans LR|AE/pdiff cl; ods output lsmeans=pertable2; run;

a simulator study evaluating the efficacy of group … simulator study evaluating the efficacy of...

Documents