Verification and Validation of Complex Systems: Human Factors Issues

NATO ASI Series Advanced Science Institutes Series

A series presenting the results of activities sponsored by the NA TO Science Committee, which aims at the dissemination of advanced scientific and technological knowledge, with a view to strengthening links between scientific communities.

The Series is published by an international board of publishers in conjunction with the NATO Scientific Affairs Division

A Life Sciences B Physics

C Mathematical and Physical Sciences

D Behavioural and Social Sciences

E Applied Sciences

F Computer and Systems Sciences

G Ecological Sciences H Cell Biology I Global Environmental

Change

NATo-pea DATABASE

Plenum Publishing Corporation London and New York

Kluwer Academic Publishers Dordrecht, Boston and London

Springer-Verlag Berlin Heidelberg New York London Paris Tokyo Hong Kong Barcelona Budapest

The electronic index to the NATO ASI Series provides full bibliographical references (with keywords and/or abstracts) to more than 30000 contributions from international scientists published in all sections of the NATO ASI Series. Access to the NATO-PCO DATABASE compiled by the NATO Publication Coordination Office is possible in two ways:

- via online FILE 128 (NATO-PCO DATABASE) hosted by ESRIN, Via Galileo Galilei, 1-00044 Frascati, Italy.

- via CD-ROM "NATO Science & Technology Disk" with user-friendly retrieval software in English, French and German (© WTV GmbH and DATAWARE Technologies Inc. 1992).

The CD-ROM can be ordered through any member of the Board of Publishers or through NATO-PCO, Overijse, Belgium.

Series F: Computer and Systems Sciences Vol. 110

Verification and Validation of Complex Systems: Human Factors Issues

Edited by

John A. Wise Center for Aviation/Aerospace Research Embry-Riddle Aeronautical University Daytona Beach, FL 32114-3900, USA

V David Hopkin United Kingdom Civil Aviation Authority Farnborough, Hampshire GU14 6SZ, United Kingdom

Paul Stager Department of Psychology York University Toronto, Ontario M3J 1 P3, Canada

Springer-Verlag Berlin Heidelberg GmbH

Proceedings ofthe NATO Advanced Study Institute on Verification and Validation of Complex and Integrated Human-Machine Systems, held in Vimeiro, Portugal, July 6-17,1992

CR Subject Classification (1991): D.2.4, C.4, J.2, C.3, J.7

ISBN 978-3-642-08155-2 ISBN 978-3-662-02933-6 (eBook) DOI 10.1007/978-3-662-02933-6

This work is subject to copyright. AII rights are reserved, whether the whole or part 01 the material is concerned, specilically the rights 01 translation, reprinting, reuse 01 illustrations, recitation, broadcast­ing, reproduction on microlilms or in any other way, and storage in data banks. Duplication 01 this publication or parts thereol is permitted only under the provisions 01 the German Copyright Law 01 September 9, 1965, in its current version, and permission for use must always be obtained Irom Springer-Verlag. Violations are liable lor prosecution under the German Copyright Law.

© Springer-Verlag Berlin Heidelberg 1993

Originally published by Springer-Verlag Berlin Heidelberg New York in 1993

Softcover reprint of the hardcover 1 st edition 1993

Typesetting: Camera ready by Embry-Riddle Aeronautical University 40/3140 - 5 43210 - Printed on acid-Iree paper

Acknowledgments

The editors would like to acknowledge the work of those individuals whose untiring effort and dedication to the Advanced Study Institute made possible the publication of this manuscript. In a way, each of them should have his or her name associated with this text because without any one of them, the text would not exist.

We must thank a large number of people and organizations for the success of the Institute, beginning with our sponsors. Only through the support of our sponsors was it possible to undertake the Institute and to bring representatives from so many countries together for a two week period. The sponsors for the Institute included:

• NATO Scientific Affairs Division • Eurocontrol • U.S. Federal Aviation Administration • Embry-Riddle Aeronautical University • U.S. Department of Defense • Research Institute for Information Science and Engineering

We must also thank the U.S. National Science Foundation for supplying financial support for several participants

We are grateful to the Institute's staff who worked hard before, during, and after the meeting, We are particularly indebted to Barbara Gibson, whose administrative skills were extraordinary. The outstanding work in video and audio recording the Institute by Kevin Norris, and the technical and photographic support by James Gibson directly contributed to the success of the meetings. The Editors owe a significant debt to Dr. Geoffrey Kain for his technical editing of the proceedings and in particular for his work with those authors who do not have English as their first language. Finally, we must thank the students who assisted in the preparation of the papers for publication: Bill Becher, Jose Gandara, Michael Graves III, Len Hennessy, Florian Jentsch, Yves Koning, Kerwin McKenzie, Martin Quinones, Paul Wassell, and Mark Wise.

The participants contributed actively in the exchange of their views and experiences which were drawn from a diversity of backgrounds and national origins. We would like to thank all of the participants for their contributions to the discussions throughout the Institute and for their preparation of their individual position papers.

As Co-Directors of the Institute, it was our good fortune to have received the support of these many individuals and the sponsoring organizations in bring this volume to publication.

John A. Wise V. David Hopkin Paul Stager

VII

Preface

Rapid advances in technology and software have provided the capability to develop very complex systems with highly interrelated components. While this capability has permitted significant increases in system efficiency and has allowed the development and operation of systems that were previously impossible (e.g., negative stability aircraft), it has also brought the inherent danger of system induced catastrophes. Perrow (1984), in his book Normal Accidents, demonstrated that systems which are highly-complex and highly-coupled (i.e., have highly interdependent components) have an inherent disposition toward massive failure.

Highly-coupled systems often create new types of failures. Interrelated components that were previously independent can cause unpredicted failures in each other. For example, the tests of wide-bodied aircraft initially used the same criteria for cabin depressurization as those for older narrow bodied aircraft. When a DC-l 0 lost a cargo door in flight because an unskilled ground crew could not apply a complex locking procedure, the insufficient means of depressurizing the cabin caused the floor to buckle and jam the controls. The unpredicted coupling of ground personnel skill, cabin pressure, and flight controls resulted in a crash and the loss of many lives.

This instability makes the verification/validation process even more important than it has been in the past, while the coupling makes traditional modular testing obsolete. As complex systems become more coupled, interdisciplinary issues also become more critical. Nowhere is this more true than in the person-machine interface. It is likely that new operational interface problems will reside in locations where disciplines (and the system components relevant to their domain) meet and interact. It is in these intellectual intersections that most new compromises and cross-discipline trade-offs are made. And it will be in these intersections, that new and unanticipated interface-induced failures will emerge.

With increasing system complexity and integrality, the employment of external independent criteria for verification and validation purposes becomes impractical, and alternative internal criteria intrinsic to the planning and design of a system must be sought. However, verification and validation methods must not only be effective, they must also be cost effective. Thus, criteria which will enable the cost effectiveness of the verification and validation procedures to be demonstrated are also required. For example, the decision to limit testing of the Hubble Space Telescope before launch was based, in part, on the cost involved in its validation.

Technically adequate testing may not even be sufficient - or in some cases even relevant - to a system becoming safely operational. The political and emotional issues associated with the acceptance of some technically adequate systems (e.g., nuclear power, totally automatic public transportation systems) must also be considered. For many systems, the evaluation must answer questions beyond safety and reliability. What type of evaluation will be acceptable to the users and the public? Likewise, how much will the public be willing to spend to test the system? What level of security and reliability will they demand from the system?

In spite of the fact that the importance of verification/validation of the interface is increasing, the processes by which it is accomplished are perhaps the most overlooked aspect of the system development. Although a considerable amount has been written about the design and development process, very little organized information is available on how to verify and validate highly-complex and highly-coupled dynamic systems. For example, a 1986 NATO Advanced Research Workshop (Wise & Debons, 1987) addressed the process of trying to determine the cause of a failure post Jacto; however, little has been done to improve the

VIII Preface

processes that will identify potential problems before they cause a failure. In fact, the inability to evaluate such systems adequately may become the limiting factor in our ability to employ systems that our technology and knowledge will allow us to design.

This volume has been developed to provide guidance for the verification and validation of all highly complex and coupled systems. In these proceedings, air traffic control was used as an exemplar in order to provide a focus (i.e., to assure the theory is described in terms that will allow its application). Air traffic control is perhaps the best current example in the western democracies. Not only is contemporary air traffic control a very complex, dynamic, and a highly coupled process, but it also has very significant social, political, and economic impacts. Air traffic system failures not only result in significant economic losses (e.g., the U.S. government estimates the current air traffic system annually induces over a $5 billion economic loss in the U.S. alone), but also can result in the loss oflives.

The Advanced Study Institute (ASI) tried to build on the accomplishments of the 1990 NATO ASI "Automation and System Issues in Air Traffic Control" (Wise, Hopkin, & Smith, 1991). That Institute provided designers and other key decision makers with the most up-to­date knowledge and theory relevant to automation issues in the design process, but it was not able to address the issues of verification and validation in the time available. The 1992 ASI represented by these proceedings continues the process by providing that knowledge.

This volume presents relevant knowledge and theory in a format that will enable readers, who may be working in diverse contexts, to apply the information to the systems for which they are responsible. The objectives of the proceedings were to describe those domains where significant advances have been made in identifying potential problems, to articulate the associated human factors issues, and to review new methodologies, especially those that address the cross-disciplinary nature of verification and validation.

References

1. A. Wise V. David Hopkin Paul Stager

Perrow, C. (1984). Normal Accidents: Living with High-Risk Technologies. New York: Basic Books.

Wise, 1. A., & Debons, A. (Eds.). (1987). Information Systems: Failure Analysis. NATO ASI Series F, Vol. 32. Berlin: Springer-Verlag.

Wise, 1. A., Hopkin, V. D., & Smith, M. L. (Eds.). (1991). Automation and Systems Issues in Air Traffic Control. NATO ASI Series F, Vol. 73. Berlin: Springer-Verlag.

Table of Contents

Editors' Summary ................................................................................................................. 1

Perspectives on Verification and Validation ............................................. 7

Verification and Validation: Concepts, Issues, and Applications .......................................... 9 V. David Hopkin

Resilience Theory and System Evaluation ............................................................................ 35 Harold D. Foster

On The Future Of Hybrid Human-Machine Systems ........................................................... 61 P. A. Hancock

Basic Considerations in Verification and Validation ............................................................. 87 John A. Wise and Mark A. Wise

Developing Definitions and Approaches ...................................................... 97

Validation in Complex Systems: Behavioral Issues .............................................................. 99 Paul Stager

Defining Human-Centered System Issues for Verifying and Validating Air Traffic Control Systems .................................................................................................. 115 Kelly Harwood

Complexity in a Systems Context ..................................................................... 131

Evaluating the Impact of New Technology on Human-Machine Cooperation ...................... 133 David D. Woods and Nadine B. Sarter

Integrating Verification and Validation with the Design of Complex Man-Machine Systems ..................................................................................... 159 William F. Stubler, Emilie M. Roth, and Randall J. Mumaw

Assessment of Complexity ................................................................................................... 173 Peter A. Wieringa and Henk G. Stassen

X Contents

Limits to Analysis and Verification ....................................................................................... 181 Ragnar Rosness

The Validation and Verification of Complex Knowledge-Based Systems ............................ 193 Robert T. Plant

Reliability, Errors, and Safety .......................................................................... 203

The Reliability Of Interactive Systems: Simulation Based Assessment.. .............................. 205 Erik Hollnagel

The Identification of Latent Organizational Failures in Complex Systems ............................ 223 James Reason

The Role ofIncident Investigation in System Validation ...................................................... 239 Sue Baker

Problems of Systematic Safety Assessments: Lessons Learned from Aircraft Accidents ........................................................................................................ 251 Florian G. Jentsch

Major Incidents, Safe and Reliable Verdicts and the Process of Verification and Validation ............................................................................................... 261 Clive John A. Andrews

Operator Capabilities and Variability ........................................................... 279

The Human Component of System Validation ..................................................................... 281 P.G.A.M. Joma

When Task Demand is Variable: Verifying and Validating Mental Workload in Complex, "Real World" Systems ..................................................................................... 305 Mark W. Smolensky and Lloyd Hitchcock

Performance Evaluation of Human-Machine Systems ......................................................... 315 A.F. Sanders & P.H.M.P. Roelofsma

Requirements Analysis for Human System Information Exchange ..................................... 333 Jeremy Clare

Working Memory and Human-Machine Systems ................................................................ 341 Robert H. Logie

Contents XI

Mental Models in Operational Systems ......................................................... 355

The Role of Verification and Validation in the Design Process of Knowledge Based Components of Air Traffic Control Systems ........................................................................ 357 Marcel Leroux

Automation and Representation in Complex Man-Machine Systems ................................... 375 Harald Kolrep

How to Fit the Man-Machine Interface and Mental Models of the Operators ....................... 381 Michael Dubois and Jose Gaussin

The Cultural Context ............................................................................................... 399

Cultures with Requisite Imagination ..................................................................................... 40 1 Ron Westrum

System Validation - A Step in a Continuous Improvement Process. ... ... ... . ................... .417 Gerd Svensson

Cultural Behavior in the Airline Cockpit System: A Theoretical Framework Proposal ...................................................................................... 423 Alejandro Perez Chavez

Involving the Users in Verification and Validation Processes .......................................................... .433

The Inclusion of Future Users in the Design and Evaluation Process .................................. .435 Patrick Dujardin

User Involvement in the Development of Highly Interactive Software Systems ................. .443 Richard Jack

Psychological Aspects of Human Factors Testing and Evaluation of Military Human-Machine Systems ....................................................................................... 453 Gerhard L. Schaad

Involving the User in the Design of Computer-Based Displays in Power Plant Control Rooms ............................................................................................. 457 E. C. Marshall

XII Contents

The Need for User Involvement... ... ................................................................. .463

Systems Theory Versus Verification and Validation ........................................................... .465 Hugh David

Controlling Factors: An Operator's Perspective ................................................................... .475 Guy C. St. Sauveur

What They Want Is What They Get? ................................................................................... .481 John Lane

Contemporary Issues in ATC System Development.. ......................................................... .489 J. Michael Tonner and Karen Kalmbach

Validation Problems in Air Traffic Control Systems ........................................................... .497 Hans-Jurgen Bangen

Simulating and Evaluating the Future - Pitfalls or Success? ................................................ 521 Anthony Smoker

The National Plan for Aviation Human Factors .................................................................... 529 Joseph Pitts, Phyllis Kay ten, and John Zalenchak III

Other Applications Contexts ................................................................................ 541

Test and Evaluation Program for a Prototype of an Advanced Computerized Control Room for Nuclear Power Plants ...................................... 543 Knut Folles¢ and Frode S. Volden

Validation Issues in Decision Support Systems for Maintenance Planning .......................... 553 Ilhan Or

Artificial Habitat for Man in Extreme Environments as an Integrated Human-Machine System ............................................................................. 575 Olga N. Zakharova

Concept of a FMS/ATC Air-Ground Data Link Testbed Employing an Airbus A340 Full Flight Simulator ............................................................... 585 G. Huttig, U. Rottmann, and A. Wattier

The Qualification of Military Aircraft Cockpits ..................................................................... 593 Peter R. Wilkinson

The Use of Video to Verify and Validate Human System Interactions: A Methodology ....................................................................... 609 Margaret T. Shaffer

Contents XIII

Potential Application of Neural Networks to Verification and Validation of Complex Systems ..................................................................................... 617 Ozer Ciftcioglu and Erdinc Turkcan

Training and Implementation ... ........................................................................... 625

Verification and Validation of the Training Components of Highly Complex Systems ....... 627 Richard S. Gibson

An Expert Air Traffic Control Teaching Machine: Critical Learning Issues ......................... 635 Vincent P. Gaiotti

Interaction of Stages in Validating and Verifying ATC Training .......................................... 651 Rod Baldwin

The Verification of Pilot Abilities as a Basis for Validating Flight Crew Competency ......... 659 Graham J. F. Hunt

Retrospect ....................................................................................................................... 671

Closing Remarks ................................................................................................................... 673 V. David Hopkin

Complex and Integrated Human-Machine Systems: Retroflections ...................................... 679 Anthony Debons and Esther E. Home

Lecturers, Participants, and Staff ........................................................................................... 687

Index ..................................................................................................................................... 697

Editors' Summary

This volume contains the main papers presented at a NATO Advanced Study Institute on 'Verification and Validation of Complex and Integrated Human-Machine Systems.' Additional papers presented at the meeting have been issued as a supplementary publication (Wise, Hopkin, & Stager, 1993).

In the context of human-machine systems, verification and validation are terms with significant inter-disciplinary connotations. Different disciplines have evolved disparate approaches to verification and validation, some of which do not readily transfer across disciplines. Interdisciplinary consensus is on the importance of verification and validation studies.

In these Proceedings, the main focus of interest has been narrowed to the human factors aspects of verification and validation in the development of complex systems. Focusing on the Human-Factors aspects enabled many examples of verification and validation in complex systems to be introduced, while retaining air traffic control as the primary exemplar.

Many of the earlier human factors studies of complex human-machine systems assumed that some form of verification and validation was necessary. Gradually attempts to verify or validate human factors recommendations in large system diminished. A consequence is that there is no widely accepted conceptual structure or organization of the subject matter of verification and validation which would have provided a ready-made framework for the Advanced Study Institute and for this text. Accordingly, the Directors did not place restrictions on the authors about the verification and validation issues that should be addressed. The organizational structure imposed by the Editors on this volume attempts to encompass both the broad general concepts associated with verification and validation within human-machine systems and the narrower concepts intrinsic to specific papers.

Perspectives on Verification and Validation

Four general introductory papers employ a diversity of perspectives to scan broader horizons and contexts of verification and validation. In the first paper, Hopkin proposes some definitions, seeks to identify actual and potential issues, suggests that verification and validation may in some circumstances serve as unifying processes, and attempts to broaden the range of discussion by drawing on some less familiar examples. Foster considers the concept of resilience and uses its many dimensions to demonstrate the heterogeneity of complex systems to which verification and validation could be applied. He also demonstrates the value of resilience when systems must be able to cope with unanticipated changes without incurring catastrophic consequences. Hancock takes human-machine relationships within systems as a main theme and identifies implications for verification and validation of design alternatives in the relationship and integration of human and machine functions, notably in hybrid systems in which both human and machines interact. He acknowledges the influence of Western ways of thought in setting the context for views on human-machine relationships. Wise & Wise consider some aspects of Western philosophical thought which have influenced the evolution of the scientific method as it is now applied, and some consequent assumptions in approaching verification and validation issues. These four papers collectively seek to extend horizons of verification and validation and to suggest the range of issues to be addressed.

2 Editors' Summary

Developing Definitions and Approaches

Two papers focus more directly on approaches to verification and validation in systems. To Stager, the concept of validation is inherent in the process of design and experimental investigation but the confidence we can place in validation depends on our ability to identify and accommodate sources of human and system variance. In the final analysis, achieving system validity is minimizing the variance not accounted for in system design. Harwood addresses human centered issues, and distinguishes between technical usability and user acceptability. She considers a cookbook approach to evaluating complex systems inappropriate, and believes that many system goals such as safety are too general to be assessed directly.

Complexity in a Systems Context

These four papers are still concerned with the frameworks within which validation and verification can occur, but put more emphasis on context and complexity in relation to the evaluation of the systems. Woods and Sarter relate evaluation to design. They note the need within the verification and validation process to detect design deficiencies, to predict system functionality, and to treat logical innovations as a form of intervention in continuous ongoing activity. They distinguish two complementary methodologies, the more rigorous one for development and the less rigorous for what has been developed. Wieringa and Stassen identify factors that influence level of complexity and the implications for its assessment, and suggest that it could be feasible to make progress by developing a form of complexity index. Rosness considers that the extrapolation of current verification and validation methods into more complex systems can at best provide only a partial solution to difficulties in applying verification and validation, and advocates 'softer' tools as more appropriate. Plant, in addressing verification and validation methodologies associated with knowledge based systems, also distinguishes between a rigorous initial methodology and a second less rigorous one which addresses the correctness of the system produced.

Reliability, Errors, and Safety

Consideration of the verification and validation issues related to these topics requires a more specific approach to functions and attributes within systems. The first two papers are concerned with etiology, prevention, and consequences. Hollnagel construes reliability as the relative lack of inexplicable variance in system performance. He notes the critical difference between hardware, software, and human influences and suggests roles for simulation. He describes two rules to study the influence of humans on incidents in complex systems. Reason's emphasis is on latent failures, a product of the design process. He suggests how such failures can be detected, and develops this construct to address the safety health of the system. Three papers discuss what can be learned about verification and validation from actual incidents that have occurred. Baker pursues the notion of system health in incident investigation, the objective being to trace the origins of specific incidents and prevent their recurrence but also to provide broader forms of feedback that could assist in the identification of inherent sources of human error in the system. Jentsch discusses verification and validation as aspects of safety

Editors' Summary 3

assessment in relation to aircraft accidents and incidents, insofar as every accident raises questions of safety. Andrews, focusing on major incidents, defines conditions to be met before satisfactory verification and validation can occur, indicates sources of relevant data in relation to techniques of human reliability assessments, and draws lessons for verification and validation from accounts of two major incidents.

Operator Capabilities and Variability

This group of papers examines inherent human attributes affecting verification and validation processes, with particular attention to operator capabilities and variability. Jorna explores the complexity of workload as a concept and suggests that attempts to match human and machine more suitably should place more emphasis on human capabilities as distinct from human limitations. He argues that the interactions of workload with modern technology can influence human performance sufficiently to induce potentially dangerous errors. Smolensky and Hitchcock note that workload in the form of task demands is not constant, and they define some of the implications of workload for verification and validation studies. Sanders and Roelofsma advocate the combination of task analysis and simulation methods to evaluate performance in large human-machine systems. Clare discusses the transfer of information between human and machine components. He draws examples from airline pilots and proposes that it should be possible to test design solutions to particular human-machine interface problems ahead of the full integration of the system. Logie addresses the measurement of human cognitive workload in the context of theories of working memory.

Mental Models in Operational Systems

Leroux describes a project to design and validate the computer components of a complex air traffic control system that are intended to optimize the productivity of the system as a whole, the aim being to ascertain the dependability and validity of the whole system rather than of its computer components only. Kolrep also takes air traffic control as an example and considers the impact of various kinds of automation on mental modeling and controller memory, suggesting simulation techniques by which such issues could be examined. Noting that the work situation is not simply a sum of its component parts, Dubois and Gaussin nevertheless suggest that the implied need for a global approach to evaluation may not in fact be feasible. They believe that the successful unification of the processes of design and evaluation is contingent on detailed analysis of the actual activities of controllers.

The Cultural Context

The papers in this group address more directly issues touched on obliquely by many other authors, for they examine the significance of cultural and organizational influences within systems, including the managerial contexts in which they function. Westrum distinguishes between three classes of organization, designated pathological, bureaucratic, and generative.

4 Editors' Summary

The different ways in which these various classes treat information indicates that verification and validation procedures may have to be modified to be compatible with the constraints inherent in different organizational cultures. Svensson also adopts a cultural context for a paper in which she implies that, when a system in which there has been an adverse incident is examined in order to improve its safety, the process is often too narrow and limited. Chavez draws comparisons between cultural aspects of the behavior in aircraft cockpits and the culture of the larger organization within which such activities take place. He views more cross cultural comparisons as a promising approach for making verification and validation processes more universally applicable in large systems.

Involving the Users in Verification and VaIidation Processes

Four papers discuss the role of the systems user in verification and validation processes. Dujardin, using air traffic control examples, concludes that users must be involved in the design team and other users should be involved as participants in the evaluations themselves. He argues that it is essential to enlist user knowledge and experience to promote the verification and validation process itself. Jack adopts a different approach, but reaches broadly similar conclusions. Jack's emphasis is on the need to involve users in the requirements analysis phase of software development for the system. Schaad describes the use of soldiers as test subjects as a means to validate recommended equipment and attainable performance. Methods to ensure that the samples selected are representative of the larger population are described. Marshall puts most emphasis on display contents and information presentation in describing means to involve users in the design and evaluation of workspaces in power stations.

The Need for User Involvement

In a related series of papers, most emphasis is on the need for user involvement rather than on the products of user involvement. David distinguishes between systems engineering, operational, and human factors approaches to systems. He notes the different kinds of thinking and different roles assigned to verification and validation in these alternative approaches, and suggests that the approaches have not always been closely or successfully integrated in the past. St. Sauveur expresses the need to put the concerns of the human user first, not because the user's point of view must always prevail but to act as a counterbalance to the concerns of engineers, scientists, and others who may be too remote from the actual system users. He believes that such user involvement introduces an element of verification and validation into the system planning and design. Lane's approach is broadly similar: he remarks that if the consequences of changes are unfavorable to those who must use the system and if the system functioning appears to be given preponderance over the interests of the user then this may engender resistance to changes introduced to improve the system. Tonner and Kalmbach see a role for the professional expert user in the promotion of validation when automation is introduced, in order to achieve high efficiency and safety and to ensure that the user's requirements are not compromised by hardware constraints. Bangen argues for greater interdisciplinary work and user involvement, for example in order to ensure that modeling of human interaction behavior is validated for the requisite decision processes in advanced command systems. Smoker sees the verification and validation of the next generation of air

Editors' Summary 5

traffic control systems as a major challenge which can only be met by active collaboration throughout the aviation community. The final broad paper in this section, by Pitts, Kay ten, and Zalenchak, describes the United States National Plan for Aviation Human Factors, including the air traffic control work and user involvement. The plan exemplifies the process of defining research issues as a prerequisite to gathering the evidence needed for planning, evaluating, and validating future air traffic control systems.

Other Applications Contexts

A set of seven papers illustrates the variety of contexts in which issues of verification and validation are addressed. FolleS!/l and Volden describe a complex and comprehensive test and evaluation program for the nuclear power plant control room that includes new operator support systems and new concepts of control tasks. Or considers decision support systems generally, and describes the development of a support system intended for maintenance planning in an electric motor production facility. Zakharova describes the evolution, development and testing of a vehicle intended to be used by humans in exploring remote habitats such as Mars. The verification and validation issues are especially critical because the environmental conditions are extreme and the concepts must be tested in relation to obstacles that can only be imagined. Huttig, Rottmann, and Wattier describe an air-to-ground data link test bed linked to an aircraft flight simulator where many of the aspects of using data link can be explored and improved. Wilkinson, taking military aircraft cockpits as an example of complex systems requiring human and machine integration, notes that the cockpit can no longer be treated simply as a human-machine interface but that the full integrality and interactions between human and machine have to be understood, and that this understanding is an essential aspect of the validation process. Shaffer describes a technique of using video recording as a method to verify human-machine interaction requirements and validate human-machine performance. Ciftcioglu and Turkcan apply recent developments in the theory and application of natural networks to study the reliability and safety of complex systems. They see natural networks as a possible tool to aid verification and validation studies.

Training and Implementation

The four papers in this group consider training and implementation issues in relation to verification and validation. Gibson addresses the problem of how to validate training. He discusses factors that influence training decisions, training management, and the application of knowledge about the long term retention of human skills. Baldwin examines interactions between the various processes and stages of air traffic control training, and considers which aspects of those processes are appropriate for some form of verification and validation. Gaiotti examines the validation issues that arise within a course of air traffic control training when teaching machines are used in conjunction with human instructors. Hunt notes that the validity of much of the curriculum content of flight training has never been substantially established, and suggests that significant improvements may be obtained by identifying patterns of abilities that successfully predict flight crew performance management. Benefits are improved prescriptions for valid competency requirements and better transference of learning across conditions.

6 Editors' Summary

Retrospect

Two retrospective papers conclude this text. Hopkin, in his closing remarks, draws attention to some verification or validation issues which were scarcely mentioned or not developed, placing particular emphasis on those which appear to have practical utility. Debons and Horne analyze the contents of two previous Advanced Study Institutes and derive a structure for them from their analyses. Their methodology could be applied to the content of this Advanced Study Institute to define how verification and validation have in fact been treated and structured within it, and could perhaps be extended to derive a structure, currently lacking, for verification and validation work as a whole.