Slide 1

CONQUER CollabOrative School of Biomedical Engineering, Science and Health Systems Drexel University, Philadelphia, PA Brain in the Loop Learning & Training Systems December 13, 2010 Training and Workload Assessment for Safe Piloting

Slide 2

Merging two major advancements Scope Understanding of human brain function Increase in digital computation and storage Dynamic, complex human automation networks

Slide 3

Scope Safe & Effective Piloting Air Traffic Controller Cognitive Workload Monitor UAV Ground Operator Training and Workload Monitor Pilot Expertise Development Monitor Brain Activity Monitor Monitor manage & maintain Crew can manage & maintain high levels of cognitive information- processing Portable brain imaging technology enables such convergence by bringing more of brain on task* Safe & effective performance *DARPA: Dylan Schmorrow, Ph.D. LCDR, MSC, US Navy, Program Manager, ITO

Slide 4

Functional Near-Infrared Monitor [fNIR] >> Cortical activation-related to hemodynamic changes in natural environments and diverse field conditions Brain Activity Monitor

Slide 5

Physiological Principles of fNIR Neural Activity & Hemodynamic Response Neurons consume energy (glucose) when activated Oxygen is required to metabolize the glucose As clusters of neurons are activated, there is an increased need for oxygen in that area Oxygen is transported to neural tissue via oxy-hemoglobin in the blood Oxy-Hb and deoxy-Hb are correlates of brain activity through oxygen consumption by neurons The oxygen exchange occurs in the capillary beds As oxy-hemoglobin gives up oxygen to the neural tissue, it is transformed into deoxygenated hemoglobin David J. Heeger & David Ress. (2002)What does fMRI tell us about neuronal activity? Nature Reviews Neuroscience 3, 142-151

Slide 6

Photons that enter the tissue undergoes two types of interaction: Scattering (cell membranes) Absorption (Hb,HbO 2 water) Source Detector Physical Principles of fNIR I Photon Migration in Tissue H Obrig, R Wenzel, M Kohl, S Horst, P Wobst, J Steinbrink, F Thomas, A Villringer. Near-infrared spectroscopy: does it function in functional activation studies of the adult brain? International Journal of Psychophysiology 35(2-3):125-142, 2000

Slide 7

Near IR Optical Window in Tissue.G Stimuli Presentation Initial Dip Peak Oxygenation Post stimulus undershoot Modified Beer-Lambert Law.G IoIo I Physical Principles of fNIR II

Slide 8

Human Performance Studies Mental Workload, Training & Learning First Study: DARPAs Augmented Cognition Study Performance scores and brain activation in dorsolateral prefrontal cortex as a function of task difficulty on the Warship Commander Task The experts brain activity levels rise as they become more effortful in completing the complex task. In the highest level of difficulty (24 planes, auditory task ON), the novices disengaged, i.e., quit the task - resulting in a large deoxygenation due to loss of attention and memory effort, whereas experts averaged 90% of the possible game score in this wave.

Slide 9

Air Traffic Controller FAA Next Generation Air Transportation System (NextGen) Study Similar to SESAR (Single European Sky ATM Research) Program FAA NextGen legacy versus new? Communication between air traffic controller (ATC) and the flight deck Transition from auditory-based communications (Voice Comm) to text-based messaging (Data Comm) systems. Allow ATC to manage more traffic with fewer cognitive resources, thereby increasing both traffic capacity and the safety of passengers Need: objective, brain-based measures of cognitive workload

Slide 10

Air Traffic Controller FAA Next Generation Air Transportation System (NextGen) Study Similar to SESAR (Single European Sky ATM Research) Program ATC part-task Communication type: Voice Comm and Data Comm Workload manipulated by number of aircrafts in the sector: n= 6,12,18 16-channel fNIR sensor

Slide 11

Air Traffic Controller FAA Next Generation Air Transportation System (NextGen) Similar to SESAR (Single European Sky ATM Research) Program Aircraft: (F 2,42 = 6.39, p < 0.05) Comm: (F 1,21 = 10.57, p < 0.05) ATC Task: fNIR Measurements (F3,69= 4.37, p < 0.05) ATC Working Memory: fNIR Measurements Changes in blood oxygenation in prefrontal cortex were shown to be associated with increasing cognitive workload. Data-based required less brain activation of the operator than legacy Voice- based communication. Results:

Slide 12

UAV Piloting US Army Medical Research and Materiel Command (USAMRMC) The Telemedicine & Advanced Technology Research Center (TATRC) UAV ground controllers expertise level, cognitive workload and situational awareness are critical. operation failures due to human error (60% - 80% ) ground flight crews mental workload increase inadequate training

Slide 13

UAV Piloting Integration of the functional brain monitoring system with training procedures ground control systems objective measure of expertise level to validate adequate training real-time monitor of mental workload safe operation of UAVs

Slide 14

UAV Operators Cognitive Workload Assessment Study Menda J, Hing JT, Ayaz H, Shewokis PA, Izzetoglu K, Oh, P, Onaral B (in print). Optical Brain Imaging to Enhance UAV Operator Training, Evaluation, and Interface Development. Journal of Intelligent and Robotic Systems. Comparison of Day 2 (first task) and Day 9 (last task) yields significant drop on average oxy-Hb. UAV Piloting Landing Task (day2 vs. day9) Approach Task (day2 vs. day9) fNIR Measures Self-reported Measures (day2 vs. day9)

Slide 15

University of Maryland (Dr. Jos L. Contreras-Vidal) University of Pittsburgh (Dr. Joseph Ricker) Albert Einstein College of Medicine (Dr. Roee Holtzer) Italy Polytechnic University of Milan (Dr. Sergio Cerutti) Spain University of Seville ((Dr. Jose Leon-Carrion) Hospital Nacional de Paraplejicos Toledo & Dr. Guglielmo Foffani) Israel Haifa University (Dr. Zvia Breznitz) Turkey Middle East Technical University (Dr. Veysi Isler) Dokuz Eylul University (Dr. Murat Ozgeren ) Japan Bukkyo University (Dr. Yasufumi Kuroda) China Wuhan National Laboratory for Optoelectronics (Dr. Britton Chance & Dr. Shoko Nioka) United Kingdom The Defence Science and Technology Laboratory (DSTL) National and International Partnership

Slide 16

WWW.BIOMED.DREXEL.EDU http://www.biomed.drexel.edu/fNIR/CONQUER/Welcome.html CONQUER Collaborative 3508 Market Street Philadelphia, PA, USA 19104 Thank you!