reliable objective cognitive assessment (roca) tiw... · from mil-hdbk-46855a, para 6.2.3....

Test and Evaluation/Science and Technology Program Advanced Instrumentation Systems Technology (AIST)

Reliable Objective Cognitive Assessment (ROCA)

20th Test Instrumentation Workshop May 11, 2016

Dr. Neil McDonald, Senior Scientist at QUASAR Dr. George Shoemaker, AIST S&T Executing Agent

Distribution Statement A: Approved for public release; distribution is unlimited.

20th Test Instrumentation Workshop 2

ROCA Project Description

Objective: Meet Test and Evaluation (T&E) requirements for assessing the mental workload that a system under test imposes on its operators.

Problem: There is a critical need for a tool that is capable of unobtrusively, objectively, and reliably measuring the mental workload exerted during task performance while using a test system.

Solution: ROCA provides a tool for objective assessment of mental workload that improves diagnosticity, validity, as well as reliability of current methods.

Approach: ROCA is developing a headset with integrated dry Electroencephalogram (EEG) and functional Near Infrared (fNIR) sensors for measuring brain activity, along with algorithms that use these signals to derive Visual, Auditory, Cognitive, and Psychomotor (VACP) mental workload in real-time.

Reliable Objective Cognitive Assessment (ROCA)


Background: What is Mental Workload Assessment?

In terms of the ISO-10075-1 standard for Mental Workload Assessment

► Mental Workload: The immediate effect of the total of all assessable influences impinging upon a human being from external sources and affecting it mentally within the individual depending on his/her individual habitual and actual preconditions including individual coping styles.


Test & Evaluation Need: Why Assess Mental Workload?

1. Because it is IMPORTANT! 30 to 50% of UAV accidents across the DoD are due to Human Factors

(Seagle, 1997, Schmidt, 1995, US Army, 1998) 2. Because it is part of Test and Evaluation Requirements

► From MIL-HDBK-46855A, para 6.2.3. "verify that personnel can safely perform tasks to time and accuracy standards

without excessive workload“

► From Test Operating Procedure (TOP) 1-2.6.10. “5.16.2 Criteria Workload criteria can address either physical workload in terms of force exertion and lifting requirements or mental workload in terms of information load, information processing, decision rate, etc. […] Mental or cognitive workload is of relevance in HFE testing in that excessive workload imposed by item tasks may exceed human capabilities and lend to errors which impact personnel safety and/or mission success.”


Test & Evaluation Need: Current Approaches and Limitations

► Military systems are increasing demands on soldiers’ mental workload ► DoD lacks objective methods for reliably assessing mental workload ► Mental workload estimation is currently performed by

– modeling operator performance using IMPRINT ► This approach to mental workload is useful in the design phase of a new system ► This approach does not provide real-world measurement of actual operators

– Empirical methods that measure human mental workload ► Surveys and questionnaires such as NASA Task Load Index (TLX) are prone to

subjective biases and do not measure workload continuously through a task ► Measurement of performance or reaction times are indirect measures that may not

accurately reflect mental workload ► Psychophysiological measures such as EEG, fNIR, GSR, ECG, HRV, EOG, pupillometry

have technical and practical tradeoffs

► Need for objective Mental Workload Assessment methods that are sensitive, valid, reliable, and diagnostic

► Tools that assess Mental Workload subtypes (e.g. Visual, Auditory, Cognitive, and Psychomotor) would allows for mitigation strategies


Science &Technology Challenge

► Integration of EEG and fNIR Measurement Technologies

– Development of integrated bimodal EEG and fNIR sensor – Development of wearable headset with integrated hybrid EEG/fNIR sensors

► Development of EEG/fNIR-Based Mental Workload Assessment Software – Acquisition software incorporating both EEG and fNIR measurements – Real-time objective, sensitive, reliable, and diagnostic mental workload assessment

► Validate ROCA Method of Measuring Mental Workload

– Develop models for visual, auditory, cognitive and psychomotor mental workload – Performance benchmarks will be those specified in ISO 10075-3 for Mental Workload

► TRL Start/Finish: 3/6


Background: Demonstration of Mental Workload Assessment


1st person Shooting Simulation

Background: Validity of EEG-based Mental Workload Assessment

00.10.20.30.40.50.60.70.80.9

1

1 2 3 4 5 6

Line

ar C

lass

ifica

tion

Out

put

Engagement

Normative Model

Individual Calibration

Drift Calibration

EO WK B1 B3 B5

00.10.20.30.40.50.60.70.80.9

1

1 2 3 4 5 6

Line

ar C

lass

ifica

tion

Out

put

Workload

Normative Model

Individual Calibration

Drift Calibration

EO WK B1 B3 B5


Background: Applications of Mental Workload Assessment

0

0.2

0.4

0.6

0.8

1

Novice 1 Novice 2 Expert 1 Expert 2

Wor

kloa

d

Cognitive Workload During X-Ray Task

Image Presentation

Answer Correct Answer Incorrect

Workload <50%

Raise proficiency score on relevant content

Workload50-75%

Gaze on Target

Gaze on Target Gaze not on Target

Mark image content for review

Attention Gauge >80%

Attention Gauge <80%

Insert TIP image nextMark imagecontent for review

Next image incrementally easier


Workload >75%

Gaze not on Target

Proceed through test Mark image content for review

Mark relevant content for review


Next Image incrementally harder

Insert TIP image next

Image Presentation

Answer Correct Answer Incorrect

Workload <50%

Raise proficiency score on relevant content

Workload50-75%

Gaze on Target

Gaze on Target Gaze not on Target

Mark image content for review

Attention Gauge >80%

Attention Gauge <80%

Insert TIP image nextMark imagecontent for review



Workload >75%

Gaze not on Target

Proceed through test Mark image content for review

Mark relevant content for review


Next Image incrementally harder

Insert TIP image next

Operator Monitoring Team Operations Resource Management

Enhanced Training Paradigms Special Forces Training


Background: ElectroEncephaloGraphy (EEG)

► EEG is a recording of electrical activity measured on the scalp ► EEG reflects the sum of the synchronous activity of multiple neurons

► Patterns of activity can reflect various mental states and conditions


Background: The problems with conventional EEG

Skin Abrasion Conductive Gel

NOT: •Portable •Practical •Ambulatory

Wired Systems Conventional “Wet” EEG:

QUASAR’s Solution for Practical EEG: Dry Sensors require no abrasion or gels in wireless headsets

•Easy to don •Comfortable •Ambulatory


Background: Introduction to fNIRS

► Functional Near Infrared Spectroscopy (fNIRS)

► Measures hemodynamic response associated with neural activation

► Dense capillary network in cortex have intrinsically high spatial resolution

► Oxygenation changes optical absorption and scattering in near infrared light


Approach: Integrated EEG/fNIRS Sensor

Combine QUASAR’s dry EEG technology with Archinoetic’s fNIR capabilities

Space for emitter or detector component

Circuit board for fNIR components

Circuit boards for EEG components

EEG sensing pins to contact the scalp

NIR light guide pins couple IR to and from the scalp

Integrated EEG/fNIR Sensor


Acquisition Electronics

Results to Date: EEG/fNIR Integrated System

Digital

Analog ADC

External power supply

dark level 760 nm 808 nm 850 nm

Emitter &

Detector

selection EEG

channels

Acquisition Software

Sensor Electronics


Results to Date: Headset Design: Localized Sensors

Visual

Auditory

Cognitive

EEG-fNIR headset ► 8 EEG/fNIR hybrid sensors

► 1 EEG only sensor

► 2 sensors per brain area

Psychomotor


Results to Date: Completed Functional Headset and Power Supply

Front Back


Results to Date: fNIR measurements at Motor Cortex and Visual Cortex

Motor task Visual task

Green: Stimulus ON

Reversing checkerboard Psychomotor: finger tapping


Results to Date: Mental Workload Algorithm Enhancements

Current Features: • Partial Least-Squares algorithm for rapid individual or group model training • Up to 3 Mental Workload models at a time • Inputs EEG, ECG, EMG, and EOG data • Monitors Quality of Data • Real-time Mental Workload measures

ROCA Enhancements: • 11 Additional EEG features • fNIR features •Improved Deflation Methods • Higher Kernel Partial Least Squares Algorithm • VACP Models

QUASAR’s Mental Workload classification tool has been developed through several DoD funded projects

V A C P


Results to Date: ISO Validity Metric Met

0.00

0.20

0.40

0.60

0.80

1.00

V A C P

Valid

ity

Task

Baseline - VACP Tasks

0.00

0.20

0.40

0.60

0.80

1.00

BF2 CA FDS N-Back

Valid

ity

Task

Baseline – Year 1 Results

► Results from 20 subjects

► Target: 0.5

► Validity metric met for VACP tasks

Confidence Intervals: Visual models 90% CI [0.84, 0.97] Auditory models 90% CI [0.72, 0.94] Cognitive models 90% CI [0.75, 0.95] Psychomotor models 90% CI [0.71, 0.94]


Results to Date: ISO Reliability Metric Met

0.00

0.20

0.40

0.60

0.80

1.00

V A C P

Valid

ity

Task


0.00

0.20

0.40

0.60

0.80

1.00

BF2 CA FDS N-Back

Valid

ity

Task



► Target: ≥ 0.9

► Reliability metric met for VACP tasks

Confidence Intervals: Visual models 90% CI [0.85, 0.97] Auditory models 90% CI [0.77, 0.95] Cognitive models 90% CI [0.80, 0.96] Psychomotor models 90% CI [0.72, 0.94]


Results to Date: ISO Sensitivity Metric Met

0.00

0.20

0.40

0.60

0.80

1.00

V A C P

Valid

ity

Task


0.00

0.20

0.40

0.60

0.80

1.00

BF2 CA FDS N-Back

Valid

ity

Task



Confidence Intervals: Visual: 90% CI [6, 6] Auditory: 90% CI [4.2, 4.8] Cognitive: 90% CI [5.5, 5.9] Psychomotor: 90% CI [5.9, 6.0]

► Target: ≥ 3 levels

► Sensitivity metric met for VACP tasks


Results to Date: Demonstrated Real-Time Classification of VACP Tasks

► VACP models working in real-time on play back of reconstructed file from VACP data individually recorded

► Task Sequence: Eyes Open (EO), Visual High, EO, Auditory High, EO, Cognitive High, EO, Psychomotor High, EO

► Each task was approximately 70 seconds

EO VH EO AH EO CH EO PH EO


Significant Program Accomplishments to Date

► Integrated EEG and fNIR systems that meet respective low noise specs

► Optimized optical path to yield robust fNIR signal

► Enabled Mayer wave removal through optimized emitter/detector spacing

► Implemented real-time Mayer wave removal algorithm

► Designed a flexible mechanical design that allows for easy sensor positioning

► Built and validated functional ROCA headset with 8 EEG/fNIR hybrid sensor pods

► Developed software to simultaneously record and display EEG and fNIR data

► Incorporated fNIR features into QStates

► Increased robustness and functionality of QStates

► Implemented real-time Mental Workload classification for VACP models

► Developed and implemented tasks for all VACP mental workload types

► Obtained VACP data from 20 participants to date with the ROCA headset

► Met the ISO Validity, Reliability and Sensitivity metrics


Upcoming Work: External Validation Plans in Relevant Environments


Acknowledgment and Disclaimer

This material is based upon work supported by the Test Resource Management Center (TRMC) Test and Evaluation/Science & Technology (T&E/S&T) Program through the U.S. Army Program Executive Office for Simulation, Training and Instrumentation (PEO STRI) under Contract No. W900KK-13-C-0033. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Test Resource Management Center (TRMC) Test and Evaluation/Science & Technology (T&E/S&T) Program and/or the U.S. Army Program Executive Office for Simulation, Training and Instrumentation (PEO STRI).


Partnership and Roles

► T&E/S&T Executing Agent: Dr. George Shoemaker, AIST ► Contracting Officer’s Representative (COR):

Dr. Sheila Jones, AIST Technical Project Lead ► Government Lead: Jim Buxton (Aberdeen Test Center) ► Program Manager and Principal Investigator:

Dr. Walid Soussou, CEO of QUASAR Inc. ► Presenter:

Dr. Neil McDonald, Senior Scientist at QUASAR, Inc. ► Email: [email protected] ► Phone: +1-858-412-1845

Project Team: ► QUASAR Team

► Archinoetics, LLC

mailto:[email protected]

reliable objective cognitive assessment (roca) tiw... · from mil-hdbk-46855a, para 6.2.3....

Documents