using information to drive decisions · 2017. 4. 26. · schema—store, acquisition, automation...

From imagination to impact

Using Information to Drive Decisions

Exploring Visual Modalities Exploring Visual Modalities Exploring Visual Modalities Exploring Visual Modalities

Dr. Fang Chen

NICTA Copyright 2010 1

Dr. Fang [email protected]

High cognitive load and human responses

• Disturbance in responses caused by high cognitive load may not be perceptible to humans, but may be machine measurable

• Teaching Athletes Cognitive Skills:Detecting Cognitive Load in Computer-based Training

LongLongLongLong----term memoryterm memoryterm memoryterm memory

ShortShortShortShort----term memoryterm memoryterm memoryterm memory Shared spaceShared spaceShared spaceShared space (7±2)

E.g. Baddeley’s Modal Model of Working Memory

Visual Visual Visual Visual

processingprocessingprocessingprocessing

(Visuospatial (Visuospatial (Visuospatial (Visuospatial

sketchpad)sketchpad)sketchpad)sketchpad)

Linguistic Linguistic Linguistic Linguistic

processingprocessingprocessingprocessing

(Phonological loop)(Phonological loop)(Phonological loop)(Phonological loop)

Central

executive

PerceptionPerceptionPerceptionPerception

ResponseResponseResponseResponse

Muscular action

Excitation + vocal

tract configuration

Gesture…

Multi-sensory

perception

. . .. . .. . .. . .

. . .. . .. . .. . .

Disturbance

e.g. latency, pitch,

jittering

Experiment design and data set

Cognitive Load and Athletic Performance

• Cognitive load is important in complex tasks– Working memory is limited in capacity and duration

• Shared by cognitive processes e.g. perception and production of responses

– When load is too high or too low

• Problematic for skill acquisition + optimal performance

– Performance is sometimes too coarse, too subjective

– Need for cognitive load measurement/assessment– Need for cognitive load measurement/assessment

• Cognitive load in elite athletes @ AIS– Perceptual cognitive/motor skills

– Decision Making

– Strategy/ Plays

Cognitive Load

Pe

rfo

rma

nce

Aim: Elicit response behaviours, using a computerAim: Elicit response behaviours, using a computerAim: Elicit response behaviours, using a computerAim: Elicit response behaviours, using a computer----based based based based training tool (AISReact), to detect differences in cognitive training tool (AISReact), to detect differences in cognitive training tool (AISReact), to detect differences in cognitive training tool (AISReact), to detect differences in cognitive load.load.load.load.

• Participants: – 12 male recreational basketball players (within-subjects)

• Longitudinal Study – Two Factors: CL and Expertise

– Pre and Post test + 6 training sessions, over 6 weeks

User Study: Basketball Cognitive Skill Learning

– Pre and Post test + 6 training sessions, over 6 weeks

• Task Design: – View a short basketball clip (10 seconds each)

– Recall the positions of players (player formations)

– Draw locations on blank court

– Think aloud responses while drawing

• Task difficulty:– Report on increasing numbers of players

– Low (3), Medium (6), High (10)

– 6 trials at each level

Data Collected

• Performance Scores– Scores were given for each mark within a radius of 8% screen distance (in pixels)

from the correct player position, as recommended by basketball experts at the AIS.

• Subjective Ratings– Likert Scale (1 extremely easy - 9 extremely difficult)

• Speech Responses– Think-aloud speech as they completed the tasks– Think-aloud speech as they completed the tasks

– Continuous, multiple word phrases.

• Pen gesture – Trajectory duration, velocity and length

– Three types of markings

• Eye-Gaze– Gaze trajectory, pupil dilation, and blinks

• Galvanic Skin Response – Baseline, rest and active readings

Performance Scores and Subjective Ratings

50%

60%

70%

80%

90%

Perf

orm

an

ce S

co

res

5

6

7

8

9

Su

bje

cti

ve R

ati

ng

s

Scores (%)

0%

10%

20%

30%

40%

LOW MED HIGH

CL Levels

Perf

orm

an

ce S

co

res

0

1

2

3

4

Su

bje

cti

ve R

ati

ng

s

Scores (%)

Ratings

Speech Responses: Classification Results

• Pre-Test: 3 Cognitive Load Levels

– 3 classes

– Each representing a single load level (low, med or high)

– 82% of samples correctly classified Classified as

Low Medium High

Testingsamplesfrom

Low 100% 0% 0%

Medium 40% 6% 54%

• Pre-Test: 2 Cognitive Load Levels

– 2 classes

– Each representing a single load level (low and high)

– 93.2% of samples correctly classified

fromMedium 40% 6% 54%

High 15% 3% 82%

Classified as

Low High

Testing samplesfrom

Low100% 0%

High15% 85%

Pen Input Features

Pen Input Features

• High cognitive load can be reflected in communicative signals (production)

• Symptoms of cognitive load, – depending on the application (e.g. blackberry, tablet etc)


– depending on the application (e.g. blackberry, tablet etc)

– Geometric and temporal features (shape and trajectory)

– Interactive features (when it is used and for what)

– Content analysis (what is being drawn)

Basketball User Study Design

• Recalling basketball player formations from 10s video clip

– Mark the position of the players on the court

• Subjective ratings (1-9 scale) and performance scores

• Longitudinal: Pre-Test, 6 Training Sessions, Post-Test

Cognitive Load Levels


Attacker Defender Ball Carrier


Low (Easy): 3 players

Medium (Med): 6 players

High (Hard): 10 players

Basketball User Study


Feature Analysis: Duration and Velocity

• Changes from Low load to High loadHypothesis: Duration will decrease and velocity will increase as subjects try to respond Hypothesis: Duration will decrease and velocity will increase as subjects try to respond Hypothesis: Duration will decrease and velocity will increase as subjects try to respond Hypothesis: Duration will decrease and velocity will increase as subjects try to respond

faster, before recall is lost.faster, before recall is lost.faster, before recall is lost.faster, before recall is lost.

Velocity increased as the tasks become

more complex in 78% of gestures

tested

Significant increase from Low to High (t-

test, p<0.05) in 44% gestures tested

Gesture duration decreased as tasks

become more complex in 82% of

gestures

Significant decrease from Low to High (t-



0

100

200

300

400

500

600

cross circle ball carrier

Subject 9 Pre-test Gesture Duration

easy

medium

hard


0

0.05

0.1

0.15

0.2

0.25

0.3

cross circle ball carrier

Subject 12 Pre-test Velocity


Feature Analysis: Duration and Velocity

50

100

150

200

250

300

350

400

Subject 12 Gesture Duration

Hypothesis: Duration will decrease even further, and velocity will increase further as subjects master the task.

Gesture duration decreased from pre-test to post test.

• Changes from Pre-test to Post-test


0

50

pre cross post cross pre ball

carrier

post ball

carrier

test to post test.

Repeated measures ANOVA shows significant effect of load (decreasing duration from low to high), and a significant effect of expertise (decreasing duration from pre to post).

No significant differences were found in the velocity feature.

Defenders (Circle) - Duration

0

100

200

300

400

500

600

700

800

LOW MED HIGH

Cognitive Load

Du

rati

on

(m

s)

PRE

POST

Pen-Input Results – Trajectory Durations

• Circles and Cross shapes

200

250

300

350

400

450

500

550

600

650

Du

rati

on

(m

s)

Pre-Test Cross

Pre-Test Circle

Pre-Test Ball

• Trajectory analysis

– Significant trends of decrease in

trajectory duration as CL ↑

– Significant trends of decrease in

trajectory velocity as CL ↑,

• Except in Ball Carrier

0.2

0.25

0.3

0.35

0.4

0.45

Low Med High

Cognitive Load

Sp

eed

(p

ixels

/ms)

Pre-Test Ball

Pre-Test Circle

Pre-Test Cross

150

Low Med High

Cognitive Load

Traffic Incident Study: Design

• Creating traffic detours and green light corridors

– Using pen and speech interaction on a tablet

– Scratchpad for ‘working out’

• Subjective ratings (1-9 scale) and performance scores




Low (Easy): 6 streets

Medium (Med): 10 streets

High (Hard): 16 streets

Selection Examples Shape Examples

Degeneration of Interactive Shapes

• Geometric analysis of trajectoryGeometric analysis of trajectoryGeometric analysis of trajectoryGeometric analysis of trajectory– 12 features from Rubine[1991] paper on single stroke pen-gesture

recognition e.g. angle at start stroke, angle and end stroke, duration, length, sharpness etc

• MalahanobisMalahanobisMalahanobisMalahanobis distance (MDISTdistance (MDISTdistance (MDISTdistance (MDIST---- a weighted Euclidean distance))))– The number of standard deviations a pen-gesture is away from the mean of

its “standard/baseline” form, captured during training.

– As load increases, the curve moves away from 0, indicating a greater degree of degeneration (statistically significant).


– As load increases, the curve moves away from 0, indicating a greater degree of degeneration (statistically significant).

Use of the Scratchpad

• Scratchpad as a cognitive tool

– Use of note-taking as an external memory aid

• High usage expected during high cognitive load

– Organisational marks for understanding, clarification, planning.

– Diagramming as a strategy for

• Results

– Significantly increased usage ad CL increases (manual freq)

– Automated trajectory frequency count and rate per second significantly increasing

– Use of diagramming doubles between low load tasks and high load tasks (manual freq)


– Diagramming as a strategy for generating and discarding hypotheses

– Content Analysis:

Alphanumeric -> Symbolic, Organisational -> Diagrammatic

load tasks (manual freq)

– Increased evidence of symbolic and organisational marks, as well as spatial representations when cognitive load is high

Cognitive Load Evaluation

Using Stroke-level Features Using Stroke-level Features

From Writing to Cognitive Load

The writing process*:The writing process*:The writing process*:The writing process*:

What is writing? What is writing? What is writing? What is writing?

~More than pure stoke creation:~More than pure stoke creation:~More than pure stoke creation:~More than pure stoke creation:Writing is a comprehensive activity integrating the close cooperation between the Writing is a comprehensive activity integrating the close cooperation between the Writing is a comprehensive activity integrating the close cooperation between the Writing is a comprehensive activity integrating the close cooperation between the

human brain, the eyes, the limb and both hands.human brain, the eyes, the limb and both hands.human brain, the eyes, the limb and both hands.human brain, the eyes, the limb and both hands.

Low Low Low Low mediummediummediummedium highhighhighhigh


Dataset

It includes the writing samples of 20 subjects, each writing 30 sentences (10 low CL, 10 med CL and 10 High CL). The sentence is composed based on a set of given words.

Dataset (cont.)

A writing sample:A writing sample:A writing sample:A writing sample:

Research Method

DecisionDecisionDecisionDecision

Pattern Set & Decision Rules

Workflow of the method:Workflow of the method:Workflow of the method:Workflow of the method:

Handwriting Tasks Pre-processing Feature Extraction Pattern Matching

DecisionDecisionDecisionDecision

•Sentence compositionSentence compositionSentence compositionSentence composition •Stroke quality evaluationStroke quality evaluationStroke quality evaluationStroke quality evaluation•Strokes/pointsStrokes/pointsStrokes/pointsStrokes/points

•Feature SelectionFeature SelectionFeature SelectionFeature Selection

•GMMGMMGMMGMM

The adoption of stroke quality evaluation is dependent on the

method of feature extraction and classification.

Stroke Quality Evaluation

� To selectively abandon some strokes from further

process based on the selection criteria.

Why not use all the strokes?

• Strokes with insufficient information

• Strokes with redundant information• Strokes with redundant information

• Complex strokes

• Writing noise

How the quality of the strokes are evaluated?

• Time criteria

• Length criteria

• Combined criteria during feature extraction

Feature Evaluation: ANOVA Test

Pressure:Pressure:Pressure:Pressure:

Velocity:Velocity:Velocity:Velocity:

Low CLLow CLLow CLLow CL High CLHigh CLHigh CLHigh CL

ANOVA test on the extracted features:ANOVA test on the extracted features:ANOVA test on the extracted features:ANOVA test on the extracted features:

Max P Min P Avg P Max V Min V Avg V

F-ratio 7.7 4.8 6.8 1.8 10.1 4.5

p Value 0.002 0.016 0.004 0.191 0.001 0.020

Velocity:Velocity:Velocity:Velocity:

Cognitive Load Classification:

Subject-Dependent GMM

Classification with leave-one-out cross-validation

9 writing tasks from each subject were used to train the GMM, and the remaining one for test; this process is repeated for every single written task. Decision of the CL level is based on the maximum probability of the test task on the three CL models.

Feature Pressure Velocity Azimuth Altitude Length Time

Accuracy 46% 39.8% 40% 39% 37.3% 37.3%

Feature Azimuth + Altitude Pressure + Velocity All

Accuracy 47.2% 52.7% 50.3%

Cognitive Load Classification (cont.)

Feature Pressure Velocity P+V Azimuth Altitude Angle All

Accuracy 61.8% 57.3% 62.8% 58.5% 53.8% 65.5% 69.8%

Two CL Level Classification:

Low vs Mid

Mid vs HighMid vs High


Accuracy 57.5% 58.3% 61.3% 56.8% 57.3% 57.8% 62.3%


Accuracy 64% 58.5% 68% 55% 56.8% 58.8% 64%

Low vs High

Result Analysis

• Pressure is relatively the most reliable feature for CL classification, followed by velocity.

• Combined features perform better than single feature, except for the time and stroke length feature.

• In the classification of two CL levels, the accuracy between Low CL and Mid CL is better than other pairs, and this may result from the unstable performance of the subjects in the High CL level tasks.performance of the subjects in the High CL level tasks.

Future Work

• GMM analysis for inter-stroke features;

• Subject-independent models for 3 CL levels;

• Other classifiers, e.g. SVM for framed stroke sequence;

• Decision level fusion for CL classification.• Decision level fusion for CL classification.

Video Based Workload

Measurement

Video Based Workload Measures

Content

� Imaging sensors

�Cognitive load theory and measurable elements

� Visual perception and cognition

� Eye activities affected by cognition

NICTA confidential

� Eye activities affected by cognition

�Results

� Summary

Imaging sensors

• Video based eye tracker

– Head-mount/remote

• Thermal infrared camera

– Skin temperature

• Brain imaging

– Magnetic resonance imaging– Magnetic resonance imaging

– Near-infrared neuroimaging

Video based measures

• Eye activity

– Eye blink

– Eye movement

– Pupillary response

• Skin temperature

• Physical behaviour• Physical behaviour

Cognitive tasks

• Single task

– Visual

– Auditory

– Arithmetic

– Memory

– Executive– Executive

– Driving

– Flight

– Traffic control

• Dual task

• Multi-tasking

Cognitive load theory and measurable elements

� Cognitive load theoryCognitive load theoryCognitive load theoryCognitive load theory (Sweller, 1994)

� Schema—store, acquisition, automation

� Total cognitive load—intrinsic, extraneous

� Element interactivity in high and low load � Element interactivity in high and low load

Cognitive load theory and measurable

elements

� Assessment dimensionAssessment dimensionAssessment dimensionAssessment dimension (Pass et. al, 2003)

� Mental load – knowledge, subject characteristics

� Mental effort – cognitive capacity allocation

� Performance – achievement� Performance – achievement

� Source of limitations affect visual bufferSource of limitations affect visual bufferSource of limitations affect visual bufferSource of limitations affect visual buffer (Kirby et al,1992)

� The amount of entities that can be retained

� The amount of entities that can be represented

� Objects’ scope and size

Visual perception and cognition

� Visual attention modelVisual attention modelVisual attention modelVisual attention model (Duchowski, 2007)

� Consists of low and high level function

� Examined by psychological basis

� Justified by neurological substrate� Justified by neurological substrate

� Measured by psychophysics


� Psychological basis for eye movementPsychological basis for eye movementPsychological basis for eye movementPsychological basis for eye movement (Duchowski, 2007)

� To know attention behaviour, factual information

Image stimuli (entire view)

Parallel through foveal and

peripheral vision

Sensory input

Low level

Bottom-up or

Feature -driven

Interesting features pop up to

attract attention

Current foveal location is

disengaged, eye repositioned.

Fast eye movement.

Foveal location is directed to

perceive the feature. Attention

is engaged. Identify meaning or

expectation.

Detection

Context-sensitive

Feature

attribution

Top-up or

goal-driven

High level

To “where“ reflects

the will. Saccade

Identification

Gaze


� Implication for visual display (Duchowski, 2007)

• Foveal vision �identify “what”, sequential pattern, “scan

path”;

• Parafovea or peripheral vision �detect “where”, parallel • Parafovea or peripheral vision �detect “where”, parallel

pattern, “spot light”;

• A distinction of sensitivity between them in spatial vision,

temporal vision, luminance and contrast.

• Salience map � distinctiveness of an object

Eye activities affected by cognition

� Basic eye movement Basic eye movement Basic eye movement Basic eye movement –––– saccade and fixationsaccade and fixationsaccade and fixationsaccade and fixation

� No systematic pattern found with large changes in

visual stimuli

• Comparing ROI determined by human and detected by • Comparing ROI determined by human and detected by

10 algorithms (Privitera et al ,2000)

•No best matching algorithms found with high accuracy to

predict the next points of fixation

� Cognitive factor affects eye movement programming

• targets viewed are more task-related than “intrinsic

salience” of objects

Eye movement

• Workload measures

– Fixation frequency

– Fixation duration

– Saccade extent

– Saccade speed

– Scan path– Scan path

– Vergence

– Spectrum

• Questionable sensitivity

– Fixation vs. saccade

– Inconsistent results


� Basic eye movement Basic eye movement Basic eye movement Basic eye movement –––– task dependenttask dependenttask dependenttask dependent

� Reading �conceptually difficult text increases fixation

duration and decrease saccade length (Rayner, 1998)

� art work � complex causes short fixation and fast � art work � complex causes short fixation and fast

saccade speed (Molnar, 1981)

� driving � shorter fixation duration and more visual

resource happened to experienced driver than novice

(Chapman, 1998)


�VisuospatialVisuospatialVisuospatialVisuospatial task experimenttask experimenttask experimenttask experiment

• Fixation frequency distinguishes target density by three levels while fixation time fails (Van Orden et al. 2001)

• Fixation time significantly differs from each workload Greef et al.(2009)

� Scan path: strategies shiftScan path: strategies shiftScan path: strategies shiftScan path: strategies shift (Marshall et al., 2003)

NICTA confidential

� Scan path: strategies shiftScan path: strategies shiftScan path: strategies shiftScan path: strategies shift (Marshall et al., 2003)

• Successful strategies � less effort;

• Unsuccessful strategies � more effort, poor performance

• Combining performance score or other measures

Eye blink


– Blink rate

– Blink interval

– Blink duration

– Startle eye blink

• Sensitive to visual demands• Sensitive to visual demands

– Blink interval: positive correlation

– Blink rate/duration: negative correlation

• Opposite effects in dual task

– Driving/listening/visual

– Memory/flight


� Eye blinksEye blinksEye blinksEye blinks

� Types �reflective, voluntary, spontaneous (dominating)

� Suppress vision � tend to be inhibited in difficult part

� Thinking affects blinks � blink less as task difficulty increased (Irwin et al. 2010)et al. 2010)

� Blink tied to motor response (Baumstimler et al. 1971)

� Indicating cognition resource released or reflecting the adequate preparation (Siegle et al., 2008)



� blink rate � lower as task difficulty increases in both visual and non-visual tasks (Irwin et al.2010)

� blink rate � higher in more difficult arithmetic task (Tanaka et al. 1993)1993)

� Blink rate pattern � (Bentivoglio et al. 1997)

conversation > rest > reading in 101 subjects (67.3%), rest > conversation > reading in 34 subjects (22.7%), conversation > reading > resting in 12 subjects (8.0%).



� blink latency � increased when short-term memory capacity reaches limit (Irwin et al.2010)

� blink duration � tend to decrease with increased visual attention demands demands

� Visuospatial task experiment � (Van Orden et al. 2001))

blink duration, blink rate have significant difference when target density increased by three levels

Pupillary response


– Pupil dilation

– Percentage change in pupil size

– Index of cognitive activity

– Power spectrum

• Effective physiological index• Effective physiological index


� PupillometryPupillometryPupillometryPupillometry

� reliable pattern � linear relationship with cognitive processing

� task-invoke or emotion-trigger? � reflect cognitive load or arousal or both?

� task-driven event � Peavler (1974)� task-driven event � Peavler (1974)


� PupillometryPupillometryPupillometryPupillometry

�by emotional arousal � Bradley et al. (2008)

�Controlling cognitive demands and arousal factors � Stanners et. al (1979)

• cognitive processing affects pupil dilation more than arousal components


� Combination of eye activitiesCombination of eye activitiesCombination of eye activitiesCombination of eye activities

�Visuospatial task experiment (Van Orden et al, 2001)

Skin temperature/physical behaviour

• Facial skin temperature

– Nose

– Forehead

• Physical behaviour

– Head

– Hand/mouse– Hand/mouse

– Mouth

• Low sensitivity

– Composite measure

Eye Activity Results

• Video-based measure

• Low to Medium load• Low to Medium load

• As cognitive load

increases,

– Blink latency ↑

– Mean pupil size ↑

– Fixation duration ↑

– Saccade size ↓

– All significantly

Eye Activity Results (2)

• As cognitive load

increases,

– Blink rate ↓

– Fixation rate ↓

– Saccade speed ↓– Saccade speed ↓

– All significantly

Noisy factors

• Person

– Engagement

– Fatigue

– Stress

– Physical

• Environment• Environment

– Illumination

• Sensor

– Distortion

Multiple measures vs. single measure

• Different effects on individual measures

• Single measures may be insufficient

• Multiple measures could be inconsistent

Multimodal data fusion

• Fusion techniques

– Linear weighting

– Neural network

– Bayesian network

– Decision tree

– Kernel learning– Kernel learning

Comprehensive modelling

• Multiple-multiple mapping

Workload Pupillary response

Environment Eye blink

Fatigue Eye movement

Stress Facial temperature

Engagement Physical behaviourEngagement Physical behaviour

Summary

� Different eye activities have different ability to

classify cognitive load levels;

� Look into critical points or specific boundary

might enhance the ability to distinguish cognitive

load levelsload levels

� The relationship between those eye activities and

consistency between different task categories

using information to drive decisions · 2017. 4. 26. · schema—store, acquisition, automation...

Documents