ppt - johnny chung lee - human computer interaction research
Post on 18-Oct-2014
622 views
DESCRIPTION
TRANSCRIPT
![Page 1: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/1.jpg)
Using a Low-Cost Electroencephalograph for Task Classification in HCI Research
Johnny C. Lee
Carnegie Mellon University
Desney S. TanMicrosoft Research
UIST 2006, Montreux Switzerland
![Page 2: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/2.jpg)
National Geographic, March 2005 NY Times Magazine, October 16, 2005
![Page 3: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/3.jpg)
Brain-Computer Interfaces (BCI)
A direct technological interface between a brain and computer not requiring any motor output from the user
Example Conferences/Journals with BCI interests:Neural Information Processing Systems (NIPS)IEEE Transactions on Biomedical EngineeringIEEE Transactions on Neural Systems and Rehabilitation Engineering
![Page 4: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/4.jpg)
Why is this relevant to UIST or HCI?
BCI research traditionally focuses on exploratory neuroscience and rehabilitation engineering.
Brain sensing could provide valuable data about:- engagement
- cognitive work load
- surprise
- satisfaction
- frustration
Potentially helpful toContext Sensitive or Evaluation Systems
![Page 5: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/5.jpg)
Values of BCI Values of HCI
VS
To use any means necessary to demonstrate that brain-computer interaction is possible.
To use reasonable means to achieve a practical benefit to many users.
use equipment costing $100K to +$1 million USD
use highly invasive surgical procedures
require hours or days of operant conditioning
remove data from poor performing subjects
It is okay to:
use fairly affordable and accessible equipment
be safe for repeated and extended use
be usable without requiring significant user training
use data from all subjects to evaluate its performance
We’d like to:
![Page 6: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/6.jpg)
Where do we start?
![Page 7: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/7.jpg)
Brain Sensing/Imaging Technologies
MRI
CT
ECoG
SPECT
PET
MEG
fMRI
EROS/fNIR
EEG
Currently Impracticalfor HCI
- Safe, easy, no medical expertise
![Page 8: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/8.jpg)
EEG – Electroencephalograph
the neurophysiological measurement of the electrical activity
of the brain by recording from electrodes placed on the scalp
(skipping the lower level neurophysiology)
- Measures the voltage difference between two locations on the scalp
- Only picks up gross, macroscopic, coordinated, and synchronized firing of neurons near the surface of the brain with perpendicular orientation to the scalp. (thus majority of activity is hidden)
Analogous to holding a thermometer up to the side of a PC case
![Page 9: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/9.jpg)
EEG Devices
Manufacturer: BioSemiChannels: 64-128Cost: ~$30K USD
Manufacturer: EGI SystemsChannels: 128-512Cost: $100K-$250K USD
![Page 10: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/10.jpg)
• Lowest cost FDA approved device• Designed for home and small clinical use.• Only $1500 USD• Specs:
– 2-channels– 8-bit at 4µV resolution– 256 samples/sec
• Has yet to be validated for BCI research work.• If it works, it lowers the entry bar for BCI research.
The Brainmaster
![Page 11: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/11.jpg)
Validating the Device (and ourselves)
1. Validate the device
Can we get useful data from such a low-end device?
2. Validate ourselves
To explore this space, we must be able to collect our own data.
![Page 12: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/12.jpg)
Validating the Device (and ourselves)
Keirn, Z., “A New Mode of Communication Between Man and His Surroundings”, IEEE Transactions on Biomedical Engineering, Vol. 37, No. 12, 1990.
• Data is available for download• Data has not been reproduced in the past 15 years
– Some computational BCI researchers have just used this data.– State of the art does is not a great deal better.
![Page 13: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/13.jpg)
Reproducing the Keirn Data
We adapted procedure from Keirn to better control potential confounds.
3 tasks:• Rest (Baseline): Relaxation and clearing of mind• Math: Mental arithmetic, prompted with “7 times 3 8 5”• Rotation: Mentally rotate an object, prompted with “peacock”
Tasks from the original paper were designed to elicit hemispheric differences.
![Page 14: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/14.jpg)
Experimental Procedure
User is instructed to keep eyes closed, minimize body movement, and not to vocalize part of the tasks.
For each task, a computer driven cue is given: Rest, Math, Rotate
Following Math and Rotate, the experimenter says either the math problem or object
![Page 15: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/15.jpg)
Experimental Procedure
Rest Math Rot
Rot Rest Math
Rest Rot Math
Rot Math Rest
Math Rest Rot
Math Rot Rest
session
task (14 seconds)
trial
Block design adapted from Kiern
![Page 16: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/16.jpg)
Experimental Procedure
Rest Math Rot
Rot Rest Math
Rest Rot Math
Rot Math Rest
Math Rest Rot
Math Rot Rest
Rest Math Rot
Rot Rest Math
Rest Rot Math
Rot Math Rest
Math Rest Rot
Math Rot Rest
Rest Math Rot
Rot Rest Math
Rest Rot Math
Rot Math Rest
Math Rest Rot
Math Rot Rest
3 sessions per subject
Many short tasks prevent correlation with EEG drift
![Page 17: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/17.jpg)
Experimental Procedure
Rest Math Rot
Rot Rest Math
Rest Rot Math
Rot Math Rest
Math Rest Rot
Math Rot Rest
Rest Math Rot
Rot Rest Math
Rest Rot Math
Rot Math Rest
Math Rest Rot
Math Rot Rest
Rest Math Rot
Rot Rest Math
Rest Rot Math
Rot Math Rest
Math Rest Rot
Math Rot Rest
Subjects:8 subjects (3 female)29-58 years of ageAll were cognitively and neurologically healthyAll right handed
![Page 18: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/18.jpg)
EEG Setup
International 10-20 EEG electrode placement system
Two channels placed on P3 and P4 with both references tied to Cz.
Electrodes are held in place using conductive paste.
5-10 minute preparation.
![Page 19: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/19.jpg)
Processing the Data
![Page 20: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/20.jpg)
Data Processing
14 secs
Rot
task (14 seconds)
![Page 21: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/21.jpg)
Data Processing
14 secs
Task Cue
Rot
task (14 seconds)
![Page 22: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/22.jpg)
Data Processing
14 secs
Experimenter Prompt
Rot
task (14 seconds)
![Page 23: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/23.jpg)
Data Processing
14 secs
Task Onset
Rot
task (14 seconds)
![Page 24: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/24.jpg)
Data Processing
14 secs
Performing Task
Rot
task (14 seconds)
![Page 25: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/25.jpg)
Data Processing
14 secs
Performing Task
~4 secs
Rot
task (14 seconds)
![Page 26: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/26.jpg)
Data Processing
10 secs
Performing Task
Rot
task (14 seconds)
![Page 27: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/27.jpg)
Removing time for machine learning
Most machine learning algorithms don’t handle time series data very well.
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
10 seconds
![Page 28: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/28.jpg)
Removing time for machine learning
• Divide the 10 seconds into 2 sec windows that overlap by 1 sec• Perform signal processing on each of the 9 windows to get our
“time less” feature set
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
2 secs
![Page 29: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/29.jpg)
Removing time for machine learning
• Divide the 10 seconds into 2 sec windows that overlap by 1 sec• Perform signal processing on each of the 9 windows to get our
“time less” feature set
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
2 secs
![Page 30: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/30.jpg)
Removing time for machine learning
• Divide the 10 seconds into 2 sec windows that overlap by 1 sec• Perform signal processing on each of the 9 windows to get our
“time less” feature set
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
2 secs
![Page 31: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/31.jpg)
Removing time for machine learning
• Divide the 10 seconds into 2 sec windows that overlap by 1 sec• Perform signal processing on each of the 9 windows to get our
“time less” feature set
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
2 secs
![Page 32: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/32.jpg)
Removing time for machine learning
• Divide the 10 seconds into 2 sec windows that overlap by 1 sec• Perform signal processing on each of the 9 windows to get our
“time less” feature set
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
2 secs
This provides486 windows
per participant
![Page 33: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/33.jpg)
Signal features for each window
Generic signal features such as mean power, peak frequency, peak frequency amplitude, etc.
Features frequently used in EEG signal analysis.
![Page 34: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/34.jpg)
Common EEG Features
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
0 5 10 15 20 25 30 35 40 45 500
1000
2000
3000
4000
5000
Raw EEG
SpectralPower
Gamma (30Hz-50Hz)
Beta High(20Hz-30Hz)
Beta Low(12Hz-20Hz)
Alpha(8Hz-12Hz)
Theta(4Hz-8Hz)
Delta(1Hz-4Hz)
![Page 35: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/35.jpg)
Feature Processing and Selection
The 39 base features from each window are mathematically combined to create 1521 total features.
We used a feature preparation and selection process similar to [Fogarty CHI’05] to reduce the number of features:
23 features for 3-task classification (486 examples)
16.4 features for pair-wise classification (324 examples)
![Page 36: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/36.jpg)
Baseline Results – 3 cognitive tasks
3 taskMath v. Rotate
Rest v. Math
Rest v. Rotate
user 1 67.9% 83.3% 88.0% 85.8%
user 2 70.6% 82.7% 91.4% 84.3%
user 3 77.6% 88.3% 93.8% 86.7%
user 4 63.6% 69.4% 84.9% 86.7%
user 5 66.5% 91.0% 81.2% 80.9%
user 6 59.3% 80.6% 80.2% 68.5%
user 7 71.4% 87.3% 90.4% 86.7%
user 8 69.8% 87.7% 82.4% 83.6%
Average 68.3% 83.8% 86.5% 82.9%
BayesNet classifier
Chance: 33.3% 50% 50% 50%
![Page 37: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/37.jpg)
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
2 secs 86.5%68.3
%
83.8% 82.9%
![Page 38: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/38.jpg)
We can do better…
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
???
![Page 39: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/39.jpg)
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
Throwing time back in…
“Math”
We can average over temporally adjacent windows
to improve classification accuracy
![Page 40: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/40.jpg)
Averaging with Task Transitions
Task transitions result in conflicting data in averaging window.
High density of transitions will result in lower accuracy.
![Page 41: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/41.jpg)
Averaging with Task Transitions
Fewer task transitions will yield better classification accuracy.
![Page 42: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/42.jpg)
Averaging with Task Transitions
No transitions and averaging over all data will be the even better.
![Page 43: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/43.jpg)
Classification Accuracy with Averaging
3 tasks
Math v. Rot
Rest v. Rot
Rest v. Math
100%
95%
90%
85%
80%
75%
70%
65%
60%
55%
50%
Mean Classification Accuracy vs. Averaging Scenarios (Mental Tasks)
Baseline No Averaging
5 windows with transitions
5 windows no transitions
9 windows no transitions
+5.1 to +15.7%for 3-tasks
Error bars represent standard deviation
![Page 44: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/44.jpg)
So, can we really read minds?
3 tasks
Math v. Rot
Rest v. Rot
Rest v. Math
100%
95%
90%
85%
80%
75%
70%
65%
60%
55%
50%
Mean Classification Accuracy vs. Averaging Scenarios (Mental Tasks)
Baseline No Averaging
5 windows with transitions
5 windows no transitions
9 windows no transitions
Error bars represent standard deviation
Possibly not, we might be really detecting subtle motor movements….
![Page 45: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/45.jpg)
Cognitive/Motor “Fabric”
Does this matter to neuroscience? Yes
Does this matter to HCI? Maybe not
Tasks of varying cognitive difficultly are involuntarily coupled with physiological responses, such as minute imperceptible motor activity. [Kramer ’91]
Therefore, it is impossible to completely isolate cognitive activity neurologically intact individuals.
![Page 46: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/46.jpg)
Cognitive/Motor “Fabric”
If motor artifacts are reliably correlated with different types of tasks or engagement, why not use those to help the classifier?
Requiring users to not move is also very impractical.
Non-Cognitive Artifacts detected by EEG:– Blinking– Eye movement– Head movement– Scalpal GSR– Jaw and facial EMG– Gross limb movements– Sensory Response Potentials
![Page 47: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/47.jpg)
Experiment 2 – Game Task
To explore this idea of using non-cognitive artifacts to classify tasks using EEG, we chose a PC-based video game task.
Halo, a PC-based first person shooter game produced by Microsoft Game Studios.
Navigate a 3D environment in an effort to shoot opponents using various weapons.
Relatively high degree of interaction with mouse and keyboard input controls.
![Page 48: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/48.jpg)
Game Tasks
Rest – baseline rest task, relax, fixate eyes on cross hairs on center of screen, do not interact with controls. Game elements do not interact with participant.
Solo – navigate environment, interact with elements in the scene, and collect ammunition. Opponent controlled by expert did not interact with participant.
Play – navigate environment and engage opponent controlled by expert. Expert instructed to play at a level just slightly above skill of participant to optimally challenge them.
![Page 49: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/49.jpg)
Game Experimental Procedure
Setup, design, and procedure was similar to first study.
• Participants had tutorial and practice time with game controls.
• 3 tasks repeated 6 times (counterbalanced)
• Tasks were 24 seconds to allow navigation time.
• Only 2 sessions were run for each participant
• Same 8 participants from first study were run in this study.
• Same data preparation and machine learning procedure.
![Page 50: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/50.jpg)
Results – Game Tasks
3 tasks
Rest v. Play
Rest v. Solo
Solo v. Play
100%
95%
90%
85%
80%
75%
70%
65%
60%
55%
50%
Mean Classification Accuracy vs. Averaging Scenarios (Game Tasks)
Baseline No Averaging
5 windows with transitions
5 windows no transitions
19 windows no transitions
Error bars represent standard deviation
93.1%
![Page 51: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/51.jpg)
Conclusion
This experimental design and data processing procedure can be applied to a much wider range of applications/tasks. Our two experiments were just two examples at different ends of a spectrum.
Compelling results can be achieved with low-cost equipment and without significant medical expertise or training.
Non-cognitive artifacts (inevitable in realistic computing scenarios) can be embraced improve classification power.
To make BCI relevant to HCI, we must challenge traditional assumptions and creatively work with its limitations.
![Page 52: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/52.jpg)
Thanks!
Johnny Chung [email protected]
Desney [email protected]
Thanks to MSR and the VIBEGroup for supporting this work.
![Page 53: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/53.jpg)
Cross-user Classifier
3 Cognitive Tasks
40.00%
45.00%
50.00%
55.00%
60.00%
65.00%
70.00%
75.00%
80.00%
85.00%
90.00%
95.00%
100.00%
Base 5 win 5win no edges Known
3class
MathRotate
RestRotate
RestMath
![Page 54: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/54.jpg)
Cross-user Classifier
3 game tasks
50.00%
55.00%
60.00%
65.00%
70.00%
75.00%
80.00%
85.00%
90.00%
95.00%
100.00%
Base 5 win 5win no edges Known
3class
SoloPlay
RestPlay
RestSolo
![Page 55: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/55.jpg)
Brain Sensing/Imaging Technologies
MRI – only anatomical data
CT – only anatomical data
ECoG
SPECT
PET
MEG
fMRI
EROS/fNIR
EEG
![Page 56: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/56.jpg)
Brain Sensing/Imaging Technologies
MRI – only anatomical data
CT – only anatomical data
ECoG – highly invasive surgery
SPECT – radiation exposure
PET – radiation exposure
MEG
fMRI
EROS/fNIR
EEG
![Page 57: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/57.jpg)
Brain Sensing/Imaging Technologies
MRI – only anatomical data
CT – only anatomical data
ECoG – highly invasive surgery
SPECT – radiation exposure
PET – radiation exposure
MEG – extremely expensive
fMRI – extremely expensive
EROS/fNIR
EEG
![Page 58: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/58.jpg)
Brain Sensing/Imaging Technologies
MRI – only anatomical data
CT – only anatomical data
ECoG – highly invasive surgery
SPECT – radiation exposure
PET – radiation exposure
MEG – extremely expensive
fMRI – extremely expensive
EROS/fNIR – currently expensive, still in infancy
EEG – safe, easy, no medical expertise
![Page 59: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/59.jpg)
Other cool things you can do with an EEG device…
![Page 60: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/60.jpg)
Event Related Potentials (ERP)
• Electrical activity related to or in response to the presentation of a stimulus
• Very well studied• Relatively robust• Used daily in clinical settings to check sensory
mechanisms, typically in infants
• Requires averaging over 30-100 windows synchronized with to see response.
![Page 61: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/61.jpg)
ERP - AEP
ERP: Auditory Evoked Potential • Used in clinics/hospitals to check hearing.• Response to clicks in the ear
Bold Lines = no clicksThin Line = with clicksAEP response
AEP response
![Page 62: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/62.jpg)
ERP - VEP
ERP: Visual Evoked Potential • Focusing on a flashing target, the visual cortex will “resonate” at the
stimulus frequency.
Stimulus Frequency
Harmonics
![Page 63: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/63.jpg)
ERP – Auditory and Visual P300
• Well known/studied potential related to “attention” or “surprise”
• Presented with 2 stimuli and instructed to count one of the stimuli
• Positive response will follow the stimulus of interest.
![Page 64: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/64.jpg)
Side note: EEG as ECG
ECG - Electrocardiogram • placing an electrode on the chest provides a clear measure of
cardiac activity.• translation to BPM is a simple autocorrelation
Heart beats
Single Beat period
0.4
µV
un
its
![Page 65: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/65.jpg)
EEG as EMG
EMG - Electromyography • Measures muscular activity
Wrist relaxation (return to straight position)
Tension holding
Wrist inward contraction (toward inner forearm)
Wrist rest state
NOTE: The magnitude of the spikes seem to be proportional to the acceleration involved with the movement.
0.4
µV
un
its
![Page 66: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/66.jpg)
EEG as Blink Detector
• Electrical activity due to muscle movement involved with eye blinks propagate through the head.
• Similarly, eye movements also affect the EEG recording
0 0.5 1 1.5 2 2.5
x 104
-200
-150
-100
-50
0
50
100
150
200
Blinks Blinks
![Page 67: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/67.jpg)
Task Classification Background
• Previous work is split primarily into two camps:
Operant Conditioning Pattern Recognition
![Page 68: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/68.jpg)
Task Classification Background
• Previous work is split primarily into two camps:
Operant Conditioning Pattern Recognition
Human learns how the machine works
Machine learns how the human works
![Page 69: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/69.jpg)
Task Classification Background
• Previous work is split primarily into two camps:
Operant Conditioning Pattern Recognition
Human learns how the machine works
Machine learns how the human works
Relatively newEarly dabbling in the late-80’s
Most work has happened in last 5 years.
![Page 70: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/70.jpg)
Pattern Recognition
Data Collection &
Experimental Design
Signal Processing &
Feature Generation
Machine Learning &
Improving Accuracy
![Page 71: ppt - Johnny Chung Lee - Human Computer Interaction Research](https://reader036.vdocuments.site/reader036/viewer/2022062613/5442ea80b1af9f2d0a8b4788/html5/thumbnails/71.jpg)
EEG Setup
International 10-20 EEG electrode placement system
Two channels placed on P3 and P4 with both references tied to Cz. Locations selected based on pilot recordings.
Attaching electrodes: Prepare the site with a cleaner, use conductive paste to improve connection and hold electrode in place.
P Paste rinses out with water, non-toxic. 5-10 minute preparation.