toward brain-computer interfacing minhye chang. contents 5. the berilin brain-computer interface:...
TRANSCRIPT
Toward Brain-Computer InterfacingMinhye Chang
Contents
5. The Berilin Brain-Computer Interface: Machine Learning-Based Detection of User Specific Brain States
7. Brain Interface Design for Asynchronous Control
Chapter 5
• Berlin Brain-Computer Interface procject
• Premovement Potentials in Executed and Phan-
tom Movements
• BCI Control-Based on Imagined Movements
• Lines of Further Improvement
Brelin Brain-Computer Interface
Brelin Brain-Computer Interface
• EEG-based BCI system with machine learning techniques– High quality feedback w/o subject training– Copes with the huge intersubject variability
• Spatial resolution of the somatotopy• Discriminability of premovement potentials in voluntary mo-
vements• Sensorimotor rhythms caused by motor imagery
– 128 channel EEG– Subjects w/ no or very little experience w/ BCI control– Information transfer rate above 35bits per minute.
Premovement Potentials
• Bereitschaftspotenzial, RP (readiness potential)– Activity in the motor cortex leading up to voluntary muscle movement– To build a classifier
• Letting healthy subjects actually perform the movements– Movement imagination poses a dual task: motor command prepara-
tion plus vetoing the actual movement
• Predictions about imminent movements– Exclude a possible confound with feedback from muscle and joint re-
ceptors– Assistance of action control in time-critical behavioral contexts
Left vs. Right Hand Finger Movements
• Readiness potential– Pronounced cortical negativation
• Left-hand vs. Right-hand finger tapping experiment– Electrodes : CCP3 and CCP4– Predominantly contralateral negativation before movement
• Self-paced finger-monements on a computer keyboard– Tap-rates of 30, 45, 60, and 120 taps per minute– 128 Ag/AgCl scalp electrodes– EMG, EOG
• Discriminate premovement potentials as fast as two taps per second
• Highly subject-specific
Left vs. Right Hand Finger Movements
Preprocessing
• Extracts the low-frequency content w/ an emphasis on the late part of the signal
Starting points: 128 samples
Emphasize the late signal content
FT coefficients : Pass-band
Four feature components per channel
Classification
• Regularized linear discriminant analysis (RLDA)– RP features normally distributed with equal covariance matrices.– The data processing preserve gaussianity
RP-Based Feedback in Asynchronous Mode
• Calibration session– The data is used to train a classifier.
• A useful continuous feedback in an asynchronous mode– Classifier must work for a broader interval of time– The system needs to detect the buildup of movement intentions
• Quite simple strategy : jittering– Extracts several with some time jitter b/w training samples– Two samples per key press: at 150 and at 50 ms before key press– Invariant to time shifts
• EMG activity at about 120ms
RP-Based Feedback in Asynchronous Mode
• Movement intention detector– Distinguishes b/w motor preparation intervals and “rest” intervals
• In a BCI feedback experiment (-160 to -80ms)
Detection of ‘Phantom Limb Commands’
• ERD(ERS) • Attenuation (amplification) of pericentral μ and β rhythms in the cor-
responding motor areas
• Lack of a time marker signal– Listened to an electronic metronome– Deep sound: rest, higher sound: perform either a finger tap or a phan-
tom movement
• 8 patients with amputations showed significant ″phantom-re-lated″ ERD/ERS of μ- and/or β -frequencies – Signed r2 values of the differences in ERD curves
• Gross somatotopic arrangement– Hand versus foot
• Finely graded representation of individual fingers• Examine the discriminability of BCI signals from close-by brain
regions– 128-channel EEGs– During self-paced movements of various limbs– Significantly reflect specific activations in sensorimotor cortices
Exploring the Limits
Exploring the Limits
• Averaged premovement potential patterns of one subject in different self-paced limb moving tasks
• Significantly reflect specific activations in sensorimotor cor-tices
• In repetitive movements, the discrimination decays already af-ter about 1s
• Modulations of sensorimotor rhythms evoked by imagined movements
• 6 subjects who had no or very little experience w/ BCI feed-back– 118-channel EEGs– Recorded EOG and EMG
• Calibration measurement (machine training)– Estimate parameters of a brain-signal to control-signal translation al-
gorithm
BCI Control-Based on Imagined Movements
• In the training sessions– Visual stimuli for 3.5s: (L) left hand, (R) right hand, or (F) right foot– Two types were selected for feedback; binary classifier
• Bias and scaling of the linear classifier– Different experimental condition of the (exciting) feedback situation
Experimental Setup
• 1st feedback application: position-controlled cursor– Classifier output translated to the horizontal position of a cursor.
Experimental Setup
Holding for 500msActivatedSuccess
• 2nd feedback application: rate-controlled cursor– At each update step a fraction of the classifier output was added to
the actual cursor position.
• 3rd feedback application: basket game– Operated in a synchronous mode– Horizontal position was controlled by
the classifier output
Experimental Setup
• Spatial filters– Optimize the discriminability of brain signals based on ERD/ERS ef-
fects of the motor rhythms– Common spatial pattern (CSP) analysis
• Feature calculation– Log of the variance in those surrogate channels– Calculated every 40 ms from sliding windows of 250 to 1000ms (sub-
ject-specific) for online operation
• Details about the processing methods and the selection of pa-rameters : Blankertz et al. (2005)
Processing and Classification
• Information transfer rate (ITR) in bits per minute (bpm)
– Compared to ROC curves, ITR considers different duration of trials and different number of classes
• Highest ITRs: rate-controlled cursor, asynchronous protocol
Results
• It can be operated at a high decision speed– Average trial length for 1st, 2nd, and 3rd was 3s, 2.5s, and 2.1~3s resp.
• The fastest subject : average speed of one decision every 1.7s• Subject who showed the most reliable performance
– only 2% of the total 200trials were misclassified at an average speed of one decision per 2.1s
– Sentences 135 characters in 30 minutes (4.5 letters per minute) in a free-spelling mode
Results
• It is possible to voluntarily modulate motorsensory rhythms w/o concurrent EMG activity (Vaughan et al., 1998)
• Squared biserial correlation coefficient, γ2
– For the classifier output and for the bandpass filtered and rectified EMG signals of the feedback sessions
– Occurrence of minimal EMG activity in some trials does not correlate with the EEG-based classifier
Investigating the Dependency
• CSP with Simultaneous Spectral Optimization– CSP strongly depends on the choice of the bandpass filter– Broadband filter for general choice– Subject-specific choices– Optimized spatial filters (usual CSP technique) + temporal finite im-
pulse response(FIR) filter : enhance the discriminability
• Significant superiority of the proposed CSSSP • The spatial and/or the spectral filter can be used for source
localization of the respective brain rhythms.
CSSSP
• Nonstationarities in EEG signals– How much of this nonstatioarity is reflected in the EEG features– How strongly is the classifier output affected– How can this be remedied
• The most serious shift occurred b/w the initial calibration measurement and online operation
• Shifts during online operation were largely compensated for by the CSP filters or the final classifier
• Simple adaption of classification bias successfully cured the problem.
Need for Adaptivity
Chapter 7
• Introduction
• Asynchronous Control
• EEG-Based Asynchronous Brain-Switches
• Asynchronous Control Design Issues
• Neil Squire Society– The only national not-for-profit organization in Canada– Neil Squire Brain Interface lab focused on BI system design specifically
for asynchronous control environments.
• Synchronized control environments where the system dictates the control of the user
• Robust multistate, asynchronous brain-controlled switch in the most natural manner in the real-world environments
Introduction
Asynchronous Control
• Output signal levels are changed or commands are issued only when control is intended– Intentional control (IC) / no control (NC) state– Remains neutral or unchanged during the NC state
• Asynchronous control– Characteristic of most real-world control applications– Most people expect from interface technology
Asynchronous Control
NC state; stable and unchanged IC; available for control
• The neutral or unchanging system output response during NC states
• Poor idling is indicated by false switch activations• How well BI transducers idle
– Rate of false activations or false positive error
• True and false activation rates as performance metrics
System Idling
With no gas (the NC state), the engine idles. Poor at idling
• Synchronized BI system– System-driven control strategy which cause significant user frustration
and fatigue
• Asynchronous TV controller– Simply change channels at any time they wish– The channel selection is stable when users are watching TV (NC state)
• Synchronous TV controller– Regularly poll the user to ask– Renders the changing of channels to specific periods
Synchronized BI System
• Two serious drawbacks– control decision regardless of whether the person is actually intending
control– User would need to wait for the system polling period to occur
Synchronized BI System
• Essence of the temporal control paradigms– Idle support and system availability
• Continuously available control– Always ready for the user to control
• Periodically available control– For Initial trial-based technology development– For restricting the signal processing complexity– Blocks a user’s attempt to control for the periods b/w control periods– “no control is possible at this time”
System availability
• Unintended action during NC states– Control paradigms that do not support idling– “Midas touch problem” by the eye-tracking community
• Four primary control paradigms– Constantly engaged mode : impractical– Asynchronous mode : most natural assistive device operation
Control Paradigms
• Specific signal processing algorithms to handle the NC state– Optimized TP rate and minimized FP rates– Trade-off b/w TP accuracy and FP rates → specific characteristics of
an application
• Brain-switch based on the outlier processing method(OPM)– Extract single-trial voluntary movement-related potentials (VMRPs)– TP rates greater than 90%– FP rates b/w 10 and 30% → limits OPM as an asynchronous switch– Multiposition asynchronous brain-switch
Asynchronous Brain-Switches
• Attempted VMRPs– Voluntary movement : existing and natural internal control system– “attempted” : subject with SCI attempt to move their fingers– By a very different neural mechanism
• The low-frequency asynchronous switch design (LF-ASD)– Relative power in the 1-4Hz band from ensemble VMRP increased– Focus on low frequency band– Wavelet analysis of the EEG signal over SMA and MI– Lower end of FP activations– TP rates of 30 to 78% during IC states with very low FP rates of 0.5 to
2% during NC
Asynchronous Brain-Switches
• Additional signal processing– EEG signal normalization– Switch output debounce– Feature set dimensionality reduction blocks– Increased the TP rate by an average of 33%
• Feature vectors navigate the feature space– System classification accuracy of more than 97%
• Levine and Huggins, Millán et al. (2004a), Yom-Tov and Inbar (2003), and Townsend et al. (2004)
Additional signal processing
• Very low FP rates during the NC state– High FP error rates cause undue user frustration– False activation appear uncontrollable to the user– Subjects would rather experience more trouble performing accurate
hits ( low TP rate)
• Evaluating asynchronous control– Receiver operating characteristic (ROC) curves– BI transducers with low FP error rates
Asynchronous Control Design Issue1
• “Tuning” the performance– ROC curve shows possible operating setups– by tuning various parameters
• Two-state LF-ASD brain-switch (on/off)– Scaling the relative magnitudes of NC state feature vectors vs. IC state
ones– FP rates under 1% during the time b/w FPs (30s or more)
Asynchronous BI Performance
• FPs typically clump together– Large periods of system idle time with free of FPs
• Switch-output jitter reduction methods– Switch debounce block– Improve error rates by reducing the FP jitter in the switch output– Trade-off : transducer availability– Debounce time ↑ → time the transducer is available ↓,
control time ↓
Switch-output jitter reduction
Borisoff et al. (2004)
• Intra-False Positive Rates– Dependent on the output classification rate of a transducer– FP rate of 1% with classification output every 1/16th sec →
FP every 1/16 * 100 = 6.3s
• Average time b/w errors greater than 30s– Reasonable design goal : FP rate of under 0.25%
• False activation rates : time rates w/ raw percentages
Asynchronous Control Design Issue2
• Temporal characteristics of NC and IC states– Environmental controllers : periods b/w the IC commands– Neural control of a robotic device : intercontrol times during periods
of intense usage
• Ubiquitous ON/OFF problem– Users with amputation have to turn the system on/off by themselves– Confirms user intent to turn the system to the awake mode– Eliminate FPs and require a simple commands
Asynchronous Control Design Issue3
• Costs associated with an FP– Multiple operating levels of the sleep mode– Full awake mode : sequence through higher modes, higher FP rates
and intentional command sequences
• How to structure tasks during different phases– Customization, training, and testing phases
Asynchronous Control Design Issue4
Sleep mode Awake mode
Low costs of FPs Easily corrected
High FP ratesLess intricate
• Initial customization– Accurate time-stamping for calibration and classifier– System-guided and system-paced tasks– Train and test in very similar control environments
• Apparatus necessary– Self guided and self-paced tasks → Self-report errors– Contamination of data – Accurate assessment of asynchronous system performance
Asynchronous Control Design Issue4
Thank you for your attention