Integration of EEG/MEG/fMRIIntegration of EEG/MEG/fMRI

Vince D. Calhoun, Ph.D.Vince D. Calhoun, Ph.D.Director, Image Analysis & MR ResearchDirector, Image Analysis & MR Research

The MIND InstituteThe MIND Institute

Associate Professor, Electrical and Computer EngineeringAssociate Professor, Electrical and Computer EngineeringThe University of New MexicoThe University of New Mexico

X A S= ×2

OutlineOutline•• fMRI/EEG datafMRI/EEG data•• Three Approaches to integration/fusionThree Approaches to integration/fusion

•• PredictionPrediction•• ConstraintsConstraints•• 22ndnd Level FusionLevel Fusion

3

EEG/ERPEEG/ERP

4

EventEvent--related Potentials (related Potentials (ERPsERPs))

Time

+

_

Vol

tage

EEG

CSF

EEG Cocktail PartyCocktail Party

Blind EEG Source Separation Blind EEG Source Separation ICAICA

Unmixes scalp channel mixing by volume conduction!

Slide borrowed from S. Slide borrowed from S. MakeigMakeig

Cortex

Skull

Local Synchrony

Local Synchrony

Skin

Electrodes

Domains of synchrony

EEG

Scalp sensors ‘mix’ the dynamics of cortical (and non-brain) sources

Spatial Source Filtering

Slide borrowed from S. Slide borrowed from S. MakeigMakeig

Slide borrowed from S. Slide borrowed from S. MakeigMakeig Slide borrowed from S. Slide borrowed from S. MakeigMakeig

Slide borrowed from S. Slide borrowed from S. MakeigMakeig Slide borrowed from S. Slide borrowed from S. MakeigMakeig

Activation of 2 diff. Muscels sets}

Horizontal eye movement

Heart beating, digital watch

Faulty sensor

}Breathing, bumps caused by overlearning

http://sccn.ucsd.edu/eeglab

EEGLAB Matlab Toolbox

Why EEGWhy EEG--fMRI IntegrationfMRI Integration

+ high temporal resolution (~ ms)– poor spatial resolution (~ cm)

+ high spatial resolution (~ mm)– low temporal resolution (~ seconds)

EEG

fMRI

simultaneous acquisition

EEG and fMRI data reflect the same neuronal activityEEG and fMRI data reflect the same neuronal activity

Signal cascade of EEG and fMRI signals

BOLD and LFP (BOLD and LFP (--> EEG) Signal result from same neuronal process> EEG) Signal result from same neuronal process

Logothetis et a. Nature 2001

EEGEEG--fMRI Integration: Two Inverse ProblemsfMRI Integration: Two Inverse Problems

Metabolic & VascularResponse

EEG fMRI

NeuronalActivation

???

???

EEGEEG--fMRI Integration: Two Inverse ProblemsfMRI Integration: Two Inverse Problems

EEG: Inverse Problem in SpaceEEG: Inverse Problem in Space

EEG

NeuronalActivation

???

???

ill-posed Problem:

•2-Dim Measurement: Electrode recordings on the skull

•3-Dim Solution: Neuronal Activation in the Brain

-> Infinite Number of Solutions-> Need for Priors

fMRI: Inverse Problem in TimefMRI: Inverse Problem in Time

Metabolic & VascularResponse

fMRI

NeuronalActivation

37

OutlineOutline•• fMRI/EEG datafMRI/EEG data•• Three Approaches to integration/fusionThree Approaches to integration/fusion

•• PredictionPrediction•• ConstraintsConstraints•• 22ndnd Level FusionLevel Fusion

•• ConclusionsConclusions

Approaches to EEG-MRI data integration

Data Integration through:

(i) Predictionsome features of EEG to predict fMRI responses.

(ii) ConstraintsSpatial Information from fMRI as Priors for Source Reconstruction

(iii) FusionCommon forward or generative model to explain EEG and fMRI data

Through Prediction: e. g. EpilepsyThrough Prediction: e. g. Epilepsy

Benar et al. Neuroimage 2002

Interictal Epileptiform Discharges predict fMRI response

Moosmann & Ritter, et al. Neuroimage 2003

Through Prediction: Alpha RhythmThrough Prediction: Alpha Rhythm

Alpha Rhythm predicts fMRI response

SummarySummary•• EEG can be recorded simultaneously with fMRIEEG can be recorded simultaneously with fMRI•• A reasonable EEG quality can be obtainedA reasonable EEG quality can be obtained•• ICA helps to disentangle otherwise overlapping ICA helps to disentangle otherwise overlapping

EEG signalsEEG signals•• EEGEEG--informed fMRI analysis supports direct informed fMRI analysis supports direct

couplingcoupling

61

OutlineOutline•• fMRI/EEG datafMRI/EEG data•• Three Approaches to integration/fusionThree Approaches to integration/fusion

•• PredictionPrediction•• ConstraintsConstraints•• 22ndnd Level FusionLevel Fusion

•• ConclusionsConclusions

Approaches to EEG-MRI data integration

Data Integration through:

(i) Predictionsome features of EEG to predict fMRI responses.

(ii) ConstraintsSpatial Information from fMRI as Priors for Source Reconstruction

(iii) FusionCommon forward or generative model to explain EEG and fMRI data

(ii) Constraints: fMRI Priors help EEG Analysis(ii) Constraints: fMRI Priors help EEG Analysis

Without fMRI priors With fMRI priors

Dale & Halgren, Curr Opin Neurobiol. 2001

Dipole Fitting

MEG Waveforms

Visual size judgment task

•Spatial Information from fMRI as Priors for Source Reconstruction

64

OutlineOutline•• fMRI/EEG datafMRI/EEG data•• Three Approaches to integration/fusionThree Approaches to integration/fusion

•• PredictionPrediction•• ConstraintsConstraints•• 22ndnd Level FusionLevel Fusion

•• ConclusionsConclusions

Approaches to EEG-MRI data integration

Data Integration through:

(i) Predictionsome features of EEG to predict fMRI responses.

(ii) ConstraintsSpatial Information from fMRI as Priors for Source Reconstruction

(iii) FusionCommon forward or generative model to explain EEG and fMRI data

66

Data integration/fusion: Previous WorkData integration/fusion: Previous Work

67

Data Fusion vs. Data IntegrationData Fusion vs. Data Integration•• Definitions Definitions [[SavopolSavopol ISPRS 2004]ISPRS 2004]

•• Data IntegrationData Integration: The use of another data: The use of another data--type to type to improve the geometry or other information improve the geometry or other information reconstructed from the first datareconstructed from the first data--type (e.g. fMRI type (e.g. fMRI constrained EEG).constrained EEG).

•• Data FusionData Fusion: Images analyzed jointly, hence allowing : Images analyzed jointly, hence allowing them to influence one another, to reveal interthem to influence one another, to reveal inter--relationships between datarelationships between data--types.types.

EEG

fMRITem

pora

l Res

olut

ion

Spatial Resolutionlower higher

low

erhi

gher

(iii) Fusion: common generative model that explains EEG and fMRI(iii) Fusion: common generative model that explains EEG and fMRI

•• Bayesian Hierarchical ModelsBayesian Hierarchical Models

Pedro Valdes-Sosa

Reverend Thomas Bayes (1702-1761)Karl Friston et al. (now)

•Multiway Partial Least Squares

•ICA or jICA

S = Ax

69

Possible approaches for joint analysesPossible approaches for joint analyses•• PointPoint--basedbased

•• Correlation Correlation [[WorseleyWorseley 2002]2002]

•• Straightforward, but difficult to visualizeStraightforward, but difficult to visualize•• RegionRegion--basedbased

•• Interregional correlation Interregional correlation [[HorovitzHorovitz, et al, 2002, Mathalon, et al, 2003], et al, 2002, Mathalon, et al, 2003]

•• Structural equation modeling or dynamic causal modeling Structural equation modeling or dynamic causal modeling [McIntosh and [McIntosh and GonzalezGonzalez--Lima 1994; Friston et al., 2003]Lima 1994; Friston et al., 2003]

•• Useful for model testing, does not take into account all brain Useful for model testing, does not take into account all brain regions/timesregions/times

•• TransformationTransformation--basedbased•• A natural set of tools for this problem include those that transA natural set of tools for this problem include those that transform form

data matrices into a smaller set of modes or componentsdata matrices into a smaller set of modes or components•• Singular value decomposition Singular value decomposition [Friston et al., 1993; Friston et al., 1996][Friston et al., 1993; Friston et al., 1996]

•• Partial Least Squares Partial Least Squares [McIntosh, [McIntosh, BooksteinBookstein, et al, 1996], et al, 1996]

•• Canonical Canonical VariatesVariates Analysis Analysis [[StrotherStrother et al, 1995]et al, 1995]

•• Independent Component Analysis Independent Component Analysis [[McKeownMcKeown et al, 1998, Calhoun et al, 2001, et al, 1998, Calhoun et al, 2001, Beckmann, et al., 2002]Beckmann, et al., 2002]

70

Separate vs. Joint EstimationSeparate vs. Joint Estimation

( ) ( ) ( ) ( ), , , ,

1 1 and

C CE E F F

i v ic c v i v ic c vc c

x a s x a s= =

= =∑ ∑

1/w a=( )( );Ep x w ( )( );Fp x w

( )( )1

*1 1arg max log ;E

ww p x w=

( )( )2

*2 2arg max log ;F

ww p x w=

( ) ( )( )* arg max log , ;E F

ww p x x w=

Linear mixtures with Linear mixtures with shared mixing parametershared mixing parameter

In a nonIn a non--joint analysis, we maximize joint analysis, we maximize the likelihood functions for each the likelihood functions for each

modality separatelymodality separately……

Resulting in two unmixing Resulting in two unmixing parameters, that then have to parameters, that then have to somehow be fused togethersomehow be fused together

In contrast: for a joint analysis we In contrast: for a joint analysis we maximize the joint likelihood maximize the joint likelihood

function, resulting in a single fused function, resulting in a single fused unmixing parameterunmixing parameter

71

Joint ICA ApproachJoint ICA Approach( ) ( ) ( ) ( )Mixing Model: and E E F F= =X AS X AS

( ) ( ) ( ) { }1 2For two sources: , where m= ,Tm m m E F⎡ ⎤= ⎣ ⎦X x x

11 12

21 22

Shared mixing matrix: a aa a⎡ ⎤

= ⎢ ⎥⎣ ⎦

A

( ) ( ) ( )1 11 1 12 2F F Fa a= +x s s ( ) ( ) ( )

1 11 1 12 2E E Ea a= +x s s

( ) ( ) ( )2 21 1 22 2F F Fa a= +x s s ( ) ( ) ( )

2 21 1 22 2E E Ea a= +x s s

The mixing equations are:

( ) ( )( ) ( ) ( )( ){ }Update equation: 2 2T TE E F FηΔ = − −W I y u y u W

( ) ( )( ) ( )where and 1/(1 e )m m xg g x −= = +y u

72

OverviewOverviewBrain 1 Brain 2 Brain N

a

b

Component #1 Component #2

…

…

c 0 ms 200 ms 300 ms

ERP

fMRI

IndependentComponents

Data

Spatiotemporal“Snapshots”

low

hiintensity

1t

1s

2t

2s

[ ]1 2=T t t

[ ]1 2=S s s

Joint ICA

recombine

73

Auditory Oddball TaskAuditory Oddball Task

Target NovelStandardStandard StandardStandard

1 kHz

tone, sweep,whistle

Standard

0.5 kHz

Standard

74

Group Averaged fMRI/ERP FeaturesGroup Averaged fMRI/ERP Features

75

PreprocessPreprocess

FeatureFeatureExtractionExtraction

fMRIfMRI EEGEEG

NormalizeNormalize NormalizeNormalize

Noise RemovalNoise Removal

PreprocessPreprocess

FeatureFeatureExtractionExtraction

VisualizationVisualization

jICAjICA

Subject 1Subject 1Subject 2Subject 2

Subject NSubject N

Subject 1Subject 1Subject 2Subject 2

Subject NSubject N

Position Position CzCz of of ““TargetTarget””--locked EEG signallocked EEG signalPeak of Peak of ““TargetTarget””

--locked fMRI signallocked fMRI signal

76

Joint ERP/fMRI ComponentsJoint ERP/fMRI Components

77

Visualization SnapshotsVisualization Snapshots……[ ]1ERP (temporal) Components: N=T t t…

[ ]1FMRI (spatial) Components: N=S s s…( ) ( )FMRI Image Snapshot: T

F t t= ×M T S

( ) ( )ERP Snapshot: TE v v= ×M T S

78

FMRI Snapshots (movie)FMRI Snapshots (movie)

79

FMRI Snapshots (stills)FMRI Snapshots (stills)

80

ERP SnapshotsERP Snapshots

Parallel ICAParallel ICA•• Impose additional interImpose additional inter--modality correlation to modality correlation to

improve ability to identify connections between improve ability to identify connections between EEG and fMRIEEG and fMRI

jica_simulationjica_simulation

neuronal sources s = (s1, s2, …, sn)T transformed by an unknown mixing matrix A to the signal mixture x = As; features of the sources and their mixture observed as scalp EEG/ERP waveforms E at timepoints t and channels c as AEs = xE= [xE1(tc), xE2(tc), …, xEn(tc)]T; hemodynamic correlates of the detected in fMRI volumes F at the respective locations v as AFs = xF = [xF1(v), xF2(v), …, xFn(v)]T;the mixture is entered into a common 2D space from which fused signals yEF = [xEF1(tcv), yEF2(tcv), …, yEFn(tcv)]T are extracted by adjusting the unmixing matrix W (the inverse of A), such that y = Wx optimally represents s. The update equation for the algorithm to compute the common unmixing matrix W and the fused EEG/ERP and fMRI sources, uEF is ΔW = η{I-2 yEF (uEF) T }W, where yEF = g(uEF), and g(x) = 1/(1+e-x) is the nonlinearity in the neural network…

s

y

x

A

W

jica_method

Eichele, Calhoun, Moosmann et al., in progress

jica_resultsjica_results

Eichele, Calhoun, Moosmann et al., in progress

136 ms

85

Fusion ICA Toolbox (FIT)Fusion ICA Toolbox (FIT)