decoding representational spaces with multivariate pattern ...€¦ · decoding representational...

Decoding representational spaces with multivariate pattern analysis (MVPA) of fMRI data

Jim HaxbyCenter for Cognitive Neuroscience, Dartmouth College

Center for Mind/Brain Sciences (CIMeC), University of Trento

Hyperalignment Swaroop Guntupalli Post-doctoral fellow

2

Haxby Lab Analysis of similarity structure, representation of biological classes Andy Connolly Post-doctoral fellow

Yaroslav Halchenko Research scientist

Face & person perception M Ida Gobbini Associate professor Ricercatrice, U Bologna

Person perception Dylan Wagner Post-doctoral fellow

Action representation, computational methods Nick Oosterhof Post-doctoral fellow

NeuroDebian

Attention Sam Nastase Graduate student

Hyperalignment Swaroop Guntupalli Post-doctoral fellow

With help from

Peter RamadgeElectrical EngineeringPrinceton University

Mert Rory Sabuncunow at MGH

Bryan ConroyPhilips Research

Alex LorbertSuperfish, Israel

and EE grad students, past and present

Hao XuGoogle

Cameron Chencurrent

•  Neural decoding: understanding representational spaces

•  Statement of the problem: capturing fine-grained distinctions in a common model

•  Conceptual framework: high-dimensional representational spaces

•  Deriving the common space and individual transformation matrices with hyperalignment

•  Validation

•  Conclusions

Decoding representational spaces with multivariate pattern analysis (MVPA) of fMRI data

5





•  Validation

•  Conclusions

A common high-dimensional linear model of representational spaces in human cortex

6

fMRI data matrix

fMRI data matrix Columns are voxel response tuning profiles

fMRI data matrix Rows are multivoxel response patterns

What is a neural representational space? A pattern of activity (distributed over cortex) can be analyzed as a

vector in an n-dimensional space, where n = number of voxels (fMRI) or neurons (single unit recordings) or …

Neural decoding using multivariate pattern analysis (MVPA)

•  Pattern classification (MVPC)

•  Representational similarity analysis (RSA)

MVP Classification divides the representational space into sectors, each of which is associated with a different category

Representational similarity analysis (RSA) indexes similarities between vectors as distances to analyze

representational geometry

MVPC of fMRI data •  Each observation (pattern) is

treated as a high-dimensional vector

•  Each dimension is a single feature - usually a voxel

Condition a (e.g chairs) Condition b (e.g. shoes)

Illustration: 2 voxel pattern classifier



•  Each dimension is a single voxel (or other feature)

•  Classifiers find a rule (e.g. decision surface) that optimally differentiates observations for different conditions

Condition a (e.g chairs) Condition b (e.g. shoes)




•  Each dimension is a single voxel (or other feature)

•  Classifiers find a rule (e.g. decision surface) that optimally differentiates observations for different conditions

•  The validity of that rule is tested on independent test data that played no role in deriving that rule


Training data

Test data

Building and Testing Pattern Classifiers

1.  Divide observations into training and test data sets

2.  Based on the training data only

a.  Select features (usually voxels)

b.  Develop decision rule

3.  Test decision rule on test data set

A new classifier is built for each individual

Multivariate Pattern Classification Example: Classifying responses during viewing of animal species

(VT cortex, SVM)

(Haxby et al. 2011; Connolly et al. 2012)

monkey lemurPrimates

warbler mallardBirds

luna moth ladybugInsects

Classifier output

Vie

wed

stim

ulus

100

90

80

70

60

50

40

30

20

10

0

Percent ofclassifications

MVP classification of animal species is significant in both early visual cortex and the ventral visual pathway (LOC) (Connolly et al. 2012)

Data-driven cluster analysis finds distinct representational geometries in the ventral visual pathway (LOC) and early visual cortex (EV) (Connolly et al. 2012)

DSMs in early visual and LOC cortices correlate highly with semantic ratings and V1 models but not with each other (Connolly et al. 2012)

Correlation with ratings model = 0.76

Correlation with V1 model = 0.78

Correlation between LOC and EV DSMs = 0.09





•  Validation

•  Conclusions


29





•  Validation

•  Conclusions


30

The problem: Building model representational spaces that are common across brains

•  MVPA detects fine distinctions carried by fine-grained patterns of neural activity

•  Anatomical alignment of brain spaces blurs these fine-grained distinctions

•  Can a model of functional brain architecture capture these fine-grained distinctions among representations in a common framework?

–  If so, how would such a model be structured?–  Will it work? Do brains share a common basis for neural coding?

The problem: Loss of fine-grained distinctions among representations after

anatomical alignment of brainsWithin-subject classification(new model for each subject)

Between-subject classification(common model based on anatomy)

WSC (1000 voxels)

BSC (1000anatomically-aligned voxels)

Chance (16.7%)





Modeling functional architecture of the human cortex: Anatomical alignment

Individual brainsTransformations

(affine or nonlinear warps) Brain atlas



(affine or nonlinear warps) Atlas brain




P2

P1



-  A pattern of activity is a response vector

-  Dimensions are local features, e.g. voxels, of the pattern of activity

-  Model space is based on features (dimensions) with common tuning profiles


•  Validation

•  Conclusions


36

Conceptual framework: High-dimensional representational spaces

Brain activation patterns Data matrix Representational space (2 voxels)

The locations of response pattern vectors for the same stimuli differ across subjects

Modeling functional architecture of the human cortex: Individual representational spaces <=> common representational space

voxel1

voxel2

voxel3, v4, …,vi

voxel1

voxel2

voxel3 v4, …,vj

voxel1

voxel2

Individual representational spaces

dim1

dim2

dim3, dim4, …, dimm

Common modelrepresentational spaceIndividual brains

Transformations(improper rotations)

voxel3 v4, …,vk

2


voxel1

voxel2

voxel3 ….

voxel1

voxel2

voxel3 ….

voxel1

voxel2

voxel3 ….

Individual brainsIndividual

representational spaces

dim1

dim2

dim3 ….

Common modelrepresentational space

1

3

123



voxel1

voxel2

voxel3 ….

voxel1

voxel2

voxel3 ….

voxel1

voxel2

voxel3 ….

Individual brainsIndividual

representational spaces

dim1

dim2

dim3 ….


2

1

3

123


2

1

3

213


dim1

dim2

dim3 ….


X

voxel1

voxel2

voxel3 ….

voxel1

voxel2

voxel3 ….

voxel1

voxel2

voxel3 ….


X

X

X

Transformations(transposed rotations)


dim1

dim2

dim3 ….


X

voxel1

voxel2

voxel3 ….

voxel1

voxel2

voxel3 ….

voxel1

voxel2

voxel3 ….


X

X

X





-  Hyperalignment algorithm based on Procrustes transformations

-  A rich sampling of response vectors using natural stimulus

•  Validation

•  Conclusions


45

Matrix math: Individual transformation matrices rotate individual brain spaces

into common model space coordinates

Individual brain spaces Transformation matrices Common model space

Individual transformation matrices are the key to building the common model: How can the parameters be derived?

Transformation matrices

49

Subject 1 Subject 2

Broad sampling of a neural representational space with a movie

Response patterns in cortex

15 response pattern vectors in individual 3D representational spaces (full exp’t has >2600 vectors in >50,000D space)

Subject 1 Subject 2



Procrustes transformations(improper rotations)

x [ ] =

=S1

S2


S1

S2

S3



x [ ]s2 =

=

x [ ]s3 =


S4

S5

S6

…



x [ ]s5 =

x [ ]s6 =

x [ ]s4 =

Movie data in Subject-specific Movie data in Brain Space Transformation Matrix Model Space

55

X =

S1

Experiment 2 data in Subject-specific Experiment 2 data in Brain Space Transformation Matrix Model Space

56

The key that unlocks an individual’s neural code


X =

Modeling representational spaces in all human cortex with searchlight hyperalignment

d1

d2

d3, d4, …, dk

Voxels in overlapping searchlights Overlapping searchlight transformation matrices are hyperaligned across subjects are aggregated into a whole cortex matrix

Data in individual brain anatomy Data in common model space




•  Validation

–  Between-subject correlations of time-series

–  Between-subject classification of movie time segments

–  Between-subject correlations of local similarity structures

–  Applying transformation matrices to data from an unrelated experiment

•  Conclusions


59

Whole-brain hyperalignment increases between-subject MVPC (bsMVPC) of 15 s movie time segments in occipital, temporal, parietal, and frontal cortices

5% 30%

Classificationaccuracy (%)

Increased bsMVPC of movie time-segmentsin visual, auditory, and cognitive regions of interest (ROIs)

(coordinates from NeuroSynth)

Whole-brain hyperalignment increases between-subject classification of 15 s movie time segments for the whole brain (after SVD dimensionality reduction)

bsMVP

C accuracy (%

± SE)

How much movie data is necessary to calculate transformation matrices? Answer – the more the better, but ~20 minutes isn’t too bad

Number of Bme-‐points for hyperalignment

Smoothing reduces BSC accuraciesIn all visual, auditory, and cognitive regions of interest (ROIs)

Smoothing filter (FWHM)

Point spread function (intersubject correlations of movie time series):Fine spatial scale of alignment of function

ROI mean

Whole-brain hyperalignment increases between-subject correlation of high-dimensional representational geometries

(correlations between movie time-points)

Correlation0.15 0.45

Increased intersubject correlations of representational geometriesin visual, auditory, and cognitive regions of interest (ROIs)

Second-order RSA: Between-ROI representational geometry dissimilarities

Dimension 1

Dim

ension

3

Dimension 1

Dim

ension

2

V1-‐le&

V1-‐right

PPA-‐right PPA-‐le&

FFA-‐le& FFA-‐right VWFA

MT-‐right MT-‐le&

Math-‐le& Math-‐right

WM-‐right WM-‐le&

Broca

A1-‐le& A1-‐right

Voice-‐le&

Music-‐le& Music-‐right Voice-‐right

V1-‐le& V1-‐right


FFA-‐le& FFA-‐right

VWFA

MT-‐right MT-‐le&

Math-‐le&

Math-‐right WM-‐right

WM-‐le&

Broca

A1-‐le&

A1-‐right Voice-‐le&

Music-‐le& Music-‐right

Voice-‐right

Multidimensional scaling (MDS) of similarity structuresin visual, auditory, and cognitive regions of interest* (ROIs)

1st subspace (dimensions 1 & 2) 2nd subspace (dimensions 1 & 3)

* ROI coordinates from Neurosynth

Dimension 1

Dim

ension

2

Dimension 1

Dim

ension

2

V1-‐le&

V1-‐right


FFA-‐le&

FFA-‐right VWFA

MT-‐right

MT-‐le&

A1-‐le&

A1-‐right

Voice-‐le&

Music-‐le&

Music-‐right

Voice-‐right

Multidimensional scaling (MDS) fit separately tovisual and auditory ROIs

MDS of visual regions only MDS of auditory regions only

Whole-brain hyperalignment increases between-subject correlation of high-dimensional representational geometries

that reflect widely divergent domains of information

Correlation0.15 0.45

A common high-dimensional linear model of representational spaces in human cortex James V Haxby1,2, J Swaroop Guntupalli1,3, Michael Hanke4, Peter J Ramadge5 1Dartmouth College, 2CIMeC, University of Trento, 3Caltech, 4University of Magdeburg, 5Princeton University

Within-subject classification (new model for each subject)

Between-subject classification (common model based on anatomy)

WSC (1000 voxels) BSC (1000 anatomically- aligned voxels) Chance (16.7%)


monkey lemur Primates

warbler mallard Birds

luna moth ladybug Insects

The problem Models based on anatomical alignment fail to capture fine-scale topographies that carry fine-grained distinctions among representations

The solution Searchlight hyperalignment of all cortex into a high-dimensional common model space

Validation highlights

Next: Connectivity hyperalignment Model dimensions have common functional connectivity profiles.

• The Procrustes transformation is used to develop the common space and to derive individual transformation matrices.

• General validity afforded by using a complex, rich stimulus to obtain a broad sample of response vectors.

Room for improvement •  General validity can be increased with better stimulus and task paradigms (e.g. motor

execution and music) •  Hyperalignment may be improved with better algorithm (e.g. regularized CCA, Xu,

Lorbert, et al. 2012)

Classification accuracy (%)

30% 5%

Between-subject classification of 15s movie time-segments (chance<1%)

Significance • Our model captures fine distinctions among neural population responses in a high-

dimensional representational space based on response tuning functions that are common across brains. •  The model is valid for diverse domains of information. •  Functional cortical topographies are modeled with individual basis functions that are

grounded in common tuning functions. •  Potential basis for a new kind of functional brain atlas. –  Report results as vectors in common model space rather than as anatomical

coordinates – Afford comparison and interpretation of results at a far more fine-grained level – Allow arbitrarily large, multi-subject data sets for MVPA

Between- and within-subject classification of 6 animal species (chance=17%) (BSC accuracy significantly higher than WSC, hyperalignment based on movie data)

Between-subject classification Within-subject classification Anatomical alignment Hyperalignment

Classification accuracy (%)

60% 30%

Mapping retinotopy by projecting other subjects’ polar angle maps into a different subject’s occipital topography (hyperalignment based on movie data)

Polar angle from subject’s own retinotopy data

Polar angle from other subjects’ retinotopy data

Correlation between measured and projected

Horizontal meridian

Vertical meridian

Intersubject correlations of connectivity vectors

Anatomical alignment

Connectivity hyperalignment

References Connolly AC, Guntupalli JS, Gors J, Hanke M, Halchenko YO, Wu YC, Abdi H, Haxby JV. (2012). The representation of biological classes in the human brain. Journal of Neuroscience. 32:2608-2618.

Guntupalli JS, Hanke M, Halchenko YO, Connolly AC, Ramadge PJ, Haxby JV. (under review). A model of representational spaces in human cortex.

Haxby JV, Guntupalli JS, Connolly AC, Halchenko YO, Conroy BR, Gobbini MI, Hanke M, Ramadge PJ. (2011). A common high-dimensional model of the representational space in human ventral temporal cortex. Neuron 72:404-416.

Haxby JV, Connolly AC, Guntupalli JS. (2014). Decoding neural representational spaces using multivariate pattern analysis. Annual Review of Neuroscience, 37, 435-456.

Xu H, Lorbert A, Ramadge PJ, Guntupalli JS, Haxby JV. (2012). Regularized hyperalignment of multi-set fMRI data. Proc. IEEE Signal Processing Workshop, Ann Arbor Michigan, 229-232

Supported by NSF1129764, “US-German collaboration: Building common high-dimensional models of neural representational spaces”

Anatomical alignment Hyperalignment

Brain connectivity patterns are better aligned in the common model space

Hyperalignment

Inter-subject correlation of connectivity vectors


X =

Whole-brain hyperalignment based on movie affords between-subject classification of responses in a visual category experiment (6 animal species)

at levels of accuracy that exceed within-subject classification

30% 60%

Classificationaccuracy (%)

Between-subject classification Within-subject classificationAnatomical alignment Hyperalignment




Similar results for other domains

•  Action observation and attention (Sam Nastase, Dartmouth)

•  Action execution (Nick Oosterhof, CIMeC, University of Trento)

•  Connectivity hyperalignment and Music (Swaroop Guntupalli, Dartmouth, Caltech)

•  Person knowledge (Dylan Wagner, Dartmouth, now Ohio State)

Further validation testing and algorithm development

Raiders of the Lost ArkLife on Earth

The Wire

Hyperalignment parameters are estimated from responses

recorded during movie viewing

Projecting group data from common model space into individual subject’s anatomy

dim1

dim2

dim3 ….


X

voxel1

voxel2

voxel3 ….

voxel1

voxel2

voxel3 ….

voxel1

voxel2

voxel3 ….


X

X

X


Modeling functional architecture of the human cortex: common model dimensions ≠ voxels


dim2

dim3 ….



Topographies of weights for three model dimensions in two subjects

Topographies for response patterns are modeled in different brains as weighted sums of individual-specific topographic basis functions

using the same weights for common model dimensions

Weights: ß1 ß2 ß3 ß4 ß5 ß6 ... ß35 ∑(ß*PC)

…

…

=>

=>

PC1 PC2 PC3 PC4 PC5 PC6 … PC35

S1

S2

Individual VT topographies for face vs object dimension in the model agrees well with the topographies of individually-defined FFAs

Subject 1 Subject 2

81

Mapping retinotopy by projecting other subjects’ polar angle maps into a different subject’s occipital topography

The topographic basis functions for PCs individually show little correspondence to category-selective face and place areas or the

domain-specific divisions for animate and inanimate stimuli

Weights: ß1 ß2 ß3 ß4 ß5 ß6 ... ß35 ∑(ß*D)

…

…

=>

=>

PC1 PC2 PC3 PC4 PC5 PC6 … PC35

S1

S2

Single dimensions (simple contrasts) are inadequate for modeling the functional architecture of cortex

Model dimensions (from PCA)

Varia

nce

acco

unte

d fo

r (%

)

The face vs object LD accounts for only 7% of movie response variance in VT cortex

•  <20% of VAF by top 3 model dimensions

•  <13% of VAF by 35 model dimensions

0

10

20

30

40

50

60

70

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35

35D common model

1D model (face versus object LD)

84

Category-selective regions!

Domain specificity! (animate vs inanimate)

Animacy continuum!

(human to bug)

It’s expertise!

Foveal versus peripheral

vision!

Stimulus size! 85

•  Our model captures fine distinctions among neural population responses in a high-dimensional representational space based on response tuning functions that are common across brains–  Valid for diverse domains of information

•  Functional cortical topographies are modeled with basis functions that are grounded in common tuning functions-  Accounts for structure-function relationships in individual

brains with high fidelity•  Single dimensions (or small numbers of dimensions) are

inadequate to capture fine distinctions and the fine-grained structure of topographies that carry these distinctions

Common model: Structure and validation testing

Why are anatomical coordinates inadequate for capturing neural representation?

•  Response tuning functions for voxels with the same anatomical coordinates are highly variable across brains.

•  The basic unit for neural representation is the population response, not the responses of single voxels (or single neurons).

Software for ROI hyperalignment and data are on PyMVPA (www.pymvpa.org)

See the NeuroDebian/PyMVPA booth in exhibits

New massive data release of 7T fMRI with natural stimulus and lots more:data website: http://www.studyforrest.orgpaper: Hanke et al. (2014) Nature Scientific Data, 1: 140003.

decoding representational spaces with multivariate pattern ...€¦ · decoding representational...

Documents