Sensory-motor integration

Bijan Pesaran

May 1st, 2008

Visual response in PMv centered on hand

Graziano and Gross (1994)

Visual response in PMv centered on hand

Graziano and Gross (1994)

Duhamel et al (1997)

Invariant visual response in VIP

Duhamel et al (1997)

PRR neurons are eye-centered

Batista et al (1999)

Batista et al (1999)

Updating of response in PRR

Mechanisms for encoding multiple variables

• Centered tuning curve)(),( HTfTHf

Eye position influences visual response of 7a neurons

Andersen et al (1985)

Mechanisms for encoding multiple variables

• Centered tuning curve

• Multiplicative encoding - gain field)()(),( EThEgTEf

Tuning curveGain field

)(),( HTfTHf

Visual receptive fields in VIP partially shift with eye position

Avillac et al (2005)

Eye positions

Mechanisms for encoding multiple variables

• Centered tuning curve

• Multiplicative encoding - gain field

• Partial shift encoding – intermediate reference frame

)()(),( EThEgTEf

)()(),( EThEgTEf

Tuning curveGain field

)(),( ETfTEf

1

Eye position encoding

• Widespread phenomenon throughout visual and motor cortices

Peaked response

Eye leftEye middleEye right

Response

Target position

Eye position encoding

• Difficulty addressing flanking responses

Peaked response Flanking response

Eye leftEye middleEye right

Response

Target position

n

nnn ThEgTEf ),(

ThEgTEf )(),(

Gain model

n

nnn ThEgTEf ),(

ThEgTEf )(),(

Gain model

n

nnn ThEgTEf ),(

)(),( ETfTEf

Shifting model

Geometry of visually-guided reaching

• Eye-centered• Hand-centered• Relative position vectors

Nonlinear3D geometry and muscle dynamics

HH

TT

EE

T-ET-E

E-HE-H

H-EH-E

T-HT-H

Geometry of visually-guided reaching

Nonlinear3D geometry and muscle dynamics

HH

TT

EE

T-ET-E

E-HE-H

H-EH-E

T-HT-H

MIP PMdBatista et al (1999)Eye-centered

Graziano et al (1997)Hand-centered

PMv

What is code?

T-ET-E

T-HT-H

Pesaran et al(2006)

Decompose the 3-D response matrix

500

exp

1000exp1

1),,(

2ETEH

THEf

Distinguish between different reference frames

500

exp

1000exp1

1),,(

2HTEH

THEf

Define multiple effector reference frames

500exp

500exp

500exp),,(

222 EHHTETTHEf

Multiple reference frames

• Direct visuomotor transformation– Supports Area 5 data (Buneo et al 2002)

Direct Transformation

Serial TransformationHead-on-body

positionEye-in-head

position

Eye-centered

target location

Head-centered

target location

Body-centered

target location

Hand-centered

target location

Body-centeredhand position

+-

Motor output

Eye-centered

target location

Hand-centered

target location

Eye-centeredhand location

+- Nonlinear

3D geometry and muscle dynamics

Motor output

Nonlinear 3D geometry and muscle dynamics

MIP PMd

Nonlinear muscle

dynamics

Encoding relative position

Nonlinear 3D geometry and muscle dynamics

Batista et al 1999

Graziano et al 1997

H

T

E

T-E

E-H

H-E

T-H

PMv

• Coordinating the hand and the eye

• Eye-centered• Hand-centered• Relative hand-

eye position• Vectors in

abstract space

Design reference frame task

0 1 2

Start Cue DelayGo

Hold

Time (sec)

Design reference frame task

0 1 2

Start Cue DelayGo

Hold

Time (sec)

Design reference frame task

• Four initial eye positions

Design reference frame task

• Four initial eye and hand positions

Design reference frame task

• Four initial eye, hand and target positions• 111 PMd neurons and 48 MIP neurons

Example PMd neuron

Example MIP neuron

Sensory-motor transformations

Head-on-bodyposition

Eye-in-headposition

Eye-centered

target location

Head-centered

target location

Body-centered

target location

Hand-centered

target location

Body-centeredhand position

+-

Serial Transformation

+-

+-

Sensory-motor transformations

Eye-centeredhand location

Eye-centeredtarget location

Hand-centeredtarget location

-+

Direct Transformation

Buneo et al.(2002)

Sensory-motor transformations

Eye-centeredtarget location

Hand-centeredtarget location-

+

Relative position Transformations

Eye-centeredtarget location

Hand-centeredtarget location

-

+

Relative location of hand and eye

Eye-centeredhand location

Hand-centeredeye location

Relative position particularly useful for coordination

• Relative hand-eye position could be important for hand-eye coordination– Can map hand and eye onto each other– Computationally efficient

• No need to code for all combinations of locations

H

T

E

T-E

E-H

H-E

T-H

Relative position particularly useful for coordination

• Relative hand-eye position could be important for hand-eye coordination– Can map hand and eye onto each other– Computationally efficient

• No need to code for all combinations of locations

H

T

E

T-E

E-H

H-E

T-H

Relative position particularly useful for coordination

• Relative hand-eye position could be important for hand-eye coordination– Can map hand and eye onto each other– Computationally efficient

• No need to code for all combinations of locations

H

T

E

T-E

E-H

H-E

T-H