motor adaptation and the timescales of memory reza shadmehr johns hopkins school of medicine ali...

Motor adaptation and the timescales of memory

Reza ShadmehrJohns Hopkins School of Medicine

Ali Ghazizadeh

Maurice Smith Konrad Koerding Haiyin Chen

Dave ZeeWilsaan Joiner

Jun Izawa

Tushar Rane

Duhamel et al. Science 255, 90-92 (1992)

The brain predicts the sensory consequences of motor commands

musclesMotor commandsforce

Body partState change

Sensory system

ProprioceptionVision

Audition

Measured sensory

consequences

Forward model

Predicted sensory consequences

Integration

Belief

What we sense depends on what we predicted

Wolpert et al. (1995)

5 10

5

Eye Position (deg)

Eye

Po

siti

on

(d

eg)

Saccade adaptation: gain decrease

McLaughlin 1967

5 10

5

Eye Position (deg)

Eye

Po

siti

on

(d

eg)

McLaughlin 1967

Saccade adaptation: gain decrease

Kojima et al. (2004) J Neurosci 24:7531.

_

Result 1: After changes in gain, monkeys exhibit recall despite behavioral evidence for washout.

+ +

Savings: when adaptation is followed by de-adaptation, motor system still exhibits recall

Saccade gain = Target displacement

Eye displacement

Result 2: Following changes in gain and a period of darkness, monkeys exhibit a “jump” in memory.

+ _ +

Offline learning: with passage of time and without explicit training, the motor system still appears to learn

Kojima et al. (2004) J Neurosci 24:7531.

( ) ( )( )1 2

( ) ( ) ( )

( 1) ( ) ( )1 11 1

( 1) ( ) ( )2 22 2

ˆ ˆ ˆ

ˆ

ˆ ˆ

ˆ ˆ

n nn

n n n

n n n

n n n

y y y

y y y

y a y b y

y a y b y

Adaptation as concurrent learning in multiple systems:A fast learning system that forgets quicklyA slow learning system that hardly forgets

Smith et al. PLOS Biology, 2006

prediction

Prediction error

Learning

Savings: de-adaptation may not erase adaptation

Task reversal periodre-adaptation

Trial number

Smith et al. PLOS Biology, 2006

Offline learning: Passage of time has asymmetric affects on the fast

and slow systems

Smith et al. PLOS Biology, 2006

Task reversal period

“dark” period

re-adaptation

Trial number

Slow stateFast state

-

( 1) ( ) ( )1 11 1

( 1) ( ) ( )2 22 2

ˆ ˆ

ˆ ˆ

n n n

n n n

y a y b y

y a y b y

Spontaneous recovery is also observed in reach adaptation

Trial number

Per

turb

atio

n

forc

e

Trial number

0

1

Per

form

ance

rel

ativ

e to

go

al

Task reversal period

Error clamp period

Smith et al. PLOS Biology 2006

Errors clamped to zero ( 1) ( ) ( )1 11 1

( 1) ( ) ( )2 22 2

ˆ ˆ

ˆ ˆ

n n n

n n n

y a y b y

y a y b y

1. Perturbations that can affect the motor plant have multiple time scales.Some perturbations are fast: muscles recover from fatigue quickly.Some perturbations are slow: recovery from disease may be slow.

2. Faster perturbations are more variable (have more noise).

3. The error that we observe is due to a contribution from all possible perturbations.

4. The problem of learning is one of credit assignment: when I observe an error, what is the time-scale of this perturbation?

The learner’s view about the cause of motor errors

( ) (1 1/ ) ( )disturbance t disturbance t

0, /N c

( ) ( )observation t disturbance t

Koerding, Tenenbaum, Shadmehr, unpublished

A

1w

2w

mw

t

Slow change

fast change

The Bayesian learner’s interpretation of motor error

y

xContext

perturbation y

x

1w

State of the variouspotential causes of error 2w

mw

tDisease state

Fatigue state

Savings: de-adaptation does not washout the

adapted system

Simulation

Koerding, Tenenbaum, Shadmehr, unpublished

Spontaneous recovery

Characteristics of long-term motor memoryData from Robinson et al. J Neurophysiol 2006

Bayesian Learner

Koerding, Tenenbaum, Shadmehr, unpublished

Motor system is disturbed by processes that have various timescale (fatigue vs. disease). Credit assignment of error depends on uncertainty regarding what is the timescale of the disturbance.

Prediction: When there are actions but the sensory consequences cannot be observed, states decay at various rates, but uncertainty grows. Increased uncertainty encourages learning.Bayesian learner

Adapting with uncertainty

Adapting with uncertainty: two predictions

Sensory deprivation Faster subsequent rate of learning.

Example: A subject that spends a bit of time in the dark will subsequently learn faster than a subject that spends that time with the lights on.

Why: In the dark, uncertainty about state of the motor system increases.

Longer inter-stimulus interval Better retention.

Example: A subject that trains on n trials with long ITI will show less forgetting than one that trains on the same n trials with short ITI.

Why: events that take place spaced in time will be interpreted as having a long timescale.

Ali Ghazizadeh Maurice Smith

Konrad Koerding

Fast and slow adaptive processes arose because disturbances to the motor system have various timescales (fatigue vs. disease). When faced with error, the brain faces a credit assignment problem: what is the timescale of the disturbance? To solve this problem, the brain likely keeps a measure of uncertainty about the timescales.

A prediction error causes changes in multiple adaptive systems. Some are highly responsive to error, but rapidly forget. Others are poorly responsive to error but have high retention. This explains savings and spontaneous recovery.

Summary

1. Internal models are supposed to help us control our movements in real-time. What are these fast and slow systems learning and how does that learning affect real-time control of movements?

2. Can we say anything about the neural structures that might be responsible for computing internal models?

What are some of the holes in these ideas?

Body +environment

State change

Sensory system

ProprioceptionVision

Audition

Measured sensory consequences

Forward model

Predicted sensory consequences

Integration

Belief about state of body

and world

Goalselector

Motor commandgenerator

Emo Todorov: Motor command generator as an optimal controller

( 1) ( ) ( ) ( )

( ) ( ) ( )

1(0) ( ) ( ) ( ) ( 1) ( 1) ( 1)

0

( ) ( ) ( )

( 1) ( ) ( ) ( ) ( ) ( )

ˆ

ˆ ˆ ˆ

k k k ku

k k ky

pk T k k k T k k

k

k k k

k k k k k k

A C

B

J L T

G

A AK B C

x x u ε

y x ε

u u y y

u x

x x y x u

Signal dependent motor noise

Signal dependent sensory noise

Actual state of the system(eye state, target state, etc.)

What we can observe about the state of the system

Motor command generator as a stochastic optimal controller

Todorov (2005)

Cost to minimize

Feedback control policy

Body +environment

State change

Sensory systemMeasured sensory

consequences

Forward modelPredicted sensory consequencesIntegration

Belief about state of body

and world

Goalspecification

Motor commandgenerator

Belief about state

eye velocity

deg

/sec

0 0.05 0.1 0.15 0.2 0.25

0

100

200

300

400

500

Time (sec)

Body +environment

State change

Sensory systemMeasured sensory

consequences

Forward modelPredicted sensory consequencesIntegration

Belief about state of body

and world

Goalspecification

Motor commandgenerator

5 10 15 30 40 50 Saccade size

The mathematical framework allows one to produce detailed trajectory of movements.

In the target jump paradigm, error is a difference between predicted and actual sensory consequences of oculomotor commands.

Therefore, the forward model must adapt.

But if that adaptation is not precisely matched by the motor command generator, the result will be sub-optimal saccades.

Prediction error

The direct and indirect output pathways from the superior colliculus (SC)

• Direct pathway

SCbrainstem

• Indirect pathway

SCcerebellumbrainstem

Cross-axis saccade adaptation

Equal rates of learning in the controller and the forward model

saccades remain straight

Learning in the forward model only

saccades become curved

Body +environment

State change

Forward modelPredicted sensory consequences

Belief about state of body

and world

Goalspecification

Motor commandgenerator

Incre

ased

tra

inin

g

T1

T2

fixation

Cross-axis saccade adaptation: Experiment design

(In complete darkness, with search coil lenses on the eyes)

Chen, Joiner, Zee, Shadmehr (unpublished)

Characteristics of primary saccades during adaptation

T1

T2

15o5o

Chen, Joiner, Zee, Shadmehr (unpublished)

Curvature of primary saccades quantified through chord slopes

Chen, Joiner, Zee, Shadmehr (unpublished)

The observation that saccades become curved, and therefore sub-optimal, is a reflection of a neural system that adaptively computes sensory consequences of motor commands, and corrects the motor commands as they are produced.

The forward model (indirect pathway) appears to adapt much more quickly than the controller (direct pathway).

Saccade curvature suggests that errors cause rapid adaptation in the forward

model

Body +environment

State change

Sensory system

Forward modelPredicted sensory consequencesIntegration

Belief about state of body

and world

Goalspecification

Motor commandgenerator

Prediction error

Haiyin Chen

Dave Zee

In saccades and reaching, performance is guided by internal models that adapt at multiple timescales:

A fast learning system that has poor retention.

A slow learning system that hardly forgets.

The observation that saccades become curved, and therefore sub-optimal, is a reflection of a neural system that adaptively computes sensory consequences of motor commands, and corrects the motor commands as they are produced.

The forward model (indirect pathway) appears to adapt much more quickly than the controller (direct pathway).

Summary:

Wilsaan Joiner

1. If learning of forward models (indirect pathway) is faster than the controller (direct pathway), the result is a sub-optimal system. Most of our movements appear optimal. What guides learning in the direct pathway so that we eventually become optimal?

2. If we learn as a Bayesian, we keep a measure of uncertainty about what we know. Does the uncertainty in the internal model affect our control policies (direct pathway)?

What are some of the holes in these ideas?

Raymond Clarence Ewry (USA)Gold Medal, 1908 Olympics

Cornelius Johnson (USA)Gold Medal, 1936 Olympics Dick Fosbury (USA)

Gold Medal, 1968 Olympics

Body +environment

State change

Sensory system

Forward modelPredicted sensory consequences

Integration

Belief about state of body and world

Goalspecification

Motor commandgenerator (control policy)

Prediction error

Learning in the direct pathway:

finding a better control policy in the high jump task

N=6

The optimal control policy

To maximize probability of arriving at target in time, I should minimize my motor commands near the end of the movement.

Over compensate for the forces early, let the robot bring you back.

Predicted trajectories under the optimal control policy

Accuracy of model

Izaw

a,

Ran

e,

Don

ch

in,

Sh

ad

meh

r (u

np

ub

lish

ed

)

Null field

Izawa, Rane, Donchin, Shadmehr (unpublished)

In performing an action, the motor commands that we generate should depend on our confidence (uncertainty) in

our models.

( )

( )

0

0

n

n

Bf x

B

9cm

FP

Screen

( 1) ( ) ( ) ( )

( ) ( ) ( )

1(0) ( ) ( ) ( ) ( 1) ( 1) ( 1)

0

( ) ( ) ( )ˆ

k k k ku

k k ky

pk T k k k T k k

k

k k k

A C

B

J L T

G

x x u ε

y x ε

u u y y

u x

Stochastic optimal control with model uncertainty

Jun Izawa

Tushar Rane

Traditional stochastic optimal control

( 1) ( ) ( ) ( )ˆ

ˆ ,

k k k kuA C

A N M V

x x u ε

Izawa, Rane, Donchin, Shadmehr (unpublished work)

Stochastic optimal control with model uncertainty: Predictions

Izawa, Rane, Donchin, Shadmehr (unpublished work)

People learn policies that depend on their model uncertainty:

Overcompensate only if you are certain of the world

N=6

High certainty

Low certainty

High certainty

Low certainty

Jun Izawa

Motor control is about solving two distinct problems:

Learning a control policy (direct pathway).Learning a forward model (indirect

pathway).

Motor learning is at multiple timescales:A fast learning system that has poor

retention.A slow learning system that hardly forgets.

The forward model (indirect pathway) adapts much more quickly than the controller (direct pathway).

Overview: Computational problem of motor control

Maurice Smith

Haiyin Chen

1. In saccade adaptation, nothing happened to the body; it was the target that was behaving strangely. When there is error, how does the brain distinguish between changes in the body vs. changes in the world? This is a second credit assignment problem.

2. What is the error signal that guides learning of control policies?

3. Are the direct and indirect pathways computational pathways or neural pathways?

What are some of the holes in these ideas?

thalamus

Motor cortex

Deep cerebellar nuclei

Pons

DBS: deep brain stimulation

Inf. Olive

Reversible disruption of cerebellar pathways in humans

Cerebellar cortex

Co

rtic

osp

inal

tra

ct

Sherwin Hua

Deep Brain Stimulation

1.5 mm electrode is implanted in the thalamus and connected via subcutaneous wires to a stimulator.

The subcutaneous stimulator and battery.

Parameter settings can be adjusted via an external device.

Fred Lenz

Stimulation of VL thalamus improves tremor but impairs adaptation

Ch

en

et

al.

Cere

bra

l C

ort

ex,

2006

Stimulation voltage

Bipolar stimUnipolar stim

Tremor during reaching

Movement onset

Thoroughman & Shadmehr, J Neurosci, 1999

EMG patterns during reach adaptation

Neural correlates of motor learning in the VL thalamus

Adaptation level was low

Behavioral performance

• Sites attempted recording ……………….. 105• Sites successfully recorded units ………. 58 (55%)• Units with more than 60 trials …………… 61

–Vim………………….35–Vim-Vop border……12–Voa/Vop…………… 14

• Single units ……………………………….. 16 (26%)• Movement related units …………………. 36 (59%)

–Vim………………….21–Vim-Vop border……5–Voa/Vop……………10

• Units showed direction selectivity ………. 18 (50%)–Vim………………….11–Vim-Vop border……1–Voa/Vop…………….6

Recording sites and neural responses

target Vmax stop hold/wait

Adaptation induces change in firing pattern before movement onset

1. The cerebellum appears to be a critical structure for motor adaptation. Is this the place where forward models are formed?

2. Speculation: cerebellar cortex may represent the “fast system”, with the cerebellar nuclei representing the “slow system”. Prediction: cerebellar patients may learn slowly, but they will also forget slowly.

3. Learning control policies depends on reward prediction errors.Is the basal ganglia the structure crucial for learning control policies?

4. Challenge ahead: To look for behavior and neural signatures of control policies and forward models in healthy individuals and patients with motor disorders.

Conclusion and speculations

The neural basis of motor adaptation

Cerebellar degeneration impaired adaptation of reaching

Huntington’s disease (HD) patients showed no deficit in adaptation

Smith and Shadmehr, J Neurophysiology 2005

early null3

Visual rotation adaptation