computational motor control: state space models for motor adaptation (jaist summer course)
TRANSCRIPT
![Page 1: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/1.jpg)
Computational Motor Control Summer School03: State space models for motor adaptation.
Hirokazu Tanaka
School of Information Science
Japan Institute of Science and Technology
![Page 2: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/2.jpg)
State-space modeling of motor adaptation.
In this lecture, we will learn:
• Motor adaptation paradigms
• Continuous-time state-space models
• Discrete-time state-space models
• Controllability
• Observability
• State-space description for motor adaptation
• Multi-rate models
• Motor memory of errors
• Mirror reversal (non-error based learning)
![Page 3: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/3.jpg)
Motor adaptation paradigms to dynamical perturbations: Force-field adaptation.
Shadmehr & Mussa-Ivaldi (1994) J Neurosci
Baseline (no field) Initial exposures
adaptation catch trials
![Page 4: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/4.jpg)
Motor adaptation paradigms to kinematical perturbations: Visuomotor rotation.
Krakauer et al. (2000) J Neurosc; Krakauer (2009) Progress in Motor Control
![Page 5: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/5.jpg)
Adaptation to prism displacements.
Martin et al. (1996) Brain ;Kitazawa et al. (1995) J Neurosci
![Page 6: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/6.jpg)
Adaptation to prism displacements.
Kitazawa et al. (1995) J Neurosci
1 1n n ne e ke 1
1
1
n
in
i
e e k e
![Page 7: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/7.jpg)
Continuous-time state-space models.
F ma mx x v
Fv am
x
v
x
Newton’s equation of dynamics
0 1 0
0 0 1/
x xF
v v m
x Ax Bu
x Ax Bu
State-space representation
A x B u
State-space vector
![Page 8: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/8.jpg)
Discrete-time state-space models.
Discrete-time representation
k k t x x
1 ( 1)k
k k
k k k
k k
k t
t
t
t t
x x
x x
x Ax Bu
I A x B u
1ˆ ˆ
k k k x Ax Bu
2
2ˆ
ˆ t
t
t t
te t
e
t
A
A
A I A
B BB
t
Δt 2Δt 3Δt (k-1)Δt kΔt (k+1)Δt0
k-1 k k+10 1 2 3
time (continuous)
time steps (discrete)
![Page 9: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/9.jpg)
Deterministic and stochastic state-space models.
1k k k
k k
x Ax Bu
z Cx
1
k
kk k
kk
k
w
v
x Ax Bu
z Cx
Deterministic Stochastic
XkXk-1 Xk+1
zk-1 zk zk+1
uk-1 uk uk+1
![Page 10: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/10.jpg)
Linear time-variant and time-invariant state-space models.
1k k k
k k
k k
k
x x u
xCz
A B
Time-variant model
1k k k
k k
x x u
xCz
A B
Time-invariant model
Throughout these lectures, we will use linear time-invariant (LTI) models for mathematical simplicity.
![Page 11: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/11.jpg)
State-space models in an explicit component form.
1k k k x Ax Bu
k kz Cx
1, 1
2, 1
, 1
k
k
N k
x
x
x
1,
2,
,
k
k
N k
x
x
x
11 12 1
21 21
11 1
N
N
a a a
a a
a a
11
21
1
1
L
N NL
b b
b
b b
1
L
u
u
= +
1,
2,
,
k
k
N k
x
x
x
1
M
z
z
11 12 1
1 112
N
MM
c c c
c c c
=
Process equation
Measurement equation
N vector N×N matrix N vector N×L matrix L vector
M vector M×N matrix N vector
![Page 12: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/12.jpg)
Controllability: the ability of driving a system into desired final state.
1k k k x Ax Bu
, , ,N L N N
k k
N L x u A B
Controllability is the ability of external inputs {uk} to drive a state from any initial condition to any final condition in a finite time. A state-space model is controllable if the N×NL controllability matrix has full row rank:
2 1n B AB A B A B
Sketch of proof:
0
1 1
2
2 2 1
1
1
0
2
N N N
N N
NN
N
N
N
x Ax Bu
A x ABu Bu
u
uA x B AB A B
uKalman (1963) SIAM J Contr
![Page 13: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/13.jpg)
Observability: determining hidden state from measurements.
k kz Cx
, ,N M N
k
M
k
x z C
Observability is the ability to determine a (latent) state from a sequence of measurements {zk}. A state space model is called observable if the MN×N observability has full rank N:
Kalman (1963) SIAM J Contr
1N
C
CA
C A
![Page 14: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/14.jpg)
State-space models for dynamic (force-field) motor adaptation.
Thoroughman & Shadmehr (2000) Nature; Donchin et al. (2003) J Neurosci
1n n n
n n n
x Ax Bu
z Cx Du
![Page 15: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/15.jpg)
State-space models for dynamic (force-field) motor adaptation.
Thoroughman & Shadmehr (2000) Nature; Donchin et al. (2003) J Neurosci
1n n n
n n n
x Ax Bu
z Cx Du
![Page 16: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/16.jpg)
State-space models for kinematic (visual rotation) motor adaptation.
Tanaka et al. (2006) J Neurophysiol
T
1
k k k k
k k k
z
z
x Ax BH
H x
![Page 17: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/17.jpg)
Trial-by-trial generalization width reflects directional tuning width.
1
1i i N
i
N
g
g
g
r r r r Rg
T
k k k rR g
1 1 1
T
k k k k k k k k R R g R g gr g
Suppose that, for target direction θ, the motor output is a weighted sum of population activity {gi(θ)} multiplied with preferred directions {ri}:
A gradient descent learning rule specifies the change of preferred directions according to the movement error Δrk and the population activity {g(θk)} :
This change affects the motor output at the next trial as:
![Page 18: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/18.jpg)
Two-rate model of motor adaptation: fast and slow learners.
Smith et al. (2006) PLoS Biol
1n n nu x Ax B
f ff f
1
s ss s
1
0
0
n n
n
n n
x xa bu
x xa b
n nz Cx
f
f s1
1 1s
1
1 1n
n n n
n
xz x x
x
f s s f,a a b b
There are two learners in the brain; the fast learner (x(f)) learns quickly but forgets quickly, while slow learner (x(s)) learns slowly but maintains its memory longer.
Motor output is a sum of the fast and slow learners.
State vector consists of fast (x(f)) and slow learners (x(s)).
![Page 19: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/19.jpg)
The model explains savings, spontaneous recovery.
Smith et al. (2006) PLoS Biol
Savings Spontaneous recovery
![Page 20: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/20.jpg)
The prediction of spontaneous recovery is confirmed in humans.
Smith et al. (2006) PLoS Biol
![Page 21: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/21.jpg)
The slow process contributes to motor memory consolidation.
Joiner & Smith (2008) J Neurophysiol
The slow process, but not the fast process, contributes to motor memory consolidation.
![Page 22: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/22.jpg)
Explicit (strategic) and implicit (error-based) learning.
Mazzoni & Krakauer (2006) J Neurosci
Strategy (aiming the adjacent target) cancels the “error” without any adaptation!
![Page 23: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/23.jpg)
Explicit (strategic) and implicit (error-based) learning.
Mazzoni & Krakauer (2006) J Neurosci
![Page 24: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/24.jpg)
Explicit (strategic) and implicit (error-based) learning.
Mazzoni & Krakauer (2006) J Neurosci
![Page 25: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/25.jpg)
What is “motor error?”: Aiming error and target error.
Taylor & Ivry (2011) PLoS Comp Biol; Taylor & Ivry (2014) Prog Brain Res
![Page 26: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/26.jpg)
State-space model for strategic and error-based learning.
Taylor & Ivry (2011) PLoS Comp Biol; Taylor & Ivry (2014) Prog Brain Res
yn: target directionrn: rotation anglexn: adaptation variablesn: strategy variable
yn
sn
sn-rn+xn
aiming
n n n nn n ne s s r x r x
target
n n n n n n nn ne y s r x y sx r
aiming
netarget
ne
![Page 27: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/27.jpg)
State-space model for strategic and error-based learning.
Taylor & Ivry (2011) PLoS Comp Biol; Taylor & Ivry (2014) Prog Brain Res
yn
sn
sn-rn+xn
aiming
n n n nn n ne s s r x r x
target
n n n n n n nn ne y s r x y sx r
aiming
netarget
ne
aiming
targ
1
e
1
t
n n n
nn n
x ax be
s cs de
a=0.99, b=0.015,
c=0.999, d=0.022
![Page 28: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/28.jpg)
Steepest descent learning rule for optimization.
Lecture 6, in Neural Networks for Machine Learning, Geoff Hinton
E( 1) ( )n n E
w w
w
optimum
E
w
E
w
Descent learning rule:
RPROP: Adjustment of learning rate.
E
wgradient
learning rate
1 1
![Page 29: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/29.jpg)
Motor memory of experienced errors.
Herzfeld et al. (2014) Science
( ) ( ) ( )
( 1) ( ) ( ) ( )
ˆ
ˆ ˆ
n n n
n n n n
e y y
x ax e
( )
( )
( )
( )
( )
: perturbation
ˆ : estimated perturbation ("belief")
: sensory consequence
ˆ : predicted sensory consequence
: control signal
n
n
n
n
n
x
x
y
y
u
State-space model: memory of environments
Population-coding model: memory of errors
( ) ( )n n
i i
i
w g e
2
2exp
2
i
i
e eg e
( 1)
( 1) ( 1) ( 1) ( )
T ( 1) ( 1)sgn
n
n n n n
n n
ee e
e e
gw w
g g error
acti
vity
w1 w2 w3 wn
η
![Page 30: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/30.jpg)
Motor memory of experienced errors.
Herzfeld et al. (2014) Science
![Page 31: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/31.jpg)
Displacement and left-right reversal: Why so different?
Martin et al. (1996) Brain; Sekiyama et al. (2000) Nature
Displacement prism
… takes only few dozen trials. … takes a few weeks.
Left-right reversed prism
Day 3
Day 34
![Page 32: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/32.jpg)
Mirror reversal: a distinct form of motor adaptation?
Taglen et al. (2014) J Neurosci; Lilicrap et al. (2013) Exp Brain Res
Movement number Movement number
Ab
solu
te e
rro
r
Ab
solu
te e
rro
r
Visual rotation Mirror reversal
![Page 33: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/33.jpg)
Summary
• A state-space model consists of a process equation (temporal transition) and an observation equation (measurement).
• Humans are flexible to a novel environment, known as motor adaptation, such as perturbations of force fields and visual transformation.
• State-space modeling has been very successful in describing trial-by-trial adaptation processes in humans.
![Page 34: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/34.jpg)
References
• Thoroughman, K. A., & Shadmehr, R. (2000). Learning of action through adaptive combination of motor primitives. Nature, 407(6805), 742-747.
• Donchin, O., Francis, J. T., & Shadmehr, R. (2003). Quantifying generalization from trial-by-trial behavior of adaptive systems that learn with basis functions: theory and experiments in human motor control. The Journal of Neuroscience, 23(27), 9032-9045.
• Tanaka, H., Sejnowski, T. J., & Krakauer, J. W. (2009). Adaptation to visuomotor rotation through interaction between posterior parietal and motor cortical areas. Journal of Neurophysiology, 102(5), 2921-2932.
• Smith, M. A., Ghazizadeh, A., & Shadmehr, R. (2006). Interacting adaptive processes with different timescales underlie short-term motor learning. PLoS Biol, 4(6), e179.
• Joiner, W. M., & Smith, M. A. (2008). Long-term retention explained by a model of short-term learning in the adaptive control of reaching. Journal of Neurophysiology, 100(5), 2948-2955.
• Inoue, M., Uchimura, M., Karibe, A., O'Shea, J., Rossetti, Y., & Kitazawa, S. (2015). Three timescales in prism adaptation. Journal of Neurophysiology, 113(1), 328-338.
• Mazzoni, P., & Krakauer, J. W. (2006). An implicit plan overrides an explicit strategy during visuomotor adaptation. The Journal of Neuroscience, 26(14), 3642-3645.
• Taylor, J. A., & Ivry, R. B. (2011). Flexible cognitive strategies during motor learning. PLoS Comput Biol, 7(3), e1001096-e1001096.
• Taylor, J. A., & Ivry, R. B. (2014). Cerebellar and prefrontal cortex contributions to adaptation, strategies, and reinforcementlearning. Progress in Brain Research, 210, 217.
• Taylor, J. A., Krakauer, J. W., & Ivry, R. B. (2014). Explicit and implicit contributions to learning in a sensorimotor adaptation task. The Journal of Neuroscience, 34(8), 3023-3032.
• Telgen, S., Parvin, D., & Diedrichsen, J. (2014). Telgen, S., Parvin, D., & Diedrichsen, J. (2014). Mirror reversal and visual rotation are learned and consolidated via separate mechanisms: Recalibrating or learning de novo?. The Journal of Neuroscience, 34(41), 13768-13779.?. The Journal of Neuroscience, 34(41), 13768-13779.
• Lillicrap, T. P., Moreno-Briseño, P., Diaz, R., Tweed, D. B., Troje, N. F., & Fernandez-Ruiz, J. (2013). Adapting to inversion of the visual field: a new twist on an old problem. Experimental Brain Research, 228(3), 327-339.
• Ostry, D. J., Darainy, M., Mattar, A. A., Wong, J., & Gribble, P. L. (2010). Somatosensory plasticity and motor learning. The Journal of Neuroscience, 30(15), 5384-5393.
![Page 35: Computational Motor Control: State Space Models for Motor Adaptation (JAIST summer course)](https://reader031.vdocuments.site/reader031/viewer/2022030313/58a784a81a28abef478b6009/html5/thumbnails/35.jpg)
Exercise
• Simulate the state-space model proposed by Taylor and Ivry.
• Mirror reversal is different from most adaptation paradigms in that learning from error worsens the performance. Can we consider a state-space model for mirror reversal?