adaptive stimulation design for the treatment of epilepsy joelle pineau school of computer science,...
TRANSCRIPT
Adaptive Stimulation Design for the Treatment of Epilepsy
Joelle PineauSchool of Computer Science, McGill University
Montreal, QC CANADA
Jointly with Robert Vincent, Aaron Courville, Massimo Avoli
SAMSI Program on Challenges in Dynamic Treatment Regimes and Multistage Decision-Making
June 21, 2007
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Background
Vagus nerve and deep brain stimulation are used to treat various neurological disorders, including epilepsy.
Images from www.epilepsyfoundation.org and www.cyberonics.com
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Project goal
• Problem: Existing devices offer limited control and do not adapt to the patient’s condition.
• Idea: Create an improved class of devices with closed-loop control.
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
A reinforcement formulation
• Objective: Minimize occurrence of seizures and overall amount of stimulation.
• The MDP model:
– States, st : recordings of electrical activity
– Actions, at : stimulation (frequency, voltage, location)
– Transitions, P(st|st-1, at) : unknown
– Rewards, rt : large cost for seizures, small cost for stimulation
st-1
at at+1
…
rt-1
st
rt
st+1
rt+1
at-1
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Technical challenges
1. Investigate supervised learning methods for automatic seizure detection
to inform choice of good state representation.
2. Design a computational (generative) model of epilepsy.
3. Run reinforcement learning using online data from the computational
model.
4. Run reinforcement learning using batch data from an in-vitro model.
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Recordings of electrical activity
• Recorded from single sensing electrode in in-vitro model of epilepsy.
• Raw data: 4096-sample frames windowed, normalized, FFT
• xt = 83 real-valued features: mean, range, energy, 80 FFT magnitudes
• yt ={normal, spike, seizure} hand-labeled for each frame.
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Seizure detection
• Long literature on seizure detection using supervised learning (e.g. RBFs, wavelets, neural networks, energy methods).
» Usually uninterpretable results.
• Related literature in time-series prediction (e.g. HMMs, CRFs).
» Requires feature selection.
• Instead, focus on ensemble boosting methods:
– Extend standard multi-class Adaboost [Schapire&Singer, 1999] to a
recurrent formulation, in which yt = f (xt , yt-1 , yt-2 , …, yt-k).
» Relatively interpretable results (assuming simple learners).
» Requires no prior model of the distribution of features.
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Classification accuracy
yt = f (xt , yt-1 )
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Feature extraction = insight into state design
• In all recurrent examples, first weak hypothesis recruited was frequency band 62 or 63 (= 76-77 Hz).
– High value normal
– Low value spike
• Frequency bands 6-8 (=7-10 Hz) also consistently recruited early.
– High value spike
– Low value normal
• Often recruited in the first 20 rounds is Energy.
– High value spike
• In recurrent Adaboost, prior label often recruited early and acts as memory.
– High spike prior spike
etc.
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Technical challenges
1. Investigate supervised learning methods for automatic seizure detection
to inform choice of good state representation.
2. Design a computational (generative) model of epilepsy.
3. Run reinforcement learning using online data from the computational
model.
4. Run reinforcement learning using batch data from an in-vitro model.
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
A computational model of epilepsy
• Aims of computational modeling:
– To understand basic mechanisms of epilepsy sufficiently to design
a good state representation.
– To understand the appropriate class of policies to consider.
– To have an inexpensive testing environment for RL algorithms.
• A word of caution:
– Epilepsy is a complex disease. Many animal models are found in
the literature. We focus on model by [Avoli et al., 2002], which
we will use for online exploration in-vitro.
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Model overview
(a) Assume a simple stochastic neuron model (leaky integrate-and-fire).
(b) Connect many neurons in small-world network configuration.
(a) Leaky integrate-and-fire neuron model (b) Network structure
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Model overview (cont’d)
(c) Define a sensor model:
– Voltage measurement over a patch of adjacent neurons (neuron
contribution falls off in inverse-square relationship with distance
from the patch center).
(d) Define the stimulation model:
– Input current applied uniformely to a “patch” of neural units.
istim
vsensor
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Traces from the computational model
# neurons
firing
sensor
voltage
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Results from fixed stimulation strategies
0 Hz
0.5 Hz
1.0 Hz
2.0 Hz
4.0 Hz
5.0 Hz
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Technical challenges
1. Investigate supervised learning methods for automatic seizure detection
to inform choice of good state representation.
2. Design a computational (generative) model of epilepsy.
3. Run reinforcement learning using online data from the computational
model.
4. Run reinforcement learning using batch data from an in-vitro model.
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Reinforcement learning agent
• States:
st = {sensor reading} x {# time steps since last stimulation}
discretized into small number of (independent) features
• Actions:
a1 = Stimulator on for 1 time-step.
a2 = Stimulator off for 1 time-step.
• Rewards:
R(s,a) = -100 for a seizure (i.e. firing count > 50)
R(s,a) = -10 for a stimulation
R(s,a) = 0 otherwise
• Training details:» Online data, finite horizon (60 x 100-sec traces), -greedy exploration, Sarsa()
with eligibility
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Results: Learning
(a) After 1 learning episode (b) After 10 learning episodes
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Results: Expected return
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Results: Policy
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Technical challenges
1. Investigate supervised learning methods for automatic seizure detection
to inform choice of good state representation.
2. Design a computational (generative) model of epilepsy.
3. Run reinforcement learning using online data from the computational
model.
4. Run reinforcement learning using batch data from an in-vitro model.
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Learning from batch in-vitro data
We have data of the type used in the classification task, but which
includes fixed-policy stimulation.
• States: 83 real-valued features (electrical signal sampled + FFT)
• Actions: {0 Hz, 0.5 Hz, 1.0 Hz, 2.0 Hz, 5.0 Hz}
• Training: batch (~30 x 60-sec traces), tree-based regression.
No results
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Wrap-up
• Preliminary evidence for:
– ability to detect seizures using in-vitro data.
– generation of synthetic data with epileptiform behavior.
– controllability of the stimulator in the computational model.
• Results apply to a specific model of epilepsy - generalization to other models is unknown.
• Issues I did not discuss today:
– Designing the reward function.
– Using the features found in Adaboost within the RL agent.
– Transferring what we have learned in the computational model to the biological model.
– Learning from very few data points.
SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy
Available data
• Recordings used for classification (no stimulation)
– http://www.cs.mcgill.ca/~jpineau/datasets/epilepsy.tar.gz
• Recordings used for batch reinforcement learning (fixed policy stimulation)
– Subject to approval by neuroscientists (1-2 weeks delay).
• Generative model of epilepsy
– Still under development, but publicly available soon (1-2 months).