mind reading: neuroscience methods and technology
TRANSCRIPT
Mind Reading: Neuroscience Methods
and Technology
David J. Heeger
New York University(http://www.cns.nyu.edu/~david/)
Classifying spatial patterns of brain activity with machine learningmethods: Application to lie detection
C. Davatzikos,a,* K. Ruparel,b Y. Fan,a D.G. Shen,a M. Acharyya,a J.W. Loughead,b
R.C. Gur,b and D.D. Langlebenb,c
aDepartment of Radiology, University of Pennsylvania, 3600 Market Street, Suite 380, Philadelphia, PA 19104, USAbDepartment of Psychiatry, University of Pennsylvania, 3400 Spruce Street, Philadelphia, PA 19104, USAcTreatment Research Center, University of Pennsylvania, 3900 Chestnut Street, Philadelphia, PA 19104, USA
Received 1 March 2005; revised 15 June 2005; accepted 4 August 2005
Available online 5 October 2005
Patterns of brain activity during deception have recently been
characterized with fMRI on the multi-subject average group level.
The clinical value of fMRI in lie detection will be determined by the
ability to detect deception in individual subjects, rather than group
averages. High-dimensional non-linear pattern classification methods
applied to functional magnetic resonance (fMRI) images were used to
discriminate between the spatial patterns of brain activity associated
with lie and truth. In 22 participants performing a forced-choice
deception task, 99% of the true and false responses were discriminated
correctly. Predictive accuracy, assessed by cross-validation in partic-
ipants not included in training, was 88%. The results demonstrate the
potential of non-linear machine learning techniques in lie detection and
other possible clinical applications of fMRI in individual subjects, and
indicate that accurate clinical tests could be based on measurements of
brain function with fMRI.
D 2005 Elsevier Inc. All rights reserved.
Introduction
A large body of functional neuroimaging literature has
elucidated relationships between structure and function, as wellas functional activity patterns during a variety of functionalactivation paradigms. Statistical parametric mapping (SPM)
(Friston et al., 1995) has played a fundamental role in thesestudies, by departing from the conventional biased ROI- andhypothesis-based methods of data analysis and enabling unbiased
voxel-by-voxel examination of all brain regions. While a great dealof knowledge has been gained during the past decade regardingbrain regions that are activated during various tasks using voxel-
based SPM analysis, the quantitative characterization of entirespatio-temporal patterns of brain activity, as opposed to voxel byvoxel examination, has received much less attention, especially asa means for deducing ‘‘the state of the mind’’ from functional
imaging data. The important distinction between a voxel-basedanalysis and the analysis of a spatio-temporal pattern is the same asthe distinction between (mass) uni-variate and multi-variate
analysis (Davatzikos, 2004). Specifically, a pattern of brain activityis not only a collection of active voxels, but carries with itcorrelations among different voxels. Notable efforts towards thefunctional activity pattern analysis have been made (Strother et al.,
1995; McIntosh et al., 1996), some of which, attempt to use thesemethods to classify complex activation patterns using machinelearning methods (Cox and Savoy, 2003; LaConte et al., 2005).
In this paper, we present an approach to the problem ofidentifying patterns of functional activity, by using a high-dimensional non-linear pattern classification method. We apply
this approach to one of the long-standing challenges in appliedpsychophysiology, namely lie detection. Deception is a sociallyand legally important behavior. The limitations of the specificityof the currently available physiological methods of lie detection
prompted the exploration of alternative methods based on thecorrelates of the central nervous system activity, such as EEG andfMRI (Rosenfeld, 2001; Spence et al., 2001; Langleben et al.,
2002). Using SPM-based analyses of multi-subject average groupdata, several recent fMRI studies demonstrated differences inbrain activation between truthful and non-truthful responses in
various experimental paradigms (Langleben et al., 2002; Langle-ben et al., in press; Ganis et al., 2003; Kozel et al., 2004a,b; Leeet al., 2002). In order to translate these data into a clinically
relevant application, discrimination between lie and truth has tobe achieved at the level of single participants and single trials(Kozel et al., 2004b), not just via group analysis. The potential ofthe SPM-based approach to achieve this goal is limited due to the
between-subject variability of regional brain activity. In thecurrent work, we have overcome this limitation using a multi-variate non-linear high-dimensionality pattern classification tech-
nique (Lao et al., 2004) applied to spatial patterns of brainactivation recorded via fMRI. Using data acquired with apreviously reported formal deception paradigm (Langleben et
al., in press), we have tested the hypothesis that truthful and non-
1053-8119/$ - see front matter D 2005 Elsevier Inc. All rights reserved.
doi:10.1016/j.neuroimage.2005.08.009
* Corresponding author. Fax: +1 215 614 0266.
E-mail address: [email protected] (C. Davatzikos).
Available online on ScienceDirect (www.sciencedirect.com).
www.elsevier.com/locate/ynimg
NeuroImage 28 (2005) 663 – 668
•Neuroscience-based lie detector technology
•Commercial venture (http://www.noliemri.com/)
•Application of functional magnetic resonance imaging (fMRI)
•Claim 88% accuracy
No Lie MRI, Inc.
• This is one of a series of recent brain imaging studies on lie detection, the most impressive to date.• The technique that they use relies on sophisticated statistical analysis of brain imaging measurements.• They conclude 88% accuracy (90% sensitivity, 86% specificity). This corresponds to 14% false alarm rate and 10% miss rate. Put another way, about 1 person
in 7 will be incorrectly identified as lying. • This approach is very different from, and potentially much more powerful than, traditional polygraph. Polygraph measures physiological responses that
indirectly reflect the activity of the autonomic nervous system: pulse, blood pressure, sweat (electrical conductance of the skin). The autonomic nervous system regulates essential bodily functions (heart, breathing, digestion). This happens automatically and subconsciously. Changes in the polygraph measures can reflect arousal during deception but also general anxiety. fMRI lie detection and EEG "brain fingerprinting" record neural activity in the brain including neural activity that is related to conscious mental state. This is very different from the polygraph which is an indirect physiological measures of automatic and unconscious regulation of bodily function.
Functional magnetic resonance imaging
Revolution in psychology and neuroscience: ~1000 papers published per month over the past 5 years!
• fMRI has revolutionized neuroscience over the past decade.• It is like regular MRI that is used clinically to take pictures of the anatomy of the body, but is used to take pictures of brain function in addition to brain anatomy.• New era of research into the function and dysfunction of the human brain.• fMRI bridges the gap between brain and behavior. Most of what we know about human behavior (perception, cognition, emotion) originates with experimental
psychology. Most of what we know about how the brain works comes from neurobiology.
Briefly, here’s how it works:1) The brain controls the flow of oxygenated blood to where it is needed (discovered over 100 years ago).2) There is iron in blood which is a magnetic metal. Oxygenated and deoxygenated blood have different magnetic properties (discovered by Linas Pauling in the 1930s).3) MRI was invented in the 1970s, based on the physics of magnetic resonance which was discovered in the 1940s.4) The MRI scanner can be reprogrammed to pick up differences in magnetization that take place when the brain ships oxygenated blood to where it is needed.
Setting up an fMRI facility
•Vendors: Siemens, GE, Phillips
•Capital investment (scanner plus renovation): $2-3 million
•Annual operating expenses (staff plus service contract): $250,000-$500,000
Mobile MRI
Genesis Medical Imaging, Inc.(http://www.genesismedicalimaging.com/)
PerceptionAttentionMemoryLanguageEmotionPersonalityDecision makingArts & aestheticsMoral reasoningSocial interaction
It’s all in the brain
Conjecture: The human mind – perceptions, emotions, memories, thoughts – can be completely explained by the electrical and chemical activity of the brain.
Neuroscience research has broad ramifications, not only in biomedical science but for our society as a whole. It’s all in the brain - everything about human behavior, human nature, and human society is controlled by the human brain. Neuroscientists have focused in the past on a handful of basic, core topics (perception, memory, emotion). But we are now beginning new interdisciplinary collaborations to explore, from a neuroscientific point of view, topics that used to be outside the purview of science.
If this conjecture is correct, then it will be possible with neuroscience methods like fMRI to measure any and all aspects of an individual’s mental state (conscious and unconscious). The current techniques for non-invasive measurement of human brain function are limited but both the techniques themselves and our ability to use them effectively will continue to improve as we understand more about brain function.
Outline
•Neurosci 101 (how the brain works)
•How neurons work
•A little bit of neuroanatomy
•Functional specialization & computational theory
•Case study: visual motion perception
•Techniques for measuring human brain activity
•Some examples of mind reading with fMRI
To understand how these neuroscience techniques and applications (like neuroscience based lie detection) work and what their limitations are, we need to take a step back and learn some of the basics of neuroscience.
Outline
•Neurosci 101 (how the brain works)
•How neurons work
•A little bit of neuroanatomy
•Functional specialization & computational theory
•Case study: visual motion perception
•Techniques for measuring human brain activity
•Some examples of mind reading with fMRI
Schematic of a neuron
• Dendrites: Inputs from other neurons arrive here, they are combined in various ways (input signals might be added, subtracted, divided by one another).• Axon: Result of this computation propagates along the axon to serve as inputs to other neurons. Some axons are very short (local connections between
neurons in one part of the brain). Some are very long (more than a meter) down the spinal cord to innervate a muscle.• Soma: cell body (in between dendrites and axon)
Cortical pyramidal cell (Golgi stain)
amplifiermicroelectrode
Time (msec)V
olta
ge (
mV
)
Electrophysiology (action potential)
Neurons represent and transmit information electrically. If you place an electrode close to a neuron's soma then you can measure and record these electrical signals. When stimulated, a neuron fires an action potential during which the voltage jumps way up, then comes back down again. It all happens very fast (note the time scale), all in a matter of a few thousandths of a second. The action potential starts at a neuron's soma and travels all the way along its axon. The shape of the action potential is basically the same anywhere along the length of the axon. Hodgkin and Huxley won the Nobel prize for work they did in the 1950s to understand what makes action potentials happen and how they propagate along axons. Action potentials are the “currency” of the brain.
Firing rate: visual neurons and
contrast
V1
Firing rate: the number of action potentials that occur per unit of time. For example, the responses of neurons in visual areas of the brain depend on the contrast of the test light. For a low contrast we would get only a few action potentials. For a high contrast, we would get many more action potentials. Any given action potential looks exactly like all the others. When we increase the contrast, the individual action potentials do not get bigger.
This graph plots the average firing rate (averaged across a large number of individual neurons) in response to series of stimuli with different stimulus contrasts.
The graph also shows that fMRI measurements of human brain activity increase in the same way with contrast. This is important because it shows that the fMRI measurements are tightly linked with the underlying firing rates of the neurons.
Neurons in MT are selective
for motion direction
Newsome et al.
How is it that we know that a particular neural signal is due to a stimulus that originated as a sound, versus knowing that another neural signal arose from some other type of physical stimulus, such as a light source? Suppose that we initiate a neural response in our visual system by electrically stimulating neurons in your eye rather than by absorbing some light. When you excite the visual neurons in any way whatsoever, the response that is evoked in the nervous system is that of a visual response. This holds true no matter what the physical cause of the excitation. The determining factor is which neuron is being excited. Likewise, no matter how it is you excite an auditory neuron, the resulting sensation will be one of hearing (cochlear implants, for example, work by direct electrical stimulation of the auditory nerve).
Neurons in different parts of the brain are selective for different things. Just keeping to neurons involved with vision, there are neurons that are selective for all kinds of properties: stimulus position, color, pattern, direction and speed of motion. There are even neurons that respond selectively to faces.
Visual area MT is one of the most studied regions of the brain. MT neurons are velocity selective; each responds best to a preferred velocity (speed and direction) of motion within particular spatial location, pretty much independent of stimulus pattern, shape, or color. Motion is the only thing that matters. This video shows a sequence of moving stimuli. A monkey watched these stimuli while neuroscientists recorded the electrical activity of a single neuron in area MT. The electrical signal from the neuron was amplified and recorded as the sound track. The popping sounds are action potentials. The neuron responded most strongly to one direction (down-left) and less to other directions (least to up-right).
Neural communication
How do neurons communicate? The axon of one neuron comes very close to (about 20 nanometers; a nanometer is a billionth of a meter) but doesn't quite touch the dendrite of its target. The narrow space between the two neurons is called a synapse.
Synapse: electron microscope picture
Highly magnified slice through the brain showing the axon terminal, dendrite, and synapse. The little circles in the axon terminal are neurotransmitter vesicles.
• Postsynaptic dendrite
• Direction of action potential
• Presynaptic axon
• Synapse
Synaptic transmission
Neurotransmitter vesicle
The electrical signal from the (pre-synaptic neuron's) axon is not transmitted directly across the synapse. Rather, when the action potential comes along, it causes the release of certain chemicals called neurotransmitters. The neurotransmitters diffuse across the synapse and bind to receptor molecules in the membrane of the (postsynaptic neuron's) dendrite. This, in turn, causes a change in the electrical properties of the postsynaptic neuron. If the postsynaptic neuron is stimulated strongly enough, then it too will fire action potentials. Bernard Katz won a Nobel prize for figuring out how synapses work.
There are a variety of different neurotransmitter molecules including: dopamine, acetylcholine, serotonin, GABA, glutamate. Some neurotransmitters are excitatory, causing the postsynaptic neuron to respond. Some are inhibitory, shutting down the responses. A single neuron receives many syanpses, some excitatory and some inhibitory. The combinations of excitatory and inhibitory inputs are what allow the neuron to perform computations. Some neurotransmitters act very quickly (just a few msec) and some more slowly so that neurons can combine information over short or long time intervals.
Outline
•Neurosci 101 (how the brain works)
•How neurons work
•A little bit of neuroanatomy
•Functional specialization & computational theory
•Case study: visual motion perception
•Techniques for measuring human brain activity
•Some examples of mind reading with fMRI
Frontallobe Parietal
lobe
Occipitallobe
Temporallobe
Cerebral cortex
Lobes: occipital in back, frontal in front, parietal on top, temporal on side.Sulcus: infoldingGyrus: outfolding
Each major sulcus and gyrus has a name (e.g., inferior temporal sulcus or inferior frontal gyrus). These anatomical landmarks are identifiable in every human brain and, as a general rule, the function and physiology of the neurons in a particular region (e.g., posterior end of the inferior temporal sulcus) will be the same in every human brain. More on this later.
cerebellum
cerebral cortex corpus collosum
brain stem
thalamus
optic nerve
cerebral cortex
Medial view
Gray matter (stained purple): folded sheet containingcell bodies, dendrites, local axons collaterals.
White matter: axons, long range connections.
Gray matter and white matter
Right: Slab of cortical tissue removed from the back of a monkey brain.
Left: This tissue is stained so that the cell bodies appear purple. All the cell bodies are in a layer near the surface of the brain, called the gray matter of the brain because it has grayish color when its not stained. The longer axons reach out of the gray matter into the central part of the brain, called the white matter of the brain because the axons altogether look white. The white matter is like a massive tangle of wires - axons connecting from one part of the cortex to another.
Gray matter is about 3-4 mm thick, filled with cell bodies (somas), dendrites, and axon terminals (synapsing on the dendrites). Gray matter is where all the interesting stuff happens, where a neuron receives inputs (via synapses) from a whole bunch of other neurons, and then computes a new output. The axons in the white matter just transmit that output to other parts of the brain.
~50,000 neurons per cubic mm~6,000 synapses per neuron~10 billion neurons & ~60 trillion synapses in cortex
Neural circuits perform computations
Neurons perform computations. For example, a given neuron’s response might depend on the sum of the responses from two of its inputs. The function of a neuron in a particular brain area are best understood in terms of the computations that they perform. There are many different types of neurons in each bit of brain tissue, connected together to form circuits. It is not unlike a computer, but neural circuits are extremely complex.
How can we possibly hope to understand such a complex process? There are two guiding principles.
Outline
•Neurosci 101 (how the brain works)
•How neurons work
•A little bit of neuroanatomy
•Functional specialization & computational
theory
•Case study: visual motion perception
•Techniques for measuring human brain activity
•Some examples of mind reading with fMRI
Two guiding principles
Functional specialization
Computational theory
Functional specialization: primary
sensory areas
Functional specialization: The brain is subdivided into a number of separate and distinct brain areas, each of which performs a different function. Particular brain areas are specialized for sensation and perception: vision, touch, olfaction, hearing.
Visual maps in the brain
V2V3
V3A/B
V7
IPS1
IPS2
V4
MT+LO2LO1
Each visual brain area contains a map of the
visual world and performs a different function.
Within each sensory modality, there is a further subdivision. The visual cortex, for example, is subdivided into a number of distinct visual cortical areas. Each visual brain area contains a map of the visual world. The neurons at one location in the brain respond to visual features at one location in the world. The neurons at a neighboring location respond to neighboring features. We routinely measure these visual maps in the brian using fMRI to identify each of the visual brain areas. In about an hour, I could measure your visual maps and identify each of your visual brain areas.
Each visual brain area performs a specific function. Some examples follow.
Functional specialization:shape perception
LO
Malach et al.
This video demonstrates that visual area LO is functionally specialized for the perception of shape.
Functional specialization:motion perception
MT
This video demonstrates that visual area MT is functionally specialized for motion perception.
How do neurons compute motion?
This video illustrates a simplified theory of how a neuron can compute visual motion. The top two ovals represent photoreceptors in the retina of the eye (specialized neurons that convert light into electrical neural responses). The photoreceptor on the left send its action potentials to another neuron that delays the transmission before reaching the output neuron at the bottom (the circle), whereas the photoreceptor on the right sends its action potentials directly without delay. When a spot of light moves at a particular speed from left to right, the delayed signal from the first photoreceptor and the non-delayed signal from the second photoreceptor both reach the output neuron (circle) at the same time. The output neuron itself responds if, and only if, it receives coincident inputs.
This is oversimplified, but illustrates the basic idea. A full-blown theory of MT physiology includes a series of neural computations along a well-defined pathway of neural connections, beginning with a particular class of neurons in the retina, and a specialized subclass of neurons in primary visual cortex, before reaching MT.
optic nerve
LGN
primary visual cortex (V1)
Functional specialized brain areas are
connected in pathways and networks
Ventral view showing visual pathways from the eye to the back of the brain.
Lateral view showing visual pathways for recognition and visually-guided action.
Action
Recognition
Neural computations are performed by local circuits within a brain area (e.g., MT) and then passed on to other brain areas. The brain areas are interconnected in well-defined pathways and networks.
The diagram on the left shows the pathway from the eye to primary visual cortex (also called V1). The diagram on the right shows the two main pathways in visual cortex. One goes to the parietal lobe and the other goes to the temporal lobe. What do these two pathways do? Clinical observations have provided us with most of the information on this. One finds different deficits in patients who have lesions in these two different areas. The deficits are very different. Parietal lesions lead to deficits in spatial orientation, attention, and visually-guided behavior (this is the "where" pathway or the “action” pathway). Temporal lobe lesions lead to deficits in object recognition (thus, this is the "what" pathway or the “recognition” pathway).
Maps of the body in the brain
The notion of functional specialization is not restricted to sensation and perception. There are, for example, areas of the brain that are functionally specialized for movement. These areas of the brain are also organized into maps, in this case maps of the body. Each location in the brain is responsible for controlling movement of a different part of the body.
Functional areas in speech and
language
Wernicke’s Areaspeech perception
Broca’s areaspeech production
Arcuate Fasciculus
bundle of fibers connecting
Broca’s and Wernicke’s areas
Left: Auditory pathways from the ear to auditory cortex.
Right: Functional specialization for speech and language. Analogous to sensory brain areas, these language areas are also interconnected in well-defined pathway. These are the brain areas, the activity of which would be most closely associated with “verbal testimony”.
prefrontalcortex
The frontal lobe and higher
cognitive function
anteriorcingulatecortex
• Purple: somatosensory cortex (contains a map of the body for touch sensation).
•Orange: primary motor cortex (contains a map of the body for movement control)
• Pink: supplementary motor cortex (contains a number of subregions involved in movement planning and control)
• Prefrontal cortex: large region in the front of the brain that is involved in higher cognitive function. Prefrontal cortex is the least well-understood region of the brain, but it is believed that it might be organized into maps of cognitive function.
• Anterior cingulate: a subregion of medial prefrontal cortex that has been implicated in deception (a critical part of the interpretation of a many lie detection studies). There is some evidence that this brain area is important for cognitive control (response selection, response inhibition, conflict resolution), which might be important for deception. However, we have a lot to learn about what this part of the brain really does and it is an active area of research. More on this later.
Outline
•Neurosci 101 (how the brain works)
•How neurons work
•A little bit of neuroanatomy
•Functional specialization & computational theory
•Case study: visual motion perception
•Techniques for measuring human brain activity
•Some examples of mind reading with fMRI
Vision is an unconscious inference
We all have the notion that vision is automatic and effortless. But in fact it is an amazing accomplishment of your brain. The light from this picture reaches your eye. It is only light, of course, that comes into your eye. The light is absorbed by specialized neurons in the retina and transformed into electrical signals. Those signals are processed by a series of neurons in the retina and then action potentials are transmitted to the visual part of the brain (at the back) where further processing is done. About a third of your brain is devoted to visual processing. The end result is your full visual experience of the world: recognition of objects, people, faces, facial expressions, color, distance, motion, and so on. All this is inferred from the changing pattern of light reaching your eye. These inferences are performed by neural computations.
Visual inference: motion perception
Visual inferences are usually quite accurate but not always. This video shows an example of a visual illusion of motion that I made a few years ago. When the video is playing, the figure appears to move while staying in the same location. It is based on Duchamp’s famous painting “Nude descending a staircase” which is all about freezing motion in time. The changing pattern of light causes the visual brain to make a mistake. Normally, such changing patterns of light would correspond to an object that is moving in the world. In this case, however, there is no motion and the inferred/perceived motion is wrong.
Another visual illusion of motion, again demonstrating that visual motion perception is an unconscious inference.
MT
Cortical area MT is specialized
for visual motion perception
•Neurons in MT are selective for motion direction.
•Neural responses in MT are correlated with the perception of motion.
•Damage to MT causes deficits in visual motion perception.
•Electrical stimulation in MT causes changes in visual motion perception.
•Computational theory quantitatively explains both the responses of MT neurons and the perception of visual motion.
•Well-defined pathway of brain areas (cascade of neural computations) underlying motion specialization in MT.
Visual motion perception is accomplished by neural computations. Cortical area MT has been shown to play a particularly important role. Importantly, it has been shown that MT is not only correlated with motion perception but also that activity in MT causes the perception of motion. We have rigorous, quantitative theories that explain both the physiology of MT neurons and the psychology motion perception (e.g., in the visual illusions shown earlier). Correlational evidence, by itself, is not enough.
• Confounding factor: some hidden variable that might be driving a correlation. Example: moving stimuli are more interesting to look at than stationary. If all we had was a correlation between between MT activity and motion perception, then it might be due to attention/engagement, not motion perception per se.
• Generalization: without a theory, can’t generalize beyond the specific circumstances of the experiments that have already been done.
Outline
•Neurosci 101 (how the brain works)
•How neurons work
•A little bit of neuroanatomy
•Functional specialization & computational theory
•Case study: visual motion perception
•Techniques for measuring human brain
activity
•Some examples of mind reading with fMRI
Functional magnetic resonance imaging
Now that we have learned some of the basics of how the brain works, let’s revisit fMRI.
Indirect measure of neural activity
In an fMRI experiment, we acquire a time-series of images. Changes in blood oxygenation (following metabolic demand of neural activity) appear as changes in image intensity. There are numerous demonstrations (including some of the above slides) that the fMRI measurement is tightly linked with underlying neural firing rates. fMRI is, however, an indirect measure of the underlying neural activity
fMRI takes advantage of the coupling between neuronal activity and hemodynamics (the local control of blood flow and oxygenation) in the brain to allow the non-invasive localization and measurement of brain activity. The ultimate success of fMRI as a measurement of brain function depends on the relationship between the fMRI signal and the underlying neuronal activity. The vascular source of the fMRI signal places important limits on the usefulness of the technique. Although we know that the fMRI signal is triggered by the metabolic demands of increased neuronal activity, the details of this process are only partially understood. Consequently, this issue has emerged as one of the most important areas in neuroscience. There are two parts to the story: 1) The physics is very well understood, but 2) The physiology is not so well understood.
Physics. The fundamental signal for blood oxygen level dependent (BOLD) functional magnetic resonance imaging (fMRI) comes from hydrogen atoms,which are abundant in the water molecules of the brain. In the presence of a magnetic field,these hydrogen atoms absorb energy that is applied at a characteristic radio frequency (~64 MHz for a standard, clinical 1.5-Tesla MRI scanner). After this step of applying radio-frequency excitation, the hydrogen atoms emit energy at the same radio frequency until they gradually return to their equilibrium state. The MRI scanner measures the sum total of the emitted radio-frequency energy. The measured radio-frequency signal decays over time, owing to various factors, including the presence of inhomogeneities in the magnetic field. Greater inhomogeneity results in decreased image intensity, because each hydrogen atom experiences a slightly different magnetic field strength, and after a short time has passed (commonly called T2*), their radio-frequency emissions cancel one another out. BOLD fMRI techniques are designed to measure primarily changes in the inhomogeneity of the magnetic field,within each small volume of tissue, that result from changes in blood oxygenation. Deoxy- and oxyhemoglobin have different magnetic properties; deoxyhemoglobin is paramagnetic and introduces an inhomogeneity into the nearby magnetic field,whereas oxyhemoglobin is weakly diamagnetic and has little effect. Hence, an increase in the concentration of deoxyhemoglobin causes a decrease in image intensity, and a decrease in deoxyhemoglobin causes an increase in image intensity.
Physiology. The brain increases the local blood flow in reaction to the demand for glucose and oxygen. The details of this process are not fully understood but one theory posits that blood flow follows directly from increased synaptic activity. Astrocytes (specialize glial cells, not neurons, in the brain) surround both synapses and capillaries, and they are responsible for neurotransmitter recycling (taking the glutamate out of the synapse quickly to stop its action on the post-synaptic membrane, causing a chemical change in the glutamate molecules to deactivate it, and handing it back to the nearby neurons for reuse). All this takes a lot of energy.
fMRI spatial and temporal resolution
Time series of fMRI images
Time
Typical pixel size: 3mm x 3mm x 3mm
Typical frame time: 2 sec
Spatial and temporal resolution of fMRI is limited by blood flow. fMRI response depends on the average activity of neurons in a little chunk of brain, and averaged over time. Best case spatial resolution is about 1mm x 1mm x 1mm. Typical spatial resolution is 3mm x 3mm x 3mm. Recall that there are about 50,000 neurons per cubic mm. That’s a lot of neurons and they could be doing different things which would get confounded with one another in the fMRI measurement. Best case temporal resolution is about 100 msec, and typical temporal resolution is 2 sec. That’s a long time compared to the time-scale of neural processing (recall that action potentials take place in just a couple msec). A series of neural processes could take place during that time period which would get confounded with one another during that time period.
Neurovascular coupling limits spatial
specificity
Spatial specificity (spread and mislocalization) depends on vein size.
I
II
III
IV
V
VI
Reina de la Torre et al Anatomical Record (1998)
Veins that drain from large regions of the brain will reflect the metabolic demand of a large number of neurons whereas small veins that drain locally will reflect the activity in a smaller population of neurons, with better spatial resolution and spatial accuracy.
Sluggishness of hemodynamics limits
temporal specificity
fMR
I re
spon
se(%
chan
ge im
age
inte
nsit
y)
Time (s)
0 2 4 6 8 10 12 14 16
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
Brief pulse of neural activity
Hemodynamics: control of blood flow and oxygenation is sluggish. A brief pulse of neural activity evokes a change in blood oxygenation that evolves over many seconds.
In spite of sluggish hemodynamics, can make fine discriminations about timing of a neural processes. I.e., can discriminate the timing of a single neural processing event. But cannot distinguish a series of neural processing events that all take place during a short time period.
PET (positron emission tomography)
A patient lies in the scanner after being injected with radioactive-labeled water. Because more blood goes to active brain regions, there is more radioactivity in those brain regions. The PET scanner detects and localizes these pockets of higher radioactivity.
PET based upon the unique radioactive decay of positrons, the anitmatter equivalent of electrons. These tiny, positively charged, radioactive particles emerge from the radioactive water. After being emitted they are attracted to nearby negatively charged electrons. When positrons and electrons come together, they are annihilated, and energy is released in the form of two photons (particles of light, no charge) that leave the point of annihilation in exactly opposite directions. The PET scanner is set up to detect the coincidental arrival of pairs of photons. The location of the positron-electron annihilation is determined by which pair of detectors are simultaneously active.
Both fMRI and PET depend on regional control of blood flow in the brain. But, current fMRI methods go way beyond the conventional PET subtraction methodology. Critically, fMRI is non-invasive - no radioactive stuff needs be injected in your blood stream.
Event-related potentials (ERP)
EEG (electroencephalography)
EEG (electroencephalograph) is also being used to study brain function. At each electrode, the electrical activity is recorded at fixed intervals following a stimulus presentation - say, every 4 milliseconds.
With EEG, you get millisecond time resolution which is good. But a huge disadvantage is localization. Don't really know exactly where these electrical signals are coming from. The EEG signal recorded with each electrode on the scalp reflects the pooled (average) electrical responses of large numbers of neurons throughout the brain. The strength of the EEG signals depend on how far the electrode is from the neural source so the array of electrodes on the scalp can be used to infer roughly where the neural source is located. This works well if there are only a few neural sources. But if there are a large number of brain regions active simultaneously then the locations of the different neural sources get confused with one another and there is no way to pull them apart.
MEG (magnetoencephalography)
MEG measures the tiny magnetic fields evoked by electrical currents in neurons. The magnetic fields are generated when electrical currents travel along the dendrites of a large enough number of adjacent neurons simultaneously. Like EEG, it has the advantage of millisecond time resolution. But like EEG, it has limited spatial resolution.
Given the importance of spatial localization of activity in the brain, fMRI has become the method of choice.
TMS (transcranial magnetic stimulation)
Transcranial magnetic stimulation (TMS) is the use of powerful rapidly changing magnetic fields to induce electric fields in the brain by electromagnetic induction without the need for surgery or external electrodes. Repetitive transcranial magnetic stimulation is known as rTMS. TMS is a powerful tool in research for mapping out how the brain functions, and has shown promise for noninvasive treatment of a host of disorders, including depression and auditory hallucinations.
One reason TMS is important in neuroscience is that it can demonstrate causality. A noninvasive mapping technique such as fMRI allows researchers to see what regions of the brain are activated when a subject performs a certain task, but this is not proof that those regions are actually used for the task; it merely shows that a region is associated with a task. If activity in the associated region is suppressed with TMS stimulation and a subject then performs worse on a task, this is much stronger evidence that the region is used in performing the task. For example, it has been shown the TMS in MT interferes with motion perception.
Outline
•Neurosci 101 (how the brain works)
•How neurons work
•A little bit of neuroanatomy
•Functional specialization & computational theory
•Case study: visual motion perception
•Techniques for measuring human brain activity
•Some examples of mind reading with
fMRI
Mind reading: attention
Gandhi, Heeger, & Boynton, PNAS (1999)
The study of attention has a very long history within psychology, dating back more than 100 years. We know that when people attend (without moving their eyes) to certain visual stimuli to perform a task, the responses of some visual neurons are enhanced compared to when attention is directed elsewhere.
This video shows a demonstration of how attention modulates brain activity in visual cortex. In the experiments, subjects fixated the center of a display while shifting their attention to the right or the left half of the display. Stimuli on the right are processed by neurons in the left hemisphere and vice versa. The stimuli were always the same and the subjects did not move their eyes, nor did they perform any other overt behavioral response to indicate where they were attending. Only their internal mental state (attention to one side or the other) varied. Attending left increased brain activity in the right hemisphere. Attending right increase activity in the left hemisphere.
Functional specialization for recognition
Fusiform face area (FFA)
Parahippocampal place area (PPA)
A series of experiments have shown that there are brain regions that are functionally specialized for recognizing different categories of objects. Here, for example, are two particularly well-studied areas. The PPA responds more strongly to pictures of houses, buildings, and indoor or outdoor scenes. The FFA responds selectively to pictures of faces.
“Woody Allen” “Great Count” “Bill Clinton” “Cary Grant” “Media Lab”
FFA
PPA
fMR
I re
spon
se
Stimulus type
presented
85% oftrials werecorrectlyidentified.
1.0
1.5
2.0
0.5
0
F F P P P F F P F
Mind reading: imagined faces and places
O’Craven & Kanwisher, J Cogn Neurosci (2000)
This experiment capitalized on the functional specializations of PPA and FFA to study what happens in the brain when you imagine something. Once every 12 seconds, subject heard the name of a person or a familiar place (a building on the MIT campus). They were instructed to imagine it. The investigator recorded fMRI signals in the "face" area (that responds strongly to pictures of faces) and the "place" area (that responds strongly to pictures of familiar places, buildings, etc.). The red curve shows the time-course of response in the "face area". The blue curve shows the time-course of response in the the "place area". fMRI response is bigger in the "face" area when imagining a face. Response is bigger in the "place" area when imagining a place.
Classifying perceived motion direction
Right vs left?
90-95% accuracy
Up until now, I have used examples in which the different states of mind (attend right vs attend left, imagine a face vs imagine a place) correspond to brain activity in widely separated locations. Often, the neural representation of a particular state of mind is interleaved with that of another state of mind. Visual motion perception is, again, a good example. The neurons (e.g., in area MT) that prefer leftward motion are interleaved with those preferring rightward motion. Hence, rightward motion evokes a distributed pattern of brain activity (blue) and leftward motion evokes a complementary distributed pattern (red). The perceived motion direction can be “classified” on any given trial by measuring the brain activity and deciding which pattern it most resembles.
How the human brain interacts with the
world in real life
Simple sensory stimuli:
The full complexity of real life:
Neuroscience research has tended to follow a reductionistic, deductive line of reasoning. For decades, neuroscientists have worked toward simplification, using simple stimuli and behavioral tasks, precisely parameterized and in highly controlled laboratory settings. This approach has obvious advantages and has served us well, as evidenced by the tremendous amount of knowledge amassed about brain structure and function. These conditions, however, are removed from natural real life situations.
Recent research has started employing empirical protocols in which we measure brain activity using fMRI during free viewing of engaging, natural sensory stimuli (e.g., movie).
How the human brain interacts with the
world in real life
Inter-subject correlation
Hasson et al., Science (2004)
Requires developing new data analysis methods that do rely on predetermined stimulation or behavioral protocols. A simple example of this is an analysis of the inter-subject correlations in the timecourse of brain activity. Using the activity at each location of one brain to predict the activity in other brains, Hasson et al (Science, 2004) found that brain activity is highly correlated (across individuals) when watching the same movie. About 30-40% of your brain does the same thing as my brain as we share the same audiovisual and emotional experience.
This characterizes the component of brain activity that is common/shared across individuals during “realistic” situations. Extensions of this approach will be able to characterize differences between groups in how their brain react to complex stimuli, and eventually individual differences as well.
“Mind reading” competition at Human Brain Mapping 2006(http://www.ebc.pitt.edu/competition.html)
fMRI responses to video from 3 segments of the Home Improvement TV series, rated for a variety of features (e.g., faces, emotions).
Utilize data from segments 1 and 2 to train classifier, then generate predicted behavior ratings for segment 3.
Brain activity interpretation competition
Combining the classifier analysis (above) with the use of complex, realistic stimuli, neuroscientists are now able to “read out” from the brain what the subject was viewing as well as some other cognitive and emotional aspects of what they were experiencing while viewing. A competition was held at a recent brain imaging conference.
Classifying spatial patterns of brain activity with machine learningmethods: Application to lie detection
C. Davatzikos,a,* K. Ruparel,b Y. Fan,a D.G. Shen,a M. Acharyya,a J.W. Loughead,b
R.C. Gur,b and D.D. Langlebenb,c
aDepartment of Radiology, University of Pennsylvania, 3600 Market Street, Suite 380, Philadelphia, PA 19104, USAbDepartment of Psychiatry, University of Pennsylvania, 3400 Spruce Street, Philadelphia, PA 19104, USAcTreatment Research Center, University of Pennsylvania, 3900 Chestnut Street, Philadelphia, PA 19104, USA
Received 1 March 2005; revised 15 June 2005; accepted 4 August 2005
Available online 5 October 2005
Patterns of brain activity during deception have recently been
characterized with fMRI on the multi-subject average group level.
The clinical value of fMRI in lie detection will be determined by the
ability to detect deception in individual subjects, rather than group
averages. High-dimensional non-linear pattern classification methods
applied to functional magnetic resonance (fMRI) images were used to
discriminate between the spatial patterns of brain activity associated
with lie and truth. In 22 participants performing a forced-choice
deception task, 99% of the true and false responses were discriminated
correctly. Predictive accuracy, assessed by cross-validation in partic-
ipants not included in training, was 88%. The results demonstrate the
potential of non-linear machine learning techniques in lie detection and
other possible clinical applications of fMRI in individual subjects, and
indicate that accurate clinical tests could be based on measurements of
brain function with fMRI.
D 2005 Elsevier Inc. All rights reserved.
Introduction
A large body of functional neuroimaging literature has
elucidated relationships between structure and function, as wellas functional activity patterns during a variety of functionalactivation paradigms. Statistical parametric mapping (SPM)
(Friston et al., 1995) has played a fundamental role in thesestudies, by departing from the conventional biased ROI- andhypothesis-based methods of data analysis and enabling unbiased
voxel-by-voxel examination of all brain regions. While a great dealof knowledge has been gained during the past decade regardingbrain regions that are activated during various tasks using voxel-
based SPM analysis, the quantitative characterization of entirespatio-temporal patterns of brain activity, as opposed to voxel byvoxel examination, has received much less attention, especially asa means for deducing ‘‘the state of the mind’’ from functional
imaging data. The important distinction between a voxel-basedanalysis and the analysis of a spatio-temporal pattern is the same asthe distinction between (mass) uni-variate and multi-variate
analysis (Davatzikos, 2004). Specifically, a pattern of brain activityis not only a collection of active voxels, but carries with itcorrelations among different voxels. Notable efforts towards thefunctional activity pattern analysis have been made (Strother et al.,
1995; McIntosh et al., 1996), some of which, attempt to use thesemethods to classify complex activation patterns using machinelearning methods (Cox and Savoy, 2003; LaConte et al., 2005).
In this paper, we present an approach to the problem ofidentifying patterns of functional activity, by using a high-dimensional non-linear pattern classification method. We apply
this approach to one of the long-standing challenges in appliedpsychophysiology, namely lie detection. Deception is a sociallyand legally important behavior. The limitations of the specificityof the currently available physiological methods of lie detection
prompted the exploration of alternative methods based on thecorrelates of the central nervous system activity, such as EEG andfMRI (Rosenfeld, 2001; Spence et al., 2001; Langleben et al.,
2002). Using SPM-based analyses of multi-subject average groupdata, several recent fMRI studies demonstrated differences inbrain activation between truthful and non-truthful responses in
various experimental paradigms (Langleben et al., 2002; Langle-ben et al., in press; Ganis et al., 2003; Kozel et al., 2004a,b; Leeet al., 2002). In order to translate these data into a clinically
relevant application, discrimination between lie and truth has tobe achieved at the level of single participants and single trials(Kozel et al., 2004b), not just via group analysis. The potential ofthe SPM-based approach to achieve this goal is limited due to the
between-subject variability of regional brain activity. In thecurrent work, we have overcome this limitation using a multi-variate non-linear high-dimensionality pattern classification tech-
nique (Lao et al., 2004) applied to spatial patterns of brainactivation recorded via fMRI. Using data acquired with apreviously reported formal deception paradigm (Langleben et
al., in press), we have tested the hypothesis that truthful and non-
1053-8119/$ - see front matter D 2005 Elsevier Inc. All rights reserved.
doi:10.1016/j.neuroimage.2005.08.009
* Corresponding author. Fax: +1 215 614 0266.
E-mail address: [email protected] (C. Davatzikos).
Available online on ScienceDirect (www.sciencedirect.com).
www.elsevier.com/locate/ynimg
NeuroImage 28 (2005) 663 – 668
No Lie MRI, Inc.
This brings us back full circle to lie detection. The methods described in this paper (and the basis for the lie detection technology that is being adopted by No Lie MRI) is based on a classifier just like the one I illustrated for “reading” a person’s motion perception.
In this study, participants were first given two cards and they were told to lie about one of them during the experiment. Then, in the MRI scanner, they were shown a series of cards, and they were instructed to press one of two buttons in reaction to each card to indicate whether or not it was one of the cards that they were given. On some trials, the participants were presented with one of the two cards that they were given initially and on other trials they were shown other cards from the deck. The investigators measured the activity throughout the brain on each individual trial (each card) and applied a classifier to distinguishing “truth” from “lie” trials.
•Correlation only (no causation, no theory): many possible confounds.
•Contrived scenario (instructed to lie for $50) versus real-world scenario (risk of severe societal, emotional, monetary damages).
•Individual differences: experimental subject population (college students) versus real-world population (socioeconomic groups, cultural backgrounds, sociopaths, etc.).
•Deception vs truth: false memory, imagination, mental illness.
•External validity and verifiability.
Generalizability of fMRI-based lie
detection
• Confounds. An example of a possible confound is the following. It turns out that in many of the lie detection studies, participants were slower in providing their response (”yes” or “no” by pressing one of two buttons) when they were lying than when they were telling the truth. There is certainly a neural correlate of this difference in behavioral response time. Might it be that some of the studies are basing their “lie detection” simply on this?
• Theory. The theoretical claims in the literature are weak at best. The interpretation of the lie detection studies rests on previous reports that specific brain areas (e.g., anterior cingulate cortex or ACC) have particular functional specializations (e.g., cognitive control, conflict resolution, response inhibition). But those previous reports are themselves just correlations between brain activity and behavior. So it is like a house of cards: a correlation, the interpretation of which rests on a correlation, all the way down. Prefrontal cortex (including ACC) is the least well-understood region of the brain; we have a lot to learn about what this part of the brain really does and it is an active area of research. For example, close to 600 papers in 2005 about ACC. In addition to cognitive control, the topics of these papers are quite diverse: stress, depression, drug abuse, bipolar disorder, pain, verbal fluency, psychosis, schizophrenia, reward encoding, motivation and reward anticipation, PTSD, fear and fear memory, borderline personality disorder, vision, disgust, social phobia, delirious state, conscious effort, social anxiety disorder, and attention. Moreover, neuropsychology studies of patients with damage to ACC have shown that there are no performance deficits for tasks that require cognitive control (Fellows & Farah, Is anterior cingulate cortex necessary for cognitive control? Brain 128, 788-96, 2005). Hence, there is no causal link between ACC activity and cognitive control.
• Individual differences. In all of the studies of lie detection, the results have been inconsistent across individual subjects’ brains. This might reflect different “strategies” for lying or it might reflect different confounding factors in different individuals. This is within a highly constrained population (typically, undergraduate college students). Indiv diffs will be even greater across different segments of the population.
• Detecting conscious deception is not the same as finding the truth. Memory can be inaccurate. Can we distinguish true memories from false memories? Can we distinguish memories for real events from memories for imagined or fantasized events? Mental illness can make all this particularly complicated. More on this from Dan Schacter tomorrow.
• Verifiability. For visual motion perception, there is an objectively correct answer for the classification on each trial and we know what that answer is. For lie detection, it is not clear that it would be possible to design a study to validate a lie detection technology. It would be relatively easy to show that it doesn't work using contrived situations. To show convincingly that it does work, however, you need to use real-world situations. But then, you don't know if someone is lying or telling the truth so there is no external validity.
Neuro-evidence checklist
Visual motion perception
90-95%
yes
yes
yes
yes
yes
yes
no
Lie detection
88%
yes
no
no
no
no
no
yes
Classification accuracy
Correlational evidence
Causal evidence
Quantitative theory
Well-defined neural pathway
Consistency across individuals
External validity & verifiability
Potential impact on law & society
Davatzikos et al., 2005:"Moreover, since the classification method is not specific to lie detection, it could ultimately be used to a very broad range of applications in which the state of mind is to be inferred from spatio-temporal patterns of brain activity."
Mind reading and cognitive freedom
•Is mind reading a new kind of evidence, is it physical evidence (the physical state of the brain), or is it like verbal testimony?
•To the extent that a brain measurement yields information about the state of the brain that is highly correlated with state-of-mind, should it be protected based on privacy considerations?
•To the extent that a brain measurement is not highly correlated with state-of-mind, it will not be reliable.
Many of the concerns about reliability might be mitigated by improving our understanding of how lying works in the brain. Even the issue of individual differences and different strategies for lying might become well understood. But that leaves us with another dilemma: should state-of-mind be considered private?
On the one hand, we should be concerned about misapplication and preadoption of a technique that is not reliable. On the other hand, we should be concerned about invading the privacy of someone’s mind. Does a person have a right to keep his or her subjective thoughts private, the right to cognitive freedom?
Selected reading
• Davatzikos C, Ruparel K, Fan Y, Shen DG, Acharyya M, Loughead JW, Gur RC, Langleben DD (2005) Classifying spatial patterns of brain activity with machine learning methods: application to lie detection. Neuroimage 28:663-668.
• Dubin M (2002) How the Brain Works. Blackwell Science.
• Gazzaniga MS, Ivry RB, Mangun GR (2002) Cognitive Neuroscience. W. W. Norton & Company.
• Hasson U, Nir Y, Levy I, Fuhrmann G, Malach R (2004) Intersubject synchronization of cortical activity during natural vision. Science 303:1634-1640.
• Heeger DJ, Ress D (2002) What does fMRI tell us about neuronal activity? Nature Reviews Neuroscince 3:142-151.
• Wolpe PR, Foster KR, Langleben DD (2005) Emerging neurotechnologies for lie-detection: promises and perils. Am J Bioeth 5:39-49.
• Cephos Corporation (http://www.cephoscorp.com/index.html)
• No Lie MRI, Inc. (http://www.noliemri.com/)
• “Mind Reading” competition (http://www.ebc.pitt.edu/competition.html)