computation in sensory processing · • adrian (1920s) • first recordings ... •learning of...

Spikes and Computation in

Sensory Processing Simon Thorpe

CerCo (Brain and Cognition Research Center) &

SpikeNet Technology SARL,

Toulouse, France

• 1974-77Physiology and Psychology (Oxford)• 1977-81 Doctorate with Edmund Rolls (Oxford)

• Recording neurons in orbitofrontal cortex, striatum, parietal cortex, lateral hypothalamus, amygdala, hippocampus, supplementary motor area (!!)

• 1982-83Post-doc with Max Cynader (Halifax, Canada)• Attempting to test Hebb's hypothesis in kitten visual cortex

• 1983-93Paris with Michel Imbert• Testing Hebb's Hypothesis (with Yves Frégnac & Elie Bienenstock)• Dynamics of responses in monkey V1 (with Simona Celebrini & Yves Trotter)

• 1993-nowCERCO Toulouse• Work on ultra-rapid scene processing

• 1999-now SpikeNet • Spike-based image processing• Bioinspired vision

Simon Thorpe

Overview

Part 1

• Ultrarapid visual categorization

• Biological vs Computer Vision

• Temporal Constraints

• Coding with Spikes

• Convolutional Neural Networks in 1999

• SpikeNet

• The current state of the art in Convolutional Neural Networks

• Supervision & GoogleNet

Part 2

• Biologically inspired learning

• Spike-Time Dependent Plasticity (STDP)

• Applications in Vision

• Applications in Audition

• Development of Neural Selectivity

• Memories that can last a lifetime

• Grandmother Cells

• Neocortical Dark Matter

Behavioural Reaction Times

Targets

Distractors

Difference

Event Related Potentials

Scene Processing in 150 ms

Ultra-Rapid Visual Processing

•Saccades towards faces

•Latency 100ms!

Crouzet, Kirchner & Thorpe, 2010


RSVP at 10 fps


Activating Memories

• “That Dalmation picture that I saw in Psych 101”

• How does the brain recognize visual objects and scenes?

• Can a machine be built that can do the same thing?

Some Questions

Biological and Computer VisionCommon Problems

• Identification and Categorising objects and events in complex dynamically changing natural scenes• As fast as possible• As reliably as possible• Using the most energy efficient hardware possible• Using the smallest size and weight footprint

Common Solutions?• David Marr (1982) “Vision : A computational

investigation into the Human Representation and Processing of Visual Information”

• Recent years - is there convergence?

Hardware ConstraintsElectronics • Nvidia GeForce GTX Titan

• 4.5 TeraFlops!

• 2668 cores

• 7.1 billion transistors

• 288 Gbytes/sec Memory bus

• $999

Brain• 1 KHz

• 86 billion processors

• 1-2 m/s conduction velocity

• 20 watts

QuestionsAre we going to be able to implement brain style computing with conventional computing?

ResponseIt depends if we can understand how the brain computes

How many teraflops does the brain need?

How much memory bandwidth?

Classic Neural Computing

• To simulate the visual system

• 4 billion neurons

• 10000 connections each

• Update at 1 kHz

• 40 Petaflops

What’s missing?

• Real brains use spikes• Simulation

• 16.7 million neurons

• 21 billion synapses

• The Brain• 86 billion neurons

• 16 billion in the cortex

• 100 trillion synapses

• Spikes• Firing rate

• 0-200 Hz

• Spontaneous activity

• 1-2 Hz?

Classes of Brain Simulation

• Connectionist Simulators (non-spiking)

• Backpropagation, ART, Kohonen Maps, Time Delay Networks,

• Computational Neuroscience Simulators

• Up to 30 000 compartments

• Lots of channel kinetics etc.

• Spike-Based Simulators

• Software

• Hardware

The Human Brain Project

• European Flagship Proposal

• Three simulation approaches

• Blue Brain

• Henry Markram (Lausanne, Switzerland)

• Analog Chips

• Karl-Heinz Meyer (Heidelburg, Germany)

• SpiNNaker

• Steve Furber (Manchester, UK)

SpiNNaker Project• A simulation system for Spiking Neural Networks

• Prof. Steve Furber, Computer Science, University of Manchester

• 18 ARM968 cores • each with 64 Kbytes of Data and 32 Kbytes of Instructions

• 128 MBytes of shared memory• 48 chips on a board• 18 boards in a 19” frame

A billion neurons in real time

IBM TrueNorth Chip

• What can be computed?

SCIENCE

• Jerry Feldman’s 100-step limit

• High level decisions in about 0.5 seconds

• Interval between spikes around 5 ms

• No more than 100 (massively parallel) computational steps

• Development of connectionist and PDP modelling

• Surely, more detail can help

Temporal Constraints – Early 1980s

Temporal Contraints - 1989

Argument• Roughly 10 layers• 10 ms per layer• Firing rates 0-100 Hz

Face selectivity at 100 ms

Food selectivity at 150 ms

Therefore• Mainly feedforward (?!) • One spike per neuron (?!)

T

T

T

TT

TRetina

LGN

V1

V2

V4

IT

Temporal Constraints 1988-1989Inferotemporal Cortex :Face selectivity at 100 ms

Perrett, Caan & Rolls, 1982

See also : Jerry Feldman & John TsotsosLeonard Uhr (1987)

How can you code with just one spike per neuron?

• Feedforward processing

• Only a few milliseconds per processing step

• One spike per neuron

• Very sparse coding

• Processing without context based help


Sensory Coding with Spikes

• Adrian (1920s)• First recordings from sensory fibres

• Hubel & Wiesel (1962)• Orientation selectivity in striate cortec

Sensory Coding with Spikes

• Wurtz & Goldberg (1972, 1976)

Raster Display Post Stimulus Time Histogram

Assumption : Firing rate is enough to describe the response

The Classic View

2

4

1

2

3

0,213

0,432

0,112

0,238

0,309

•Spikes don't really matter

•Neurons send floating point numbers

•The floating point numbers are transformed into spikes trains using a Poisson process

•God plays dice with spike generation0,375

Temporal Coding Option

• Spikes do really matter

• The temporal patterning of spikes across neurons is critical for computation

• Synchrony• Repeating patterns• etc

• The apparent noise in spiking is unexplained variation

Simon Thorpe's Version

Weak StimulusMedium StimulusStrong

Stimulus

Time

Threshold

•Ordering of spikes is critical•The most activated neurons fire first•Temporal coding is used even for stimuli that are not temporally structured•Computation theoretically possible even when each neuron emits one spike

Sensory Coding with Spikes• Adrian (1920s)

• First recordings from sensory fibres

Low intensity

High intensity

Higher maintained firing

Higher peak firing

Shorter Latency!

Coding in the Optic Nerve

1,000,000 fibres

Intensity

n

n

n

n

n

n

n

n

nn

n

n

n

n

n

n


Intensity

n

n

n

n

n

n

n

n

nn

n

n

n

n

n

n

A mini retina32 x 32 pixels


Coding with Spike Ordering

Less than 1% of cells need to fire for recognition!

Example• A toy retina

Early Studies

• Face identification directly from the output of oriented filters

Early Studies• Virtually all the faces correctly

identified

• Very robust to low contrast

• Very robust to noise

SpikeNet Technology

• Created in 1999

• Simon Thorpe, Rufin VanRullen & Arnaud Delorme

• Currently 12 employees

SpikeNet - Invariance

• Luminance

• Contrast

• Blurring

• Noise


• Size

• Rotation • Perspective


• 3D Rotation

• Identity

•What is the current state of the art?

Biological vs Computer HardwareComputer

• Nvidia GeForce GTX Titan

• 4.5 TeraFlops!

• 2668 cores

• 7.1 billion transistors

• 288 Gbytes/sec Memory bus

• 200 watts

• $999

• 86 billion neurons

• 16 billion in the cortex

• 4 billion in the visual system

• 1 KHz

• 1-2 m/s conduction velocity

• 20 wattsIs that enough to reproduce human performance?

Brain

The ImageNet Challenge

• 10,000,000 training images

• 10,000+ labels

• Systems tested on new images, with 1000 possible labels

• ECCV 2012 Firenze

• The state of the art was beaten by a “simple” feedforward convolutional neural network trained with Back-Propagation

SuperVision

• 650,000 “neurons”

• 60,000,000 parameters

• 630,000,000 “synapses”

Output Layer

Convolutional Layers

Fully Connected Layers

253440 186624 64896 64896 43264

Animals

SuperVision

And then….• Geoff Hinton and his two

students launched a start-up (DNNresearch)

• bought by Google…

• Yann LeCun, a pioneer of feed-forward convolutional networks since the end of the 1980s

• Hired by Facebook…

Supervision vs Primate Vision

Convergent Evolution!

? ? ? ?

Coding at higher levels

Basic Methodology

•Computer models• 57 parameters• 1-3 layers

•Neural Activity• Parallel Multielectrode Recordings • V4 and IT• Activity (70-170 ms)

•Human performance• Crowd sourced data• Amazon Mechanical Turk• 104 subjects

Comparison of Models

• HMO - Hierarchical Modular Optimisation

• 4 layer CNN with 1250 output units

• Close match between models, humans and neural signals

Human vs Machine

• Sorry, Andrej!

• What about faces?

Feb 2015

Human vs Machine

A range of architectures• SpikeNet (1999)

• Identification immediately after V1

• Supervision (2012)• 7 layers

• GoogleNet (2014)• 30 layers

• 12 times less parameters

The Surprises• Human levels of performance are possible despite

• No feedback• No horizontal connections• No top-down control• No need for context• No dendrites• Complex channel dynamics• Only positive and negative weights• No dynamics• No memory• No oscillations• No “binding”• No attention!!!• No Spikes!!!

Question : Is Spike Timing Precise Enough?

Fred Rieke, David Warland, Rob de Ruyter van Steveninck and William Bialek (1999)

Increasing evidence that the timing of spikes carries a great deal of information

First Spike Coding : Evidence

Coding motion direction

Coding stimulus shape

Experimental proof in the visual system!

Salamander Retinal Ganglion cellsInformation in latency vs Information in the spike count

Overview

Part 1






• SpikeNet



Part 2









The Learning Problem

•SuperVision uses Backpropagation

•Billions of training examples with labeled data

• 1 ms per image!• 100 images in 100 ms

•Totally unbiological

Spike-Based Processing

Song, Miller & Abbott, 2000

• Synapses that fire before the target neuron get strengthened

• Synapses that fire after the target neuron get weakened

Spike Time Dependent Plasticity (STDP)

Simplified STDP rule• All synapses are depressed after each output neuron spike

• Except those that fire in just before

• Natural Consequence• High synaptic weights concentrate on early firing inputs

Threshold 3

Short time constant

12 inputs

Initial synaptic weight = 0.25

Threshold = 3.0

All inputs must fire to reach threshold

Learning of a repeating pattern

Threshold 3

Short time constant

Presentation 1

All synapses activated before the output

neuron

STDP reinforces all synapses


Threshold 3

Short time constant

Presentation 2

9 synapses reinforced

3 depressed


Threshold 3

Short time constant

Presentation 3


6 depressed


Threshold 3

Short time constant

Presentation 4


9 depressed


Threshold 3

Short time constant

Presentation 5


Threshold 3

Short time constant

Presentation 6

3 fully potentiated synapses

9 set to zero


Threshold 3

Short time constant

In six presentations STDP has found the first three inputs that

fire duing the repeating pattern

Multiple Units

Mutual inhibition

Competitive Learning Mechanism

Multiple Units

Multiple Units

Neurons can detect patterns embedded in continous activity

Neurons can detect patterns embedded in continous activity

Multiple Units

Add a second layer to detect combinations of combinations

Can this be scaled up beyond a toy demo?

Learning Spike Sequences with STDP

• Initial State

• DuringLearning

• After Learning

Learning Spike Sequences with STDP

Learning Faces with STDP

Learning Cars with STDP

Tobi Delbruck’s Spiking Retina

STDP Rule

12 seconds 30 seconds 90 seconds

• The system has learned to recognize cars

• No supervision!

“STDP” Rule

Tobi Delbruck’s Spiking Retina

Learning Cars with STDP

Building a complete visual system

1920

1080

• Add more neurons

• Add more layers of processing

• Increase the input resolution

• Use more sophisticated preprocessing

• Mexican hat convolutions

• Different types of input signal

Auditory Noise Learning in Humans

•Learning of random noise patterns

•Roughly 10 repetitions are sufficient!

•Learning appears to be all-or-none

•Lasts for weeks at least

Auditory Noise Learning with STDP

Olivier Bichler, Thesis

Modified STDP rule•Post synaptic spike - depresses all synapses except those activated recently

Auditory Noise Learning with STDP

Auditory Noise Learning -ERP data

Perceptual learning of acoustic noise generates memory-‐evoked potentials Thomas Andrillon, Sid Kouider, Trevor Agus & Daniel Pressnitzer (Currrent Biology, accepted)

Selective Brain Responses to Meaningless noise in just 5 presentations

Selectivity MechanismsThreshold

Controlling Sparsity

• With Temporal Coding

• Easy to control the percentage of neurons that fire

• 1%, 2%, 5%, 10%

12345678910

Controlling Sparsity

• With Temporal Coding

• Easy to control the percentage of neurons that fire

• 1%, 2%, 5%, 10%

1,00E+00&

1,00E+02&

1,00E+04&

1,00E+06&

1,00E+08&

1,00E+10&

1,00E+12&

1,00E+14&

1,00E+16&

1,00E+18&

1,00E+20&

1,00E+22&

1,00E+24&

1,00E+26&

1,00E+28&

1,00E+30&

0& 10& 20& 30& 40& 50& 60& 70& 80& 90& 100&

Combina(ons*with*M*=*100*

12345678910

N of M coding

• Steve Furber

• With 100 neurons• 2100 binary patterns• ~1030

• What about controlling the proportion of active neurons?

1,00E+024,95E+031,62E+053,92E+067,53E+071,19E+091,60E+101,86E+111,90E+121,73E+13

Generating Selectivity• Suppose that the neuron has 10 synapses

• Suppose that the percentage of active inputs is fixed at 10%

• What is the likelihood of having a given number of synapses active?

• With a threshold of 4 the neuron would only have a 1% chance of firing with a random input

• With a threshold of 5, the probability drops to 0.1%

Interim Conclusion - Learning

• Error back-propagation is an unrealistic way to determine the selectivity of neurons in the visual system

• But simple spike-based learning mechanisms could allow the system to generate the appropriate selectivity in an unsupervised way

• Can we build a visual system without label based training??

Memory Mechanisms in Man & Machine

1. Humans can recognise visual and auditory stimuli that they have not experienced for decades.

2. Recognition after very long delays is possible without ever reactivating the memory trace in the intervening period.

3. These very long term memories require an initial memorisation phase, during which memory strength increases roughly linearly with the number of presentations

4. A few tens of presentations can be enough to form a memory that can last a lifetime.

5. Attention-related oscillatory brain activity can help store memories efficiently and rapidly

10 provocative claims

Memory Mechanisms in Man & Machine

6. Storing such very long-term memories involves the creation of highly selective "Grandmother Cells" that only fire if the original training stimulus is experienced again.

7. The neocortex contains large numbers of totally silent cells ("Neocortical Dark Matter") that constitute the long-term memory store.

8. Grandmother Cells can be produced using simple spiking neural network models with Spike-Time Dependent Plasticity (STDP) and competitive inhibitory lateral connections.

9. This selectivity only requires binary synaptic weights that are either "on" or "off", greatly simplifying the problem of maintaining the memory over long periods.

10. Artificial systems using memristor-like devices can implement the same principles, allowing the development of powerful new processing architectures that could replace conventional computing hardware.

10 provocative claims

STDP and Spiking Neurons

• “Intelligent” sensory processing with small numbers of neurons connected to spiking sensory devices• 2 layers of neurons connected to a spiking retina

• Unsupervised learning of car counting

• A single neuron connected to a spiking cochlear

• Unsupervised learning of auditory noise patterns

• What would happen with 16 billion neurons and multiple sensory inputs?

16 billion neurons

2 million optic nerve fibres

60,000 auditory nerve fibres

Grandmother Cells?

• Jerzy Konorski (1967)

• “gnostic neurons”

• Horace Barlow (1972)

• “cardinal cells”

• Alberta Gilinsky (1984)

• “cognons”

A Jennifer Anniston Cell !

Grandmother Cells in Man?

• Halle Berry• The Taj Mahal• Bill Clinton• Saddam Hussein• The Simpsons• The patient’s brother• Members of the research

team• ...

Temporal Lobe Recordings in Humans

“Josh Brolin” “Marilyn Munroe”

• The hit rate (0.54%) is too high

• Assume 109 neurons in the hippocampus

• Assume 10-30 000 identifiable objects (Irv Biederman)

• Each neuron should respond to 100-150 different objects

• Number of identifiable objects probably much higher : 105 - 106

• Rafi Malach : “Totem Pole Cells”

• Alternative Hypothesis

• The hippocampus keeps track of a few thousand objects and events that have been experiences recently

Grandmother Cells in Man?

How to make a Grandmother Cell

• STPD allows neurons to become rapidly selective to repeated stimuli• This can happen in a few tens of presentations• Oscillatory activity could allow this to happen in a single presentation

• Controlling the percentage of active inputs guarantees that the neuron will not fire to random inputs

• Setting a high threshold means that the neuron will never fire unless the stimulus used for training was presented again

• It could maintain its selectivity for several decades

• Cortical Dark Matter?

Dark Matter in Cortex?• Question

• What is the true distribution of firing rates in the cortex?

• Are there neurons that never fire for very long periods of time?

• Problem• Nearly all single unit neurophysiological studies are biased towards neurons that

have spontaneous activity

• The number of neurons recorded seems far lower than would be expected• Only 5-15 neurons recorded per descent (out of hundreds)

• Possible methods• Implanted arrays

• 2-photon optical imaging

• Patch clamp recording

• Roughly 50% of output neurons have firing rates below 0.1 Hz!• Maybe the output neurons are a special class??

• Dark matter in cortex??

Very low firing rate neurons

Invisible Cells?

Overview

Part 1






• SpikeNet



Part 2









Final Conclusions

• Temporal Constraints on Vision

• Feedforward Neural Networks

• SpikeNet


• Importance of Spikes

• Ultrafast coding with one spike per neuron

• Very efficient STDP schemes for unsupervised learning

• Biological Vision has been and will continue to be a source of inspiration for technology

computation in sensory processing · • adrian (1920s) • first recordings ... •learning of...

Documents