![Page 1: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/1.jpg)
MIT
December 15, 2017
Jeff [email protected]
Have We Missed Half of What the Neocortex Does?
Allocentric Location as the Basis of Perception
![Page 2: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/2.jpg)
1) Reverse engineer the neocortex- an ambitious but realizable goal- seek biologically accurate theories- test empirically and via simulation
2) Enable technology based on cortical theory- active open source community- basis for Machine Intelligence
![Page 3: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/3.jpg)
![Page 4: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/4.jpg)
L2
L3aL3b
L4
L6a
L6b
L6 ipL6 mpL6 bp
L5 ttL5 ccL5 cc-ns
L2/3
L4
L6
L5
Input
The Cortical Column
1) Cortical columns are complex- Twelve or more excitatory cellular layers- Two parallel FF pathways- Parallel FB pathways (not shown)- Numerous intra- and inter-column connections (not shown)- Inhibitory neurons/circuits are equally complex
2) The function of a cortical column must also be complex.
3) Whatever a column does applies to everything the cortex does.
L5: Calloway et. al, 2015L6: Zhang and Deschenes, 1997
Simple
Output, via thalamus
50%10%
CortexThalamus
Output, direct
L5 CTC: Guillery, 1995Constantinople and Bruno, 2013
A Couple of Thoughts
Output
![Page 5: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/5.jpg)
Observation:
The neocortex is constantly predicting its inputs.
How do networks of neurons, as seen in the neocortex,learn predictive models of the world?
Research:
![Page 6: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/6.jpg)
1) How does the cortex learn predictive models of extrinsic sequences?
2) How does the cortex learn predictive models of sensorimotor sequences?
Current research: How do columns compute allocentric location?
- Grid cells in entorhinal cortex solve a similar problem- Big Idea: cortical columns contain analogs of grid cells and head direction cells- Starting to understand the function of numerous layers and connections
“Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in the Neocortex”Hawkins and Ahmad, Frontiers in Neural Circuits, 2016/03/30
- Big Idea: Pyramidal neuron model for prediction- A single layer network model for sequence memory- Properties of sparse activations
“A Theory of How Columns in the Neocortex Learn the Structure of the World”Hawkins, Ahmad, and Cui, Frontiers in Neural Circuits, 2017/10/25
- Extension of sequence memory model- Big Idea: Columns compute “allocentric” location of input- By moving sensor, columns learn models of complete objects
![Page 7: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/7.jpg)
Proximal synapses: Cause somatic spikesDefine classic receptive field of neuron
Distal synapses: Cause dendritic spikesPut the cell into a depolarized, or “predictive” state
Depolarized neurons fire sooner, inhibiting nearby neurons.
A neuron can predict its activity in hundreds of unique contexts.
5K to 30K excitatory synapses- 10% proximal- 90% distal
Distal dendrites are pattern detectors- 8-15 co-active, co-located synapses
generate dendritic spike- sustained depolarization of soma
HTM Neuron Model
Prediction Starts in the Neuron
Pyramidal Neuron
Major, Larkum and Schiller 2013
![Page 8: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/8.jpg)
Properties of Sparse Activations
L2
L3aL3b
L4
L6a
L6b
L6 ipL6 mpL6 bp
L5 ttL5 ccL5 cc-ns
Example: One layer of cells, 5,000 neurons, 2% (100) active
1) Representational capacity is virtually unlimited(5,000 choose 100) = 3x10211
2) Randomly chosen representations have minimal overlap
3) A neuron can robustly recognize an activation pattern by forming 10 to 20 synapses
4) Unions of patterns do not cause errors in recognition
Hypothesis: Cellular layers use unions to represent uncertainty
Hawkins, Ahmad, 2016Ahmad, Hawkins, 2015
Pattern 1 (100 active cells)
Cell robustly recognizes pattern1by forming synapses to small sub-sample of active cells
UnionPatterns 1-10 (1,000 active cells)
Cell still robustly recognizes pattern 1
![Page 9: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/9.jpg)
A Single Layer Network Model for Sequence Memory
- Neurons in a mini-column learn same FF receptive field.- Neurons forms distal connections to nearby cells.
No prediction Predicted input
(Hawkins & Ahmad, 2016)
(Cui et al, 2016)
- High capacity (learns up to 1M transitions)
- Learns high-order sequences: “ABCD” vs “XBCY”
- Makes simultaneous predictions: “BC…” predicts “D” and “Y”
- Extremely robust (tolerant to 40% noise and faults)
- Learning is unsupervised, continuous, and local
- Satisfies many biological constraints
- Multiple open source implementations (some commercial)
t=0
t=1
Predicted cells fire first and inhibit neighbors
Next prediction t=2
t=0
t=1
![Page 10: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/10.jpg)
1) How does the cortex learn predictive models of extrinsic sequences?
2) How does the cortex learn predictive models of sensorimotor sequences?
Current research: How do columns compute allocentric location?
- Grid cells in entorhinal cortex solve a similar problem- Hypothesis: cortical columns contain analogs of grid cells and head direction cells- Starting to understand the function of numerous layers and connections
“Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in the Neocortex”Hawkins and Ahmad, Frontiers in Neural Circuits, 2016/03/30
- Pyramidal neuron model- A single layer network model for sequence memory- Properties of sparse activations
“A Theory of How Columns in the Neocortex Learn the Structure of the World”Hawkins, Ahmad, and Cui, Frontiers in Neural Circuits, 2017/10/25
- Extension of sequence memory model- Big Idea: Columns compute “allocentric” location of input- By moving sensor, columns learn models of complete objects
![Page 11: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/11.jpg)
How Could a Layer of Neurons Learn a Predictive Model of Sensorimotor Sequences?
Sequence memory
Sensorimotor sequences
SensorMotor-related context
Hypothesis:By adding motor-related context, a cellular layer can predictits input as the sensor moves.
What is the correct motor-related context?
L2
L3aL3b
L4
L6a
L6b
L6 ipL6 mpL6 bp
L5 ttL5 ccL5 cc-ns
50%
Sensoryfeature
![Page 12: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/12.jpg)
![Page 13: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/13.jpg)
Two Layer Model of Sensorimotor Sequence Memory
Feature @ location
Object Stable over movement of sensor
With allocentric location input, a column can learn models of complete objects by sensing different locations on object over time.
SensorFeature
AllocentricLocation
Pooling
Seq Mem
Changes with each movement
![Page 14: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/14.jpg)
Object
Feature @ LocationLocationon object
Column 1 Column 2 Column 3
Sensorfeature
Sensorimotor Inference With Multiple Columns
Each column has partial knowledge of object.
Long range connections in object layer allow columns to vote.
Inference is much faster with multiple columns.
![Page 15: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/15.jpg)
FeatureFeatureFeatureLocationLocationLocation
Output
Input
Objects Recognized By Integrating Inputs Over Time
![Page 16: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/16.jpg)
FeatureLocationFeatureLocationFeatureLocation
Column 1 Column 2 Column 3
Output
Input
Recognition is Faster with Multiple Columns
![Page 17: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/17.jpg)
Yale-CMU-Berkeley (YCB) Object Benchmark (Calli et al, 2017)
- 80 objects designed for robotics grasping tasks
- Includes high-resolution 3D CAD files
YCB Object Benchmark
We created a virtual hand using the Unity game engineCurvature based sensor on each fingertip4096 neurons per layer per column
98.7% recall accuracy (77/78 uniquely classified)
Convergence time depends on object, sequence of sensations, number of fingers.
Simulation using YCB Object Benchmark
![Page 18: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/18.jpg)
Pairwise confusion between objects after 1 touch
Convergence 1 finger 1 touch
![Page 19: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/19.jpg)
Pairwise confusion between objects after 2 touches
Convergence 1 finger 2 touches
![Page 20: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/20.jpg)
Pairwise confusion between objects after 6 touches
Convergence 1 finger 6 touches
![Page 21: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/21.jpg)
Pairwise confusion between objects after 10
touches
Convergence 1 finger 10 touches
![Page 22: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/22.jpg)
Convergence Time vs. Number of Columns
This is why we can infer complex objects in a single grasp or single visual fixation.
![Page 23: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/23.jpg)
1) How does the cortex learn predictive models of extrinsic sequences?
2) How does the cortex learn predictive models of sensorimotor sequences?
Current research: How do columns compute allocentric location?
- Hypothesis: cortical columns contain analogs of grid cells and head direction cells- Starting to understand the function of numerous layers and connections
“Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in the Neocortex”Hawkins and Ahmad, Frontiers in Neural Circuits, 2016/03/30
- Pyramidal neuron model- A single layer network model for sequence memory- Properties of sparse activations
“A Theory of How Columns in the Neocortex Learn the Structure of the World”Hawkins, Ahmad, and Cui, Frontiers in Neural Circuits, 2017/10/25
- Extension of sequence memory model- Big Idea: Columns compute “allocentric” location of input- By moving sensor, columns learn models of complete objects
![Page 24: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/24.jpg)
Entorhinal Cortexenvironments
A
B C
X
Y Z
R
S T
Room 3
Room 2Room 1
Location- Encoded by Grid Cells- Unique to location in room AND room- Location is updated by movement
Orientation (of head to room)- Encoded by Head Direction Cells- Anchored to room- Orientation is updated by movement
Location- Unique to location on object AND object- Location is updated by movement
Orientation (of sensor patch to object)- Anchored to object- Orientation is updated by movement
Cortical Columnobjects
Hypothesis:Cortical columns contain analogs of grid cells and head direction cells
A
C
B
X
Y
Z
Stensola, Solstad, Frøland, Moser, Moser: 2012
Location and Orientation are both necessary to learn the structure of rooms and predict sensory input.
Location and Orientation are both necessary to learn the structure of objects and predict sensory input.
![Page 25: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/25.jpg)
L3
L4
L6a
L6b
L5a
L5b
Mapping Orientation and Location to a Cortical Column (most complex slide)
Sensation
Orientation
1) A column is a two-stage sensorimotor model for learning and inferring structure.
2) A column usually cannot infer a Feature or Object in one sensation.- Integrate over time (sense, move, sense, move, sense..)- Vote with neighboring columns
3) This system is most obvious for touch, but it applies to vision and other sensory modalities.
Because this architecture exists throughout the neocortex, it suggests we learn, infer,and manipulate abstract concepts the same way we manipulate objects in the world.
Location
Sensation @ Orientation
Feature
Feature @ Location
Object
Motor updated (HD cell-like)
Motor updated (grid cell-like)
Seq mem
Pooling
Seq mem
Pooling
Meaning Operation
![Page 26: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/26.jpg)
Rethinking Hierarchy
Every column learns complete models of objects. They operate in parallel.
Inputs project to multiple levels at once. Columns operate at different scales of input.
Sense
Simple features
Complex features
Objects
Classic
Objects
Objects
Objects
Sensor array
Proposed
Region 3
Region 2
Region 1
![Page 27: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/27.jpg)
Rethinking Hierarchy
Every column learns complete models of objects. They operate in parallel.
Inputs project to multiple levels at once. Columns operate at different scales of input.
Non-hierarchical connections allow columns to vote on shared elements such as “object” and “feature”.
Sense
Simple features
Complex features
Objects
Classic
Sensor array
Objects
Objects
Objects
Sensor array
vision touch
Proposed
Region 3
Region 2
Region 1
![Page 28: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/28.jpg)
Summary
Goal: Understand the function and operation of the laminar circuits in the neocortex.
Method: Study how cortical columns make predictions of their inputs.
Proposals
1) Pyramidal neurons are the substrate of prediction.Each neuron predicts its activity in hundreds of contexts.
2) A single layer of neurons forms a predictive memory of high-order sequences.(sparse activations, mini-columns, fast inhibition, and lateral connections)
3) A two-layer network forms a predictive memory of sensorimotor sequences.(add motor-derived context and a pooling layer)
4) Columns need motor-derived representations of location and orientation, of the
sensor relative to the object. These are analogous to grid and head direction cells.
5) A framework for the cortical column.- Columns learn complete models of objects as “features at locations”, using twosensorimotor inference stages.
6) The neocortex contains thousands of parallel models, that resolve uncertainty byassociative linking and/or movement of the sensors.
![Page 29: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/29.jpg)
Open Issues
Behaviors: how are they learned, encoded, and applied to objects?
Detailed model of hierarchy including thalamus
How can the model be applied to “Where” pathways, and how do “What” and “Where” pathways work together
Collaborations
There are many testable predictions in this model, a “green field”. We welcome collaborations and discussions.
We are always interested in hosting visiting scholars and interns.
![Page 30: Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)](https://reader031.vdocuments.site/reader031/viewer/2022021815/5a649b6d7f8b9a2c568b6391/html5/thumbnails/30.jpg)
Numenta Team
Subutai AhmadVP Research
Marcus Lewis
Thank You