replicating the human brain: deep learning in action
TRANSCRIPT
Replicating the human brain:Deep learning in action
Marc Garcia
PyData Mallorca - January 25th, 2017
1 / 25Replicating the human brain: Deep learning in action
N
About me
http://datapythonista.github.io
2 / 25Replicating the human brain: Deep learning in action
N
The human brain
3 / 25Replicating the human brain: Deep learning in action
N
The human brain
Visual perception example
4 / 25Replicating the human brain: Deep learning in action
N
The human brain
Visual perception
5 / 25Replicating the human brain: Deep learning in action
N
The human brain
Neuron synapse
Neuron doctrine, Santiago Ramón y Cajal
6 / 25Replicating the human brain: Deep learning in action
N
The human brain
Hubel and Torsten cat experiment
Receptive fields of single neurones in the cat’s striate cortexDavid H Hubel and Torsten N Wiesel, 1959, The Journal of physiology
7 / 25Replicating the human brain: Deep learning in action
N
The human brain
Hebbian theory
Learning The capacity of a neuron to activate another changes over time
Memory This capacity of activation can recall previous activations
Let us assume that the persistence or repetition of a reverberatory activity (or"trace") tends to induce lasting cellular changes that add to its stability.[...] Whenan axon of cell A is near enough to excite a cell B and repeatedly or persistentlytakes part in firing it, some growth process or metabolic change takes place in oneor both cells such that A’s efficiency, as one of the cells firing B, is increased.
The Organization of Behavior, Donald Hebb, 1949
8 / 25Replicating the human brain: Deep learning in action
N
Deep learning: structure
9 / 25Replicating the human brain: Deep learning in action
N
Deep learning: structure
McCulloch-Pitts neuron
x2 w2 Σ f
Activatefunction
y
Output
x1 w1
x3 w3
Weights
Biasb
Inputs
A logical calculus of the ideas immanent in nervous activityWarren S. McCulloch and Walter Pitts, 1943, Bulletin of Mathematical Biophysics
10 / 25Replicating the human brain: Deep learning in action
N
Deep learning: structure
Activate function
11 / 25Replicating the human brain: Deep learning in action
N
Deep learning: structure
Hopfield networks
Neural networks and physical systems with emergent collective computational abilitiesJohn J Hopfield, 1982, Proceedings of the National Academy of Sciences of the USA
12 / 25Replicating the human brain: Deep learning in action
N
Deep learning: structure
Hopfield networks
13 / 25Replicating the human brain: Deep learning in action
N
Deep learning: structure
Hopfield networks
14 / 25Replicating the human brain: Deep learning in action
N
Deep learning: structure
Boltzman machines
Optimal Perceptual InferenceGeoffrey E. Hinton and Terrence J. Sejnowski, 1983Proceedings of the IEEE conference on Computer Vision and Pattern Recognition
15 / 25Replicating the human brain: Deep learning in action
N
Deep learning: structure
Restricted Boltzman machines
Information Processing in Dynamical Systems: Foundations of Harmony TheoryPaul Smolensky, 1986, Parallel Distributed Processing, Volume 1, Chapter 6
16 / 25Replicating the human brain: Deep learning in action
N
Deep learning: structure
Deep belief networks
A fast learning algorithm for deep belief netsYee-Whye Teh et al., 2006, Neural computation
17 / 25Replicating the human brain: Deep learning in action
N
Deep learning: structure
Deep belief networks
18 / 25Replicating the human brain: Deep learning in action
N
Deep learning: training
19 / 25Replicating the human brain: Deep learning in action
N
Deep learning: training
Loss function
Error in our predictions, compared to truth
error = y − θ0 + θ1 · x1 (1)
20 / 25Replicating the human brain: Deep learning in action
N
Deep learning: training
Stochastic gradient descent
21 / 25Replicating the human brain: Deep learning in action
N
Deep learning: training
Backpropagation
22 / 25Replicating the human brain: Deep learning in action
N
Deep learning: training
Deep learning in practise
Not for everyone, just makes sense at a scale
Many layers to extract features
Curse of dimensionality
The key is to compute derivatives very fast:
Theano
Tensorflow
Torch
...
23 / 25Replicating the human brain: Deep learning in action
N
Deep learning: training
Google Youtube experiment
Building High-level Features Using Large Scale Unsupervised LearningQuoc V. Le, 2012, Proceedings of the 29th International Conference on ML
24 / 25Replicating the human brain: Deep learning in action
N
Deep learning: training
Questions?
@datapythonista
25 / 25Replicating the human brain: Deep learning in action
N