machine learning neural networks. introduction artificial neural network is based on the biological...
TRANSCRIPT
![Page 1: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/1.jpg)
Machine Learning
Neural Networks
![Page 2: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/2.jpg)
Introduction
Artificial Neural Network is based on the
biological nervous system as Brain
It is composed of interconnected computing
units called neurons
ANN like human, learn by examples
![Page 3: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/3.jpg)
3
Why Artificial Neural Networks?There are two basic reasons why we are interested in building artificial neural networks (ANNs):
Technical viewpoint: Some problems such as character recognition or the prediction of future states of a system require massively parallel and adaptive processing.
Biological viewpoint: ANNs can be used to replicate and simulate components of the human (or animal) brain, thereby giving us insight into natural information processing.
![Page 4: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/4.jpg)
4
Science: Model how biological neural systems, like human brain, work?
How do we see? How is information stored in/retrieved
from memory? How do you learn to not to touch fire? How do your eyes adapt to the amount
of light in the environment? Related fields: Neuroscience,
Computational Neuroscience, Psychology, Psychophysiology, Cognitive Science, Medicine, Math, Physics.
![Page 5: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/5.jpg)
5
Brief HistoryOld Ages: Association (William James; 1890) McCulloch-Pitts Neuron (1943,1947) Perceptrons (Rosenblatt; 1958,1962) Adaline/LMS (Widrow and Hoff; 1960) Perceptrons book (Minsky and Papert; 1969)
Dark Ages: Self-organization in visual cortex (von der Malsburg; 1973) Backpropagation (Werbos, 1974) Foundations of Adaptive Resonance Theory (Grossberg; 1976) Neural Theory of Association (Amari; 1977)
![Page 6: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/6.jpg)
6
History
Modern Ages: Adaptive Resonance Theory (Grossberg; 1980) Hopfield model (Hopfield; 1982, 1984) Self-organizing maps (Kohonen; 1982) Reinforcement learning (Sutton and Barto; 1983) Simulated Annealing (Kirkpatrick et al.; 1983) Boltzmann machines (Ackley, Hinton, Terrence; 1985) Backpropagation (Rumelhart, Hinton, Williams; 1986) ART-networks (Carpenter, Grossberg; 1992) Support Vector Machines
![Page 7: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/7.jpg)
7
Hebb’s Learning Law In 1949, Donald Hebb formulated William James’ principle of
association into a mathematical form.
• If the activation of the neurons, y1 and y2 , are both on (+1) then the weight between the two neurons grow. (Off: 0)
• Else the weight between remains the same.
• However, when bipolar activation {-1,+1} scheme is used, then the weights can also decrease when the activation of two neurons does not match.
![Page 8: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/8.jpg)
8
Real Neural Learning
Synapses change size and strength with experience.
Hebbian learning: When two connected neurons are firing at the same time, the strength of the synapse between them increases.
“Neurons that fire together, wire together.”
![Page 9: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/9.jpg)
9
Biological Neurons Human brain = tens of thousands
of neurons Each neuron is connected to
thousands other neurons A neuron is made of:
– The soma: body of the neuron– Dendrites: filaments that provide
input to the neuron– The axon: sends an output signal– Synapses: connection with other
neurons – releases certain quantities of chemicals called neurotransmitters to other neurons
![Page 10: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/10.jpg)
10
Modeling of Brain Functions
![Page 11: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/11.jpg)
11
The biological neuron
The pulses generated by the neuron travels along the axon as an electrical wave.
Once these pulses reach the synapses at the end of the axon open up chemical vesicles exciting the other neuron.
![Page 12: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/12.jpg)
12
How do NNs and ANNs work? Information is transmitted as a series of
electric impulses, so-called spikes.
The frequency and phase of these spikes encodes the information.
In biological systems, one neuron can be connected to as many as 10,000 other neurons.
Usually, a neuron receives its information from other neurons in a confined area
![Page 13: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/13.jpg)
13
Navigation of a car
Done by Pomerlau. The network takes inputs from a 34X36 video image and a 7X36 range finder. Output units represent “drive straight”, “turn left” or “turn right”. After training about 40 times on 1200 road images, the car drove around CMU campus at 5 km/h (using a small workstation on the car). This was almost twice the speed of any other non-NN algorithm at the time.
![Page 14: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/14.jpg)
14
Automated driving at 70 mph on a public highway
Camera image
30x32 pixelsas inputs
30 outputsfor steering
30x32 weightsinto one out offour hiddenunit
4 hiddenunits
![Page 15: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/15.jpg)
15
Computers vs. Neural Networks
“Standard” Computers Neural Networks
one CPU highly parallelprocessing
fast processing units slow processing units
reliable units unreliable units
static infrastructure dynamic infrastructure
![Page 16: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/16.jpg)
Neural Network
![Page 17: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/17.jpg)
Neural Network Application
•Pattern recognition can be implemented using NN
•The figure can be T or H character, the network should identify each class of T or H.
![Page 18: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/18.jpg)
![Page 19: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/19.jpg)
![Page 20: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/20.jpg)
![Page 21: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/21.jpg)
Simple Neuron
X1
X2
Xn
OutputInputs
b
![Page 22: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/22.jpg)
An Artificial Neuron
x1
x2
xn
…
Wi,1Wi,2
…
Wi,n
n
jjjii txtwt
1, )()()(net
xi
neuron i
net input signal
synapses
output ))(()(x tnetft iii
![Page 23: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/23.jpg)
Neural Network
Input Layer Hidden 1 Hidden 2 Output Layer
![Page 24: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/24.jpg)
Network Layers
The common type of ANN consists of three
layers of neurons: a layer of input neurons
connected to the layer of hidden neuron
which is connected to a layer of output
neurons.
![Page 25: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/25.jpg)
Architecture of ANN
Feed-Forward networks
Allow the signals to travel one way from input to output
Feed-Back Networks
The signals travel as loops in the network, the output is connected to the input of the network
![Page 26: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/26.jpg)
![Page 27: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/27.jpg)
![Page 28: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/28.jpg)
![Page 29: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/29.jpg)
How do NNs and ANNs Learn?
NNs are able to learn by adapting their connectivity patterns so that the organism improves its behavior in terms of reaching certain (evolutionary) goals.
The NN achieves learning by appropriately adapting the states of its synapses.
![Page 30: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/30.jpg)
Learning Rule
The learning rule modifies the weights of
the connections.
The learning process is divided into
Supervised and Unsupervised learning
![Page 31: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/31.jpg)
Supervised Network
Which means there exists an external
teacher. The target is to minimization of the
error between the desired and computed
output
![Page 32: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/32.jpg)
![Page 33: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/33.jpg)
![Page 34: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/34.jpg)
![Page 35: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/35.jpg)
Unsupervised Network
Uses no external teacher and is based upon
only local information.
![Page 36: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/36.jpg)
![Page 37: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/37.jpg)
![Page 38: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/38.jpg)
![Page 39: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/39.jpg)
Perceptron
It is a network of one neuron and hard limit transfer function
Inputs f
X1
X2
Xn
Output
W1
W2
Wn
![Page 40: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/40.jpg)
Perceptron
The perceptron is given first a randomly
weights vectors
Perceptron is given chosen data pairs (input
and desired output)
Preceptron learning rule changes the
weights according to the error in output
![Page 41: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/41.jpg)
![Page 42: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/42.jpg)
Perceptron Learning Rule
W new = W old + (t-a) X
Where W new is the new weight
W old is the old value of weight
X is the input value
t is the desired value of output
a is the actual value of output
![Page 43: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/43.jpg)
Example
Let – X1 = [0 0] and t =0– X2 = [0 1] and t=0– X3 = [1 0] and t=0– X4 = [1 1] and t=1
W = [2 2] and b = -3
![Page 44: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/44.jpg)
AND Network
This example means we construct a network for AND operation. The network draw a line to separate the classes which is called Classification
![Page 45: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/45.jpg)
Perceptron Geometric ViewThe equation below describes a (hyper-)plane in the input space
consisting of real valued m-dimensional vectors. The plane splits the input space into two regions, each of them describing one class.
0 wxw 0
m
1iii
x2
C1
C2x1
decisionboundary
w1x1 + w2x2 + w0 = 0
decisionregion for C1
w1x1 + w2x2 + w0 >= 0
![Page 46: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/46.jpg)
Problems
Four one-dimensional data belonging to two classes are
X = [1 -0.5 3 -2]
T = [1 -1 1 -1]
W = [-2.5 1.75]
![Page 47: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/47.jpg)
Boolean Functions
Take in two inputs (-1 or +1) Produce one output (-1 or +1) In other contexts, use 0 and 1 Example: AND function
– Produces +1 only if both inputs are +1 Example: OR function
– Produces +1 if either inputs are +1 Related to the logical connectives from F.O.L.
![Page 48: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/48.jpg)
The First Neural Neural Networks
AND Function
1
1X1
X2
Y
AND
X1 X2 Y
1 1 1
1 0 0
0 1 0
0 0 0
Threshold(Y) = 2
![Page 49: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/49.jpg)
Simple Networks
t = 0.0
y
x
W = 1.5
W = 1
-1
![Page 50: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/50.jpg)
Exercises
Design a neural network to recognize the problem of
X1=[2 2] , t1=0 X=[1 -2], t2=1 X3=[-2 2], t3=0 X4=[-1 1], t4=1
Start with initial weights w=[0 0] and bias =0
![Page 51: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/51.jpg)
Perceptron: Limitations
The perceptron The perceptron can only model linearly separable classes, linearly separable classes, like (those described by) the following Boolean functions:
ANDAND OROR COMPLEMENTCOMPLEMENT It cannot cannot model the XORXOR.
You can experiment with these functions in the Matlab practical lessons.
![Page 52: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/52.jpg)
Types of decision regions
022110 xwxww
022110 xwxww
x1
1
x2 w2
w1
w0
Convexregion
L1L2
L3L4 -3.5
Networkwith a singlenode
One-hidden layer network that realizes the convex region
1
1
1
1
1
x1
x2
1
![Page 53: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/53.jpg)
Gaussian NeuronsAnother type of neurons overcomes this problem by using a Gaussian activation function:
11
00
11
ffii(net(netii(t))(t))
netnetii(t)(t)-1-1
2
1)(net
))(net(
t
ii
i
etf
![Page 54: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/54.jpg)
54
Gaussian NeuronsGaussian neurons are able to realize non-linear functions.
Therefore, networks of Gaussian units are in principle unrestricted with regard to the functions that they can realize.
The drawback of Gaussian neurons is that we have to make sure that their net input does not exceed 1.
This adds some difficulty to the learning in Gaussian networks.
![Page 55: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/55.jpg)
55
Sigmoidal NeuronsSigmoidal neurons accept any vectors of real numbers as input, and they output a real number between 0 and 1.
Sigmoidal neurons are the most common type of artificial neuron, especially in learning networks.
A network of sigmoidal units with m input neurons and n output neurons realizes a network function f: Rm (0,1)n
![Page 56: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/56.jpg)
56
Sigmoidal Neurons
The parameter controls the slope of the sigmoid function, while the parameter controls the horizontal offset of the function in a way similar to the threshold neurons.
11
00
11
ffii(net(netii(t))(t))
netnetii(t)(t)-1-1
/))(net(1
1))(net(
tii ietf
= = 11
![Page 57: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/57.jpg)
57
Sigmoidal NeuronsThis leads to a simplified form of the sigmoid function:
)(1
1)(
netenetS
We do not need a modifiable threshold , because we will use “dummy” inputs as we did for perceptrons.
The choice = 1 works well in most situations and results in a very simple derivative of S(net).
![Page 58: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/58.jpg)
58
Sigmoidal Neurons
This result will be very useful when we develop the backpropagation very useful when we develop the backpropagation algorithm.algorithm.
xexS
1
1)(
2)1(
)()('
x
x
e
e
dx
xdSxS
22 )1(
1
1
1
)1(
11xxx
x
eee
e
))(1)(( xSxS
![Page 59: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/59.jpg)
Multi-layers Network
Let the network of 3 layers– Input layer– Hidden layer– Output layer
Each layer has different number of neurons The famous example to need the multi-layer
network is XOR unction
![Page 60: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/60.jpg)
![Page 61: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/61.jpg)
![Page 62: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/62.jpg)
![Page 63: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/63.jpg)
![Page 64: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/64.jpg)
Learning rule
The perceptron learning rule can not be
applied to multi-layer network
We use BackPropagation Algorithm in
learning process
![Page 65: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/65.jpg)
65
Feed-forward + Backpropagation
Feed-forward: – input from the features is fed forward in the network from input
layer towards the output layer Backpropagation:
– Method to asses the blame of errors to weights– error rate flows backwards from the output layer to the input
layer (to adjust the weight in order to minimize the output error)
![Page 66: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/66.jpg)
Backprop
Back-propagation training algorithm illustrated:
Backprop adjusts the weights of the NN in order to minimize the network total mean squared error.
Network activationError computationForward Step
Error propagationBackward Step
![Page 67: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/67.jpg)
Correlation LearningHebbian Learning (1949):
“When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes place in firing it, some growth process or metabolic change takes place in one or both cells such that A’s efficiency, as one of the cells firing B, is increased.”
Weight modification rule:
wi,j = cxixj
Eventually, the connection strength will reflect the correlation between the neurons’ outputs.
![Page 68: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/68.jpg)
68
Competitive Learning• Nodes compete for inputs
• Node with highest activation is the winner
• Winner neuron adapts its tuning (pattern of weights) even further towards the current input
• Individual nodes specialize to win competition for a set of similar inputs
• Process leads to most efficient neural representation of input space
• Typical for unsupervised learning
![Page 69: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/69.jpg)
69
Backpropagation LearningSimilar to the Adaline, the goal of the Backpropagation learning algorithm is to modify the network’s weights so that its output vector
op = (op,1, op,2, …, op,K)
is as close as possible to the desired output vector
dp = (dp,1, dp,2, …, dp,K)
for K output neurons and input patterns p = 1, …, P.
The set of input-output pairs (exemplars) {(xp, dp) | p = 1, …, P} constitutes the training set.
![Page 70: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/70.jpg)
Bp Algorithm
The weight change rule is
Where is the learning factor <1 Error is the error between actual and trained
value f’ is is the derivative of sigmoid function =
f(1-f)
)('.. ioldij
newij inputferror
![Page 71: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/71.jpg)
Delta Rule
Each observation contributes a variable amount to the output
The scale of the contribution depends on the input Output errors can be blamed on the weights A least mean square (LSM) error function can be
defined (ideally it should be zero)
E = ½ (t – y)2
![Page 72: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/72.jpg)
Example
For the network with one neuron in input layer and one neuron in hidden layer the following values are givenX=1, w1 =1, b1=-2, w2=1, b2 =1, =1 and t=1
Where X is the input valueW1 is the weight connect input to hidden W2 is the weight connect hidden to outputB1 and b2 are biasT is the training value
![Page 73: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/73.jpg)
Exercises
Design a neural network to recognize the problem of
X1=[2 2] , t1=0 X=[1 -2], t2=1 X3=[-2 2], t3=0 X4=[-1 1], t4=1
Start with initial weights w=[0 0] and bias =0
![Page 74: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/74.jpg)
Exercises
Perform one iteration of backprpgation to network of two layers. First layer has one neuron with weight 1 and bias –2. The transfer function in first layer is f=n2
The second layer has only one neuron with weight 1 and bias 1. The f in second layer is 1/n.
The input to the network is x=1 and t=1
![Page 75: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/75.jpg)
Neural NetworkConstruct a neural network to solve the problem
X1 X2 Output
1.0 1.0 1
9.4 6.4 -1
2.5 2.1 1
8.0 7.7 -1
0.5 2.2 1
7.9 8.4 -1
7.0 7.0 -1
2.8 0.8 1
1.2 3.0 1
7.8 6.1 -1
Initialize the weights 0.75 , 0.5, and –0.6
![Page 76: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/76.jpg)
Neural NetworkConstruct a neural network to solve the XOR problem
X1 X2 Output
1 1 0
0 0 0
1 0 1
0 1 1
Initialize the weights –7.0 , -7.0, -5.0 and –4.0
![Page 77: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/77.jpg)
-0.5
-0.5
-2
3
-1
1
1
1-1
1
0.5
The transfer function is linear function.
![Page 78: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/78.jpg)
Consider a transfer function as f(n) = n2. Perform one iteration of BackPropagation with a= 0.9 for neural network of two neurons in input layer and one neuron in output layer. The input values are X=[1 -1] and t = 8, the weight values between input and hidden layer are w11 = 1, w12 = - 2, w21 = 0.2, and w22 = 0.1. The weight between input and output layers are w1 = 2 and w2= -2. The bias in input layers are b1 = -1, and b2= 3.
X1
X2
W11
W22
W12
W1
W21 W2
![Page 79: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/79.jpg)
79
Some variations
True gradient descent assumes infinitesmall learning rate (). If is too small then learning is very slow. If large, then the system's learning may never converge.
Some of the possible solutions to this problem are:– Add a momentum term to allow a large learning rate.– Use a different activation function– Use a different error function– Use an adaptive learning rate– Use a good weight initialization procedure.– Use a different minimization procedure
![Page 80: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/80.jpg)
Problems with Local Minima
Backpropagation is gradient descent search– Where the height of the hills is determined by error– But there are many dimensions to the space
• One for each weight in the network
Therefore backpropagation– Can find its ways into local minima
One partial solution:– Random re-start: learn lots of networks
• Starting with different random weight settings
– Can take best network– Or can set up a “committee” of networks to categorise examples
Another partial solution: Momentum
![Page 81: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/81.jpg)
Adding Momentum Imagine rolling a ball down a hill
Without Momentum With Momentum
Gets stuck here
![Page 82: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/82.jpg)
Momentum in Backpropagation For each weight
– Remember what was added in the previous epoch
In the current epoch– Add on a small amount of the previous Δ
The amount is determined by – The momentum parameter, denoted α– α is taken to be between 0 and 1
![Page 83: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/83.jpg)
How Momentum Works If direction of the weight doesn’t change
– Then the movement of search gets bigger
– The amount of additional extra is compounded in each epoch
– May mean that narrow local minima are avoided
– May also mean that the convergence rate speeds up Caution:
– May not have enough momentum to get out of local minima
– Also, too much momentum might carry search
• Back out of the global minimum, into a local minimum
![Page 84: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/84.jpg)
84
Momentum
Weight update becomes:
wij (n+1) = (pj opi) + wij(n) The momentum parameter is chosen
between 0 and 1, typically 0.9. This allows one to use higher learning rates. The momentum term filters out high frequency oscillations on the error surface.
What would the learning rate be in a deep valley?
![Page 85: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/85.jpg)
Problems with Overfitting
Plot training example error versus test example error:
Test set error is increasing!– Network is overfitting the data
– Learning idiosyncrasies in data, not general principles
– Big problem in Machine Learning (ANNs in particular)
![Page 86: Machine Learning Neural Networks. Introduction Artificial Neural Network is based on the biological nervous system as Brain It is composed of interconnected](https://reader035.vdocuments.site/reader035/viewer/2022062313/56649c775503460f9492bdec/html5/thumbnails/86.jpg)
Avoiding Overfitting Bad idea to use training set accuracy to terminate One alternative: Use a validation set
– Hold back some of the training set during training– Like a miniature test set (not used to train weights at
all)– If the validation set error stops decreasing, but the
training set error continues decreasing• Then it’s likely that overfitting has started to
occur, so stop Another alternative: use a weight decay factor
– Take a small amount off every weight after each epoch
– Networks with smaller weights aren’t as highly fine tuned (overfit)