artificiel neural networks 2 morten nielsen department of systems biology, dtu
TRANSCRIPT
![Page 1: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/1.jpg)
Artificiel Neural Networks 2
Morten NielsenDepartment of Systems Biology,
DTU
![Page 2: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/2.jpg)
Outline
• Optimization procedures – Gradient decent (this you already know)
• Network training– back propagation– cross-validation– Over-fitting– Examples– Deeplearning
![Page 3: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/3.jpg)
Neural network. Error estimate
I1 I2
w1 w2
Linear function
o
![Page 4: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/4.jpg)
Neural networks
![Page 5: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/5.jpg)
Gradient decent (from wekipedia)
Gradient descent is based on the observation that if the real-valued function F(x) is defined and differentiable in a neighborhood of a point a, then F(x) decreases fastest if one goes from a in the direction of the negative gradient of F at a. It follows that, if
for > 0 a small enough number, then F(b)<F(a)
![Page 6: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/6.jpg)
Gradient decent (example)
![Page 7: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/7.jpg)
Gradient decent (example)
![Page 8: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/8.jpg)
Gradient decent. Example
Weights are changed in the opposite direction of the gradient of the error
I1 I2
w1 w2
Linear function
o
![Page 9: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/9.jpg)
Network architecture
Input layer
Hidden layer
Output layer
Ik
hj
Hj
o
O
vjk
wj
![Page 10: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/10.jpg)
What about the hidden layer?
![Page 11: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/11.jpg)
Hidden to output layer
![Page 12: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/12.jpg)
Hidden to output layer
![Page 13: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/13.jpg)
Hidden to output layer
![Page 14: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/14.jpg)
Input to hidden layer
![Page 15: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/15.jpg)
Summary
![Page 16: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/16.jpg)
Or
![Page 17: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/17.jpg)
Or
Ii=X[0][k]
Hj=X[1][j]
Oi=X[2][i]
![Page 18: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/18.jpg)
Can you do it your self?
v22=1v12=1
v11=1v21=-1
w1=-1 w2=1
h2
H2
h1
H1
oO
I1=1 I2=1
What is the output (O) from the network?What are the wij and vjk values if the target value is 0 and =0.5?
![Page 19: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/19.jpg)
Can you do it your self (=0.5). Has the error decreased?
v22=1v12=1
v11=1v21=-1
w1=-1 w2=1
h2=H2=
h1=
H1=
o=O=
I1=1 I2=1
v22=.
v12=V11=
v21=
w1= w2=
h2=H2=
h1=H1=
o=O=
I1=1 I2=1
Before After
![Page 20: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/20.jpg)
h1=2H1=0.88
Can you do it your self (=0.5). Has the error decreased?
v22=.
v12=V11=
v21=
w1= w2=
h2=H2=
h1=H1=
o=O=
I1=1 I2=1
Before After
v22=1v12=1
v11=1v21=-1
w1=-1 w2=1
h2=0H2=0.5
o=-0.38O=0.41
I1=1 I2=1
![Page 21: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/21.jpg)
Can you do it your self?
v22=1v12=1
v11=1v21=-1
w1=-1 w2=1
h2=0H2=0.5
h1=2H1=0.88
o=-0.38O=0.41
I1=1 I2=1
• What is the output (O) from the network?
• What are the wij and vjk values if the target value is 0?
![Page 22: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/22.jpg)
Can you do it your self (=0.5). Has the error decreased?
v22=1
v12=1v11=1
v21=-1
w1=-1 w2=1
h2=0H2=0.5
h1=2H1=0.88
o=-0.38O=0.41
I1=1 I2=1
v22=.0.99v12=1.005
v11=1.005
v21=-1.01
w1=-1.043w2=0.975
h2=-0.02H2=0.495
h1=2.01H1=0.882
o=-0.44O=0.39
I1=1 I2=1
![Page 23: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/23.jpg)
Sequence encoding
• Change in weight is linearly dependent on input value
• “True” sparse encoding (i.e 1/0) is therefore highly inefficient
• Sparse is most often encoded as– +1/-1 or 0.9/0.05
![Page 24: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/24.jpg)
Sequence encoding - rescaling
• Rescaling the input values
If the input (o or h) is too large or too small, g´ is zero and the weights are not changed. Optimal performance is when o,h are close to 0.5
![Page 25: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/25.jpg)
Training and error reduction
![Page 26: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/26.jpg)
Training and error reduction
![Page 27: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/27.jpg)
Training and error reduction
Size matters
![Page 28: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/28.jpg)
Example
![Page 29: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/29.jpg)
Neural network training. Cross validation
Cross validation
Train on 4/5 of dataTest on 1/5=>Produce 5 different neural networks each with a different prediction focus
![Page 30: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/30.jpg)
Neural network training curve
Maximum test set performanceMost cable of generalizing
![Page 31: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/31.jpg)
5 fold training
Which network to choose?
![Page 32: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/32.jpg)
5 fold training
![Page 33: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/33.jpg)
Conventional 5 fold cross validation
![Page 34: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/34.jpg)
“Nested” 5 fold cross validation
![Page 35: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/35.jpg)
When to be careful
• When data is scarce, the performance obtained used “conventional” versus “Nested” cross validation can be very large
• When data is abundant the difference is small, and “nested” cross validation might even be higher than “conventional” cross validation due to the ensemble aspect of the “nested” cross validation approach
![Page 36: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/36.jpg)
Do hidden neurons matter?
• The environment matters
NetMHCpan
![Page 37: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/37.jpg)
Context matters
• FMIDWILDA YFAMYGEKVAHTHVDTLYVRYHYYTWAVLAYTWY 0.89 A0201• FMIDWILDA YFAMYQENMAHTDANTLYIIYRDYTWVARVYRGY 0.08 A0101• DSDGSFFLY YFAMYGEKVAHTHVDTLYVRYHYYTWAVLAYTWY 0.08 A0201• DSDGSFFLY YFAMYQENMAHTDANTLYIIYRDYTWVARVYRGY 0.85 A0101
![Page 38: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/38.jpg)
Example
![Page 39: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/39.jpg)
Summary
• Gradient decent is used to determine the updates for the synapses in the neural network
• Some relatively simple math defines the gradients– Networks without hidden layers can be solved on
the back of an envelope (SMM exercise)– Hidden layers are a bit more complex, but still ok
• Always train networks using a test set to stop training– Be careful when reporting predictive performance
• Use “nested” cross-validation for small data sets
• And hidden neurons do matter (sometimes)
![Page 40: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/40.jpg)
And some more stuff for the long cold winter nights
• Can it might be made differently?
![Page 41: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/41.jpg)
Predicting accuracy
• Can it be made differently?
Reliability
![Page 42: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/42.jpg)
• Identification of position specific receptor ligand interactions by use of artificial neural network decomposition. An investigation of interactions in the MHC:peptide system
Master thesis by Frederik Otzen Bagger and Piotr Chmura
Making sense of ANN weights
![Page 43: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/43.jpg)
Making sense of ANN weights
![Page 44: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/44.jpg)
Making sense of ANN weights
![Page 45: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/45.jpg)
Making sense of ANN weights
![Page 46: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/46.jpg)
Making sense of ANN weights
![Page 47: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/47.jpg)
Making sense of ANN weights
![Page 48: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/48.jpg)
Making sense of ANN weights
![Page 49: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/49.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 50: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/50.jpg)
Deep(er) Network architecture
![Page 51: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/51.jpg)
Deeper Network architecture
Il Input layer, l
1. Hidden layer, k
Output layer
h1k
H1k
h3
H3
ukl
wj
2. Hidden layer, j
vjk
h2j
H2j
![Page 52: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/52.jpg)
Network architecture (hidden to hidden)
![Page 53: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/53.jpg)
Network architecture (input to hidden)
![Page 54: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/54.jpg)
Network architecture (input to hidden)
![Page 55: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/55.jpg)
Speed. Use delta’s
j
k
l
dj
vjk
ukl
dk
Bishop, Christopher (1995). Neural networks for pattern recognition. Oxford: Clarendon Press. ISBN 0-19-853864-2.
![Page 56: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/56.jpg)
Use delta’s
![Page 57: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/57.jpg)
Deep learning – time is not an issue
17000 17500 18000 18500 19000 19500 200000
50
100
150
200
250
Number of weights
CP
U (
u)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.516000
16500
17000
17500
18000
18500
19000
19500
N layer
# w
eig
hts
![Page 58: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/58.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 59: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/59.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 60: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/60.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 61: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/61.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 62: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/62.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
Auto encoder
![Page 63: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/63.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 64: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/64.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 65: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/65.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 66: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/66.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 67: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/67.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 68: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/68.jpg)
Pan-specific prediction methods
NetMHC NetMHCpan
![Page 69: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/69.jpg)
ExamplePeptide Amino acids of HLA pockets HLA Aff VVLQQHSIA YFAVLTWYGEKVHTHVDTLVRYHY A0201 0.131751SQVSFQQPL YFAVLTWYGEKVHTHVDTLVRYHY A0201 0.487500SQCQAIHNV YFAVLTWYGEKVHTHVDTLVRYHY A0201 0.364186LQQSTYQLV YFAVLTWYGEKVHTHVDTLVRYHY A0201 0.582749LQPFLQPQL YFAVLTWYGEKVHTHVDTLVRYHY A0201 0.206700VLAGLLGNV YFAVLTWYGEKVHTHVDTLVRYHY A0201 0.727865VLAGLLGNV YFAVWTWYGEKVHTHVDTLLRYHY A0202 0.706274VLAGLLGNV YFAEWTWYGEKVHTHVDTLVRYHY A0203 1.000000VLAGLLGNV YYAVLTWYGEKVHTHVDTLVRYHY A0206 0.682619VLAGLLGNV YYAVWTWYRNNVQTDVDTLIRYHY A6802 0.407855
![Page 70: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/70.jpg)
Going Deep – One hidden layer
0 50 100 150 200 250 300 350 400 450 5000
0.02
0.04
0.06
0.08
20
# Ietrations
MSE
0 50 100 150 200 250 300 350 400 450 5000
0.010.020.030.040.050.060.07
# Iterations
MSE
0 50 100 150 200 250 300 350 400 450 5000
0.2
0.4
0.6
0.8
1
# Iterations
PC
C
Train
Test
Test
![Page 71: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/71.jpg)
Going Deep – 3 hidden layers
0 50 100 150 200 250 300 350 400 450 5000
0.05
0.1
20 20+20 20+20+20
# Ietrations
MSE
0 50 100 150 200 250 300 350 400 450 5000
0.020.040.060.080.1
# Iterations
MSE
0 50 100 150 200 250 300 350 400 450 5000
0.2
0.4
0.6
0.8
1
# Iterations
PC
C
Train
Test
Test
![Page 72: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/72.jpg)
Going Deep – more than 3 hidden layers
0 50 100 150 200 250 300 350 400 450 5000
0.02
0.04
0.06
0.08
0.1
20 20+20 20+20+20 20+20+20+20 20+20+20+20+20
# Ietrations
MSE
0 50 100 150 200 250 300 350 400 450 5000
0.05
0.1
# Iterations
MSE
0 50 100 150 200 250 300 350 400 450 5000
0.5
1
# Iterations
PC
C
Train
Test
Test
![Page 73: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/73.jpg)
Going Deep – Using Auto-encoders
0 50 100 150 200 250 300 350 400 450 5000
0.05
0.1
20 20+20 20+20+2020+20+20+20 20+20+20+20+20 20+20+20+20+Auto
# Ietrations
MSE
0 50 100 150 200 250 300 350 400 450 5000
0.02
0.04
0.06
0.08
0.1
# Iterations
MSE
0 50 100 150 200 250 300 350 400 450 5000
0.5
1
# Iterations
PC
C
Train
Test
Test
![Page 74: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/74.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 75: Artificiel Neural Networks 2 Morten Nielsen Department of Systems Biology, DTU](https://reader036.vdocuments.site/reader036/viewer/2022062423/5697c0041a28abf838cc49f2/html5/thumbnails/75.jpg)
Conclusions -2
• Implementing Deep networks using deltas1 makes CPU time scale linearly with respect to the number of weights – So going Deep is not more CPU intensive
than going wide• Back-propagation is an efficient
method for NN training for shallow networks with up to 3 hidden layers
• For deeper network, pre-training is required using for instance Auto-encoders
1Bishop, Christopher (1995). Neural networks for pattern recognition. Oxford: Clarendon Press. ISBN 0-19-853864-2.