artificial neural networks 1 morten nielsen department of systems biology, dtu · 2014. 6. 17. ·...
TRANSCRIPT
![Page 1: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/1.jpg)
Artificial Neural Networks 1
Morten Nielsen Department of Systems Biology,
DTU
![Page 2: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/2.jpg)
Biological Neural network
![Page 3: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/3.jpg)
Biological neuron structure
![Page 4: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/4.jpg)
Artificial neural networks. Background
![Page 5: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/5.jpg)
Higher order sequence correlations
• Neural networks can learn higher order correlations! – What does this mean?
S S => 0 L S => 1 S L => 1 L L => 0
Say that the peptide needs one and only one large amino acid in the positions P3 and P4 to fill the binding cleft How would you formulate this to test if a peptide can bind?
=> XOR function
![Page 6: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/6.jpg)
Neural networks
• Neural networks can learn higher order correlations
XOR function: 0 0 => 0 1 0 => 1 0 1 => 1 1 1 => 0
(1,1)
(1,0) (0,0)
(0,1)
No linear function can separate the points OR
AND
XOR
![Page 7: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/7.jpg)
Error estimates
XOR 0 0 => 0 1 0 => 1 0 1 => 1 1 1 => 0
(1,1)
(1,0) (0,0)
(0,1) Predict 0 1 1 1
Error 0 0 0 1
Mean error: 1/4
![Page 8: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/8.jpg)
Linear methods and the XOR function
v1 v2
Linear function
€
y = x1 ⋅ v1 + x2 ⋅ v2
![Page 9: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/9.jpg)
Neural networks with a hidden layer
w12
v1
w21 w22
v2
1
wt2 wt1 w11
1
vt
Input 1 (Bias) {
€
O =1
1+ exp(−o)
o = xii=1
N
∑ ⋅wi + t = xii=1
N+1
∑ ⋅wi
xN =1
![Page 10: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/10.jpg)
How does it work? Ex. Input is (0 0)
0 0 6
-9
4 6
9
1
-2 -6 4
1
-4.5
Input 1 (Bias) {o1=-6 O1=0
o2=-2 O2=0
y1=-4.5 Y1=0
€
o = xi∑ ⋅ wi
![Page 11: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/11.jpg)
![Page 12: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/12.jpg)
Gradient decent (from wekipedia)
Gradient descent is based on the observation that if the real-valued function F(x) is defined and differentiable in a neighborhood of a point a, then F(x) decreases fastest if one goes from a in the direction of the negative gradient of F at a. It follows that, if for ε > 0 a small enough number, then F(b)<F(a)
€
b = a −ε ⋅ ∇F(a)
![Page 13: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/13.jpg)
Gradient decent. Example
Weights are changed in the opposite direction of the gradient of the error
wi' = wi +Δwi
E = 12 ⋅ (O− t)
2
O = wii∑ ⋅ Ii
Δwi = −ε ⋅∂E∂wi
= −ε ⋅∂E∂O
⋅∂O∂wi
= −ε ⋅ (O− t) ⋅ Ii
I1 I2
w1 w2
Linear function
€
O = I1 ⋅ w1 + I2 ⋅ w2
O
![Page 14: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/14.jpg)
ANNs - Hidden to output layer
∂E∂wj
=∂E∂O
⋅∂O∂o
⋅∂o∂wj
∂E(O(o(wj )))∂wj
=
E = 12 ⋅ (O− t)
2
O = g(o)
o = wii∑ ⋅ Ii
![Page 15: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/15.jpg)
Hidden to output layer
∂E∂wj
=∂E∂O
⋅∂O∂o
⋅∂o∂wj
= (O− t) ⋅ g '(o) ⋅H j
∂E∂O
= (O− t)
∂O∂o
=∂g∂o
= g '(o)
∂o∂wj
=1∂wj
wj ⋅l∑ H j = H j
o = wii∑ ⋅H j
![Page 16: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/16.jpg)
What about the hidden layer?
Δvjk = −ε ⋅∂E∂vjk
o = wjj∑ ⋅H j
hj = vjkk∑ ⋅ Ik
€
E = 12 ⋅ (O− t)
2
O = g(o),H = g(h)
g(x) =1
1+ e−x
![Page 17: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/17.jpg)
Input to hidden layer
∂E∂vjk
=∂E(O(o(H j (hj (vjk ))))
∂vjk
=∂E∂O
⋅∂O∂o
⋅∂o∂H j
⋅∂H j
∂hj⋅∂hj∂vjk
= (O− t) ⋅ g '(o) ⋅wj ⋅ g '(hj ) ⋅ Ik
![Page 18: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/18.jpg)
Summary
∂E∂wj
= (O− t) ⋅ g '(o) ⋅H j
∂E∂vjk
= (O− t) ⋅ g '(o) ⋅wj ⋅ g '(hj ) ⋅ Ik
![Page 19: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/19.jpg)
Or
∂E∂wj
= (O− t) ⋅ g '(o) ⋅H j = δ ⋅H j
∂E∂vjk
= (O− t) ⋅ g '(o) ⋅wj ⋅ g '(hj ) ⋅ Ik = δ ⋅wj ⋅ g '(hj ) ⋅ Ik
δ = (O− t) ⋅ g '(o)
![Page 20: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/20.jpg)
Neural networks and the XOR function
![Page 21: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/21.jpg)
Deep(er) Network architecture
€
E = 12 ⋅ (O− t)
2
O = g(o),H = g(h)
g(x) =1
1+ e−x
o = wjj∑ ⋅H
j
2
hj
2 = vjkk∑ ⋅H
k
1
hk
1 = ukll∑ ⋅ Il
Δwi = −ε ⋅∂E∂wi
![Page 22: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/22.jpg)
Deeper Network architecture Il Input layer, l
1. Hidden layer, k
Output layer
h1k
H1k
h3
H3
ukl
wj
2. Hidden layer, j
vjk
h2j
H2j
∂E∂wj
=∂E(H 3(h3(wj )))
∂wj
=∂E∂H 3 ⋅
∂H 3
∂h3⋅∂h3
∂wj
= (H 3 − t) ⋅ g '(h3) ⋅H j2
![Page 23: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/23.jpg)
Network architecture (hidden to hidden)
∂E∂vjk
=∂E∂H 3 ⋅
∂H 3
∂h3⋅∂h3
∂Hj
2 ⋅∂H
j
2
∂hj
2 ⋅∂h
j
2
∂vjk= (H 3 − t) ⋅ g '(h3) ⋅wj ⋅ g '(hj
2 ) ⋅Hk1
![Page 24: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/24.jpg)
Network architecture (input to hidden)
∂E∂ukl
=∂E∂H 3 ⋅
∂H 3
∂h3⋅
∂h3
∂Hj
2 ⋅∂H
j
2
∂hj
2 ⋅∂h
j
2
∂Hk1
j∑ ⋅
∂Hk1
∂hk1 ⋅∂h
k
1
∂ukl
= (H 3 − t) ⋅ g '(h3) ⋅ wj ⋅ g '(hj
2 ) ⋅ vjkj∑ ⋅ g '(h
k
1 ) ⋅ Il
![Page 25: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/25.jpg)
Network architecture (input to hidden)
∂E∂ukl
=∂E∂H 3 ⋅
∂H 3
∂h3⋅
∂h3
∂Hj
2 ⋅∂H
j
2
∂hj
2 ⋅∂h
j
2
∂Hk1
j∑ ⋅
∂Hk1
∂hk1 ⋅∂h
k
1
∂ukl
= (H 3 − t) ⋅ g '(h3) ⋅ wj ⋅ g '(hj
2 ) ⋅ vjkj∑ ⋅ g '(h
k
1 ) ⋅ Il
![Page 26: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/26.jpg)
Use delta’s
hj
q = wjii∑ H
i
q−1
Hi
q−1 = g(hi
q−1)
j
k
l
δj
vjk
ukl
δk
Bishop, Christopher (1995). Neural networks for pattern recognition. Oxford: Clarendon Press. ISBN 0-19-853864-2.
![Page 27: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/27.jpg)
Use delta’s
∂E∂w
ji
q =∂E∂h
j
q ⋅∂hj
q
∂wji
q = δ j
q ⋅Hi
q−1
δj
q =∂E∂h
j
q
δ3 =∂E∂h3
=∂E∂H 3 ⋅
∂H 3
∂h3= (H 3 − t) ⋅ g '(h3)
δ j2 =
∂E∂h
j
2 =∂E∂h3
⋅∂h3
∂hj2 =
∂E∂h3
⋅∂h3
∂H j2 ⋅∂H j
2
∂hj2 = g '(hj
2 ) ⋅δ3 ⋅ vjk
δk1 =
∂E∂hk
1 =∂E∂hj
2j∑ ⋅
∂hj2
∂hk1 =
∂E∂hj
2j∑ ⋅
∂hj2
∂Hk1 ⋅∂Hk
1
∂hk1 = g '(hk
1) ⋅ δ j2
j∑ ⋅ vjk
hj = wjii∑ Hi
Hi = g(hi )
![Page 28: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/28.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 29: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/29.jpg)
Deep learning – time is not an issue
0
50
100
150
200
250
17000 17500 18000 18500 19000 19500 20000
CPU (u)
Number of weights
17000
17500
18000
18500
19000
19500
0 1 2 3 4 5 6
# w
eigh
ts
N layer
![Page 30: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/30.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 31: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/31.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 32: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/32.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 33: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/33.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 34: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/34.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
Auto encoder
![Page 35: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/35.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 36: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/36.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 37: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/37.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 38: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/38.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 39: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/39.jpg)
Deep learning
http://www.slideshare.net/hammawan/deep-neural-networks
![Page 40: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/40.jpg)
Pan-specific prediction methods
NetMHC NetMHCpan
![Page 41: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/41.jpg)
Example Peptide Amino acids of HLA pockets HLA Aff VVLQQHSIA YFAVLTWYGEKVHTHVDTLVRYHY A0201 0.131751 SQVSFQQPL YFAVLTWYGEKVHTHVDTLVRYHY A0201 0.487500 SQCQAIHNV YFAVLTWYGEKVHTHVDTLVRYHY A0201 0.364186 LQQSTYQLV YFAVLTWYGEKVHTHVDTLVRYHY A0201 0.582749 LQPFLQPQL YFAVLTWYGEKVHTHVDTLVRYHY A0201 0.206700 VLAGLLGNV YFAVLTWYGEKVHTHVDTLVRYHY A0201 0.727865 VLAGLLGNV YFAVWTWYGEKVHTHVDTLLRYHY A0202 0.706274 VLAGLLGNV YFAEWTWYGEKVHTHVDTLVRYHY A0203 1.000000 VLAGLLGNV YYAVLTWYGEKVHTHVDTLVRYHY A0206 0.682619 VLAGLLGNV YYAVWTWYRNNVQTDVDTLIRYHY A6802 0.407855
![Page 42: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/42.jpg)
Going Deep – One hidden layer
0
0.02
0.04
0.06
0.08
0 50 100 150 200 250 300 350 400 450 500
MSE
# Ietrations
20
0
0.02
0.04
0.06
0.08
0 50 100 150 200 250 300 350 400 450 500
MSE
# Iterations
0
0.2
0.4
0.6
0.8
1
0 50 100 150 200 250 300 350 400 450 500
PCC
# Iterations
Train
Test
Test
![Page 43: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/43.jpg)
Going Deep – 3 hidden layers
0
0.05
0.1
0 50 100 150 200 250 300 350 400 450 500
MSE
# Ietrations
20 20+20 20+20+20
0
0.05
0.1
0 50 100 150 200 250 300 350 400 450 500
MSE
# Iterations
0 0.2 0.4 0.6 0.8
1
0 50 100 150 200 250 300 350 400 450 500
PCC
# Iterations
Train
Test
Test
![Page 44: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/44.jpg)
Going Deep – more than 3 hidden layers
0 0.02 0.04 0.06 0.08
0.1
0 50 100 150 200 250 300 350 400 450 500
MSE
# Ietrations
20 20+20 20+20+20 20+20+20+20 20+20+20+20+20
0
0.05
0.1
0 50 100 150 200 250 300 350 400 450 500
MSE
# Iterations
0
0.5
1
0 50 100 150 200 250 300 350 400 450 500
PCC
# Iterations
Train
Test
Test
![Page 45: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/45.jpg)
Going Deep – Using Auto-encoders
0
0.05
0.1
0 50 100 150 200 250 300 350 400 450 500
MSE
# Ietrations
20 20+20 20+20+20 20+20+20+20 20+20+20+20+20 20+20+20+20+Auto
0 0.02 0.04 0.06 0.08
0.1
0 50 100 150 200 250 300 350 400 450 500
MSE
# Iterations
0
0.5
1
0 50 100 150 200 250 300 350 400 450 500
PCC
# Iterations
Train
Test
Test
![Page 46: Artificial Neural Networks 1 Morten Nielsen Department of Systems Biology, DTU · 2014. 6. 17. · DTU. Biological Neural network . Biological neuron structure ... L S => 1! S L =>](https://reader036.vdocuments.site/reader036/viewer/2022070212/6105553f51f7ac6c6a5de53a/html5/thumbnails/46.jpg)
Conclusions
• Implementing Deep networks using deltas1 makes CPU time scale linearly with respect to the number of weights – So going Deep is not more CPU intensive than
going wide • Back-propagation is an efficient method
for NN training for shallow networks with up to 3 hidden layers
• For deeper network, pre-training is required using for instance Auto-encoders
1Bishop, Christopher (1995). Neural networks for pattern recognition. Oxford: Clarendon Press. ISBN 0-19-853864-2.