back propagation learning algorithm

Lecture 5

Introduction to Neural Networksand Fuzzy Logic

President University Erwin Sitompul NNFL 5/1

Dr.-Ing. Erwin SitompulPresident University

http://zitompul.wordpress.com

2 0 1 4


Back Propagation Learning AlgorithmNeural Networks

Backwardpropagation

f(.)

f(.)

f(.)

• Set the weights• Calculate output

1

1

( )( ) ( ) ( )

pl l lk k jl

ikj

Ei f net i y i

w

w

1 21

1

( )( ) ( ) ( ) ( )

pl l l l lk k kj j il

iji

Ei f net i w f net i y i

w

w

Forwardpropagation

1( ), ( )l lj ky i y i

• Calculate error• Calculate gradient

vector

• Update the weights

1

1

( ) ( )

( ) ( )

l lk k

ml l lk kj j

j

y i f net i

net i w x i

1 1

1 1 2

1

( ) ( )

( ) ( )

l lj j

nl l lj ji i

i

y i f net i

net i w y i

1

( ) ( ),

l lkj ji

E E

w w

w w

( )lk i

Multi Layer Perceptrons


Learning with Momentum In an effort to speed up the learning process, a weight

update is made to be based on the previous weight update. This is called momentum, because it tends to keep the error rolls down the error surface.

Because many updates to a particular weight are in the same direction, adding momentum will typically result in a speed up of learning time in many applications.

Multi Layer PerceptronsNeural Networks

When using momentum, the update rule for the network weights is modified to be:

( )( ) ( 1)kj kj

kj

Ew n w n

w

w

where α (typically 0.05) is the momentum and n is the number of iteration.


Learning with MomentumNeural Networks Multi Layer Perceptrons

The momentum can be seen to be practically increasing the learning rate.

This is in accordance with several heuristics that should be used in neural network design to improve the result:Each weight should have a local learning rateEach learning rate should be allowed to vary over

timeConsecutive updates with the same sign to a weight

should increase the weights’ learning rateConsecutive updates with alternating sign to a

weight should decreases the weights’ learning rate.


Learning with Weighted MomentumNeural Networks Multi Layer Perceptrons

Various efforts have focused on deriving additional forms of momentum.

One such method is to relate the momentum of the previous update and the current calculation of error gradient.

By doing so, the momentum can be preserved when an iteration attempts to update contrary to the most recent updates.

The update rule for the network weights in this case is given by:

( )( ) (1 ) ( 1)kj kj

kj

Ew n w n

w

w


Learning with Variable Learning RateNeural Networks Multi Layer Perceptrons

The basic idea: speed up the convergence by increasing the learning rate on flat surface and decreasing it when the slop increases.

If error increases by more than a predefined percentage θ (i.e. 1-5%) then: Weight update is discarded Learning rate is decreased by a factor 0< γ <1, i.e. γ = 0.7 Set momentum to zero

If error increases by less than θ: Weight update is accepted Learning rate is unchanged If momentum has been set to zero, it is reset to original value

If the error decreases: Weight update is accepted Learning rate is increased by some factor β >1, i.e. β = 1.05 If momentum has been set to zero, it is reset to its original

value


Feedforward Network

InputNeuronLayer

NeuronLayer

Output

f(.)

f(.)

f(.)

MLP for System ModelingNeural Networks


Feedforward Network

01 2 1 0 2 0y

02 3 1 3 1 0y

21 17 3 9 9 0d

01 ( )y i

02 ( )y i

21 ( )y i



Recurrent NetworksExternal Recurrence

Internal Recurrence

Input NeuronLayer

NeuronLayer

Output

Time Delay

Element

Time Delay

Element

Input NeuronLayer

NeuronLayer

Output

Time Delay

Element



InputDynamicSystem

Output

( )u k ( )y k

Dynamic System

( ) ( , )y k m g

a b( 1), , ( ), ( 1), , ( )y k y k n u k u k n g

System parameter

Input-output data vector



InputDynamic

Model

Output

( )u k ˆ( )y k

Dynamic Model

ˆ( ) ( , , )y k w b g

a b( 1), , ( ), ( 1), , ( )y k y k n u k u k n g

weightsbias

input-output data vector



Neural Network Dynamic Model

Feedforward

ˆ( )y k : model output,estimate of system output

( )y k : system output. . .

. . .

. . .

ˆ( )y k

. . .

( 1)u k

b( )u k n

( 1)y k

a( )y k n



Neural Network Dynamic Model

Recurrent

. . .

. . .

. . .

ˆ( )y k

. . .

( )u k

1z

anz

1z

bnz



Tapped Delay Line (TDL)

( )u k

( 1)u k ( 2)u k

( 3)u k ( )u k n

1z 1z 1z 1z .....

( )u k

( 1)u k ( )u k n

T D L

.....


Unit 1 Unit 2 Unit 3 Unit n


Implementation

InputDynamicSystem

Output

( )u k ( )y k

ˆ( )y k. . .

. . .

T D L T D L

feedforward

external recurrence



ExampleSingle Tank System

2

20.4 m0.012 m

Aa

outq

inq

hLearning Data Generation

A : cross-sectional area of the tanka : cross-sectional area of the pipe

Area of operation

Save data to workspace


in a

1 ah q v

A A

in

12

ah q gh

A A


Example

( 1)u k ( )y k

( 1)y k

Data size : 201 from 200 seconds of

simulation

0 20 40 60 80 100 120 140 160 180 2000

0.02

0.04

0.06

0.08

0.1

0.12

0 20 40 60 80 100 120 140 160 180 2000

0.02

0.04

0.06

0.08

0.1

0.12

Feedforward Network External Recurrent Network


2–2–1 Network


Homework 5

( 1)u k

y k( 2)u k

( 1)y k

( 2)y k

0 20 40 60 80 100 120 140 160 180 200-0.025

-0.02

-0.015

-0.01

-0.005

0

0.005

0.01

0.015

0.02

Delta of 2–2–1 network

4–4–1 Network


A neural network with 2 inputs and 2 hidden neurons seems not to be good enough to model the Single Tank System. Now, design a neural network with 4 inputs and 4 hidden neurons to model the system. Use bias in all neurons and take all a = 1.

Be sure to obtain decreasing errors.

Submit the hardcopy and softcopy of the m-file.


Homework 5A (smaller Student-ID)

( 1)u k

y k( 3)u k

( 1)y k

( 4)y k 4–4–1 Network


For 4 students with smaller Student ID, redo Homework 5 with the following changes:The network inputs are u(k–1), u(k–3), y(k–1), and y(k–4)

The activation functions for the neurons in the hidden layer are:


Compare the result with the previous result of HW5.



Homework 5B (greater Student-ID)MLP for System ModelingNeural Networks


Compare the result with the previous result of HW5.


( 1)u k

( )y k( 1)y k

( 2)y k

( 3)y k 4–4–1 Network

For 4 students with greater Student ID, redo Homework 5 with the following changes:The network inputs are u(k–1), y(k–1), y(k–2), and y(k–3)

The activation functions for the neurons in the hidden layer are:

back propagation learning algorithm

Documents