Project: Back Propagation Algorithm

2016. 11. 9

• Data prepare

• Cell / Struct

• Feed Forward

• Training schema

• Back Propagation / Weight update

• Result

• Load data

Data prepare

Premable

Clear all data

Load form

‘mnist_uint8_matlab.mat’

MNIST dataset (each row means 1 data)

Test data: 10000

Trainig data: 50000

Input size: 28 × 28 size image

Output size: 10 × 1 size label

• Input & output data

Data prepare

Output

Data

Input

Data

• Input & output data• Test_x(10000 × 784) → one data (1 × 784) → reshape (28 × 28)

Data prepare

2 1 9

0 1 2 3 4 5 6 7 8 9

• Data type of Matlab

Cell / Struct

Most used types Included in the code

• Cell• Array of indexed cells, each capable of storing array of differen

dimension and data type

• Struct• C-like structure, named fields capable of storing of different

dimension and data type

Cell / Struct

𝑎. 𝑛𝑢𝑚 = 12; 𝑎. 𝑐𝑜𝑙𝑜𝑟 =′ 𝑟𝑒𝑑′; 𝑎.𝑚𝑎𝑡 = 𝑜𝑛𝑒𝑠 4 ;

𝑎 = 𝑐𝑒𝑙𝑙 1,3 ; 𝑎 1,1 = 10; 𝑎 1,2 =′ 𝑎𝑏𝑐′ 𝑎 1,3 = 𝑜𝑛𝑒𝑠 4 ;

• 3 struct: init, training, net• init: for initialize weight and bias

• training: for training schema

• net: all information of network (num of layer&neuron, layer, weight, bias …)

Cell / Struct

• Toy example• 3 layer (2, 3, 2 neurons) network

Cell / Struct

• initialize_network.m

Cell / Struct

3 layer

neuron number: [784, 10, 10]

3 layer

neuron number: [2, 3, 2]

Initialize each layer

by zero

net.layer

𝟐 × 𝟏

𝟑 × 𝟏

𝟐 × 𝟏

• initialize_network.m

Cell / Struct

net.weight

[ ]

𝟑 × 𝟐

𝟐 × 𝟑

net.bias

[ ]

𝟑 × 𝟏

𝟐 × 𝟏

• initialize_network.m

Cell / Struct

𝒏𝟏𝟏

𝒏𝟐𝟏

𝒏𝟏𝟐

𝒏𝟐𝟐

𝒏𝟑𝟐

𝒏𝟏𝟑

𝒏𝟐𝟑

net.layer

𝟐 × 𝟏

𝟑 × 𝟏

𝟐 × 𝟏

net.weight

[ ]

𝟑 × 𝟐

𝟐 × 𝟑

net.bias

[ ]

𝟑 × 𝟏

𝟐 × 𝟏

𝑛11

𝑛21

𝑛12

𝑛22

𝑛32

𝑤1,12

𝑤2,12

𝑤3,12

𝑤1,12 𝑤1,2

2

𝑤2,12

𝑤3,12

𝑤2,22

𝑤3,22

𝑏12

𝑏22

𝑏32

𝑏12

𝑏22

𝑏32

• feed_forward.m

Feed foward

𝑛12

𝑛22

𝑛32

= logistic

𝑤1,12 𝑤1,2

2

𝑤2,12

𝑤3,12

𝑤2,22

𝑤3,22

𝑛11

𝑛21 +

𝑏12

𝑏22

𝑏32

𝒏𝟏𝟏

𝒏𝟐𝟏

𝒏𝟏𝟐

𝒏𝟐𝟐

𝒏𝟑𝟐

𝒏𝟏𝟑

𝒏𝟐𝟑

𝑛13

𝑛23 = logistic

𝑤1,13 𝑤1,2

3 𝑤1,33

𝑤2,13 𝑤2,2

3 𝑤2,33

𝑛12

𝑛22

𝑛32

+𝑏13

𝑏23

• training

Training schema

• back_propagation.m• Calculate error in each layer & neuron

• Calculate delta in each layer & neuron

• weight_update.m• Update weight and bias of each neuron by using delta

Back Propagation / Weight update

𝒏𝟏𝟏

𝒏𝟐𝟏

𝒏𝟏𝟐

𝒏𝟐𝟐

𝒏𝟑𝟐

𝒏𝟏𝟑

𝒏𝟐𝟑

• Result file• my_result.mat

• performance.fig

• performance.png

Result