parallel self-adaptive parallel processing neural networks with irregular nodal processing powers...

8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks

1/33


2/33

Department of Information Technology

Indian Institute of Information Technology and Management Gwalior Rahul Kala

Problems with Back PropagationProblems with Back Propagation Algorithm Algorithm

Increasing training data

Increasing dimensionality

Increasing Problem Complexity

Limited Computational Power

No Loss of Generality D esired


3/33



The Basic IdeaThe Basic Idea

Divide the computationof Back Propagation Algorithm into many Processing Elements

for higher speed


4/33



Motivations from HumanMotivations from HumanBrainBrain

Parallel Architecture Learning Evolution

Adaptation ComputationalDisparity

Flexibility


5/33

Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala

Basic Partitioning TechniquesBasic Partitioning Techniques

Basic PartitioningTechniques

Layer

D atasetNode


6/33


D ataset PartitioningD ataset Partitioningy D ivide data into PEs

y Each PE trains its set of data

y Weights are exchanged andaggregated


7/33


Basic TechniqueBasic Technique

PE 3 PE NPE 1 PE 2

PE 3 PE NPE 1 PE 2

Data, ANN

Weights


8/33


CommunicationCommunication

PE 3

PE 4

PE N

PE 1

PE 2


9/33


Layer PartitioningLayer Partitioning - -11y D ivide layers into PEs

y Each PE performs its part of computation

y D ifferent PEs store different weights


10/33


Layer PartitioningLayer Partitioning - -IIIIy Feed forward Basic Equation

y Feed back


11/33



PE 2PE 2 PE NPE NPE 1PE 1

i1i1

i2i2

inin


12/33



L ayer 3L ayer 2L ayer 1 L ayer N


13/33


WorkingWorking

T1

T2

T N

T1

T2

T N

T1

T2

T N

L ayer 1 L ayer 2 L ayer N

Tasks


14/33


Node PartitioningNode Partitioning - - IIy D ivide nodes into PEs

y Each PE performs its part of computation

y D ifferent PEs store different weights


15/33


Node PartitioningNode Partitioning - -IIIIy Feed forward

BasicEquation

y Feed Back


16/33



PE 2

PE N

PE 1i1

i2

in


17/33



PE 1

PE 2

PE N

PE 3


18/33


WorkingWorking

T1

T2

T N

T1

T2

T N

T1

T2

T N

L ayer 1 L ayer 2 L ayer N

Tasks


19/33


Hierarchical PartitioningHierarchical Partitioningy Mixture of three partitioning at different

levels

y Level 1: D ata Sety Level 2: Node or Layer


20/33


General techniqueGeneral technique

Dataset1

L ayer 2

L ayer 3

L ayer 1 L ayer 2

L ayer 3

Node 1 Node 2

Node 3

Node 1

Node 2 Node 3

L ayer 1

Node 3 Node 2

Node 1

Dataset2

Dataset3

Dataset

4

Dataset N


21/33


Self AdaptationSelf Adaptationy Balance computational load among

PEs as per their capability

y Works for data set partitioning

y Computation reallocated with some

frequency


22/33


Self Adaptation Server/ClientSelf Adaptation Server/ClientModelModel

PE 4

PE N

PE 3

PE 1 PE 2Server


23/33


CommunicationsCommunications

Node3

Node2

L ayer3

L ayer2

Node1L ayer1

Server DataSet 1

DataSet 2

DataSet 3

L ayer3

L ayer2

DataSet N

L ayer1

Node3

Node2

Node1


24/33

RESULTSRESULTS


25/33


Speedup v/s No of PEs (InputSpeedup v/s No of PEs (Input1)1)

0

1

2

3

4

5

6

7

8

9

10

2 4 6 8 10 20

Speedup v/s No of PEs


26/33


Speedups for Input 1Speedups for Input 1

PENo of inputs

Network

Architectur e Iterations

Server

Sync after iterations Time Serial

TimeParallel Speedup

2 500 8-15-1 150000 50000 823737 982521 0.838391

4 500 8-15-1 150000 50000 823737 530306 1.5533246 500 8-15-1 150000 50000 823737 384294 2.143507

8 500 8-15-1 150000 50000 823737 263511 3.126006

10 500 8-15-1 150000 50000 823737 211249 3.899365

20 500 8-15-1 150000 50000 823737 89225 9.232132


27/33


Speedup v/s No of PEs (Input 1)Speedup v/s No of PEs (Input 1)without self adaptive approachwithout self adaptive approach

0

1

2

3

4

5

6

7

8

9

10

2 4 6 8 10 20



28/33


Speedups for Input 1 without self Speedups for Input 1 without self adaptationadaptation

PENo of inputs

Network


Server



2 500 8-15-1 150000 NA 823737 993483 0.829141

4 500 8-15-1 150000 NA 823737 674358 1.2215136 500 8-15-1 150000 NA 823737 473899 1.738212

8 500 8-15-1 150000 NA 823737 342174 2.407363

10 500 8-15-1 150000 NA 823737 303214 2.716685

20 500 8-15-1 150000 NA 823737 97180 8.476405


29/33



Speedup v/s No of PEs (InputSpeedup v/s No of PEs (Input1I)1I)

0

1

2

3

4

5

6

7

8

9

10

2 4 6 8 10 20



30/33



Speedups for Input 1ISpeedups for Input 1I

PENo of inputs

Network


Server



2 500 11-25-1 150000 50000 1733690 1829312 0.947728

4 500 11-25-1 150000 50000 1733690 984654 1.760716 500 11-25-1 150000 50000 1733690 644070 2.691773

8 500 11-25-1 150000 50000 1733690 486311 3.564982

10 500 11-25-1 150000 50000 1733690 429221 4.039155

20 500 11-25-1 150000 50000 1733690 179378 9.665009


31/33



Speedup v/s No of PEs (InputSpeedup v/s No of PEs (Input

1I) without self adaptive1I) without self adaptiveapproachapproach

0

1

2

3

4

5

6

7

8

9

10

2 4 6 8 10 20



32/33



Speedups for Input 1I withoutSpeedups for Input 1I withoutself adaptationself adaptation

PENo of inputs

Network


Server



2 500 11-25-1 150000 NA 1733690 1911289 0.907079

4 500 11-25-1 150000 NA 1733690 1175223 1.4752016 500 11-25-1 150000 NA 1733690 683619 2.536047

8 500 11-25-1 150000 NA 1733690 548465 3.160986

10 500 11-25-1 150000 NA 1733690 471293 3.678582

20 500 11-25-1 150000 NA 1733690 196229 8.835035


33/33



parallel self-adaptive parallel processing neural networks with irregular nodal processing powers...

Documents