a hierarchical self-organizing associative memory for machine
TRANSCRIPT
Fourth International Symposium on Neural Networks (ISNN) June 3-7, 2007, Nanjing, China
A Hierarchical Self-organizing Associative Memory forMachine Learning
Janusz A. Starzyk, Ohio UniversityHaibo He, Stevens Institute of Technology
Yue Li, O2 Micro Inc
2/23
Outline
Introduction;
Associative learning algorithm;
Memory network architecture and operation;
Simulation analysis;
Conclusion and future research;
3/23
Introduction: A biological point of view
Source: “The computational brain” by
P. S. Churchland and T. J. Sejnowski
Memory is a critical component for understanding and developing natural intelligent machines/systems
The question is: How???
4/23
Introduction: self-organizing learning array(SOLAR)
Characteristics:
* Self-organization
* Sparse and local interconnections
* Dynamically reconfigurable
* Online data-driven learning
Other Neurons
Nearest neighbour neuron
Remote neurons System clock
ID: information deficiency
II: information index
5/23
Introduction: from SOLAR to AM
Characteristics: Self-organization; Sparse and local interconnections; Feedback propagation; Information inference; Hierarchical organization; Robust and self-adaptive; Capable of both hetero-associative (HA) and auto-associative (AA)
Feed forward only Feed forward
Feed backward
6/23
Outline
Introduction;
Associative learning algorithm;
Memory network architecture and operation;
Simulation analysis;
Conclusion and future research;
7/23
Basic learning element
Self-determination of the function value:
An example:
8/23
Signal strength (SS)
Signal strength (SS) =| Signal value – logic threshold|
(SS range: [0, 1])
Provides a coherent way to determine when to trigger an association; Helps to resolve multiple feedback signals;
9/23
Three types of associations
IOA: Input only association;
OOA: Output only association;
INOUA: Input-output association;
10/23
Probability based associative learning algorithm
Case 1: Given the values of both inputs, decide the output value;
0021
2110
21
21
01
21
2111
21
21
)0,0(
)1,0,0(
)0,1(
)1,0,1(
)1,0(
)1,1,0(
)1,1(
)1,1,1()(
VIIp
FIIpV
IIp
FIIp
VIIp
FIIpV
IIp
FIIpOV
•==
===+•==
===+
•==
===+•==
====
)1)(1();1(
;)1(;
0010
0111
nmVnmV
nmVmnV
−−=−=−==
11/23
Probability based associative learning algorithm
Case 2: Given the values of one input and an un-defined output, decide the value of the other input;
( ) ( )( )1
1
211
1
212 1
)0(
)1,0(
)1(
)1,1()( IV
Ip
IIpIV
Ip
IIpIV −•
===+•
====
01001
11101
)0(
)1(
ppIp
ppIp
+==+==
For instance:
12/23
Probability based associative learning algorithm
Case 3: Given the values of the output, decide the values of both inputs;
( ) ( )( )
( ) ( )( )OVFp
IFpOV
Fp
IFpIV
OVFp
IFpOV
Fp
IFpIV
−•=
==+•=
===
−•=
==+•=
===
1)0(
)1,0(
)1(
)1,1()(
1)0(
)1,0(
)1(
)1,1()(
222
111
13/23
Probability based associative learning algorithm
Case 4: Given the values of one input and the output, decide the other input value;
For instance:
00
1
2110
1
21
011
2111
1
212
ˆ)0,0(
)1,0,0(ˆ)0,1(
)1,0,1(
ˆ)1,0(
)1,1,0(ˆ)1,1(
)1,1,1()(
VFIp
IFIpV
FIp
IFIp
VFIp
IFIpV
FIp
IFIpIV
•==
===+•==
===+
•==
===+•==
====
14/23
Outline
Introduction;
Associative learning algorithm;
Memory network architecture and operation;
Simulation analysis;
Conclusion and future research;
15/23
Network operations
Feedback operation Feed forward operation
Depth
Input data Input data
Depth
?
.
?
16/23
2I
1I
Memory operation
1
2
3
4 5
1O
1I
2I
2I
2O
3O
4O
fI2
fI1
fI2
5O
fI2
fO
fOfI2
KT = 1+= KT 2+= KT
Undefined signal
Defined signal
Recovered signal Input data
Signal resolved
based on SS
17/23
Outline
Introduction;
Associative learning algorithm;
Memory network architecture and operation;
Simulation analysis;
Conclusion and future research;
18/23
M 3M
M 3M
Class 1
Class 2
M 3M
Class 3
Hetero-associative memory: Iris database classification
N-bits sliding-bar coding mechanism:
Features:
Class identity labels:
In our simulation:
N=80, L=20, M=30
3 classes, 4 numeric attributes, 150 instances
19/23
Neuron association pathway
Classification accuracy: 96%
20/23
Auto-associative memory: Panda image recovery
30% missing pixels
Original image64x64 binary image
Error: 0.4394%
Block half Error: 2.42%
64 x 64 binary panda image:
( ) 4096,...21 == nxxxp ni 1=ix 0=ix for a black pixel; for a white pixel;
21/23
Outline
Introduction;
Associative learning algorithm;
Memory network architecture and operation;
Simulation analysis;
Conclusion and future research;
22/23
Conclusion and future research
Hierarchical associative memory architecture;
Probabilistic information processing, transmission, association and prediction;
Self-organization;
Self-adaptive;
Robustness;
23/23
It’s all about design natural intelligent machines !
Future research
Multiple-inputs (>2) association mechanism;
Dynamically self-reconfigurable;
Hardware implementation;
Facilitate goal-driven learning;
Spatio-temporal memory organization;
How far are we???
“Brain On Silicon” will not just be a
dream or scientific fiction in the future!
3DANN
Picture source: http://www.cs.utexas.edu/users/ai-lab/fai/; and Irvine Sensors Corporation (Costa Mesa, CA)