what is unsupervised learning? learning without a teacher. no feedback to indicate the desired...
TRANSCRIPT
What is Unsupervised Learning?
• Learning without a teacher.
• No feedback to indicate the desired
outputs.
• The network must by itself discover the
relationship of interest from the input
data.
The Nearest Neighbor Classifier
11 22
33 44
x(1) x(2)
x(3)x(4)
The Nearest Neighbor Classifier
11 22
33 44
x(1) x(2)
x(3)x(4)
?Class
The Hamming Networks
• Stored a set of classes represented by a set of binary prototypes.
• Given an incomplete binary input, find the class to which it belongs.
• Use Hamming distance as the distance measurement.
• Distance vs. Similarity.
The Hamming Net
Similarity Measurement
MAXNET Winner-Take-All
x1 x2 xn
The Hamming Distance
y = 1 1 1 1 1 1 1
x = 1 1 1 1 1 1 1
Hamming Distance = ?Hamming Distance = ?
The Hamming Distance
y = 1 1 1 1 1 1 1
x = 1 1 1 1 1 1 1
Hamming Distance = ?Hamming Distance = ?
y = 1 1 1 1 1 1 1
x = 1 1 1 1 1 1 1
The Hamming Distance
Hamming Distance = 3Hamming Distance = 3
y = 1 1 1 1 1 1 1
The Hamming Distance
1 1 1 1 1 1 1
Sum=1
12( , ) (7 1) 3HD x y
x = 1 1 1 1 1 1 1
The Hamming Distance
1 2( , , , ) {1, 1}Tm iy y y y y
1 2( , , , ) {1, 1}Tm ix x x x x
( , ) ?HD x y
( , ) ?Similarity x y
The Hamming Distance
1 2( , , , ) {1, 1}Tm iy y y y y
12( , ) ( )THD m x y x y
12
1 12 2
( , ) ( )
T
T
Similarity m m
m
x y x y
x y
1 2( , , , ) {1, 1}Tm ix x x x x
The Hamming Distance
1 2( , , , ) {1, 1}Tm iy y y y y
12( , ) ( )THD m x y x y
12
1 12 2
( , ) ( )
T
T
Similarity m m
m
x y x y
x y
1 2( , , , ) {1, 1}Tm ix x x x x
The Hamming Net
Similarity Measurement
MAXNET Winner-Take-All
11 22 n1n1 nn
x1 x2 xm1 xm
11 22 n1n1 nn
y1 y2 yn1 yn
The Hamming Net
Similarity Measurement
MAXNET Winner-Take-All
11 22 n1n1 nn
x1 x2 xm1 xm
11 22 n1n1 nn
y1 y2 yn1 yn
WS=?WS=?
WM=?WM=?
The Stored Patterns
Similarity Measurement
MAXNET Winner-Take-All
11 22 n1n1 nn
x1 x2 xm1 xm
11 22 n1n1 nn
y1 y2 yn1 yn
WS=?WS=?
WM=?WM=?
1 2
1 2( , , ) { 1,
Stored patterns
with 1 }.
n
k k k k T km i
n , , ,
s s s s
s s s
s
1 2
1 2( , , ) { 1,
Stored patterns
with 1 }.
n
k k k k T km i
n , , ,
s s s s
s s s
s
kTk mSimilarity sxsx 21
21),( kTk mSimilarity sxsx 2
121),(
1 12 2
1
( , )m
k ki i
i
Similarity m x s
x s 1 12 2
1
( , )m
k ki i
i
Similarity m x s
x s
The Stored Patterns
Similarity Measurement
1 2
1 2( , , ) { 1,
Stored patterns
with 1 }.
n
k k k k T km i
n , , ,
s s s s
s s s
s
1 2
1 2( , , ) { 1,
Stored patterns
with 1 }.
n
k k k k T km i
n , , ,
s s s s
s s s
s
),( kSimilarity sx ),( kSimilarity sxk
x1 x2 xm. . .
112ks 1
22ks
12
kms
m/2
kTk mSimilarity sxsx 21
21),(
kTk mSimilarity sxsx 21
21),(
1 12 2
1
( , )m
k ki i
i
Similarity m x s
x s 1 12 2
1
( , )m
k ki i
i
Similarity m x s
x s
• Weight update: – Method 1: Method 2
In each method, is moved closer to il
– Normalize the weight vector to unit length after it is updated
– Sample input vectors are also normalized
– Distance
)( jlj wiw lj iw
jjj www
jjj www /
wj
il
il – wj
η (il - wj)
wj + η(il - wj)
jw
il
wjwj + ηil
ηil
il + wj
lll iii /
i ijiljljl wiwiwi 2,,2)(
• is moving to the center of a cluster of sample vectors after repeated weight updates – Node j wins for three training
samples: i1 , i2 and i3
– Initial weight vector wj(0)– After successively trained
by i1 , i2 and i3 ,the weight vector
changes to wj(1),
wj(2), and wj(3),
jw
i2
i1
i3
wj(0)
wj(1)
wj(2)
wj(3)
Example
will always win no matter
the sample is from which class
is stuck and will not participate
in learning
unstuck:
let output nodes have some conscience
temporarily shot off nodes which have had very high
winning rate (hard to determine what rate should be
considered as “very high”)
2w
1ww1
w2
Example
Results depend on the sequenceof sample presentation
w1
w2
Solution:Initialize wj to randomly selected input vector il that are far away from each other
w1
w2
